Short text understanding method based on reusable general CNN model

Abstract

This invention provides a short text understanding method based on a reusable general CNN model. The method employs a CNN framework for parsing short text content to perform short text understanding. The CNN framework for parsing short text content includes an upstream component and a downstream component. The upstream component uses a pre-trained language model to convert the original text input into dense word embeddings. The downstream component uses a task-specific CNN model to receive the dense word embeddings and generate a final prediction for a specific NLP task.

Description

Technical Field

[0001] This invention belongs to the technical fields of natural language processing and big data analysis, and specifically relates to a short text understanding method based on a reusable general CNN model. Background Technology

[0002] With the development of the internet and communication technologies (such as the Internet of Things and 5G), the forms of online information resources have gradually evolved from single (such as plain text or plain images) to diversified (such as multimodal formats). For example, an online learning course may contain text information (such as student discussions), audio and video information (such as recorded course content), and images (such as accompanying slides). In the e-commerce field, product listings on Amazon or Taobao may contain text, images, and even short video information. Therefore, processing information in a single form is no longer sufficient to fully understand these online resources. These changes bring new challenges to intelligent multimodal information processing tools in various big data application areas (including online learning, digital health, and traffic information systems).

[0003] Research on the processing and fusion of information in different formats remains relatively isolated. For example, there is a lack of robust, general models capable of handling tasks involving different types of information (i.e., multimodal information). Therefore, it is often necessary to use multiple models in a processing system to process and interpret multimodal information. For instance, in the work on open educational resources in the medical field (reference: B. Zhao, S. Xu, S. Lin, X. Luo, and L. Duan, "A new visual navigation system for exploring biomedical open educational resource (OER) videos," Journal of the American Medical Informatics Association, vol. 23, no. e1, pp. e34–e41, 2016.), three different models were used to process text, video streams, and images. Therefore, a reusable, general model, or a single model that requires only minor adjustments for different types of information, can significantly reduce the complexity of the overall system. Summary of the Invention

[0004] Tracing back to the ultimate goal of Natural Language Processing (NLP) or Computer Vision (CV) problems, we can see that most existing research aims to build a model to understand the content of a text or image. For ease of training, the content of images or text is usually represented digitally (integers or decimals). The model training process typically involves modeling the input based on some mathematical process and narrowing the gap between the predicted results and the true values. In short, CV and NLP tasks have similar goals, and the data representation is similar in both. This invention focuses on the common underlying mechanisms between NLP and CV models, proposing a reusable and general solution using CV techniques to solve NLP problems.

[0005] To address the limitations and shortcomings of existing technologies, this invention proposes a short text understanding method based on a reusable, general-purpose CNN model. This invention solves the problems of isolation in multimodal information processing, lack of a general model, difficulty in data fusion, and inconsistent training methods for NLP and CV tasks in existing technologies. By proposing an innovative method for using computer vision (CV) technology to process natural language processing (NLP) tasks, this invention demonstrates the feasibility of constructing a general model applicable to multiple information forms, reducing system complexity, improving model scalability and application potential, and filling a key gap in existing technologies.

[0006] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0007] A short text understanding method based on a reusable general CNN model is proposed. The method employs a CNN framework for parsing short text content to perform short text understanding. The CNN framework for parsing short text content includes an upstream component and a downstream component. The upstream component uses a pre-trained language model to convert the original text input into dense word embeddings. The downstream component uses a task-specific CNN model to receive the dense word embeddings and generate a final prediction for a specific NLP task.

[0008] Furthermore, the upstream component uses a pre-trained language model L to generate a dense word embedding e of dimension m; for short text t=(w1,w2,...,w i For each word w in the process, the mapping is defined as L(w) i )=>e i ,e i ∈R m , where e i R is an m-dimensional real-valued vector. m It is a set of vectors.

[0009] 1. The short text understanding method based on a reusable general CNN model according to claim 2, characterized in that: the short text is represented as a two-dimensional 'image' P, which is formed by stacking all embeddings together; this process is represented as P = Stack(e1, e2, ..., e...). i To prevent information loss during this process, the embedding order is kept consistent with the original text sequence.

[0010] Furthermore, the downstream component uses a CNN-based downstream model to receive the generated text "image" P and make a final prediction y; the goal of the downstream component is to learn the following function F, as shown in equation (1):

[0011]

[0012] The text content is interpreted using a CNN-based network; the generated two-dimensional text image is processed by multiple convolutional kernels k1-k i The system scans several times to extract semantic information; different convolutional kernels have different widths d. i And they share the same height h; the height h of the convolution kernel is equal to the size of the word embedding; so that the convolution kernel summarizes d at each step. i The information of each consecutive word is convolved as shown in Equation (2);

[0013]

[0014] Where h is the input signal, g is the convolution process, and the specific process of the downstream component is shown in equation (3);

[0015]

[0016] Where w ij For the corresponding input signal x ij The weights are denoted by b, and the bias term is denoted by b. After a series of consecutive convolutional operations, pooling operations and fully connected layers are applied to summarize all the extracted information and generate the final prediction.

[0017] Furthermore, the loss function uses a one-dimensional bounding box to label key information; the bounding box has only one dimension, and the corresponding formula is B(c index ,1), the first term is the center index of the selected text sequence, and the second term is the length of the selected text; the loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in equation (4);

[0018]

[0019] MSE is the mean squared error loss; the first term measures the difference between the predicted center and the true center, and the second term measures the difference between the length of the predicted keyword / phrase and the true length.

[0020] Furthermore, the specific process of performing short text understanding using a CNN framework for parsing short text content includes:

[0021] Data input includes short text data and short text metadata; the short text metadata includes the short text's timestamp, user information, and additional contextual information;

[0022] Data preprocessing includes data cleaning, data normalization, and feature extraction; the data cleaning includes noise removal and processing or imputing missing data.

[0023] Data normalization: This includes converting text data into a uniform format, and lexical normalization: performing stemming or word form restoration to ensure that the same words appear in a consistent form;

[0024] Feature extraction includes: generating word embeddings: converting short texts into dense word embedding vectors using a pre-trained language model; and extracting key information: identifying key information and sentiment features in short texts, including labeling key information using 1D bounding box techniques.

[0025] Model training: Select a pre-trained language model as input; then select a downstream model based on the task.

[0026] Finally, the trained model is used to perform short text understanding.

[0027] Furthermore, 1D bounding boxes are used to locate and label key sentiment words or information fragments in short texts to achieve keyword localization.

[0028] Furthermore, a short text understanding system based on a reusable general CNN model, based on a CNN framework for parsing short text content, includes: an upstream component and a downstream component; the upstream component employs a pre-trained language model to convert the original text input into dense word embeddings; the downstream component employs a task-specific CNN model to receive dense word embeddings and generate a final prediction for a specific NLP task.

[0029] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the short text understanding method based on a reusable general CNN model as described above.

[0030] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the short text understanding method based on a reusable general CNN model as described above.

[0031] Compared with the prior art, the present invention and its preferred embodiments have at least the following advantages:

[0032] 1. High versatility:

[0033] This method can handle both Natural Language Processing (NLP) and Computer Vision (CV) tasks simultaneously without large-scale adjustments. This potential for cross-domain application makes the model more versatile, enabling it to adapt to different types of multimodal data (such as text, images, audio, and video).

[0034] 2. Efficiently process short texts:

[0035] Compared to traditional RNN models (such as Bi-LSTM and Bi-GRU), Convolutional Neural Networks (CNNs) are more efficient at processing short texts (such as short Twitter texts). CNNs extract semantic information at different levels through convolutional kernels, enabling them to process large amounts of short text data in a shorter time, making them particularly suitable for online sentiment analysis.

[0036] 3. Key information location capability:

[0037] By using a method similar to "bounding boxes" in computer vision, this model can not only classify text but also accurately locate words or phrases that express key information within the text. This advantage makes it particularly effective when processing texts that require high-precision analysis, such as identifying specific sentiment words in sentiment analysis.

[0038] 4. Avoid efficiency issues in processing long sequences:

[0039] Because CNNs operate based on local convolutions, they do not rely on global temporal information. Therefore, unlike RNNs, they do not suffer from gradient vanishing or exploding problems when processing long texts or long sequences of data. Consequently, CNNs perform more stably in text understanding tasks, especially when processing long sequences.

[0040] 5. Lower computing costs:

[0041] Compared to RNN models that require long-term dependency, CNNs have advantages in parallel computing. Their computational cost and resource requirements are usually lower, so they can improve running speed and reduce computational resource consumption when processing large-scale data.

[0042] Its application scenarios include, but are not limited to:

[0043] 1. Sentiment Analysis:

[0044] The model can be applied to sentiment analysis tasks, helping businesses, researchers, or marketing teams analyze sentiment trends on social media and understand users' attitudes toward a particular event, product, or topic. By accurately locating sentiment words in short texts, the model can provide more interpretable and accurate sentiment classification results.

[0045] 2. Disaster Prediction:

[0046] This model can be applied to data analysis on social media platforms (such as Twitter) to help identify short texts related to real disasters, thereby providing early warnings for emergency management and disaster response. The model can quickly locate disaster-related short texts within a large amount of social media content, helping governments and agencies obtain critical information in a timely manner.

[0047] 3. Multimodal data processing:

[0048] The model's versatility enables it to handle various multimodal data, not just text. It can be used in fields such as e-commerce, online learning, and digital health to analyze content containing multiple data formats, including text, images, and videos, thus enhancing the ability to process complex information.

[0049] 4. Academic research and interdisciplinary applications:

[0050] This method bridges the gap between computer vision and natural language processing, making it suitable for exploring interdisciplinary applications in academic research. Researchers can leverage this model to share algorithms and techniques across different fields (such as image processing and text analysis), achieving knowledge transfer.

[0051] In summary, the main advantages of this invention lie in its efficient processing of short texts, strong ability to locate key information, low computational resource requirements, and cross-domain versatility. Its applications are wide-ranging, including disaster prediction, sentiment analysis, multimodal data processing, and interdisciplinary applications in academic research. This makes the product suitable for use in various scenarios such as social media data analysis, emergency response, and market research. Attached Figure Description

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0053] Figure 1 This is a flowchart illustrating the overall process of an embodiment of the present invention.

[0054] Figure 2 This is a diagram illustrating the algorithm framework of an embodiment of the present invention.

[0055] Figure 3 This is an example diagram illustrating the operation of embedding stacking in an embodiment of the present invention.

[0056] Figure 4 This is a network structure diagram of the downstream component in an embodiment of the present invention. Detailed Implementation

[0057] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0060] This invention presents a CNN framework for parsing short text content. Its main process is as follows: Figure 1 As shown, the following is a detailed introduction:

[0061] 1. Frame structure

[0062] The proposed framework comprises two components: an upstream component and a downstream component. The overall architecture of the framework is as follows: Figure 2 As shown, the upstream component is a pre-trained language model that transforms the raw text input into dense word embeddings. The downstream component is a task-specific model that receives dense word embeddings and generates a final prediction for a specific NLP task.

[0063] 2. Problem Definition

[0064] To clearly define the short text content comprehension task, it is defined as follows:

[0065] (1) Upstream components:

[0066] In this invention, a pre-trained language model L is used to generate dense word embeddings e of dimension m. For a short text t = (w1, w2, ..., w...), ... i For each word w in the process, the mapping can be defined as L(w). i )=>e i ,e i ∈R m , where e i R is an m-dimensional real-valued vector. mIt is the set of these vectors.

[0067] (2) Short text representation:

[0068] In this invention, short text is represented as a two-dimensional 'image' P, formed by stacking all embeddings together. This process is represented as P = Stack(e1, e2, ..., e...). i To prevent information loss during this process, the embedding order remains consistent with the original text sequence. See [link to process details] for details. Figure 3 .

[0069] (3) Downstream components:

[0070] Since different NLP tasks have different objectives, the design of downstream components is usually tailored to the specific task. Typically, recurrent neural networks and their variants are used to capture temporal information. However, this invention uses a CNN-based downstream model. This downstream component receives the generated text "image" P and makes a final prediction y. Therefore, the goal of the downstream component is to learn the following function F, as shown in equation (1):

[0071]

[0072] Inspired by computer vision research and n-gram models, this invention designs a CNN-based network using convolutional techniques to interpret text content. The generated two-dimensional text image is processed by multiple convolutional kernels k1-k. i The system scans several times to extract semantic information. Different convolutional kernels have different widths d. i However, they share the same height h. The height h of the convolution kernel is equal to the size of the word embeddings. With these settings, the convolution kernel can summarize d at each step. i The information of a series of words is used in this process, which is similar to generating n-gram samples. The convolution process is shown in Equation (2).

[0073]

[0074] Where h is the input signal and g is the convolution process. This convolution operation extracts fine-grained information at different levels in a similar manner. The specific process can be further shown in equation (3).

[0075]

[0076] Where w ij For the corresponding input signal x ij The weights are denoted by , and b is the bias term. After a series of consecutive convolutional operations, pooling and fully connected layers are applied to summarize all extracted information and generate the final prediction. The proposed CNN-based downstream component network structure is as follows: Figure 4As shown.

[0077] (4) One-dimensional bounding box:

[0078] Furthermore, in this invention, besides enabling the proposed model to understand short text content, the loss function is improved to allow the model to identify which phrases or words in the short text express key information. In text processing, a one-dimensional "bounding box" is used to label key information. Therefore, the proposed model not only classifies a given text into a predefined category but also highlights which words / phrases express the key information that triggers the model's decision. The main difference between the bounding box used in this invention and those in common computer vision tasks is that the bounding box has only one dimension. The formula for a 1D bounding box is B(c index ,1), where the first term is the center index of the selected text sequence and the second term is the length of the selected text. The loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in Equation (4).

[0079]

[0080] MSE stands for Mean Squared Error Loss. The first term measures the difference between the predicted center and the true center, and the second term measures the difference between the predicted keyword / phrase length and the true length.

[0081] The following further presents the application process based on the above design:

[0082] 1. Data Input

[0083] (1) Short text data

[0084] The system receives short text data from social media platforms (such as Twitter), including short text content from users.

[0085] (2) Short text metadata

[0086] This includes the short text's timestamp, user information (such as username, geolocation, etc.), and possible additional contextual information (such as keywords).

[0087] 2. Data Preprocessing

[0088] (1) Data cleaning

[0089] Noise Removal: Remove unnecessary information such as URLs, emojis, and stop words from short texts.

[0090] Handling missing values: Processing or filling in missing data.

[0091] (2) Data normalization

[0092] Standardized format: Convert text data into a standardized format for easier subsequent processing.

[0093] Lexical standardization: Stem extraction or word form restoration to ensure that the same words appear in a consistent form.

[0094] (3) Feature extraction

[0095] Generate word embeddings: Use pre-trained language models (such as BERT, Word2Vec) to convert short texts into dense word embedding vectors.

[0096] Extracting key information: Identifying key information and sentiment features in short texts may include using 1D bounding box techniques to label key information.

[0097] 3. Model Training

[0098] (1) Model selection

[0099] Choose a pre-trained model: Use different pre-trained language models (such as BERT, GloVe, Word2Vec) as input.

[0100] Selecting downstream models: Choose different downstream models based on the task, such as Bi-LSTM, Bi-GRU, CNN, etc.

[0101] (2) Training process

[0102] Adjust model parameters such as learning rate, batch size, and number of training epochs to optimize model performance.

[0103] Cross-validation: Use cross-validation to evaluate the model's generalization ability.

[0104] Test set evaluation: Evaluate the final performance of the model on the test set and make iterative adjustments.

[0105] 4. Real-time applications

[0106] (1) Sentiment Analysis

[0107] Analyze short text sentiment: Use a trained model to analyze the sentiment tendency (e.g., positive or negative) of new short texts in real time.

[0108] (2) Disaster Prediction

[0109] Identify disaster-related information: Use models to detect in real time whether short texts involve disaster events and make corresponding predictions.

[0110] (3) Keyword Targeting

[0111] Highlight key information: Use 1D bounding box technology to locate and mark key sentiment words or information fragments in short texts.

[0112] 5. Results Display

[0113] (1) Recommendations and Alerts

[0114] Display analysis results: Present the results of sentiment analysis, disaster prediction, etc. to users or relevant departments.

[0115] Generate alerts: For disaster prediction tasks, issue alerts in a timely manner to facilitate response.

[0116] (2) User feedback

[0117] Collect feedback: Further optimize the model and system based on user feedback.

[0118] Test Example

[0119] 1. Dataset

[0120] The experiment used two short text datasets, both from the Twitter platform and publicly available. Both datasets consist of informal short texts collected on Twitter. The reasons for choosing these two datasets are:

[0121] (1) Processing informal text to demonstrate that the model proposed in this invention is general enough to handle complex NLP problems.

[0122] (2) Use short texts to demonstrate the excellent information extraction capabilities of the model proposed in this invention.

[0123] The first dataset contains over 10,000 short texts, some of which discuss real disaster events. The ratio of disaster-related to non-disaster-related short texts is 43:57. This dataset includes the original text, keywords, and the location information of the short texts. Keyword and location information may be empty; only the original text is used to train the model. The second dataset contains 30,000 short texts with or without sentiment information. The ratio of neutral, positive, and negative samples is 41:31:28. This dataset includes the original text as well as text fragments with sentiment. Again, only the original text is used to train the model. The original texts were collected directly from the Twitter platform and have not been preprocessed. Table 1 shows examples of these two datasets.

[0124] Table 1

[0125]

[0126] 2. Evaluation Indicators

[0127] To reflect the model's performance from different perspectives, three different types of evaluation metrics were used in the experiment. The first evaluation metric is accuracy, as shown in Equation (5), which directly reflects the proportion of correct predictions generated by each model. However, for imbalanced real data distributions, this metric may not be suitable for comparing model effectiveness.

[0128]

[0129] The second evaluation metric is the area under the curve (AUC). AUC reflects the model's ability to distinguish between different types of information (i.e., whether it is a disaster in Task 1, or different emotions in Task 2). In handling multi-class classification tasks (sentiment analysis tasks), a "one-to-one" strategy was used in the experiment. The AUC value is the area under the receiver operating characteristic (ROC) curve.

[0130] The final evaluation metric is the F-score, as shown in Equation (6), which is the harmonic mean of recall and precision. Since there is a trade-off between recall and precision, a model's quality cannot be judged solely by high recall or high precision. Therefore, using the F-score is a better choice for model comparison.

[0131]

[0132] 3. Baseline Model

[0133] In the experiments, different pre-trained models were used to investigate the effectiveness of applying computer vision (CV) solutions to natural language processing (NLP) problems. Specifically, for the upstream components, the following pre-trained language models were introduced:

[0134] (1) Word2Vec: This model has high optimization efficiency, but it can only model local semantic information within a predefined window.

[0135] (2) GloVe: This model combines the advantages of LSA and Word2Vec. It uses a co-occurrence matrix to model both local and global semantic information.

[0136] (3) Bert: Bert and its variants can dynamically model semantic information. According to the original research, multi-task fine-tuning methods can be used to train models, which will further improve performance.

[0137] For downstream components, the effectiveness of the following models in understanding short text content was compared:

[0138] (1)Bi-GRU: Bidirectional gated recurrent unit neural network.

[0139] (2) Bi-LSTM: Bidirectional Long Short-Term Memory Neural Network.

[0140] (3) The CNN-based model proposed in this invention.

[0141] 4. Experimental Setup

[0142] In this invention, all models are implemented using the PyTorch framework. The BERT and Word2Vec pre-trained language models are implemented using Transformer and Gensim, respectively; the GloVe model is reproduced using its pre-trained word vectors. The CNN-based model proposed in this embodiment uses six different convolutional kernels with 128 output channels each. Each convolutional output uses ReLU as the activation function, and there are four consecutive convolutional layers. The dimensions of the Bi-LSTM and Bi-GRU hidden layers are set to 256. The disaster prediction task uses the sigmoid function to generate the final prediction, and the sentiment analysis task uses the softmax function. All other settings strictly follow the original work or directly use the default settings of the PyTorch framework. To prevent overfitting, early stopping is employed in all training processes.

[0143] Before using a language model to convert short text content into dense vectors, the original text undergoes multiple stages of NLP data cleaning and normalization, including stop word removal and lemmatization, removal of URLs and emojis, etc.

[0144] 5. Experimental Results and Discussion

[0145] Table 2 shows a comparison of the effectiveness of different downstream components in two different NLP tasks. Table 3 reports the effectiveness of different upstream components. Since similar results were obtained from the two datasets, only the experimental results for the disaster prediction task are shown here (Table 3). Table 4 shows the one-dimensional bounding boxes used to locate key information.

[0146] Table 2

[0147]

[0148] (1) The effectiveness of the proposed CNN model

[0149] Based on the results in Table 2, it is easy to conclude that, with the same BERT upstream component, the proposed CNN-based downstream component performs competitively with mainstream NLP solutions across all criteria for both tasks (highlighted in bold). For the disaster prediction task, the Bi-LSTM model slightly outperforms the Bi-GRU model, while in the sentiment analysis task, the Bi-GRU model significantly outperforms the Bi-LSTM model. This improvement is due to the structural differences between LSTM and GRU units. According to original research on LSTM and GRU, LSTM tends to remember longer semantic information. Therefore, including too much information in the modeling process does not always improve model performance across different tasks. The disaster prediction task requires understanding the entire short text to predict whether a given short text is related to a disaster. It is difficult to infer whether a short text is related to a disaster based solely on short text fragments or phrases. For example: “All residents asked to ‘shelter inplace’ are being notified by officers. No other evacuation or shelter in place orders are expected.”

[0150] Remembering more words or longer sequences of text helps in understanding the context of a short text. Therefore, for such tasks, LSTM-based models outperform GRU-based models.

[0151] As for sentiment analysis tasks, in most cases, sentiment is only a part of the short text content. Here is an example of a short text with negative sentiment: a user posted about an event and expressed their feelings: "Grrr...stupid internet connection ruined a great scrabble game."

[0152] The interpretation of user sentiment relies on short text sequences or phrases. Inference from longer text sequences can negatively impact model performance, as in the example above, the phrase "a great scrabble game" expresses positive sentiment, while the short text itself contains negative sentiment. Therefore, under the same settings, the Bi-GRU model significantly outperforms the Bi-LSTM model in the second task.

[0153] By using the proposed CNN-based model, the aforementioned problem of modeling long and short sequences can be avoided. When using the CNN model, the length of the word sequence to be modeled in each step can be flexibly controlled by setting the kernel size of the first convolutional layer. The configuration of the convolutional kernel size can be determined based on domain knowledge or preliminary experiments.

[0154] (2) The importance of upstream language models

[0155] The tests also investigated the impact of using different language models in upstream components on framework performance. Table 3 shows that the framework using the pre-trained BERT model outperforms the frameworks using the GloVe or Word2Vec models (highlighted in bold). The BERT model captures more semantic information. A better language model means downstream components can access more useful information. From a computer vision perspective, a better language model can generate "higher resolution text images," which is crucial for extracting details from these images. As the literature points out, short texts often contain more information per character, likely due to Twitter's unique abbreviations and inconsistent writing style. Therefore, using a powerful language model to preserve semantic information to the maximum extent is necessary for better interpreting short text content.

[0156] (3) Locating key information

[0157] Using a second dataset, the tests further investigated whether the proposed model could identify and locate key information (i.e., keywords). Keywords were selected using a designed one-dimensional bounding box (which is typically done using attention mechanisms for NLP tasks). Due to space limitations, this example only randomly presents five positive and five negative results, as shown in Table 4. The "True Value Keywords" column shows the labeled true keywords containing sentiment information; the "Selected Text" column shows the keywords selected by the bounding box. It is clear that the proposed model can not only understand the sentiment meaning of short text content but also identify which words express this meaning. The only drawback of the bounding box is that it tends to select text that is longer than the true values ​​(by comparing the second and third columns). This is because longer text usually contains more information and provides more support for the bounding box's decision.

[0158] Table 3

[0159] Model accuracy F1 AUC Bert+ proposed a model 0.8238 0.8169 0.8798 GloVe+ proposes a model 0.8173 0.8145 0.8723 Word2Vec+ proposes a model 0.6852 0.6762 0.7271 Bert+Bi-GRU 0.8108 0.8044 0.8698 GloVe+Bi-GRU 0.7667 0.7598 0.8097 Word2Vec+Bi-GRU 0.6819 0.6760 0.7348 Bert+Bi-LSTM 0.8172 0.8128 0.8695 GloVe+Bi-LSTM 0.8107 0.8039 0.8588 Word2Vec+Bi-LSTM 0.6770 0.6625 0.7283

[0160] Table 4

[0161]

[0162]

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0168] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0169] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of short text understanding methods based on reusable general CNN models under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A short text understanding method based on a reusable general CNN model, characterized in that: A CNN framework for parsing short text content is used to perform short text understanding; the CNN framework for parsing short text content includes an upstream component and a downstream component; the upstream component uses a pre-trained language model to convert the original text input into dense word embeddings; the downstream component uses a task-specific CNN model to receive dense word embeddings and generate a final prediction for a specific NLP task. The upstream component uses a pre-trained language model L to generate a dense word embedding e of dimension m; for a short text t = (w1, w2, ..., w i For each word w in the process, the mapping is defined as L(w) i )=>e i ,e i ∈R m , where e i R is an m-dimensional real-valued vector. m It is a set of vectors; The short text is represented as a two-dimensional 'image' P, formed by stacking all the embeddings together; this process is represented as P = Stack(e1, e2, ..., e...). i To prevent information loss during this process, the embedding order remains consistent with the original text sequence. The downstream component uses a CNN-based downstream model to receive the generated text "image" P and make a final prediction y; the goal of the downstream component is to learn the following function F, as shown in equation (1): The text content is interpreted using a CNN-based network; the generated two-dimensional text image is processed by multiple convolutional kernels k1-k i The system scans several times to extract semantic information; different convolutional kernels have different widths d. i And they share the same height h; the height h of the convolution kernel is equal to the size of the word embedding; so that the convolution kernel summarizes d at each step. i The information of each consecutive word is convolved as shown in Equation (2); Where h is the input signal, g is the convolution process, and the specific process of the downstream component is shown in equation (3); Where w ij For the corresponding input signal x ij The weights are denoted by b, and the bias term is denoted by b. After a series of consecutive convolutional operations, pooling operations and fully connected layers are applied to summarize all the extracted information and generate the final prediction. Keyword localization is achieved by using 1D bounding boxes to locate and mark key sentiment words or information fragments in short texts.

2. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The loss function used employs a one-dimensional bounding box to label key information; the bounding box has only one dimension, and the corresponding formula is B(c index ,1), the first term is the center index of the selected text sequence, and the second term is the length of the selected text; the loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in equation (4); MSE is the mean squared error loss; the first term measures the difference between the predicted center and the true center, and the second term measures the difference between the length of the predicted keyword / phrase and the true length.

3. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The specific process of performing short text understanding using a CNN framework that parses short text content includes: Data input includes short text data and short text metadata; the short text metadata includes the short text's timestamp, user information, and additional contextual information; Data preprocessing includes data cleaning, data normalization, and feature extraction; the data cleaning includes noise removal and processing or imputing missing data. Data normalization includes converting text data into a uniform format and lexical normalization, which involves stemming or word form restoration to ensure that the same words appear in a consistent form. Feature extraction includes: generating word embeddings: converting short texts into dense word embedding vectors using a pre-trained language model; and extracting key information: identifying key information and sentiment features in short texts, including labeling key information using 1D bounding box techniques. Model training: Select a pre-trained language model as input; then select a downstream model based on the task. Finally, the trained model is used to perform short text understanding.

4. A short text understanding system based on a reusable general CNN model, used to implement the method as described in claim 1, characterized in that: The CNN framework based on parsing short text content includes an upstream component and a downstream component. The upstream component uses a pre-trained language model to convert the original text input into dense word embeddings. The downstream component uses a task-specific CNN model to receive the dense word embeddings and generate the final prediction for a specific NLP task.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the short text understanding method based on a reusable general CNN model as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the short text understanding method based on a reusable general CNN model as described in any one of claims 1-3.

Images