Short text understanding method based on reusable general CNN model

Abstract

The invention provides a short text understanding method based on a reusable general CNN model. The short text understanding method comprises the following steps: executing short text understanding by adopting a CNN framework for analyzing short text contents; the CNN framework for analyzing the short text content comprises an upstream component and a downstream component; the upstream component adopts a pre-trained language model and is used for converting original text input into dense word embedding; the downstream component employs a task-specific CNN model for receiving dense word embedding and generating a final prediction for a particular NLP task.

Description

Technical Field

[0001] The present invention belongs to the technical fields of natural language processing, big data analysis, etc., and specifically relates to a short text understanding method based on a reusable general CNN model. Background Art

[0002] With the development of the Internet and communication technologies (such as the Internet of Things and 5G), the form of online information resources has gradually evolved from a single form (such as pure text or pure pictures) to a diversified form (such as multimodal forms). For example, an online learning course can contain text information (such as student discussions), audio and video information (such as recorded course content), and images (such as attached slides). In the field of e-commerce, product listings on Amazon or Taobao can contain text, images, and even short video information. Therefore, processing a single form of information alone is no longer sufficient to fully understand these online resources. These changes have brought new challenges to intelligent multimodal information processing tools in various big data application fields (including online learning, digital health, traffic information systems, etc.).

[0003] Research on the processing and fusion of information in different formats remains relatively isolated. For example, there is a lack of robust general models that can handle tasks involving different types of information (i.e., multimodal information). Therefore, it is often necessary to use multiple models in a processing system to process and interpret multimodal information. For example, in the work on open educational resources in the medical field (reference: B. Zhao, S. Xu, S. Lin, X. Luo, and L. Duan, "A new visual navigation system for exploring biomedical open educational resource (OER) videos," Journal of the American Medical Informatics Association, vol. 23, no. e1, pp. e34–e41, 2016.), three different models were used to process text, video streams, and images. Therefore, a reusable general model, or a single model that only requires minor adjustments for different types of information, can greatly reduce the complexity of the overall system. Summary of the invention

[0004] When we look back to the ultimate goal of natural language processing (NLP) or computer vision (CV) problems, we can see that most of the existing research aims to build a model to understand the content of a text or a picture. In order to facilitate the training process, the content of the picture or text is usually represented in digital form (integer or decimal). The training process of the model is usually based on some mathematical process to model the input and narrow the gap between the predicted result and the true value. In short, CV tasks and NLP tasks have similar goals, and the data representation is similar in CV and NLP tasks. The focus of the present invention is to propose a reusable general solution for solving NLP problems using CV technology based on the common underlying mechanism between NLP and CV models.

[0005] In view of the above limitations and shortcomings of the prior art, the present invention proposes a short text understanding method based on a reusable general CNN model. The present invention solves the problems of isolation of multimodal information processing, lack of general models, difficulty in data fusion, and inconsistency in training methods for NLP and CV tasks in the prior art. By proposing an innovative method for processing natural language processing (NLP) tasks using computer vision (CV) technology, the present invention demonstrates the feasibility of building a general model that is applicable to a variety of information forms, reduces system complexity, improves the scalability and application potential of the model, and fills the key gaps in the prior art.

[0006] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0007] A short text understanding method based on a reusable general CNN model uses a CNN framework for parsing short text content to perform short text understanding; the CNN framework for parsing short text content includes: an upstream component and a downstream component; the upstream component uses a pre-trained language model to convert raw text input into dense word embeddings; the downstream component uses a task-specific CNN model to receive dense word embeddings and generate final predictions for specific NLP tasks.

[0008] Furthermore, the upstream component uses the pre-trained language model L to generate a dense word embedding e of dimension m; for the short text t=(w1,w2,...,w i ) for each word w, the mapping process is defined as L(w i )=>e i ,e i ∈R m , where e i is an m-dimensional real-valued vector, R m is a collection of vectors.

[0009] 1. The short text understanding method based on a reusable general CNN model according to claim 2 is characterized in that: the short text is represented as a two-dimensional 'image' P, which is formed by stacking all embeddings together; this process is represented as P = Stack (e1, e2, ..., e i ); To prevent information loss in this process, the order of embeddings remains consistent with the original text sequence.

[0010] Furthermore, the downstream component uses a CNN-based downstream model to receive the generated text “image” P and make a final prediction y; the goal of the downstream component is to learn the following function F, as shown in formula (1):

[0011]

[0012] A CNN-based network is used to interpret the text content; the generated two-dimensional text image is processed through multiple convolution kernels k1-k i Scan several times to extract semantic information; different convolution kernels have different widths d i , and share the same height h; the height h of the convolution kernel is equal to the size of the word embedding; so that the convolution kernel summarizes d at each step i The information of consecutive words, the convolution process is shown in formula (2);

[0013]

[0014] Where h is the input signal, g is the convolution change process, and the specific process of the downstream component is shown in formula (3);

[0015]

[0016] where w ij Corresponding to the input signal x ij After a series of consecutive convolution operations, pooling operations and fully connected layers are applied to summarize all the extracted information and generate the final prediction.

[0017] Furthermore, the adopted loss function uses a one-dimensional bounding box to mark key information; the bounding box has only one dimension, and the corresponding formula is B(c index ,1), the first item is the center index of the selected text sequence, and the second item is the length of the selected text; the loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in formula (4);

[0018]

[0019] Where MSE is the mean squared error loss; the first term is used to measure the difference between the predicted center and the true center, and the second term is used to measure the difference between the length of the predicted keyword / phrase and the true length.

[0020] Furthermore, the specific process of using the CNN framework for parsing short text content to perform short text understanding includes:

[0021] Data input, including short text data and short text metadata; the short text metadata includes a timestamp of the short text, user information, and additional context information;

[0022] Data preprocessing, including data cleaning, data normalization and feature extraction; data cleaning includes removing noise and processing or filling missing data;

[0023] Data normalization: including converting text data into a unified format, and vocabulary normalization: performing stemming or lemmatization so that the same words appear in a consistent form;

[0024] Feature extraction, including: generating word embeddings: using a pre-trained language model to convert short texts into dense word embedding vectors; and extracting key information: identifying key information and sentiment features in short texts, including marking key information using 1D bounding box techniques;

[0025] Model training: select a pre-trained language model as input; then select a downstream model based on the task;

[0026] Finally, the trained model is used to perform short text understanding.

[0027] Furthermore, 1D bounding boxes are used to locate and mark key sentiment words or information fragments in short texts to achieve keyword localization.

[0028] And, a short text understanding system based on a reusable general CNN model, based on a CNN framework for parsing short text content, including: an upstream component and a downstream component; the upstream component uses a pre-trained language model to convert raw text input into dense word embeddings; the downstream component uses a task-specific CNN model to receive dense word embeddings and generate final predictions for specific NLP tasks.

[0029] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the short text understanding method based on a reusable general CNN model as described above are implemented.

[0030] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the short text understanding method based on a reusable general CNN model as described above.

[0031] Compared with the prior art, the present invention and its preferred embodiments have at least the following advantages:

[0032] 1. Strong versatility:

[0033] This method can handle both natural language processing (NLP) and computer vision (CV) tasks without large-scale adjustments. This cross-domain application potential makes the model more versatile and able to adapt to different types of multimodal data (such as text, pictures, audio and video, etc.).

[0034] 2. Efficient processing of short texts:

[0035] Compared with traditional RNN models (such as Bi-LSTM and Bi-GRU), convolutional neural networks (CNNs) are more efficient in processing short texts (such as Twitter short texts). CNN extracts semantic information at different levels through convolution kernels and can process a large amount of short text data in a short time, which is particularly suitable for online public opinion analysis.

[0036] 3. Key information positioning capability:

[0037] By using a method similar to the "bounding box" in computer vision, the model can not only classify text, but also accurately locate words or phrases in the text that express key information. This advantage makes it particularly effective when processing text that requires high-precision analysis, such as the identification of specific sentiment words in sentiment analysis.

[0038] 4. Avoid efficiency issues in long sequence processing:

[0039] Since CNN is based on local convolution, it does not rely on global temporal information, so it does not have the problem of gradient vanishing or gradient exploding when processing long text or long sequence data like RNN. Therefore, CNN is more stable in text understanding tasks, especially when processing long sequences.

[0040] 5. Lower computing cost:

[0041] Compared with the RNN model that needs to save long-term dependencies, CNN has advantages in parallel computing. Its computing cost and resource requirements are usually lower. Therefore, it can increase the running speed and reduce computing resource consumption when processing large-scale data.

[0042] Its application scenarios include but are not limited to:

[0043] 1. Sentiment Analysis:

[0044] The model can be applied to sentiment analysis tasks to help companies, researchers or marketing teams analyze sentiment trends on social media and understand users' attitudes towards an event, product or topic. By accurately locating sentiment words in short texts, the model can provide more interpretable and accurate sentiment classification results.

[0045] 2. Disaster prediction:

[0046] The model can be applied to data analysis on social media platforms (such as Twitter) to help identify short texts about real disasters, thereby providing early warnings for emergency management and disaster response. The model can quickly locate short texts related to disasters in a large amount of social media content, helping governments and institutions obtain key information in a timely manner.

[0047] 3. Multimodal data processing:

[0048] The versatility of this model enables it to process a variety of multimodal data, not just text. It can be used in e-commerce, online learning, digital health and other fields to analyze content in multiple data forms such as text, images, and videos, and improve the ability to process complex information.

[0049] 4. Academic research and interdisciplinary applications:

[0050] This method provides a bridge between computer vision and natural language processing, which is suitable for the exploration of interdisciplinary applications in academic research. Researchers can use this model to share algorithms and technologies between different fields (such as image processing, text analysis, etc.) to achieve knowledge transfer.

[0051] In summary, the main advantages of the present invention are its efficient processing of short texts, strong key information location capabilities, low computing resource requirements, and cross-domain versatility. It has a wide range of uses, including disaster prediction, sentiment analysis, multimodal data processing, and interdisciplinary applications in academic research. This makes the product suitable for use in multiple scenarios such as social media data analysis, emergency response, and market research. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0053] Figure 1 The figure is an overall flow chart of an embodiment of the present invention.

[0054] Figure 2 This is a diagram of the algorithm framework of an embodiment of the present invention.

[0055] Figure 3 This is a diagram showing an example of the operation of embedding stacking according to an embodiment of the present invention.

[0056] Figure 4 4 is a network structure diagram of the downstream components of an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0058] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0060] The embodiment of the present invention designs a CNN framework for parsing short text content. Its main process is as follows Figure 1 As shown, the following is a detailed introduction:

[0061] 1. Framework structure

[0062] The proposed framework consists of two components: upstream component and downstream component. The overall architecture of the framework is as follows Figure 2 As shown in Figure 2. The upstream component is a pre-trained language model, which is used to convert raw text input into dense word embeddings. The downstream component is a task-specific model, which receives dense word embeddings and generates final predictions for a specific NLP task.

[0063] 2. Problem Definition

[0064] In order to clearly define the short text content understanding task, the following definition is made:

[0065] (1) Upstream components:

[0066] In the present invention, a pre-trained language model L is used to generate a dense word embedding e with dimension m. For a short text t=(w1,w2,...,w i ) for each word w, the mapping process can be defined as L(w i )=>e i ,e i ∈R m , where e i is an m-dimensional real-valued vector, R mis the set of these vectors.

[0067] (2) Short text representation:

[0068] In this invention, a short text is represented as a two-dimensional 'image' P, which is formed by stacking all embeddings together. This process is represented as P = Stack (e1, e2, ..., e i ). To prevent information loss during this process, the order of embedding is kept consistent with the original text sequence. Figure 3 .

[0069] (3) Downstream components:

[0070] Since different NLP tasks have different goals, the design of downstream components is usually designed according to the different tasks. Usually, recursive neural networks and their variants are used to capture temporal information. However, the present invention uses a CNN-based downstream model. The downstream component receives the generated text "image" P and makes a final prediction y. Therefore, the goal of the downstream component is to learn the following function F, as shown in formula (1):

[0071]

[0072] Inspired by computer vision research and the n-gram model, this paper uses convolutional technology to design a CNN-based network to interpret text content. The generated two-dimensional text image is processed through multiple convolution kernels k1-k i Scan several times to extract semantic information. Different convolution kernels have different widths d i , but share the same height h. The height h of the convolution kernel is equal to the size of the word embedding. With these settings, the convolution kernel can summarize d at each step i The convolution process is similar to generating n-gram samples. The convolution process is shown in formula (2).

[0073]

[0074] Where h is the input signal and g is the convolution process. The convolution operation extracts fine-grained information at different levels in a similar way. The specific process can be further shown in formula (3).

[0075]

[0076] where w ij Corresponding to the input signal x ij The weight of , b is the bias term. After a series of consecutive convolution operations, pooling operations and fully connected layers are applied to summarize all the extracted information and generate the final prediction. The network structure of the proposed CNN-based downstream components is shown in Figure 4shown.

[0077] (4) One-dimensional bounding box:

[0078] Furthermore, in the present invention, in addition to enabling the proposed model to understand the content of short texts, the loss function is improved so that the model can identify which phrases or words in the short texts express key information. In text processing, a one-dimensional "bounding box" is used to mark key information. Therefore, the proposed model not only classifies a given text into a predefined category, but also highlights which words / phrases express key information that trigger the model to make a decision. The main difference between the bounding box used in the present invention and the bounding box in common computer vision tasks is that the bounding box has only one dimension). The formula for the 1D bounding box is B(c index ,1), the first term is the center index of the selected text sequence, and the second term is the length of the selected text. The loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in formula (4).

[0079]

[0080] Where MSE is the mean squared error loss. The first term measures the difference between the predicted center and the true center, and the second term measures the difference between the length of the predicted keyword / phrase and the true length.

[0081] The following further proposes the application process based on the above design:

[0082] 1. Data Input

[0083] (1) Short text data

[0084] The system receives short text data from social media platforms (such as Twitter), including users' short text contents.

[0085] (2) Short text metadata

[0086] Includes the timestamp of the short text, user information (such as username, geographic location, etc.) and possible additional contextual information (such as keywords).

[0087] 2. Data Preprocessing

[0088] (1) Data cleaning

[0089] Noise removal: Remove unnecessary information such as URLs, emoticons, stop words, etc. from short texts.

[0090] Handling missing values: Processing or filling in missing data.

[0091] (2) Data normalization

[0092] Unified format: Convert text data into a unified format for easy subsequent processing.

[0093] Lexical normalization: perform stem extraction or lemmatization so that the same words appear in a consistent form.

[0094] (3) Feature extraction

[0095] Generate word embeddings: Use pre-trained language models (such as BERT, Word2Vec) to convert short texts into dense word embedding vectors.

[0096] Extracting key information: Identifying key information and sentiment features in short texts, which may include using 1D bounding box techniques to mark key information.

[0097] 3. Model training

[0098] (1) Model selection

[0099] Choose a pre-trained model: Use different pre-trained language models (such as BERT, GloVe, Word2Vec) as input.

[0100] Select downstream models: Select different downstream models according to the task, such as Bi-LSTM, Bi-GRU, CNN, etc.

[0101] (2) Training process

[0102] Adjust model parameters: such as learning rate, batch size, training rounds, etc., to optimize model performance through tuning.

[0103] Cross-validation: Use cross-validation method to evaluate the generalization ability of the model.

[0104] Test set evaluation: Evaluate the final performance of the model on the test set and make iterative adjustments.

[0105] 4. Real-time Application

[0106] (1) Sentiment Analysis

[0107] Analyze short text sentiment: Use the trained model to analyze the sentiment tendency (such as positive, negative) of new short text in real time.

[0108] (2) Disaster prediction

[0109] Identify disaster-related information: Use the model to detect in real time whether a short text involves a disaster event and make corresponding predictions.

[0110] (3) Keyword Targeting

[0111] Highlight key information: Use 1D bounding box technology to locate and mark key sentiment words or information fragments in short texts.

[0112] 5. Results display

[0113] (1) Recommendations and alerts

[0114] Display analysis results: Display the results of sentiment analysis, disaster prediction, etc. to users or relevant departments.

[0115] Generate alerts: For disaster prediction tasks, issue alerts in a timely manner to facilitate response.

[0116] (2) User Feedback

[0117] Collect feedback: Further optimize the model and system based on user feedback.

[0118] Test Examples

[0119] 1. Dataset

[0120] Two short text datasets were used in the experiment, both of which are from the Twitter platform and open to the public. Both datasets consist of informal short texts collected from the Twitter platform. The reasons for choosing these two datasets are:

[0121] (1) Processing informal text to demonstrate that the model proposed in this paper is general enough to handle complex NLP problems.

[0122] (2) Use short texts to demonstrate the excellent ability of the model proposed in this paper in information extraction.

[0123] The first dataset contains more than 10,000 short texts, some of which discuss real disaster events. The ratio of disaster-related to non-disaster-related short texts is 43:57. The dataset contains the original text, keywords, and the location information of the short text. The keywords and location information may be empty; only the original text is used to train the model. The second dataset contains 30,000 short texts with or without sentiment information. The ratio of neutral, positive, and negative samples is 41:31:28, respectively. The dataset contains the original text, as well as text fragments with sentiment. Again, only the original text is used to train the model. The original text is collected directly from the Twitter platform and has not been preprocessed. Table 1 shows examples of the two datasets.

[0124] Table 1

[0125]

[0126] 2. Evaluation Metrics

[0127] In order to reflect the performance of the model from different perspectives, three different types of evaluation indicators are used in the experiment. The first evaluation indicator is accuracy, as shown in formula (5), which directly reflects the proportion of correct predictions generated by each model. However, for unbalanced real data distribution, this indicator may not be suitable for comparing the effectiveness of the model.

[0128]

[0129] The second evaluation metric is the area under the curve (AUC) value. AUC reflects the model's ability to distinguish different types of information (i.e., whether it is a disaster in task one, and whether it is a different emotion in task two). When dealing with multi-class classification tasks (sentiment analysis tasks), the "one-to-one" strategy was used in the experiment. The AUC value is the area under the receiver operating characteristic curve (ROC).

[0130] The last evaluation metric is F-score, which is the harmonic mean of recall and precision as shown in formula (6). Since there is a trade-off between recall and precision, the quality of a model cannot be judged solely by high recall or high precision. Therefore, using F-score is a better choice for model comparison.

[0131]

[0132] 3. Baseline Model

[0133] In the experiments, different pre-trained models were used to study the effectiveness of applying computer vision (CV) solutions to natural language processing (NLP) problems. Specifically, for the upstream components, the following pre-trained language models were introduced:

[0134] (1) Word2Vec: This model has high optimization efficiency, but can only model local semantic information within a predefined window.

[0135] (2) GloVe: This model combines the advantages of LSA and Word2Vec. It uses the co-occurrence matrix to simultaneously model local and global semantic information.

[0136] (3) Bert: Bert and its variants can dynamically model semantic information. According to the original research, multi-task fine-tuning methods can be used to train the model, which will further improve the performance.

[0137] For downstream components, the following models are compared in terms of their effectiveness in understanding the content of short texts:

[0138] (1) Bi-GRU: Bidirectional Gated Recurrent Unit Neural Network.

[0139] (2) Bi-LSTM: Bidirectional long short-term memory neural network.

[0140] (3) The CNN-based model proposed in this invention.

[0141] 4. Experimental Setup

[0142] In the present invention, all models are implemented using the PyTorch framework. The Bert and Word2Vec pre-trained language models are implemented through Transformer and Gensim, respectively; the GloVe model is reproduced through its pre-trained word vector. The CNN-based model proposed in this embodiment uses six different convolution kernels with 128 output channels. Each convolution output uses ReLU as the activation function, and there are four consecutive convolution layers. The dimensions of the Bi-LSTM and Bi-GRU hidden layers are set to 256. The disaster prediction task uses the sigmoid function to generate the final prediction, and the sentiment analysis task uses the softmax function to generate the final prediction. All other settings strictly follow the original work, or directly use the default settings of the PyTorch framework. To prevent overfitting, an early stopping mechanism is used in all training processes.

[0143] Before using the language model to convert short text content into dense vectors, the raw text undergoes multiple stages of NLP data cleaning and normalization, including removing stop words and lemmatization, removing URLs and emoticons, etc.

[0144] 5. Experimental results and discussion

[0145] Table 2 shows the effectiveness comparison of different downstream components in two different NLP tasks. Table 3 reports the effectiveness of different upstream components. Since similar results were obtained from the two datasets, only the experimental results of the disaster prediction task are shown here (Table 3). Table 4 shows the one-dimensional bounding box used to locate key information.

[0146] Table 2

[0147]

[0148] (1) Effectiveness of the proposed CNN model

[0149] According to the results in Table 2, it can be easily concluded that under the same BERT upstream component, the proposed CNN-based downstream component performs competitively with mainstream NLP solutions in all criteria of two different tasks (marked with bold text). For the disaster prediction task, the Bi-LSTM model slightly outperforms the Bi-GRU model, while in the sentiment analysis task, the Bi-GRU model greatly outperforms the Bi-LSTM model. This improvement is due to the structural differences between LSTM units and GRU units. According to the original research on LSTM and GRU, LSTM tends to remember longer semantic information. Therefore, in different tasks, including too much information in the modeling process does not always improve model performance. The disaster prediction task requires understanding the entire short text to predict whether a given short text is related to a disaster. It is difficult to infer whether a short text is related to a disaster based on a short text fragment or phrase alone. As shown in the following example: "All residents asked to'shelter inplace'are beingnotifiedby officers.No other evacuation or shelter in place orders areexpected."

[0150] Remembering more words or longer text sequences helps to understand the context of the entire short text. Therefore, for such tasks, LSTM-based models are better than GRU-based models.

[0151] As for sentiment analysis tasks, in most cases, sentiment is only a part of the content of a short text. The following is an example of a short text with negative sentiment, where a user posted an event and expressed his feelings: "Grrr..stupidinternet connection ruined a great scrabble game."

[0152] The interpretation of user sentiment relies on short text sequences or phrases. Inferring from longer text sequences may have a negative impact on model performance, because in the above example, the phrase "a great scrabble game" expresses a positive sentiment, while the short text itself contains negative sentiment. Therefore, under the same setting, the Bi-GRU model far outperforms the Bi-LSTM model in the second task.

[0153] By using the proposed CNN-based model, the above-mentioned long and short sequence modeling problem can be avoided. When using the CNN model, the length of the word sequence to be modeled in each step can be flexibly controlled by setting the convolution kernel size of the first convolution layer. The configuration of the convolution kernel size can be determined based on domain knowledge or preliminary experiments.

[0154] (2) Importance of upstream language models

[0155] The impact of using different language models in upstream components on the performance of the framework was also studied in the test. From Table 3, it can be concluded that the framework using the pre-trained Bert model outperforms the framework using the GloVe model or the Word2Vec model (marked in bold text). The Bert model can capture more semantic information. A better language model means that downstream components can access more useful information. From a CV perspective, a better language model can generate "higher resolution text images", which is crucial for mining details from "text images". As pointed out in the literature, short texts usually contain more information per character, which may be the result of Twitter's unique abbreviations and less consistent writing style. Therefore, in order to better interpret the content of short texts, it is necessary to use a powerful language model to maximize the retention of semantic information.

[0156] (3) Locating key information

[0157] Using the second dataset, the test also further studied whether the proposed model can identify and locate key information (i.e., keywords). Keywords are selected by a designed one-dimensional bounding box (for NLP tasks, this is usually done by using an attention mechanism). Due to space limitations, this example only randomly shows five positive and five negative results, as shown in Table 4. The "True Value Keywords" column shows the annotated true keywords containing sentiment information; the "Selected Text" column shows the keywords selected by the bounding box. It can be clearly seen that the proposed model is not only able to understand the sentiment meaning of short text content, but also to identify which words express this meaning. The only drawback of the bounding box is that it tends to select text that is longer than the true value (by comparing the second and third columns). This is because longer text usually contains more information and provides more support for the bounding box to make a decision.

[0158] Table 3

[0159] Model Accuracy F1 AUC Bert+ proposed model 0.8238 0.8169 0.8798 GloVe+ proposed model 0.8173 0.8145 0.8723 Word2Vec+ proposed model 0.6852 0.6762 0.7271 Bert+Bi-GRU 0.8108 0.8044 0.8698 GloVe+Bi-GRU 0.7667 0.7598 0.8097 Word2Vec+Bi-GRU 0.6819 0.6760 0.7348 Bert+Bi-LSTM 0.8172 0.8128 0.8695 GloVe+Bi-LSTM 0.8107 0.8039 0.8588 Word2Vec+Bi-LSTM 0.6770 0.6625 0.7283

[0160] Table 4

[0161]

[0162]

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0168] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0169] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of short text understanding methods based on a reusable general CNN model under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A short text understanding method based on a reusable general CNN model, characterized by: A CNN framework for parsing short text content is used to perform short text understanding; the CNN framework for parsing short text content includes: an upstream component and a downstream component; the upstream component uses a pre-trained language model to convert raw text input into dense word embeddings; the downstream component uses a task-specific CNN model to receive dense word embeddings and generate final predictions for specific NLP tasks.

2. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The upstream component uses the pre-trained language model L to generate a dense word embedding e of dimension m; for a short text t=(w1,w2,...,w i ) for each word w, the mapping process is defined as L(w i )=>e i ,e i ∈R m , where e i is an m-dimensional real-valued vector, R m is a collection of vectors.

3. The short text understanding method based on the reusable general CNN model according to claim 2 is characterized in that: The short text is represented as a two-dimensional 'image' P, formed by stacking all embeddings together; this process is represented as P = Stack(e1, e2, ..., e i ); To prevent information loss in this process, the order of embeddings remains consistent with the original text sequence.

4. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The downstream component uses a CNN-based downstream model to receive the generated text "image" P and make a final prediction y; the goal of the downstream component is to learn the following function F, as shown in formula (1): A CNN-based network is used to interpret the text content; the generated two-dimensional text image is processed through multiple convolution kernels k1-k i Scan several times to extract semantic information; different convolution kernels have different widths d i , and share the same height h; the height h of the convolution kernel is equal to the size of the word embedding; so that the convolution kernel summarizes d at each step i The information of consecutive words, the convolution process is shown in formula (2); Where h is the input signal, g is the convolution change process, and the specific process of the downstream component is shown in formula (3); where w ij Corresponding to the input signal x ij After a series of consecutive convolution operations, pooling operations and fully connected layers are applied to summarize all the extracted information and generate the final prediction.

5. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The loss function used uses a one-dimensional bounding box to mark key information; the bounding box has only one dimension, and the corresponding formula is B(c index ,1), the first item is the center index of the selected text sequence, and the second item is the length of the selected text; the loss function formula used to measure the difference between the predicted value and the true value of the bounding box is shown in formula (4); Where MSE is the mean squared error loss; the first term is used to measure the difference between the predicted center and the true center, and the second term is used to measure the difference between the length of the predicted keyword / phrase and the true length.

6. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: The specific process of using the CNN framework to parse short text content to perform short text understanding includes: Data input, including short text data and short text metadata; the short text metadata includes a timestamp of the short text, user information, and additional context information; Data preprocessing, including data cleaning, data normalization and feature extraction; data cleaning includes removing noise and processing or filling missing data; Data normalization: including converting text data into a unified format, and vocabulary normalization: performing stemming or lemmatization so that the same words appear in a consistent form; Feature extraction, including: generating word embeddings: using a pre-trained language model to convert short texts into dense word embedding vectors; and extracting key information: identifying key information and sentiment features in short texts, including marking key information using 1D bounding box techniques; Model training: select a pre-trained language model as input; then select a downstream model based on the task; Finally, the trained model is used to perform short text understanding.

7. The short text understanding method based on a reusable general CNN model according to claim 1, characterized in that: 1D bounding boxes are used to locate and mark key sentiment words or information fragments in short texts to achieve keyword localization.

8. A short text understanding system based on a reusable general CNN model, characterized by: A CNN framework based on parsing short text content includes: an upstream component and a downstream component; the upstream component adopts a pre-trained language model to convert raw text input into dense word embeddings; the downstream component adopts a task-specific CNN model to receive dense word embeddings and generate final predictions for specific NLP tasks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the short text understanding method based on a reusable general CNN model as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the short text understanding method based on a reusable general CNN model as described in any one of claims 1 to 7 are implemented.

Images