Social text enhancement method and system based on multimodal retrieval and keyword extraction

Abstract

This application proposes a social text enhancement method based on multimodal retrieval and keyword extraction, comprising: S1, extracting keywords from sample sentences of different categories using a category keyword extraction algorithm; S2, training a sentence generation model using an RNN model combined with a self-attention mechanism, and controlling the sentence generation direction of the generation model according to the keywords of the corresponding categories of the training samples; S3, inputting the original sentence into the generation model to generate a first generated sentence with text enhancement; S4, based on a multimodal retrieval algorithm, determining whether the first generated sentence contains keywords from the keyword file; if so, finding the keyword to be replaced in the first generated sentence, and retrieving synonyms of the keyword to be replaced for replacement, thereby generating multiple second generated sentences with data enhancement. This application has the effect of controlling the generation direction and number of generated sentences of the generation model.

Description

Technical Field

[0001] This application relates to the technical field of text data enhancement, specifically to a social text enhancement method and system based on multimodal retrieval and keyword extraction. Background Technology

[0002] With the advancement of technology and the development of artificial intelligence, AI models are gradually permeating people's lives. Among them, text multi-classification tasks, as an important branch of natural language processing, have significant applications in various aspects of production and life. From the training process of text multi-classification models, data collection plays a crucial guiding role in the final model results. The quality and richness of the data, to a certain extent, determine the final model's performance. However, the following problems are often encountered during data collection: firstly, the problem of insufficient data, and secondly, the problem of imbalanced data categories. The former can lead to overfitting during model training, while the latter may result in poor classification performance of the model on data with few samples.

[0003] To address this type of problem, researchers have used text data augmentation to solve it. Data augmentation is generally divided into three categories: (1) EDA (easy data augmentation), which mainly uses two methods to augment data: word replacement (synonym replacement) and sentence noise addition (random word insertion, random word swapping, random word deletion, syntax tree operations). (2) Data generation, such as seq-to-seq, Language Models and other generative models. (3) Adversarial generation, which uses adversarial models such as GAN to generate sentences.

[0004] The three data augmentation methods mentioned above can be summarized in two aspects. The first is supervised data augmentation, namely EDA data augmentation. This type of augmentation method has the characteristics of rapid data generation and controllable generation direction. However, since adding noise to words is only a modification of words based on the original text, the generated sentences are characterized by monotonous content and high repetition rate. The second is unsupervised data augmentation, namely the adversarial generation method and seq-to-seq method mentioned above. The advantage of this generation method is that the generated sentences are rich in content and easy to train. The disadvantage is that the number of generated sentences is in a one-to-one form, that is, one sentence is generated from one input sample, and the direction of sentence generation is uncontrollable.

[0005] Therefore, to address the above problems, this application proposes a novel data augmentation method based on multimodal retrieval and topic RNN models, which solves the problem of uncontrollable generation direction and generation quantity in sentence generation models. Summary of the Invention

[0006] To address the aforementioned technical issues, this application proposes a social text enhancement method and system based on multimodal retrieval and keyword extraction.

[0007] According to the first aspect of this application, a social text enhancement method based on multimodal retrieval and keyword extraction is proposed, comprising the following steps:

[0008] S1. Use the category keyword extraction algorithm to extract keywords from sample sentences of different categories to obtain a keyword file containing sets of keywords of different categories;

[0009] S2. Using an RNN model combined with a self-attention mechanism, a sentence generation model is trained using training samples, and the sentence generation direction of the generation model is controlled according to the keywords in the keyword set corresponding to the category of the training samples.

[0010] S3. Input the original sentence into the generation model to generate a first text-enhanced generated sentence; and

[0011] S4. Based on the multimodal retrieval algorithm, determine whether the first generated statement contains the keyword in the keyword file. If so, find the keyword that needs to be replaced in the first generated statement and retrieve the synonym of the keyword that needs to be replaced for replacement, thereby generating multiple data-enhanced second generated statements.

[0012] Preferably, step S1 specifically includes:

[0013] S11. Divide the sample sentences into 2-character words and 3-character words, calculate the cohesion of each word using the cohesion algorithm, and perform word segmentation operation again on words with cohesion below a preset threshold to obtain the final word set.

[0014] S12. Use the information entropy algorithm and TF-IDF algorithm to measure the effectiveness of each word in the word set, and select keywords based on the measurement results.

[0015] Preferably, the generation model sequence includes an embedding layer, a bidirectional LSTM model, an attention layer, and a unidirectional LSTM model, and step S2 specifically includes:

[0016] S21. After the training sample is segmented into words in the embedding layer, it is converted into a first word vector. Then, keywords are randomly selected from the keyword set corresponding to the category of the training sample and converted into a second word vector.

[0017] S22. Use the bidirectional LSTM model to bidirectionally concatenate the first word vector and the second word vector for output.

[0018] S23. The attention layer performs dimensionality reduction on the output first word vector and second word vector using an attention mechanism and outputs a first output vector and a second output vector, respectively.

[0019] S24. The first output vector and the second output vector are concatenated to obtain an encoded vector, and the unidirectional LSTM model decodes the encoded vector to generate a sentence.

[0020] Preferably, in step S22, the last encoded output of the first word vector and the second word vector are jointly encoded using a self-attention mechanism.

[0021] Preferably, step S4 specifically includes:

[0022] S41. Use the collected training corpus to segment words and convert them into third word vectors. Use the Skip-Gram model to train the third word vectors to obtain a word vector model.

[0023] S42. Input the first generated statement into the word vector model, determine and find the keyword that needs to be replaced, and find the corresponding specific word vector in the word vector model according to the keyword that needs to be replaced.

[0024] S43. Calculate the similarity between the specific word vector and the other third word vectors using cosine similarity, and find the synonyms based on the calculation results.

[0025] Preferably, step S43 specifically includes: dividing the third word vector in the word vector model into different categories according to the keyword file, using a hash function to save the third word vectors of different categories into hash addresses at different locations, calculating the similarity between the specific word vector and the third word vector in the corresponding hash address using cosine similarity, and finding the synonyms based on the calculation results.

[0026] Preferably, step S12, which involves selecting keywords based on the measurement results, specifically includes: performing a weighted summation of the information entropy value and TF-IDF value of the words in the word set to obtain the feature score of the words in the word set, and selecting keywords based on the feature score.

[0027] Preferably, in step S4, according to a first preset number, the first preset number of keywords to be replaced are randomly sampled from all the keywords contained in the first generated statement, and according to a second preset number, the second generated statement of the second preset number is generated.

[0028] According to a second aspect of this application, a social text enhancement system based on multimodal retrieval and keyword extraction is proposed, comprising:

[0029] The keyword file acquisition module is configured to extract keywords from sample sentences of different categories using a category keyword extraction algorithm, and obtain a keyword file containing sets of keywords of different categories.

[0030] The generative model building module is configured to use an RNN model combined with a self-attention mechanism to train a sentence generation model using training samples, and to control the sentence generation direction of the generative model according to the keywords in the keyword set corresponding to the category of the training samples.

[0031] The text enhancement module is configured to input the original statement into the generation model and generate a first generated statement with text enhancement.

[0032] The data augmentation module is configured to determine whether the first generated statement contains keywords from the keyword file based on a multimodal retrieval algorithm. If so, it finds the keyword to be replaced in the first generated statement and retrieves synonyms of the keyword to be replaced for replacement, thereby generating multiple data-augmented second generated statements.

[0033] According to a third aspect of this application, a computer-readable storage medium is proposed that stores a computer program, which, when executed by a processor, implements the social text enhancement method based on multimodal retrieval and keyword extraction as described in the first aspect of this application.

[0034] This application proposes a social text enhancement method and system based on multimodal retrieval and keyword extraction. It constructs a sentence generation model by using an RNN model combined with a self-attention mechanism, and incorporates keywords as auxiliary judgments to control the sentence generation direction of the generation model. At the same time, it uses a multimodal retrieval algorithm to supplement the data enhancement by using synonym replacement text generation, which to some extent solves the problem of insufficient generation quantity of the generation model. Attached Figure Description

[0035] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of this application. Other embodiments and many anticipated advantages of these embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0036] Figure 1 This is a flowchart of a social text enhancement method based on multimodal retrieval and keyword extraction according to an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the encoder structure according to a specific embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the decoder according to a specific embodiment of this application;

[0039] Figure 4 This is a block diagram of a social text enhancement system based on multimodal retrieval and keyword extraction according to an embodiment of this application.

[0040] Figure labeling: 1. Keyword file acquisition module; 2. Generative model construction module; 3. Text enhancement module; 4. Data enhancement module. Detailed Implementation

[0041] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0043] According to the first aspect of this application, a social text enhancement method based on multimodal retrieval and keyword extraction is proposed. Figure 1 A flowchart illustrating a social text enhancement method based on multimodal retrieval and keyword extraction according to an embodiment of this application is shown, such as... Figure 1 As shown, the method includes the following steps:

[0044] S1. Use a category keyword extraction algorithm to extract keywords from sample sentences of different categories to obtain a keyword file containing sets of keywords of different categories.

[0045] Specifically, step S1 includes:

[0046] S11. Divide the sample sentences into 2-character words and 3-character words, calculate the cohesion of each word using the cohesion algorithm, and perform word segmentation again on words with cohesion below the preset threshold to obtain the final word set.

[0047] S12. Use the information entropy algorithm and TF-IDF algorithm to measure the effectiveness of each word in the word set, and select keywords based on the measurement results.

[0048] In a specific embodiment, the principle of the category keyword extraction algorithm is as follows:

[0049] To ensure the controllability and quality of the subsequent generative model, keywords are first extracted from the sample sentences of each category. This application employs the cohesion algorithm, information entropy algorithm, and TF-IDF algorithm to extract keywords. This application uses a combination of word cohesion and information entropy to organize words. The advantage of this approach is that it does not rely on existing dictionaries, and the extracted words are more representative. Assuming w is a candidate word, and the set of characters in w is wc = {w1, w2, ... wn}, then the formula for calculating the cohesion of word w is:

[0050]

[0051] The sample sentences are divided into 2-character and 3-character words using subset partitioning. A cohesion degree algorithm is used to calculate the cohesion degree of each word, and words with a cohesion degree below a preset threshold are segmented again. To further filter the importance of words after cohesion degree segmentation, this application uses the information entropy algorithm and the TF-IDF algorithm to measure word effectiveness. The formula for calculating information entropy is:

[0052]

[0053]

[0054] E = min(EL, ER)

[0055] Information entropy represents the viscosity between a word and its surrounding characters. The lower the information entropy, the higher the viscosity between the word and its surrounding characters, and the greater the likelihood that the word will become a complete word. Here, wl represents the left neighbor word, wr represents the right neighbor word, EL represents the information entropy between the word and its left neighbor, and ER represents the information entropy between the word and its right neighbor. TF-IDF is also an algorithm for measuring the importance of words; a higher TF-IDF value indicates a more unique word. The formula for calculating TF-IDF is:

[0056]

[0057] Where, n k Let represent the number of times the k-th word appears in the article, n represent the sum of the occurrences of all words, and N represent the total number of articles. k This represents the number of articles containing the k-th word.

[0058] The calculated word information entropy and TF-IDF values ​​are weighted and summed with a weight of 0.5 to obtain the word feature score. The words are then sorted in descending order of feature score, and the top 150 words from each category are manually selected as keywords, thus constructing a keyword file containing sets of keywords from different categories. Table 1 below shows some of the category keywords:

[0059]

[0060] Table 1

[0061] Continue to refer to Figure 1 After step S1,

[0062] S2. An RNN model combined with a self-attention mechanism is used to train a sentence generation model using training samples, and the sentence generation direction of the generation model is controlled by the keywords in the keyword set of the corresponding category of the training samples.

[0063] Specifically, the generation model sequence includes an embedding layer, a bidirectional LSTM model, an attention layer, and a unidirectional LSTM model. Step S2 specifically includes:

[0064] S21. After segmenting the training samples into words in the embedding layer, convert them into the first word vector. Randomly select keywords from the keyword set corresponding to the category of the training samples and convert them into the second word vector.

[0065] S22. Use a bidirectional LSTM model to concatenate the first word vector and the second word vector in both directions for output.

[0066] S23. The attention layer performs dimensionality reduction on the first and second word vectors after output using the attention mechanism and outputs the first and second output vectors respectively.

[0067] S24. The first output vector and the second output vector are concatenated to obtain the encoded vector. The unidirectional LSTM model decodes the encoded vector to generate a sentence.

[0068] In a specific embodiment, the principle of the generative model is as follows:

[0069] RNN models, as a common seq-to-seq model, have good data generation performance. However, a drawback of this generation method is the uncontrollability of sentence generation direction. To compensate for this drawback, this application controls the sentence generation direction by adding keywords to the generative model. The first step involves collecting similar sentence corpora as training samples for the generative model. This application uses a back-translation method (Chinese-English-Chinese) on the target dataset (classified data) and calls Google Translate to generate 10 million similar sentence pairs as training samples. The second step involves constructing the generative model, which mainly consists of an encoder and a decoder. The encoder mainly includes an embedding layer, a bidirectional LSTM model (Bi-LSTM), and an attention layer. The decoder mainly includes a unidirectional LSTM model (LSTM).

[0070] Figure 2 A schematic diagram of the encoder structure according to a specific embodiment of this application is shown, such as... Figure 2 As shown, the encoder first uses an embedding layer to segment the training samples (w1, w2…w…). n The first word vector (h1, h2, ..., hn) is transformed into a first word vector. Simultaneously, m keywords are randomly selected from the keyword set corresponding to the category of the training samples and transformed into a second word vector (k1, k2, ..., km). Then, a bidirectional LSTM model (Bi-LSTM) is used to concatenate the first and second word vectors bidirectionally. The last encoded output of the first word vector and the second word vector are jointly encoded using a self-attention mechanism, effectively connecting the original text and the feature words. An attention layer performs dimensionality reduction on the output first and second word vectors using an attention mechanism, outputting a first output vector c1 and a second output vector c2 respectively. The first and second output vectors are then concatenated to obtain the encoded vector s1. The advantage of this approach is that the weights of keywords relevant to the training samples increase with each training iteration, while the weights of keywords unrelated to the training samples gradually decrease due to self-attention. This ensures, to a certain extent, that the final generated sentence is related to both the current input sentence and the relevant keywords.

[0071] Figure 3A schematic diagram of the decoder according to a specific embodiment of this application is shown, such as... Figure 3 As shown, the decoder uses a one-way LSTM model for sentence decoding. <eos>This is an identifier used to stop the model generation process. Generally, to prevent the model generation iterations from failing to stop, it is set during training. <eos>Identifiers are placed in the space of Tag word vectors, and the length of generated words is controlled to be within 32. The model stops when: ① the length generated by the model is greater than 32; ② the softmax result of the model shows... <eos>Identifier.

[0072] Continue to refer to Figure 1 After step S2,

[0073] S3. Input the original statement into the generative model to generate the first generated statement with text enhancement.

[0074] In a specific embodiment, the original statement is input into the generative model to generate the first generated statement, and the results are shown in Table 2 below:

[0075]

[0076] Table 2

[0077] Continue to refer to Figure 1 After step S3,

[0078] S4. Based on the multimodal retrieval algorithm, determine whether the first generated statement contains the keyword in the keyword file. If so, find the keyword that needs to be replaced in the first generated statement and retrieve the synonym of the keyword that needs to be replaced for replacement, thereby generating multiple data-enhanced second generated statements.

[0079] Specifically, step S4 includes:

[0080] S41. Use the collected training corpus to segment words and convert them into third word vectors. Use the Skip-Gram model to train the third word vectors to obtain the word vector model.

[0081] S42. Input the first generated sentence into the word vector model, determine and find the keywords that need to be replaced, and find the corresponding specific word vectors in the word vector model according to the keywords that need to be replaced.

[0082] S43. Calculate the similarity between a specific word vector and other third-party word vectors using cosine similarity, and find synonyms based on the calculation results.

[0083] In a specific embodiment, the principle of the multimodal retrieval algorithm is as follows:

[0084] Multimodal retrieval algorithms are a method for retrieving associations between different categories. The entire process consists of the following three steps:

[0085] Step 1: Word Vector Model Training. The collected training corpus is segmented into words and converted into third-order word vectors. These third-order word vectors are then trained using a Skip-Gram model to obtain the final word vector model. The Skip-gram model iteratively trains the word vector of the current word based on the words in the predicted context and using a loss function for the predicted words. This application constructs a word vector model using the collected training corpus through word segmentation and Skip-gram word vector training. After inputting the first generated sentence into the word vector model, the model can find the specific word vectors corresponding to the keywords that need to be replaced in the second generated sentence. Simultaneously, the word vector model acts as a thesaurus, allowing the subsequent search for synonyms for the specific word vectors.

[0086] Step 2: Multimodal Retrieval Construction. Based on the keyword file, the third-order word vectors in the word vector model are divided into different categories. A hash function is used to store the third-order word vectors of different categories into different hash addresses. For example, third-order word vectors of the same category are converted into 4-bit binary codes, and then the binary codes are converted into decimal and stored in 15 addresses, thus speeding up word vector retrieval. The hash transformation function is a transformation matrix, the purpose of which is to reduce the dimensionality of the word vectors and transform each value of the word vector into a 01 vector. The specific hash formula is as follows:

[0087]

[0088] Among them, w N For the input word vectors, W 4×N T As a hash transition matrix, after the word vector is converted to 01 encoding, it will be further converted to decimal for binning. For example, if the word vector w1 is converted to 0011 by the hash function, it will be assigned to address 3.

[0089] Storing third-word vectors of different categories separately can speed up word retrieval. For example, a word originally needs to search for synonyms among 10,000 third-word vectors in a thesaurus, but by building a position index using a hash function, the word only needs to search for synonyms among a few or dozens of third-word vectors of the same category at the corresponding hash address, thus greatly speeding up the retrieval.

[0090] After determining the specific word vectors and third-category word vectors corresponding to the keywords to be replaced, the cosine similarity score between the specific word vectors and the third-category word vectors is calculated. These scores are then arranged in descending order, and words with a similarity score greater than 0.8 are selected as the set of similar words. Let w1 be word vector 1 and w2 be word vector 2. The formula for calculating their cosine similarity is as follows:

[0091]

[0092] Step 3: Perform data augmentation using multimodal retrieval based on the preset number of keyword replacements (first preset number) and the number of generated texts (second preset number). First, input the first and second preset numbers. Then, perform word segmentation on the input first generated sentence using the word vector model from Step 1, and count the number of keywords contained in the first generated sentence against the keyword file. Randomly sample all keywords contained in the first generated sentence according to the first preset number to obtain the keywords to be replaced and their corresponding specific word vectors. Perform synonym replacement on the sampled specific word vectors according to the method in Step 2, thereby generating the second preset number of second generated sentences. It should be noted that when the number of synonyms corresponding to a specific word vector is less than the second preset number, synonyms will be randomly repeated to ensure that the second preset number of second generated sentences can be generated.

[0093] Multimodal retrieval technology and a synonym dictionary (word vector model) are used for word replacement. When a word cannot be found in the synonym dictionary, multimodal technology is used to supplement it with synonyms. Multimodal retrieval achieves sub-second performance on datasets with tens of millions of records, which well meets the efficiency requirements of synonym replacement. The sentence generation results are shown in Table 3.

[0094]

[0095] Table 3

[0096] In summary, sentence generation using multimodal retrieval algorithms, as a supplementary method to generative models, has the following characteristics: First, since the word vectors of multimodal retrieval are trained based on proprietary data, the replaced synonyms are more suitable for the current task; second, the synonym dictionary and multimodal retrieval can achieve accurate word replacement generation, and the generated sentences have greater interpretability; third, data generation based on multimodal retrieval can effectively control the proportion and number of sentence variations, achieving controllable data enhancement.

[0097] According to the second aspect of this application, a social text enhancement system based on multimodal retrieval and keyword extraction is proposed, which is built based on the above-mentioned method. Figure 4 A block diagram of a social text enhancement system based on multimodal retrieval and keyword extraction according to an embodiment of this application is shown, such as... Figure 4 As shown, the system includes:

[0098] Keyword file acquisition module 1 is configured to use a category keyword extraction algorithm to extract keywords from sample sentences of different categories, and obtain a keyword file containing sets of keywords of different categories;

[0099] Generative model building module 2 is configured to use an RNN model combined with a self-attention mechanism to train a sentence generation model using training samples, and to control the sentence generation direction of the generative model based on the keywords in the keyword set of the corresponding category of the training samples.

[0100] Text enhancement module 3 is configured to input the original sentence into the generative model and generate the first generated sentence with text enhancement.

[0101] Data augmentation module 4 is configured to determine whether the first generated statement contains keywords from the keyword file based on a multimodal retrieval algorithm. If so, it finds the keyword to be replaced in the first generated statement and retrieves synonyms of the keyword to be replaced for replacement, thereby generating multiple data-augmented second generated statements.

[0102] According to a third aspect of this application, a computer-readable storage medium is proposed that stores a computer program, which, when executed by a processor, implements the social text enhancement method based on multimodal retrieval and keyword extraction as described in the first aspect of this application.

[0103] This application proposes a social text augmentation method and system based on multimodal retrieval and keyword extraction. A sentence generation model is constructed using an RNN model combined with a self-attention mechanism. Keywords are incorporated into the generation model as auxiliary judgments to control the direction of sentence generation. Simultaneously, a synonym replacement text generation method using a multimodal retrieval algorithm is employed as a supplementary data augmentation technique, which to some extent addresses the problem of insufficient sample size generated by the generation model. This application is applicable to text classification preprocessing tasks, especially for samples with imbalanced class distributions, as this data augmentation method effectively reduces the overall impact of uneven sample size on the model.

[0104] In the embodiments of this application, it should be understood that the disclosed technical content can be implemented in other ways. The device / system / method embodiments described above are merely illustrative. For example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0108] It is evident that those skilled in the art can make various modifications and alterations to the embodiments of this application without departing from the spirit and scope of this application. In this manner, this application also aims to cover such modifications and alterations if they fall within the scope of the claims and their equivalents. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are described in mutually different dependent claims does not indicate that a combination of these measures cannot be used for profit. Any reference numerals in the claims should not be considered as limiting the scope.< / eos> < / eos> < / eos>

Claims

1. A social text enhancement method based on multimodal retrieval and keyword extraction, characterized in that, Includes the following steps: S1. Use a category keyword extraction algorithm to extract keywords from sample sentences of different categories, resulting in a keyword file containing sets of keywords from different categories; specifically including: S11. Divide the sample sentences into 2-character words and 3-character words, calculate the cohesion of each word using the cohesion algorithm, and perform word segmentation operation again on words with cohesion below a preset threshold to obtain the final word set. S12. Use the information entropy algorithm and TF-IDF algorithm to measure the effectiveness of each word in the word set, and select keywords based on the measurement results; S2. An RNN model combined with a self-attention mechanism is used to train a sentence generation model using training samples. The sentence generation direction of the generation model is controlled according to the keywords in the keyword set corresponding to the category of the training samples. The generation model sequence includes an embedding layer, a bidirectional LSTM model, an attention layer, and a unidirectional LSTM model, specifically including: S21. After the training sample is segmented into words in the embedding layer, it is converted into a first word vector. Then, keywords are randomly selected from the keyword set corresponding to the category of the training sample and converted into a second word vector. S22. Use the bidirectional LSTM model to bidirectionally concatenate the first word vector and the second word vector for output. S23. The attention layer performs dimensionality reduction on the output first word vector and second word vector using an attention mechanism and outputs a first output vector and a second output vector, respectively. S24. The first output vector and the second output vector are concatenated to obtain an encoded vector, and the unidirectional LSTM model decodes the encoded vector to generate a sentence. S3. Input the original sentence into the generation model to generate a first text-enhanced generated sentence; and S4. Based on the multimodal retrieval algorithm, determine whether the first generated statement contains the keyword in the keyword file. If so, find the keyword that needs to be replaced in the first generated statement and retrieve the synonym of the keyword that needs to be replaced for replacement, thereby generating multiple data-enhanced second generated statements.

2. The method according to claim 1, characterized in that, In step S22, the last encoded output of the first word vector and the second word vector are jointly encoded using a self-attention mechanism.

3. The method according to claim 1, characterized in that, Step S4 specifically includes: S41. Use the collected training corpus to segment words and convert them into third word vectors. Use the Skip-Gram model to train the third word vectors to obtain a word vector model. S42. Input the first generated statement into the word vector model, determine and find the keyword that needs to be replaced, and find the corresponding specific word vector in the word vector model according to the keyword that needs to be replaced. S43. Calculate the similarity between the specific word vector and the other third word vectors using cosine similarity, and find the synonyms based on the calculation results.

4. The method according to claim 3, characterized in that Step S43 specifically includes: dividing the third word vector in the word vector model into different categories according to the keyword file; using a hash function to save the third word vectors of different categories into hash addresses at different locations; calculating the similarity between the specific word vector and the third word vector in the corresponding hash address using cosine similarity; and finding the synonyms based on the calculation results.

5. The method according to claim 1, wherein The step S12 of selecting keywords based on the measurement results specifically includes: weighting and summing the information entropy value and TF-IDF value of the words in the word set to obtain the feature score of the words in the word set, and selecting keywords based on the feature score.

6. The method according to claim 1, characterized in that, In step S4, based on a first preset quantity, the first preset quantity of keywords to be replaced are randomly sampled from all the keywords contained in the first generated statement, and based on a second preset quantity, the second generated statement of the second preset quantity is generated.

7. A social text enhancement system based on multimodal retrieval and keyword extraction, characterized in that, Including the method as described in any one of claims 1-6, further comprising: The keyword file acquisition module is configured to use a category keyword extraction algorithm to extract keywords from sample sentences of different categories, and obtain a keyword file containing sets of keywords of different categories. The generative model building module is configured to use an RNN model combined with a self-attention mechanism to train a sentence generation model using training samples, and to control the sentence generation direction of the generative model according to the keywords in the keyword set corresponding to the category of the training samples. The text enhancement module is configured to input the original sentence into the generation model and generate a first generated sentence with text enhancement. The data augmentation module is configured to determine whether the first generated statement contains keywords from the keyword file based on a multimodal retrieval algorithm. If so, it finds the keyword to be replaced in the first generated statement and retrieves synonyms of the keyword to be replaced for replacement, thereby generating multiple data-augmented second generated statements.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images