Text information completion method and system
By performing semantic role annotation and pre-training language model processing on text information, reconstructing and fusing word vector features, the problem of discretization and fragmentation of text data in complex network environments is solved, and the accuracy of text information prediction and data processing efficiency are improved.
Patent Information
- Application Number
- CN202411509873.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-03-18
AI Technical Summary
In complex network environments, the text data parsed from network traffic is discrete and fragmented, resulting in the detection model based on artificial intelligence that cannot effectively use this type of text features, reducing the efficiency and accuracy of data processing.
By annotating text information in semantic roles, using a pre-trained language model to obtain sub-word vectors, reconstructing them into word vectors, aligning semantic labels, fusing word vector features with semantic labels, and training through prediction models to achieve the completion of text information.
It significantly improves the accuracy of text information prediction, improves the efficiency of data processing, provides richer semantics and structured knowledge, and builds more accurate text representations.
Smart Images
Figure CN119647416A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information processing technology, and in particular to a text information completion method; in addition, the present invention also relates to a text information completion system. Background Art
[0002] In a complex network environment, the target text data parsed from network traffic is discrete and fragmented, which makes it impossible for artificial intelligence-based detection models to effectively use such text features, thereby reducing the efficiency and accuracy of data processing. Summary of the invention
[0003] In order to solve the problems existing in the prior art, at least one embodiment of the present invention provides a text information completion method, which greatly improves the accuracy of text information prediction and the efficiency of data processing. To this end, at least one embodiment of the present invention also provides a text information completion system.
[0004] In a first aspect, an embodiment of the present invention provides a text information completion method, comprising:
[0005] Perform semantic role labeling on text information and obtain subword vectors through pre-trained language model;
[0006] Reconstruct the subword vectors into word vectors and perform semantic label alignment;
[0007] Fusion of the extracted word vector features and their corresponding semantic labels;
[0008] The fused feature vector is input into the prediction model for training to obtain a text information prediction model;
[0009] The text information is completed using the text information prediction model.
[0010] In some embodiments, the present invention provides a method for completing text information, wherein reconstructing the subword vector into a word vector comprises:
[0011] The subword vectors are subjected to feature extraction through a convolutional neural network, and the extracted subword feature vectors are average-pooled and aggregated into an overall global word-level representation.
[0012] In some embodiments, the present invention provides a method for completing text information, wherein the semantic label alignment includes:
[0013] Create an embedding vector for each semantic tag and concatenate the pooled word-level representation with the corresponding embedding tag.
[0014] In some embodiments, a text information completion method provided by the present invention, fusing the extracted word vector features with their corresponding semantic labels, includes:
[0015] The extracted word vector features and their corresponding semantic labels are fused through a multi-layer perceptron.
[0016] In some embodiments, the present invention provides a method for completing text information, the method further comprising:
[0017] Each semantic label sequence is input into the BiGRU layer to obtain the semantic label sequence vector, and all semantic label sequence vectors are connected and input into the fully connected layer.
[0018] In a second aspect, an embodiment of the present invention further provides a text information completion system, including:
[0019] Annotation module, used to annotate text information with semantic roles;
[0020] Pre-training module, used to obtain subword vectors through pre-training language model;
[0021] A reconstruction module, used to reconstruct the subword vectors into word vectors and perform semantic label alignment;
[0022] The fusion module is used to fuse the extracted word vector features with their corresponding semantic labels;
[0023] A training module is used to input the fused feature vector into the prediction model for training to obtain a text information prediction model;
[0024] The text completion module is used to complete the text information through the text information prediction model.
[0025] In some embodiments, the present invention provides a text information completion system, the system further comprising:
[0026] The sequence encoding module is used to input each semantic label sequence into the BiGRU layer to obtain a semantic label sequence vector, and then connect all the semantic label sequence vectors and input them into the fully connected layer.
[0027] In a third aspect, an embodiment of the present invention further provides a text information completion device, comprising at least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, and the executable instructions, when executed by the at least one processor, implement the steps of the method described in any one of the first aspects above.
[0028] In a fourth aspect, an embodiment of the present invention further provides a chip for executing the steps of the method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the steps of the method in the first aspect.
[0029] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects above are implemented.
[0030] It can be seen that a text information completion method and system of an embodiment of the present invention converts text information into subword vectors through a pre-trained language model, then reconstructs the subword vectors into word vectors, and performs semantic label alignment, and fuses the extracted word vector features with their corresponding semantic labels and trains them through a prediction model, thereby providing rich semantic and structured knowledge for language expression, constructing a more accurate text representation, and greatly improving the accuracy of text information prediction and the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0032] Figure 1 Shown is a flowchart of a text information completion method in an embodiment of the present invention
[0033] Figure 2 Shown is a schematic diagram of the framework of a text information completion system in an embodiment of the present invention. Specific implementation plan
[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0036] [Example 1]
[0037] The inventor of this solution found that in the prior art, in a complex network environment, the text data of the target parsed from the network traffic is discretized and fragmented, which makes the detection model based on artificial intelligence unable to effectively use such text features, thereby reducing the efficiency and accuracy of data processing. Embodiment 1 of the present invention provides the following solution:
[0038] like Figure 1 As shown, this embodiment provides a text information completion method, including:
[0039] Step 101, semantic role labeling is performed on the text information, and subword vectors are obtained through a pre-trained language model.
[0040] It should be noted that semantic role labeling of text information means labeling the predicate-argument structure of the input sentence. Semantic role labeling aims to discover the predicate-argument structure of a sentence. It focuses on the predicate of the sentence and analyzes the relationship between the components and the predicate in the sentence, that is, the predicate-argument structure of the sentence. The predicate is a statement or description of the subject, indicating "what to do", "what it is" or "how it is", representing the core of an event. The noun paired with the predicate is called an argument. The semantic role refers to the role played by the argument in the event referred to by the verb. The main ones are: agent, patient, object, experiencer, beneficiary, tool, place, goal and source, etc.
[0041] In some embodiments, data cleaning is performed before the text information is input into the pre-trained language model to remove redundant spaces, punctuation, special characters, etc. The pre-trained language model uses the Bert model, and the original text sequence is first divided into sub-words by Bert's sub-word segmenter. Then it is input into the self-attention mechanism and Transformer for text processing. The self-attention mechanism adjusts the vector representation of each word according to other words in the context, thereby generating a vector containing rich semantic information. In the multi-layer Transformer structure, multiple context information integrations are performed.
[0042] Step 102, reconstruct the subword vectors into word vectors and perform semantic label alignment.
[0043] In some embodiments, features are extracted from sub-word vectors through a convolutional neural network, and the extracted sub-word feature vectors are average-pooled and aggregated into an overall global word-level representation.
[0044] It should be noted that for each word, multiple subword vectors belonging to the same word are combined together through convolution operations. Specifically, a convolution layer is defined to convolve the stacked subword vectors using multiple filters. The convolution layer automatically learns features to capture the pattern of subword combinations. After the convolution layer, maximum pooling is added to extract important features from the output of the convolution layer and reduce the dimension to generate a more compact word-level vector representation.
[0045] In some embodiments, an embedding vector is created for each semantic tag, and the pooled word-level representation is concatenated with the corresponding embedded tag.
[0046] It should be noted that the word embedding method can be used to generate label embeddings, such as a lookup table. The pooled word-level representation is connected with the corresponding label embedding to enhance the model's ability to understand the relationship between words and their labels.
[0047] In some embodiments, each semantic label sequence is input into the BiGRU layer to obtain a semantic label sequence vector, and all semantic label sequence vectors are connected and input into the fully connected layer.
[0048] It should be noted that for m label sequences corresponding to the number of predicates, each sequence T = {t1,…,t m The length of the tag vector is equal to the length of the original sentence. The tag vector is input into the BiGRU layer to obtain the representation of the above m tag sequences in the vector space, and then the m tag sequences are concatenated and input into the fully connected layer to obtain the fusion representation of multiple tag sequences.
[0049] Step 103: Fuse the extracted word vector features with their corresponding semantic labels.
[0050] In some embodiments, the extracted word vector features and their corresponding semantic labels are fused through a multi-layer perceptron, and can also be fused through simple concatenation or weighted averaging. It is understandable that simple concatenation is to directly concatenate the sentence vector and the label vector, and weighted averaging is to weighted average the sentence vector and the label vector.
[0051] Step 104: input the fused feature vector into a prediction model for training to obtain a text information prediction model.
[0052] In some embodiments, the fused feature vector is input into a regression model to predict the missing part of the text, or it can also be input into a support vector machine or a random forest model for prediction. The model is then trained through supervised learning, and the weights of text features and label representations are optimized using the annotated data, so that the model can eventually effectively fill in the missing parts of the text.
[0053] Step 105: Complete the text information using a text information prediction model.
[0054] [Example 2]
[0055] like Figure 2 As shown, this embodiment provides a text information completion system, including:
[0056] The labeling module 201 is used to label the text information with semantic roles.
[0057] It should be noted that semantic role labeling of text information means labeling the predicate-argument structure of the input sentence. Semantic role labeling aims to discover the predicate-argument structure of a sentence. It focuses on the predicate of the sentence and analyzes the relationship between the components and the predicate in the sentence, that is, the predicate-argument structure of the sentence. The predicate is a statement or description of the subject, indicating "what to do", "what it is" or "how it is", representing the core of an event. The noun paired with the predicate is called an argument. The semantic role refers to the role played by the argument in the event referred to by the verb. The main ones are: agent, patient, object, experiencer, beneficiary, tool, place, goal and source, etc.
[0058] The pre-training module 202 is used to obtain sub-word vectors through a pre-trained language model.
[0059] In some embodiments, data cleaning is performed before the text information is input into the pre-trained language model to remove redundant spaces, punctuation, special characters, etc. The pre-trained language model uses the Bert model, and the original text sequence is first divided into sub-words by Bert's sub-word segmenter. Then it is input into the self-attention mechanism and Transformer for text processing. The self-attention mechanism adjusts the vector representation of each word according to other words in the context, thereby generating a vector containing rich semantic information. In the multi-layer Transformer structure, multiple context information integrations are performed.
[0060] The reconstruction module 203 is used to reconstruct the subword vectors into word vectors and perform semantic label alignment.
[0061] In some embodiments, features are extracted from sub-word vectors through a convolutional neural network, and the extracted sub-word feature vectors are average-pooled and aggregated into an overall global word-level representation.
[0062] It should be noted that for each word, multiple subword vectors belonging to the same word are combined together through convolution operations. Specifically, a convolution layer is defined to convolve the stacked subword vectors using multiple filters. The convolution layer automatically learns features to capture the pattern of subword combinations. After the convolution layer, maximum pooling is added to extract important features from the output of the convolution layer and reduce the dimension to generate a more compact word-level vector representation.
[0063] In some embodiments, an embedding vector is created for each semantic tag, and the pooled word-level representation is concatenated with the corresponding embedded tag.
[0064] It should be noted that the word embedding method can be used to generate label embeddings, such as a lookup table. The pooled word-level representation is connected with the corresponding label embedding to enhance the model's ability to understand the relationship between words and their labels.
[0065] The sequence encoding module 204 is used to input each semantic label sequence into the BiGRU layer to obtain a semantic label sequence vector, and connect all the semantic label sequence vectors and input them into the fully connected layer.
[0066] It should be noted that for m label sequences corresponding to the number of predicates, each sequence T = {t1,…,t m The length of the tag vector is equal to the length of the original sentence. The tag vector is input into the BiGRU layer to obtain the representation of the above m tag sequences in the vector space, and then the m tag sequences are concatenated and input into the fully connected layer to obtain the fusion representation of multiple tag sequences.
[0067] The fusion module 205 is used to fuse the extracted word vector features with their corresponding semantic labels.
[0068] In some embodiments, the extracted word vector features and their corresponding semantic labels are fused through a multi-layer perceptron, and can also be fused through simple concatenation or weighted averaging. It is understandable that simple concatenation is to directly concatenate the sentence vector and the label vector, and weighted averaging is to weighted average the sentence vector and the label vector.
[0069] The training module 206 is used to input the fused feature vector into the prediction model for training to obtain a text information prediction model.
[0070] In some embodiments, the fused feature vector is input into a regression model to predict the missing part of the text, or it can also be input into a support vector machine or a random forest model for prediction. The model is then trained through supervised learning, and the weights of text features and label representations are optimized using the annotated data, so that the model can eventually effectively fill in the missing parts of the text.
[0071] The text completion module 207 is used to complete the text information through the text information prediction model.
[0072] [Example 3]
[0073] This embodiment provides a text information completion device, including:
[0074] At least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, wherein the executable instructions, when executed by the at least one processor, implement the method steps of embodiment 1 of the present invention.
[0075] In an apparatus for completing text information provided by an embodiment of the present invention, a processor and a memory may be provided separately or integrated together.
[0076] For example, the memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.
[0077] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDR SDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0078] In some embodiments, the memory stores the following elements, upgrade packages, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and an application program.
[0079] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application program.
[0080] In the embodiment of the present invention, the processor calls a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and the processor is used to execute the method steps of the embodiment 1 of the present invention.
[0081] [Example 4]
[0082] This embodiment provides a chip for executing the method of the above-mentioned embodiment 1 of the present invention. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the method of the embodiment 1 of the present invention.
[0083] [Example 5]
[0084] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of Embodiment 1 of the present invention are implemented.
[0085] For example, machine-readable storage media may include, but are not limited to, various known and unknown types of non-volatile memory.
[0086] In summary, embodiments 1-5 of the present invention provide a method of converting text information into subword vectors through a pre-trained language model, reconstructing the subword vectors into word vectors, and performing semantic label alignment, fusing the extracted word vector features with their corresponding semantic labels and training them through a prediction model, thereby providing rich semantic and structured knowledge for language expression, constructing a more accurate text representation, and greatly improving the accuracy of text information prediction and the efficiency of data processing.
[0087] It will be appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0088] In the embodiments of the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system. In addition, the coupling between the various units can be direct coupling or indirect coupling. In addition, the various functional units in the embodiments of the present application can be integrated into a processing unit, or can be separate physical existences, etc.
[0089] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0090] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a machine-readable storage medium. Therefore, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a machine-readable storage medium, which can include several instructions to enable an electronic device to perform all or part of the technical solution described in the embodiments of the present application. The above-mentioned storage medium may include various media that can store program codes, such as ROM, RAM, removable disk, hard disk, magnetic disk or optical disk.
[0091] The above contents are only specific implementation methods of the present application, and the protection scope of the present application is not limited thereto. Those skilled in the art may make changes or substitutions within the technical scope disclosed in the present application, and these changes or substitutions shall be within the protection scope of the present application.
Claims
1. A text information completion method, characterized in that: include: Perform semantic role labeling on text information and obtain subword vectors through pre-trained language model; Reconstruct the subword vectors into word vectors and perform semantic label alignment; Fusion of the extracted word vector features and their corresponding semantic labels; The fused feature vector is input into the prediction model for training to obtain a text information prediction model; The text information is completed using the text information prediction model.
2. The text information completion method according to claim 1, characterized in that: The reconstructing the subword vector into a word vector comprises: The subword vectors are subjected to feature extraction through a convolutional neural network, and the extracted subword feature vectors are average-pooled and aggregated into an overall global word-level representation.
3. The text information completion method according to claim 1, characterized in that: The semantic label alignment comprises: Create an embedding vector for each semantic tag and concatenate the pooled word-level representation with the corresponding embedding tag.
4. The text information completion method according to claim 1, characterized in that: The fusing of the extracted word vector features and their corresponding semantic labels includes: The extracted word vector features and their corresponding semantic labels are fused through a multi-layer perceptron.
5. The text information completion method according to claim 1, characterized in that: The method further comprises: Each semantic label sequence is input into the BiGRU layer to obtain the semantic label sequence vector, and all semantic label sequence vectors are connected and input into the fully connected layer.
6. A text information completion system, characterized in that: include: Annotation module, used to annotate text information with semantic roles; Pre-training module, used to obtain subword vectors through pre-training language model; A reconstruction module, used to reconstruct the subword vectors into word vectors and perform semantic label alignment; The fusion module is used to fuse the extracted word vector features with their corresponding semantic labels; A training module is used to input the fused feature vector into the prediction model for training to obtain a text information prediction model; The text completion module is used to complete the text information through the text information prediction model.
7. The text information completion system according to claim 6, characterized in that: The system further comprises: The sequence encoding module is used to input each semantic label sequence into the BiGRU layer to obtain a semantic label sequence vector, and then connect all the semantic label sequence vectors and input them into the fully connected layer.
8. A text information completion device, comprising at least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, characterized in that: The executable instructions, when executed by the at least one processor, enable the steps of the method according to any one of claims 1 to 5 to be implemented.
9. A chip, characterized in that: It comprises a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the steps of the method as claimed in any one of claims 1 to 5.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.