Text labeling method and device, equipment and storage medium

Abstract

The invention relates to the technical field of natural language processing, and discloses a text labeling method and device, equipment and a storage medium. The method comprises the steps of obtaining to-be-labeled text data; segmenting the to-be-labeled text data to obtain multiple pieces of text fragment data; classifying the text fragment data and selecting a plurality of pieces of text fragment data closest to the corresponding text fragment category according to a classification result to obtain a plurality of pieces of candidate text fragment data; performing similarity analysis on the to-be-labeled text data and the candidate text fragment data, and selecting the candidate text fragment data with the highest similarity to obtain target text fragment data; and labeling the to-be-labeled text data according to the target text fragment data. According to the embodiment of the invention, the complete semantics of the text content can be reserved in the automatic labeling process, and the label quality is improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text annotation method, device, equipment and storage medium. Background Art

[0002] With the widespread application of large-scale pre-trained language models in the field of natural language processing, retrieval-enhanced generation technology has gradually become a key means to improve text generation and knowledge application capabilities. This vectorized knowledge base relies on high-quality labels to improve retrieval accuracy.

[0003] In related technologies, label setting relies on manual annotation, but with the expansion of knowledge base scale and the complexity of application scenarios, manual annotation becomes extremely difficult and costly. For automatic annotation of unlabeled knowledge base, the text of the document is sliced ​​and these slices are directly used as retrieval labels through vector encoding. However, when dealing with complex documents, the text slicing method has obvious limitations. As the length of the document increases, the quality of the label further decreases, affecting the efficiency and effect of vector retrieval. Summary of the invention

[0004] The purpose of this application is to provide a text annotation method, device, equipment and storage medium, which can retain the complete semantics of text content during automatic annotation and improve the quality of labels.

[0005] The present application provides a text annotation method, including: Get the text data to be annotated; Segmenting the text data to be annotated to obtain a plurality of text segment data; Classifying the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data; Performing similarity analysis on the text data to be annotated and the candidate text segment data, selecting the candidate text segment data with the highest similarity, and obtaining the target text segment data; The text data to be annotated is annotated according to the target text segment data.

[0006] In some embodiments, segmenting the text data to be annotated to obtain a plurality of text segment data includes: The text data to be annotated is segmented to obtain a plurality of text segmented data; the text contents recorded in two adjacent text segmented data are partially the same; Encoding the text intercepted data to obtain text encoded data; The text encoding data is projected into the target space to obtain the text segment data.

[0007] In some embodiments, the classifying of the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data includes: Clustering the text segment data to obtain a number of cluster centers; Calculating the distance between the text segment data and the cluster center to obtain a cluster distance; According to the cluster distance, the text segment data closest to the cluster center is selected to obtain the candidate text segment data.

[0008] In some embodiments, the performing similarity analysis on the text data to be annotated and the candidate text segment data, selecting the candidate text segment data with the highest similarity, and obtaining the target text segment data includes: Vectorizing the text data to be annotated and the candidate text segment data to obtain vector data of the text to be annotated and vector data of the candidate text segment; Calculating the similarity between the to-be-annotated text vector data and the candidate text segment vector data to obtain text vector similarity; According to the text vector similarity, the text data to be annotated that is most similar to the candidate text segment data is selected to obtain the target text segment data.

[0009] In some embodiments, the step of labeling the to-be-labeled text data according to the target text segment data includes: Extracting subject information from the target text segment data to obtain annotated text data; The annotated text data is embedded into the text data to be annotated, so as to annotate the text data to be annotated.

[0010] In some embodiments, before classifying the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data, the method further includes: The text segment data is input into a preset text processing model to perform vectorization processing on the text segment data to obtain vectorized text segment data; the text processing model is obtained by training a neural network model using sample text segment data and sample text segment vector data.

[0011] In some embodiments, the text processing model training method includes: Randomly masking the sample text segment data to obtain masked text segment data; Inputting the masked text segment data into the neural network model to obtain prediction vector data; Determining model loss information; the model loss information represents the deviation between the predicted vector data and the sample text segment vector data; Determining whether the model loss information is within a loss threshold range; If not, adjusting the weight parameters of the neural network model; returning to the step of inputting the masked text segment data into the neural network model to obtain prediction vector data; If so, end the training and obtain the text processing model.

[0012] The present application also provides a text annotation device, including: The first module is used to obtain the text data to be annotated; The second module is used to segment the text data to be annotated to obtain multiple text segment data; The third module is used to classify the text segment data and select a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data; The fourth module is used to perform similarity analysis on the text data to be annotated and the candidate text segment data, select the candidate text segment data with the highest similarity, and obtain the target text segment data; The fifth module is used to annotate the text data to be annotated according to the target text segment data.

[0013] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned text annotation method when executing the computer program.

[0014] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned text annotation method is implemented.

[0015] The beneficial effects of the present application are as follows: first, the text segment data obtained from the segmentation of the text data to be annotated are classified and several text segment data closest to the corresponding text segment category are selected according to the classification results to obtain candidate text segment data, and then similarity analysis is performed on the text data to be annotated and the candidate text segment data, and the candidate text segment data with the highest similarity is selected to obtain target text segment data, and finally the text data to be annotated is annotated according to the target text segment data. Since several candidate text segment data closest to the category that can represent the text data to be annotated are first determined, and then the one most similar to the text data to be annotated is selected from each candidate text segment data as the target text segment data, the semantic integrity of the text data to be annotated is retained, and the text data to be annotated participates in the annotation generation process as a whole, ensuring that the semantic connection of the context is maintained during the extraction of the target text segment data, the complete semantics of the text content can be retained during the automatic annotation process, and the quality of the label is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the text annotation method provided in an embodiment of the present application.

[0017] Figure 2 It is a flowchart of the specific method of step S102 provided in an embodiment of the present application.

[0018] Figure 3 It is a flowchart of the specific method of step S103 provided in an embodiment of the present application.

[0019] Figure 4 It is a flowchart of the method before step S103 provided in an embodiment of the present application.

[0020] Figure 5 It is a flowchart of the specific method of step S104 provided in an embodiment of the present application.

[0021] Figure 6 It is a structural diagram of a text annotation device provided in an embodiment of the present application.

[0022] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown can be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second" and the like in the specification, claims and drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0026] Figure 1 is a flowchart of a text annotation method provided in an embodiment of the present application. Figure 1 In some embodiments, the method may include but is not limited to steps S101 to S105.

[0027] Step S101, obtaining text data to be annotated.

[0028] The text data to be annotated refers to the data recording the corresponding text to be annotated.

[0029] In some embodiments, the text to be annotated may include words, sentences, paragraphs, and articles, etc., and the actual language of the text to be annotated may include English, Japanese, Korean, and Chinese, etc.

[0030] In some embodiments, the method for obtaining the text data to be annotated may be to obtain it through a terminal device, or to obtain it from a network by crawling, and the method for obtaining the text data to be annotated is not limited herein. For example, the terminal device may obtain the text data to be annotated through an input device connected thereto (such as a keyboard, a mouse, a handwriting tablet, a touch screen, etc.). For another example, the terminal device may also receive audio electronic data, and perform speech recognition on the audio electronic data to obtain the text data to be annotated.

[0031] Step S102, segmenting the text data to be annotated to obtain a plurality of text segment data.

[0032] The text segment data refers to data that records part of the text content in the text to be annotated after segmenting the text data to be annotated and undergoing data processing.

[0033] In some embodiments, after obtaining the text data to be annotated, the text data to be annotated can be segmented to obtain a text segment data set corresponding to the text data to be annotated, and the text segment data set includes multiple text segment data obtained by the segmentation process. Specifically, a preset text segmentation length can be obtained, the original arrangement order of each character in the text data to be annotated is kept unchanged, and the text segmentation length is used to perform sliding segmentation and data processing on the text data to be annotated to obtain multiple text segment data, and the character length of the text content in the text segment data is the text segmentation length.

[0034] Figure 2 is a flowchart of a specific method of step S102 provided in an embodiment of the present application. Figure 2 In some embodiments, the method includes but is not limited to steps S201 to S203.

[0035] Step S201 , segmenting the text data to be annotated to obtain a plurality of text segmentation data.

[0036] The text interception data refers to data recording part of the text content in the text to be annotated that is directly obtained after segmenting and intercepting the text data to be annotated.

[0037] The text contents recorded in two adjacent text interception data are partially the same. It can be understood that the text contents recorded in two adjacent text interception data are partially the same, which means that when the text data to be annotated is segmented, the text contents selected by the two adjacent segmentation interception operations overlap, that is, the front end part of the text content selected by the latter segmentation interception operation is the back end part of the text content selected by the latter segmentation interception operation.

[0038] In some embodiments, a preset text segmentation length and a sliding step length can be obtained, the text segmentation length is greater than the sliding step length, the original arrangement order of each character in the text data to be annotated is maintained unchanged, the text segmentation length and the sliding step length are used to perform sliding segmentation on the text data to be annotated, each segmentation interception operation obtains text content with a character length equal to the text segmentation length, and obtains text interception data, after each segmentation interception operation is completed, slide backward according to the sliding step length, and perform the next segmentation interception operation until the text to be annotated in the text data to be annotated is completely segmented and intercepted. Exemplarily, the text to be annotated in the text data to be annotated is "ABCDEFGH", the text segmentation length is 4 characters, the sliding step length is 2 characters, the first segmentation interception operation obtains text interception data of the text content of ABCD with a total of 4 characters, the second segmentation interception operation obtains text interception data of the text content of CDEF with a total of 4 characters, and the third segmentation interception operation obtains text interception data of the text content of EFGH with a total of 4 characters.

[0039] Step S202, encoding the text interception data to obtain text encoding data.

[0040] In some embodiments, one-hot encoding is performed on the text content in the text interception data, so that the text content in the text interception data is converted into a vector form to obtain text encoding data.

[0041] Step S203, projecting the text encoding data into the target space to obtain text segment data.

[0042] In some embodiments, a random projection technique is used to generate a random matrix in which elements are independently sampled from a standard normal distribution. The text encoding data and the random matrix are fitted to project the text encoding data into a target space to obtain text fragment data.

[0043] Step S103 , classifying the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data.

[0044] In some embodiments, after obtaining the text segment data, each text segment data is classified into a plurality of text segment categories according to a preset classification criterion, and then the text segment data that best matches the text segment category in the corresponding text segment category is selected as the candidate text segment data. Specifically, firstly, the text segment data is subjected to category analysis to obtain a category analysis result, and then the text segment data is determined to be classified into a target number of text segment categories according to the category analysis result, and then each text segment data under each text segment category is analyzed, and the text segment data that best matches the text segment category in the corresponding text segment category is selected, and all text segment categories are traversed to obtain a plurality of candidate text segment data.

[0045] Figure 3 is a flowchart of a specific method of step S103 provided in an embodiment of the present application. Figure 3 In some embodiments, the method includes but is not limited to steps S301 to S303.

[0046] Step S301, clustering the text segment data to obtain a number of cluster centers.

[0047] In some embodiments, the text segment data is clustered using a k-means clustering method.

[0048] In some embodiments, keyword extraction and context semantic extraction are performed on text fragment data to obtain keyword features and context semantic features of the text fragment data, and a corresponding clustering algorithm is used to calculate the similarity between the keyword features of each text fragment data and the similarity between the context semantic features and perform weighted fitting to obtain text feature information of the text fragment data, and then the corresponding cluster center is determined according to the mutual distance and density between the text feature information of the text fragment data.

[0049] Step S302, calculating the distance between the text segment data and the cluster center to obtain the cluster distance.

[0050] Step S303: selecting text segment data closest to the cluster center according to the cluster distance to obtain candidate text segment data.

[0051] In step S302 and step S303, the text segment data is vectorized to obtain text segment vector data, all cluster centers are traversed, the distances between all cluster centers and each text segment vector data are calculated, the cluster distances between all cluster centers and each text segment vector data are obtained, and the text segment data closest to the cluster center is selected as the candidate text segment data for the cluster center.

[0052] In some embodiments, before step S103, the method further includes: inputting the text segment data into a preset text processing model to vectorize the text segment data to obtain vectorized text segment data. The text processing model is obtained by training a neural network model using sample text segment data and sample text segment vector data. Specifically, the text processing model is obtained by training the sample text segment data and the sample text segment vector data in advance, and the text segment data is vectorized using the text processing model to convert the text segment data into a vector form to obtain vectorized text segment data.

[0053] Figure 4 is a flowchart of the method before step S103 provided in the embodiment of the present application. Figure 4 In some embodiments, the method includes but is not limited to steps S401 to S406.

[0054] Step S401, randomly masking the sample text segment data to obtain masked text segment data.

[0055] Step S402, inputting the masked text segment data into the neural network model to obtain predicted vector data.

[0056] Step S403, determining model loss information.

[0057] The model loss information represents the deviation between the predicted vector data and the sample text segment vector data.

[0058] Step S404, determining whether the model loss information is within a loss threshold range.

[0059] If not, execute step S405; if so, execute step S406.

[0060] Step S405, adjusting the weight parameters of the neural network model. Return to step S402.

[0061] Step S406, end the training and obtain the text processing model.

[0062] In step S401 to step S406, a loss threshold interval is preset as a training end condition, and then the neural network model is trained. When the model loss information is within the loss threshold interval, the training is terminated, and the neural network model of the last iteration is obtained, which is the text processing model. Specifically, the sample text segment data is randomly masked so that a certain proportion of the text content of the sample text segment data is selected and converted into a special mask mark to obtain masked text segment data, and the masked text segment data is input into the neural network model. The neural network model outputs corresponding prediction vector data according to the current weight parameters, and the current model loss information is determined according to the prediction vector data and the corresponding sample text segment vector data to determine whether the model loss information is within the loss threshold interval. If not, the weight parameters of the neural network model are adjusted and the neural network model is generated again. The prediction vector data is generated so that the model loss information gradually approaches the loss threshold interval during the iteration process and is finally within the loss threshold interval. If it is, the training is terminated to obtain the text processing model. In this way, the trained text processing model can convert the text segment data into a vector form that meets the expected requirements.

[0063] Step S104 , performing similarity analysis on the text data to be annotated and the candidate text segment data, selecting the candidate text segment data with the highest similarity, and obtaining the target text segment data.

[0064] In some embodiments, a similarity analysis is performed on the text data to be annotated and each candidate text segment data to obtain a similarity analysis result between the text data to be annotated and each candidate text segment data, and the corresponding candidate text segment data is selected based on the similarity analysis result. The selected candidate text segment data is the candidate text segment data that is most similar to the text data to be annotated, and the target text segment data is obtained.

[0065] Figure 5 is a flowchart of a specific method of step S104 provided in an embodiment of the present application. Figure 5In some embodiments, the method includes but is not limited to steps S501 to S503.

[0066] Step S501 , vectorizing the text data to be annotated and the candidate text segment data to obtain vector data of the text to be annotated and vector data of the candidate text segment.

[0067] Step S502 , calculating similarity between the text vector data to be annotated and the candidate text segment vector data to obtain text vector similarity.

[0068] Step S503: According to the text vector similarity, the text data to be annotated that is most similar to the candidate text segment data is selected to obtain the target text segment data.

[0069] In some embodiments, the similarity between the text vector data to be annotated and the candidate text segment vector data can be obtained by calculating the angular distance, cosine distance, Euclidean distance or Manhattan distance between the text vector data to be annotated and the candidate text segment vector data.

[0070] In the embodiment of the present application, the cosine distance between the vector data of the text to be annotated and the vector data of the candidate text segment is calculated as the text vector similarity. The calculation formula of the cosine distance is: , in, is the cosine distance between the vector data of the text to be annotated and the vector data of the candidate text segment, is the text vector data to be annotated, is the candidate text segment vector data, is the Euclidean norm of the text vector data to be annotated, is the Euclidean norm of the candidate text segment vector data.

[0071] Step S105 , annotating the text data to be annotated according to the target text segment data.

[0072] In some embodiments, step S105 includes: extracting subject information from the target text segment data to obtain annotated text data; and embedding the annotated text data into the text data to be annotated to annotate the text data to be annotated.

[0073] See also Figure 6 The present application also provides a text annotation device, which can implement the above text annotation method, and the device includes: The first module 601 is used to obtain text data to be annotated; The second module 602 is used to segment the text data to be annotated to obtain multiple text segment data; The third module 603 is used to classify the text segment data and select a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data; The fourth module 604 is used to perform similarity analysis on the text data to be annotated and the candidate text segment data, select the candidate text segment data with the highest similarity, and obtain the target text segment data; The fifth module 605 is used to annotate the text data to be annotated according to the target text segment data.

[0074] The specific implementation of the text annotation device is basically the same as the specific implementation of the above-mentioned text annotation method, and will not be repeated here.

[0075] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment.

[0076] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0077] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), a display unit 740, etc.

[0078] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above text annotation method section of this specification.

[0079] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0080] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0081] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0082] The electronic device 700 may also communicate with one or more external devices 700' (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 760. The network adapter 760 may communicate with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0083] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned text annotation method when executed by a processor.

[0084] The text annotation method, apparatus, device and storage medium provided in the embodiments of the present application first classify the text segment data obtained from the segmentation of the text data to be annotated and select several text segment data that are closest to the corresponding text segment category according to the classification result to obtain candidate text segment data, then perform similarity analysis on the text data to be annotated and the candidate text segment data, select the candidate text segment data with the highest similarity to obtain target text segment data, and finally annotate the text data to be annotated according to the target text segment data. Since several candidate text segment data that are closest to the category that can represent the text data to be annotated are first determined, and then the one that is most similar to the text data to be annotated is selected from each candidate text segment data as the target text segment data, the semantic integrity of the text data to be annotated is retained, and the text data to be annotated participates in the annotation generation process as a whole, ensuring that the semantic connection of the context is maintained during the extraction of the target text segment data, and the complete semantics of the text content can be retained during the automatic annotation process, thereby improving the quality of the label.

[0085] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation of the present disclosure.

[0086] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0087] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0088] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0089] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A text annotation method, characterized in that: include: Get the text data to be annotated; Segmenting the text data to be annotated to obtain a plurality of text segment data; Classifying the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data; Performing similarity analysis on the text data to be annotated and the candidate text segment data, selecting the candidate text segment data with the highest similarity, and obtaining the target text segment data; The text data to be annotated is annotated according to the target text segment data.

2. The text annotation method according to claim 1, characterized in that: The step of segmenting the text data to be annotated to obtain a plurality of text segment data includes: The text data to be annotated is segmented to obtain a plurality of text segmented data; the text contents recorded in two adjacent text segmented data are partially the same; Encoding the text intercepted data to obtain text encoded data; The text encoding data is projected into the target space to obtain the text segment data.

3. The text annotation method according to claim 1, characterized in that: The text segment data is classified and a number of text segment data closest to the corresponding text segment category are selected according to the classification result to obtain a number of candidate text segment data, including: Clustering the text segment data to obtain a number of cluster centers; Calculating the distance between the text segment data and the cluster center to obtain a cluster distance; According to the cluster distance, the text segment data closest to the cluster center is selected to obtain the candidate text segment data.

4. The text annotation method according to claim 1, characterized in that: The performing similarity analysis on the text data to be annotated and the candidate text segment data, selecting the candidate text segment data with the highest similarity, and obtaining the target text segment data includes: Vectorizing the text data to be annotated and the candidate text segment data to obtain vector data of the text to be annotated and vector data of the candidate text segment; Calculating the similarity between the to-be-annotated text vector data and the candidate text segment vector data to obtain text vector similarity; According to the text vector similarity, the text data to be annotated that is most similar to the candidate text segment data is selected to obtain the target text segment data.

5. The text annotation method according to claim 1, characterized in that: The step of marking the to-be-marked text data according to the target text segment data includes: Extracting subject information from the target text segment data to obtain annotated text data; The annotated text data is embedded into the text data to be annotated, so as to annotate the text data to be annotated.

6. The text annotation method according to claim 1, characterized in that: Before classifying the text segment data and selecting a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data, the method further includes: The text segment data is input into a preset text processing model to perform vectorization processing on the text segment data to obtain vectorized text segment data; the text processing model is obtained by training a neural network model using sample text segment data and sample text segment vector data.

7. The text annotation method according to claim 6, characterized in that: The training method of the text processing model comprises: Randomly masking the sample text segment data to obtain masked text segment data; Inputting the masked text segment data into the neural network model to obtain prediction vector data; Determining model loss information; the model loss information represents the deviation between the predicted vector data and the sample text segment vector data; Determining whether the model loss information is within a loss threshold range; If not, adjusting the weight parameters of the neural network model; returning to the step of inputting the masked text segment data into the neural network model to obtain prediction vector data; If so, end the training and obtain the text processing model.

8. A text annotation device, characterized in that: include: The first module is used to obtain the text data to be annotated; The second module is used to segment the text data to be annotated to obtain multiple text segment data; The third module is used to classify the text segment data and select a number of text segment data closest to the corresponding text segment category according to the classification result to obtain a number of candidate text segment data; The fourth module is used to perform similarity analysis on the text data to be annotated and the candidate text segment data, select the candidate text segment data with the highest similarity, and obtain the target text segment data; The fifth module is used to annotate the text data to be annotated according to the target text segment data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the text annotation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text annotation method according to any one of claims 1 to 7 is implemented.

Images