Data labeling method, device, equipment and storage medium

Abstract

The present application relates to the field of artificial intelligence technology. The present application discloses a method, apparatus, device and storage medium for data annotation. The method comprises the following steps: vectorizing operation-class captured data, visual-class captured data and voice-class captured data corresponding to a file to be annotated to obtain an operation-class annotation file, a visual-class annotation file and a voice-class annotation file; using a data annotation model to mine semantic similarity between the operation-class annotation file, the visual-class annotation file and the voice-class annotation file, and reducing the Euclidean distance between the annotation files to obtain a primary annotation file; classifying the character jump relationship set in the primary annotation file to obtain a final annotation file with a valid character jump relationship set, and using the data annotation model to improve the accuracy of the final annotation file of the data annotation in downstream classification tasks.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment and storage medium for data labeling. Background Art

[0002] Data labeling is the premise of supervised learning, and high-quality labeling is the guarantee of the effectiveness of supervised models. The labeling system currently used is strongly bound to the task, which makes a large number of labels achieve certain results on specific tasks. However, when changing to a new task, the original labeled data will have a lot of deviations. In the field of intelligent medical care, data needs to be labeled by physicians to complete the design of a specific labeling system, such as named entity recognition (MER) in existing medical technology. By making physician annotators over-emphasize data labeling, the specific data that physicians pay attention to in clinical judgment is ignored, resulting in differences between data labeling results and diagnostic ideas, resulting in low classification judgment accuracy of the diagnostic ideas learned by the model, which in turn leads to low accuracy of the labeled data in downstream tasks. Summary of the invention

[0003] The present application provides a method, apparatus, computer device and storage medium for data labeling, which solves the problem of insufficient accuracy of data labeling in the prior art in downstream classification tasks.

[0004] In a first aspect, an embodiment of the present application provides a method for data labeling, including:

[0005] According to the file to be annotated, converting the operation-type captured data, the visual-type captured data and the voice-type captured data corresponding to the file to be annotated into a vectorized operation-type annotation file, a visual-type annotation file and a voice-type annotation file;

[0006] Mining the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortening the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtaining a primary annotation file;

[0007] The character jump relationship sets in the primary annotation file are classified to obtain a final annotation file having a valid character jump relationship set.

[0008] In a second aspect, an embodiment of the present application further provides a data labeling device, including:

[0009] A vectorization module converts the operation-type captured data, visual-type captured data, and voice-type captured data corresponding to the file to be annotated into a vectorized operation-type annotation file, a visual-type annotation file, and a voice-type annotation file according to the file to be annotated;

[0010] A semantic annotation module, which mines the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortens the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtains a primary annotation file;

[0011] The association marking module classifies the character jump relationship set in the primary annotation file to obtain a final annotation file with a valid character jump relationship set.

[0012] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data labeling method when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned data labeling method are implemented.

[0014] The present application provides a method, apparatus, computer equipment and storage medium for data annotation, wherein a method for data annotation is provided, wherein the method vectorizes the operation-class captured data, visual-class captured data and voice-class captured data corresponding to the file to be annotated to obtain an operation-class annotation file, a visual-class annotation file and a voice-class annotation file; uses a data annotation model to mine the semantic similarity between the operation-class annotation file, the visual-class annotation file and the voice-class annotation file, and reduces the Euclidean distance between each annotation file to obtain a primary annotation file; classifies the character jump relationship set in the primary annotation file to obtain a final annotation file with a valid character jump relationship set, and uses the data annotation model to improve the accuracy of the final annotation file of the data annotation in the downstream classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 This is a schematic diagram of an application environment of a data labeling method provided by an embodiment of the present application;

[0017] Figure 2 This is a flowchart of a method for data annotation provided in one embodiment of the present application;

[0018] Figure 3 This is a flowchart of step S20 in a data labeling method provided in one embodiment of the present application;

[0019] Figure 4 It is a structural schematic diagram of a data labeling device provided in an embodiment of the present application;

[0020] Figure 5 It is a schematic diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The data annotation method provided in the embodiment of the present application can be applied in the following aspects: Figure 1 In the application environment shown. Figure 1 As shown, the client (computer device) communicates with the server through the network. The client (computer device) includes but is not limited to various personal computers, laptops, smart phones, tablets, cameras and portable wearable devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0023] The data labeling method provided in this embodiment can be executed by the server. For example, the client sends the information to be labeled to the server. The server executes the data labeling method provided in this embodiment based on the information to be labeled, and then obtains the final labeling file after the data labeling processing, and finally sends the final labeling file to the client.

[0024] exist Figure 1 In some other scenarios, the data labeling method can also be executed by the client, directly according to the information to be labeled of the client, by executing the data labeling method provided in this embodiment, to obtain the final labeling file after data labeling processing, and then send the final labeling file to the server for storage.

[0025] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0026] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0027] Figure 2 FIG. 1 shows a flowchart of a method for data annotation provided by an embodiment of the present application. Figure 2 As shown, a method for data labeling is provided, and its technical solution mainly includes the following steps S10-S30:

[0028] Step S10, converting the operation-type captured data, vision-type captured data and voice-type captured data corresponding to the file to be annotated into vectorized operation-type annotation files, vision-type annotation files and voice-type annotation files according to the file to be annotated.

[0029] In step S10, by capturing the user's behavior when reading or browsing the file to be annotated, the user is ensured to be in a normal reading or browsing environment, and the user's actual reading logic is more accurately obtained, and the rationality and accuracy of the annotation of the file to be annotated are ensured from multiple dimensions, thereby improving the accuracy of the final annotated file in downstream tasks. Compared with the traditional annotation method, the method of annotating based on the user's active recording of behavior will destroy the user's normal logic in actual reading or browsing, thereby reducing the accuracy and rationality of the annotation of the annotated file.

[0030] In one embodiment, when doctors consult with patients through electronic medical records, they have a complete set of diagnostic logic, and judge the patient's condition based on the relationship between different items in the resume to complete the diagnosis. However, in actual applications, such as medical named entity recognition (MER), doctors need to follow the specific annotation method designed by the system, so that doctors can accept a kind of annotation thinking, and their own clinical thinking is easily destroyed and fragmented, resulting in the annotation results not being conducive to downstream tasks to learn the diagnosis logic of doctors in case browsing.

[0031] In this embodiment, by capturing the diagnostic physician's oral words or sentences, the number and time of eye annotations, and the position and time of cursor movement during the case diagnosis process, the corresponding text weights in the resume and the jump relationships between texts are obtained from the three dimensions of vision, voice, and operation.

[0032] In step S10 of this embodiment, according to the file to be annotated, the operation-type captured data, the visual-type captured data and the voice-type captured data corresponding to the file to be annotated are converted into vectorized operation-type annotation files, visual-type annotation files and voice-type annotation files, wherein step S10 also includes:

[0033] Step S101, converting the operation-class captured data into the operation-class annotation file including character vectors, operation-class character weight data and operation-class character jump relations;

[0034] Step S102, converting the visual captured data into the visual annotation file including the character vector, visual character weight data and visual character jump relationship;

[0035] Step S103, converting the voice-type captured data into the voice-type annotation file including the character vector, voice-type character weight data and voice-type character jump relationship.

[0036] In one embodiment, the obtained operational capture data, visual capture data and voice capture data are all based on the text of the file to be annotated. The corresponding capture data is obtained by capturing the user's spoken words or sentences while browsing the text, the number and time of eye annotations, and the position and time of cursor movement, thereby ensuring that the user implements an automatic capture method under the normal logic.

[0037] In one embodiment, the operational capture data, visual capture data, and voice capture data captured from the user's browsing behavior in the file to be annotated are vectorized and converted into character vectors based on the file to be annotated, as well as the respective weights of each character in different behaviors obtained according to different operations and the jump relationships between different characters.

[0038] In step S101, based on the weight data of the operation class characters corresponding to the character vectors and the operation class annotation file of the jump relationship of the operation class characters in the operation class capture data, the operation cursor track of the user when browsing the text content is obtained as the operation class capture data. The operation cursor track in the operation class capture data includes the time for the cursor corresponding to the weight data of the operation class characters to be determined to stay on the text, and also includes the jump relationship of the cursor in the text browsing for recording the jump relationship of the operation class characters.

[0039] In step S102, based on the visual character weight data corresponding to the character vector and the visual character jump relationship in the visual capture data, the visual movement trajectory of the user when browsing the text content is obtained as the visual capture data. The visual movement trajectory in the visual capture data includes the visual character weight data, the corresponding visual focus stay time on the character, and also includes the jump relationship of the visual focus in text browsing for recording the visual character jump relationship.

[0040] In step S103, based on the speech-type character weight data corresponding to the character vector and the speech-type character jump relationship in the speech-type captured data, the spoken content of the user when browsing the text content is obtained as the speech-type captured data.

[0041] In one embodiment, the browsing oral content in the voice-type captured data is the voice captured data, the visual captured data and the operation-type captured data synchronized according to the timestamp, and the browsing oral content having browsing relevance with the operation cursor track and the visual movement track when browsing the text content is used as the voice-type captured data; wherein the browsing relevance is having the same vocabulary as the text content browsed by the operation cursor track and the visual movement track within a certain time range.

[0042] When browsing the spoken content, corresponding weights are set for the spoken voice content and the words with browsing relevance to the operation cursor track and the visual movement track, and the jump relationship in the spoken voice content is recorded as the voice character jump relationship.

[0043] In another embodiment, when the physician makes a diagnostic judgment based on the case during the diagnosis process, the physician's visual capture data during the case reading behavior is obtained through an eye tracker for visual focus capture, the physician's voice capture data during the case reading behavior is obtained through a recording device, and the physician's operational capture data during the case reading behavior is obtained through a screen capture program.

[0044] Assume that in the scenario of traditional Chinese medicine, a case of a patient who caught a cold due to wind and cold is taken as a document to be annotated. The doctor first notices that the patient had aversion to cold and sore throat yesterday. At this time, the doctor can choose to highlight "aversion to cold" and "sore throat" with the mouse, or capture it by moving and pausing the mouse on the text content; at the same time, the doctor's voice behavior corresponds to the description of the condition "after being caused by wind and cold, do you still feel aversion to cold?" The doctor finds that the patient did not have the word "aversion to cold" today, but had words such as "thirst", "cough", and "red tongue coating". The doctor continues to voice-narrate "the appearance of red tongue coating is considered to be a transition from external cold to internal heat, and there is cough. The disease is located in the lungs, so the throat feels sore, so it may be due to intense lung heat." The words with a larger weight in the captured visual behavior may be "aversion to cold, sore throat", and the results with a larger weight in the captured operational behavior are "aversion to cold" and "sore throat", and the captured voice behavior results are the words "aversion to cold" and "sore throat" that appeared in the case in the doctor's explanation.

[0045] If the doctor finds that the patient has a history of surgery while browsing the patient's medical history, the doctor will jump to the last diagnosis result for review, and will look in detail at the various reports in the last diagnosis process: imaging reports (CT, MRI), surgical records, pathology reports, etc. The above are all text jump behaviors during the diagnosis process.

[0046] Step S20, mining the semantic similarity among the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortening the Euclidean distance among the operation annotation file, the visual annotation file and the voice annotation file, and obtaining a primary annotation file.

[0047] In step S20, the weight parts of the obtained operation-class annotation files, visual-class annotation files and speech-class annotation files are used to annotate the files to be annotated using a data annotation model based on an artificial neural network, and the Euclidean distances between the operation-class annotation files, visual-class annotation files and speech-class annotation files are calculated and shortened, thereby obtaining a primary annotation file with higher accuracy.

[0048] In one embodiment, the data annotation model is composed of an artificial neural network, which can be but not limited to using a long short-term memory network (Long Short-Term Memory, LSTM). Long short-term memory (Long short-term memory, LSTM) is a special RNN, which is mainly used to solve the gradient disappearance and gradient explosion problems in the long sequence training process. By training and learning the artificial neural network in advance, the sample set labeled in this field is input into the artificial neural network, so that the artificial neural network learns the vocabulary and semantics corresponding to the input sample field.

[0049] Understandably, the Euclidean distance is also called the Euclidean metric. The process of training an artificial neural network is to minimize its Euclidean distance and maximize its cosine value, that is, the Euclidean distance and cosine value can be used to represent word similarity.

[0050] like Figure 3 As shown, step S20 of this embodiment is shown, the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file is mined through the data annotation model, the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file is shortened, and a primary annotation file is obtained. Step S20 also includes:

[0051] Step S201, matching the corresponding operation character weight data, the visual character weight data and the voice character weight data according to the character vector, and mining the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file by using the data annotation model;

[0052] Step S202, calculating the Euclidean distance values ​​between the operation annotation file, the visual annotation file, and the speech annotation file;

[0053] Step S203: if the Euclidean distance value is greater than a preset Euclidean distance threshold, shorten the Euclidean distances between the operation annotation file, the visual annotation file, and the speech annotation file using a Euclidean distance shortening strategy;

[0054] Step S204: if the Euclidean distance value is less than or equal to a preset Euclidean distance threshold, output the primary annotation file.

[0055] In step S201, since the annotation is performed according to the file to be annotated, the corresponding character vectors are the same. According to the operation character weight data, visual character weight data and voice character weight data corresponding to the character vector, the trained artificial neural network data annotation model is used to mine the semantic similarity between the operation annotation files, the visual annotation files and the voice annotation files. At this time, the annotation files are directly used for downstream classification tasks, and there will be certain differences from the true values.

[0056] In step S202, first, the Euclidean distance value between the operation-class annotation file and the visual-class annotation file is calculated by a loss function algorithm, so as to be used for reducing the Euclidean distance between the same character vectors between subsequent vectors.

[0057] In one embodiment, each annotation file is processed by a loss function algorithm, and the cosine similarity between the annotation files is calculated. According to the synonymous interchangeability principle of word2vec, if the context characters of two characters are the same, the trained word vectors of the two characters are also similar, and the Euclidean distance and cosine similarity can be used to represent the degree of association between words.

[0058] In one embodiment, the loss function may be, but is not limited to, a mean square error function (MSE) and a cross entropy function (Cross Entropy). The corresponding loss function is used according to actual needs. The mean square error function is mainly used for regression, and the cross entropy function is often used for logistic regression (logistic regression) and is mainly used for classification. Both are non-negative functions, and the extreme value is at the bottom.

[0059] It can be understood that the loss function is a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. The loss function is mainly used in the training phase of the model. After each batch of training data is fed into the model, the predicted value is output through forward propagation, and then the loss function calculates the difference between the predicted value and the true value, which is the loss value. After obtaining the loss value, the model updates each parameter through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model moves closer to the true value, thereby achieving the purpose of learning.

[0060] In step S203, if the Euclidean distance value is greater than the preset Euclidean distance threshold, the Euclidean distance between each annotation file is reduced by the Euclidean distance shortening strategy. Wherein, when the Euclidean distance is greater than the preset Euclidean distance threshold, the Euclidean distance shortening strategy is to use the data annotation model to perform semantic similarity mining on the operation annotation file and the visual annotation file again, thereby reducing the Euclidean distance of the character vector between the operation annotation file and the visual annotation file; use the data annotation model to perform semantic similarity mining on the operation annotation file and the voice annotation file again, thereby reducing the Euclidean distance of the character vector between the operation annotation file and the voice annotation file, and shortening the Euclidean distance between the annotations, thereby reducing the overall Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file. Wherein, the preset Euclidean distance threshold is obtained according to pre-training. According to the actual business environment, when the positive sample set is used, when the sample file output annotation result and the positive sample set meet the accuracy of downstream task use, the Euclidean distance value is used as the preset Euclidean distance threshold.

[0061] In step S204, when the Euclidean distance value is less than or equal to a preset Euclidean distance threshold, a primary annotation file having the semantic similarity between the character vectors is finally obtained and outputted, thereby ensuring accuracy during use in downstream tasks.

[0062] Step S30, classifying the character jump relationship sets in the primary annotation file to obtain a final annotation file with a valid character jump relationship set.

[0063] In step S30, after the operation-type captured data, the visual-type captured data, and the voice-type captured data are initially labeled, the obtained primary annotation file includes a character jump relationship set having an operation-type character jump relationship, a visual-type character jump relationship, and a voice-type character jump relationship. The character jump relationship set includes valid character jump relationships that affect the classification result, and invalid character jump relationships that are not conducive to classification prediction. By determining limited jump relationships, a final annotation file with a valid character jump relationship set is obtained for classification and prediction of downstream tasks.

[0064] In step S30 of this embodiment, the character jump relationship set in the primary annotation file is classified to obtain a final annotation file with a valid character jump relationship set. In step S30, the following is further included:

[0065] Step S301, matching the corresponding operation character jump relationship, the visual character jump relationship and the voice character jump relationship according to the character vector, and integrating them into the character jump relationship set after the semantic similarity mining process;

[0066] Step S302, classifying the character jump relations in the character jump relation set of the primary annotation file by using the data annotation model, and screening out valid character jump relations as the valid character jump relation set;

[0067] Step S303: output a final annotation file having a valid character jump relationship set.

[0068] In one embodiment, according to the character vector matching corresponding to the operation-type character jump relationship, the visual-type character jump relationship and the voice-type character jump relationship, the character jump relationship set corresponding to the primary annotation file is obtained after the semantic similarity mining process. At the same time, by using the trained data annotation model, the character jump relationship in the character jump relationship set is classified, the valid character jump relationship is screened out as the valid character jump relationship set, and the final annotation file with the valid character jump relationship set is output.

[0069] Among them, the operation character jump relationship, the visual character jump relationship and the voice character jump relationship are jump relationships recorded based on characters according to different annotation behaviors, and the jump relationships are converted into corresponding character jump relationships.

[0070] In one embodiment, when a physician browses a medical record containing "frequent diarrhea", the visual focus jumps to "age". In the downstream diagnosis prediction classification, "age" has a weight when judging the condition. After the data annotation model is trained with the downstream diagnosis sample data, it can be known that the physician's diagnosis has a valid character jump relationship when browsing the character jump relationship. When the physician browses a medical record containing "frequent diarrhea", the visual focus jumps to "name". In the downstream diagnosis prediction classification, "name" does not have a weight when judging the condition, and the character jump relationship is not an invalid character jump relationship.

[0071] In one embodiment, the artificial neural network in the data annotation model is pre-trained and learns based on the jump relationships of the annotated positive sample files and negative sample files, thereby identifying the corresponding valid jump relationships and invalid jump relationships.

[0072] The embodiment shown below is different in that, in step S30, the character jump relationship set in the primary annotation file is classified to obtain a final annotation file with a valid character jump relationship set. Specifically, steps S41-S43:

[0073] Step S41, using the final labeled file for prediction and classification processing of downstream tasks;

[0074] Step S42, obtaining the downstream task through multiple prediction and classification processes to obtain multiple classification results;

[0075] Step S43, based on the classification result with the smallest confidence weight among the classification results, the final annotation file corresponding to the classification result with the smallest confidence weight is annotated again.

[0076] In step S41, the final annotation file obtained will be used for predictive classification processing of downstream tasks. After fine-tuning according to the specific task, it will be input into the corresponding text processing model, which can be used for sequence annotation tasks (such as named entity recognition such as symptoms, drugs, treatment methods, etc.), normalization tasks (such as standardized coding of surgical names), classification tasks (such as disease diagnosis), and similarity retrieval (such as similar case retrieval) tasks.

[0077] In step S42, the downstream task corresponding to the final labeled file is obtained through multiple prediction and classification processes to obtain multiple classification results, and the classification results are collected for sample learning to improve the accuracy of subsequent files to be labeled.

[0078] In one embodiment, when the final annotation file reaches a preset threshold number of sample annotations, model training is started based on the final annotation file. When the annotation threshold is reached, a new round of training is triggered. The training results share artificial neural network parameters and set multiple prediction target tasks at the same time to achieve optimization of the annotation file.

[0079] In step S43, the probability of the predicted category of the classification result is used as the confidence. The larger the confidence, the more confident the model is in the prediction. The classification result with the smallest confidence weight in the classification results is obtained, and the final annotation file corresponding to the classification result with the smallest confidence weight is re-annotated to optimize the final annotation file, thereby improving the confidence in downstream tasks.

[0080] In one embodiment, a data labeling device is provided, and the data labeling device corresponds one-to-one to the data labeling method in the above embodiment. Figure 4 As shown, the data annotation device includes a vectorization module 11, a semantic annotation module 12 and an association marking module 13, and each functional module is described in detail as follows:

[0081] A vectorization module 11 converts the operation-type captured data, visual-type captured data and voice-type captured data corresponding to the file to be annotated into a vectorized operation-type annotation file, a visual-type annotation file and a voice-type annotation file according to the file to be annotated;

[0082] The semantic annotation module 12 mines the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortens the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtains a primary annotation file;

[0083] The association marking module 13 classifies the character jump relationship set in the primary annotation file to obtain a final annotation file with a valid character jump relationship set.

[0084] For the specific definition of the data annotation device, please refer to the definition of the data annotation method above, which will not be repeated here. Each module in the above-mentioned data annotation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0085] In one embodiment, a computer device is provided. The computer device may be a client or a server. The internal structure diagram thereof may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the readable storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for data labeling is implemented.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the method for data labeling in the above embodiment is implemented when the processor executes the computer program.

[0087] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for data labeling in the above embodiment is implemented.

[0088] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0089] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0090] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for data annotation, characterized in that: include: According to the file to be annotated, the operation-class captured data, the visual-class captured data and the voice-class captured data corresponding to the file to be annotated are converted into vectorized operation-class annotation files, visual-class annotation files and voice-class annotation files, wherein the operation-class captured data is the operation cursor track of the user when browsing the text content, the visual-class captured data is the visual movement track of the user when browsing the text content, and the voice-class captured data is the browsing oral content of the user when browsing the text content. The operation-class annotation file includes character vectors, operation-class character weight data and operation-class character jump relations, the visual-class annotation file includes character vectors, visual-class character weight data and visual-class character jump relations, and the voice-class annotation file includes character vectors, voice-class character weight data and voice-class character jump relations; Mining the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortening the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtaining a primary annotation file; The character jump relationship sets in the primary annotation file are classified to obtain a final annotation file having a valid character jump relationship set, wherein the character jump relationship set includes operation-related character jump relationships, visual-related character jump relationships, and voice-related character jump relationships.

2. The data labeling method according to claim 1, characterized in that: The method of converting the operation-type captured data, the visual-type captured data and the voice-type captured data corresponding to the file to be annotated into a vectorized operation-type annotation file, a visual-type annotation file and a voice-type annotation file according to the file to be annotated includes: Converting the operation-type captured data into the operation-type annotation file; Converting the visual captured data into the visual annotation file; The voice capture data is converted into the voice annotation file.

3. The data labeling method according to claim 2, characterized in that: The method of mining the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through the data annotation model, shortening the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtaining a primary annotation file includes: Matching the corresponding operation character weight data, the visual character weight data and the voice character weight data according to the character vector, and mining the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file by using the data annotation model; Calculating the Euclidean distance values ​​between the operation annotation file, the vision annotation file, and the speech annotation file; If the Euclidean distance value is greater than a preset Euclidean distance threshold, shortening the Euclidean distances between the operation-related annotation files, the visual-related annotation files, and the speech-related annotation files by using an Euclidean distance shortening strategy; If the Euclidean distance value is less than or equal to a preset Euclidean distance threshold, the primary annotation file is output.

4. The data labeling method according to claim 2, characterized in that: The step of classifying the character jump relationship set in the primary annotation file to obtain a final annotation file having a valid character jump relationship set includes: Matching the corresponding operation character jump relationship, the visual character jump relationship and the voice character jump relationship according to the character vector, and integrating them into the character jump relationship set after mining the semantic similarity; Classifying the character jump relations in the character jump relation set of the primary annotation file by using the data annotation model, and screening out valid character jump relations as the valid character jump relation set; Output the final annotation file with a valid set of character jump relations.

5. The data labeling method according to claim 1, characterized in that: After classifying the character jump relationship sets in the primary annotation file to obtain a final annotation file having a valid character jump relationship set, the method further includes: Using the final labeled file for predictive classification processing of downstream tasks; Obtaining the downstream task through multiple prediction and classification processes to obtain multiple classification results; According to the classification result with the smallest confidence weight among the classification results, the final annotation file corresponding to the classification result with the smallest confidence weight is annotated again.

6. The data labeling method according to claim 1, characterized in that: The voice capture data is obtained in the following manner: Synchronize the voice capture data, the visual capture data and the operation capture data according to the timestamp, and use the browsing oral content that has browsing relevance to the operation cursor track and the visual movement track when browsing the text content as the voice capture data; The browsing relevance refers to the same vocabulary as the text content browsed by the operation cursor track and the visual movement track within a certain time range.

7. A data labeling device, characterized in that: include: a vectorization module, which converts the operation-class captured data, visual-class captured data and voice-class captured data corresponding to the file to be annotated into vectorized operation-class annotation files, visual-class annotation files and voice-class annotation files according to the file to be annotated, wherein the operation-class captured data is the operation cursor trajectory of the user when browsing the text content, the visual-class captured data is the visual movement trajectory of the user when browsing the text content, and the voice-class captured data is the browsing oral content of the user when browsing the text content. The operation-class annotation file includes character vectors, operation-class character weight data and operation-class character jump relations, the visual-class annotation file includes character vectors, visual-class character weight data and visual-class character jump relations, and the voice-class annotation file includes character vectors, voice-class character weight data and voice-class character jump relations; A semantic annotation module, which mines the semantic similarity between the operation annotation file, the visual annotation file and the voice annotation file through a data annotation model, shortens the Euclidean distance between the operation annotation file, the visual annotation file and the voice annotation file, and obtains a primary annotation file; The association marking module classifies the character jump relationship set in the primary annotation file to obtain a final annotation file with a valid character jump relationship set, wherein the character jump relationship set includes an operation-type character jump relationship, a visual-type character jump relationship, and a voice-type character jump relationship.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for data labeling according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for data labeling as claimed in any one of claims 1 to 6 is implemented.

Images