A conversation text annotation method and system for characterizing doctor feedback

Abstract

The present invention discloses a method and system for annotating dialogue texts for characterizing doctor feedback. The method comprises: obtaining a set of instruction texts to be classified and annotated, classifying the instruction texts into departments through a preset instruction text department classification model, and storing the results into the annotation task allocation pool; the annotation task allocation pool automatically assigns annotation tasks to annotation doctors: if an instruction text annotation task is assigned, the annotation doctor annotates the classified instruction text set to be annotated; a preset medical dialogue model is used to generate a set of reply texts to be annotated based on a screened and standardized set of annotated instruction texts, forming a single-instruction multi-reply data set, and then performing difference evaluation and verification through a preset pre-trained doctor feedback model, and storing the results into the annotation task allocation pool; if a reply text annotation task is assigned, the annotation doctor annotates the reply text set to be annotated, and after verification, re-annotates or exports the reply text set annotation results based on the results.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method and system for annotating dialogue text representing doctor feedback. Background Art

[0002] With the rapid development of artificial intelligence, pre-trained dialogue generation models have begun to be widely used in the medical field, including intelligent customer service, health management, medical education, and medical auxiliary diagnosis. In these medical applications, pre-trained dialogue generation models can interact with doctors, patients, or medical students, providing relevant medical knowledge and assistance, thereby improving the efficiency and quality of medical services.

[0003] However, the application of pre-trained dialogue generation models in the medical field faces unique challenges and requirements. First, the language and knowledge in the medical field are highly specialized and complex, requiring pre-trained dialogue generation models to possess a higher level of semantic understanding and expertise. Second, medical conversations often involve sensitive privacy information and strict legal requirements, requiring pre-trained dialogue generation models to maintain high confidentiality and security. Furthermore, as the application of pre-trained dialogue generation models in the medical field continues to expand, the need for their annotation becomes increasingly urgent, requiring large quantities of professional manual annotation and performance evaluation to improve performance and better adapt to practical needs.

[0004] A medical image annotation method and system (CN115272742A). The method includes: obtaining a medical image annotation task; pre-annotating the medical image using a preset pre-annotation model to obtain a pre-annotation result; obtaining at least one annotator's annotation result corresponding to each of the at least one annotators based on the modification and confirmation of the pre-annotation result by at least one annotator; automatically verifying the at least one annotator's annotation result according to preset verification rules to determine whether there are any annotator's annotation results that fail the verification; if there are any annotator's annotation results that fail the verification, returning the unverified annotation results to the corresponding annotator for confirmation; if there are no annotator's annotation results that fail the verification, determining a final annotation result based on the at least one annotator's annotation results. Although this patent can reduce the time consumption of manual annotation to a certain extent in medical image annotation, improve annotation efficiency and annotation quality, it still relies heavily on the professional knowledge of the annotators. There is a lack of a complete annotation process and annotation system to guide the annotation work, and there is a lack of a reasonable screening mechanism and verification mechanism to intervene in the annotation quality. When the annotation object is medical text, a large amount of medical-related instruction text and reply text needs to be processed during the annotation process. The annotation task is huge and has a long annotation cycle, which can easily lead to problems such as decreased annotation accuracy and low annotation efficiency. There is still a lot of room for improvement. More importantly, this patent does not convert the annotation results into training data to improve model performance. In addition, although existing annotation methods and annotation systems can improve manual annotation to a certain extent, these methods and systems rarely focus on the field of medical dialogue generation models, and it is difficult to meet the professionalism and accuracy requirements in the medical field. The above problems greatly limit the efficiency and quality of medical text annotation. Summary of the Invention

[0005] The purpose of the present invention is to address the above-mentioned defects in the prior art and to provide a method and system for annotating conversational text representing doctor feedback, which improves the efficiency and quality of medical text annotation.

[0006] The present invention is achieved through at least one of the following technical solutions.

[0007] A method for annotating a conversation text to represent doctor feedback includes the following steps:

[0008] S1. Obtain a set of instruction texts to be classified and labeled, initialize a labeling task allocation pool, classify the set of instruction texts to be classified and labeled by department using an instruction text department classification model, obtain a classified set of instruction texts to be labeled and its classification labels, and store the classified set of instruction texts to be labeled and its classification labels in the labeling task allocation pool;

[0009] S2. The labeling task allocation pool automatically allocates labeling tasks to labeling doctors, wherein the labeling tasks include instruction text labeling tasks and reply text labeling tasks, wherein the reply text labeling tasks are allocated to labeling doctors in corresponding departments according to the classification labels of the corresponding instruction texts; the instruction text labeling tasks include the instruction text labeling system and the classified instruction text set to be labeled, and the reply text labeling tasks include the reply text labeling system and the reply text set to be labeled;

[0010] S3. If the annotation task assignment pool assigns an instruction text annotation task to the annotation doctor, the annotation doctor annotates the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtains an annotation result of the instruction text set, and selects a standardized annotated instruction text set according to the instruction text annotation result screening rule;

[0011] S4. The medical dialogue model generates a set of reply texts to be annotated based on a single annotated instruction text in the standardized annotated instruction text set, wherein the single annotated instruction text and the generated set of reply texts to be annotated form a single-instruction multiple-reply dataset, and performs a difference evaluation on the single-instruction multiple-reply dataset using a pre-trained doctor feedback model to obtain a difference evaluation result.

[0012] S5. Perform a difference check on the difference evaluation results of the single-command multiple-reply data set. If the check fails, return to step S4 until the check passes. If the check passes, obtain a differentiated set of reply texts to be annotated and store them in the annotation task allocation pool.

[0013] S6. If the annotation task allocation pool assigns a reply text annotation task to the annotation doctor, the annotation doctor annotates the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain an annotation result for the reply text set;

[0014] S7. Automatically verify the reply text set annotation results corresponding to the annotating doctor according to the reply text annotation result verification rules, and re-annotate or export the reply text set annotation results according to the verification results.

[0015] Furthermore, the set of instruction texts to be classified and annotated originates from an Internet hospital or an offline hospital; the classified set of instruction texts to be annotated is subdivided into first-level categories and second-level categories according to its classification labels, and the second-level categories will be further subdivided based on the first-level categories.

[0016] Furthermore, the instruction text department classification model is a two-layer pre-trained classification model based on the random forest model, wherein the first layer pre-trained classification model is used to classify the first-level departments, and the second layer pre-trained classification model is used to classify the second-level departments. The training steps of the instruction text department classification model are as follows:

[0017] 1) Manual classification to obtain a training text set T: The annotating doctor manually classifies the instruction text set to be classified and obtains a classified instruction text set N of texts as the training text set T;

[0018] 2) Randomly extracting samples and training a decision tree: randomly extracting N samples with replacement from the training text set T, extracting one sample each time, and finally forming a sample set D with N samples. A decision tree is trained using the formed sample set D as the sample at the root node of the decision tree;

[0019] 3) Randomly select features as node splitting features and divide the sample set: First, extract a keyword group that can characterize the department type from a single instruction text sample in the sample set D as the sample feature. Each sample has M features. When each node of the decision tree needs to be split, randomly select m features from the M features. Then, use the CART algorithm to select one feature from these m features as the splitting feature of the node. Divide the sample set D into two parts based on the splitting feature, and establish the left and right child nodes of the current split node. The sample set D of the left child node is D1, and the sample set D of the right child node is D2.

[0020] 4) The decision tree repeats step 3) to recursively split the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node split, then the node is a leaf node, and the classification corresponding to the leaf node is the final classification prediction;

[0021] 5) Following steps 1) to 4), a large number of decision trees are built in parallel to construct a random forest. At this point, the instruction text department classification model training is completed;

[0022] A new instruction text sample is input into the random forest in the instruction text department classification model. Each decision tree in the random forest judges and classifies the input instruction text sample respectively. Each decision tree will obtain the classification result of a certain department. The department classification model will use the department result with the most department categories in the classification results of the decision trees in the random forest as the final department classification result.

[0023] Furthermore, the instruction text annotation system includes the following indicators: the sensitivity of the personal identity information of the instruction text, the closed domain of the instruction text, the user intention of the instruction text, and the harmlessness of the content of the instruction text;

[0024] The instruction text annotation result screening rules include: protection of personal identity information of the instruction text annotation result, clarity of user intention of the instruction text annotation result, and harmlessness of the content of the instruction text annotation result.

[0025] Furthermore, the doctor feedback model is a model of a pre-trained Transformer encoder structure, wherein the Transformer encoder structure includes an encoder, a linear layer and a softmax layer, and the encoder includes a feedforward layer and a self-attention layer;

[0026] The difference evaluation results include the single-command multiple-reply data set and the difference between each reply text in the single-command multiple-reply data set, wherein the difference represents the difference between a certain reply text and other reply texts. The smaller the difference, the more similar the reply text is to the other reply texts.

[0027] Furthermore, in step S5, the specific steps of the difference verification are:

[0028] 1) Obtaining differential evaluation results of the doctor feedback model;

[0029] 2) Reading the difference of each reply text from the difference evaluation result, performing verification according to a preset difference interval, and generating a difference verification result for each reply text.

[0030] 3) If the difference of each reply text read from the difference evaluation result is within the preset difference range, the difference check of each reply text is passed, and a differentiated reply text set to be annotated is generated according to the single-instruction multiple-reply data set in the difference evaluation result; if the difference of a reply text read from the difference evaluation result is outside the preset difference range, there is a reply text that fails the difference check, and the reply text that fails the check is discarded, and the instruction text in the single-instruction multiple-reply data set in the difference evaluation result is read, and re-input into the preset medical dialogue model to generate a new reply text, forming a new single-instruction multiple-reply data set, and repeating steps S4 to S5.

[0031] Furthermore, the assigned reply text annotation system includes:

[0032] Annotating attributes of individual reply texts in the differentiated set of reply texts to be annotated, wherein the attributes include an overall evaluation of the reply text, the medical relevance of the reply text, the medical accuracy of the reply text, the medical safety of the reply text, the medical helpfulness of the reply text, and the continuity of the reply text;

[0033] Sorting and annotating the differentiated set of response texts to be annotated, wherein the sorting and annotation indicators include medical safety, medical relevance, medical accuracy, medical helpfulness, and medical authenticity in descending order of importance, and determining the order of all response texts in the differentiated set of response texts to be annotated based on the sorting and annotation indicators;

[0034] The reply text set annotation result includes: assigning scores to various attributes of each reply text in the differentiated reply text set to be annotated and the order of all reply texts in the differentiated reply text set to be annotated.

[0035] Furthermore, step S7 includes the following sub-steps:

[0036] S701. Automatically verify the annotation results of the reply text set corresponding to the annotating doctor according to a preset reply text annotation result verification rule to obtain an automatic verification result, wherein the preset reply text annotation result verification rule includes a annotation integrity verification and a annotation standardization verification;

[0037] S702: Determine whether there is a reply text annotation result that fails verification based on the automatic verification result, wherein the automatic verification result includes an annotation integrity verification result and an annotation standardization verification result. If any of the automatic verification results fails, the verification is deemed to have failed.

[0038] S703. When there are reply text annotation results that fail the verification, the reply text annotation results that fail the verification are returned to the corresponding annotation doctor for re-annotation, and steps S701 to S703 are repeated until all reply text annotation results pass the verification;

[0039] S704. When there are no doctor annotation results that fail the verification, a doctor feedback data set is formed based on the annotation results of the reply text set that pass the verification, the doctor feedback model is trained using the doctor feedback data set, and the reply text annotation result file in the specified JSON format can be exported using the annotation results of the reply text set that pass the verification.

[0040] A system for implementing the aforementioned method for annotating conversation text representing doctor feedback includes:

[0041] An instruction text set preprocessing module is used to obtain instruction text sets to be annotated from Internet hospitals and offline hospitals and a preset instruction text department classification model, and classify the instruction text sets to be annotated using the preset instruction text department classification model to obtain a classified instruction text set to be annotated and classification labels. The classified instruction text set to be annotated is subdivided into primary and secondary categories according to its classification labels, and the secondary categories are further subdivided based on the primary categories;

[0042] Annotation task assignment module, used to store and assign instruction text annotation tasks and reply text annotation tasks;

[0043] An instruction text set annotation and screening module is used to annotate the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtain the instruction text set annotation results, and screen out a standardized annotated instruction text set according to the preset instruction text annotation result screening rules;

[0044] a reply text set generation module, configured to obtain a preset medical dialogue model, generate a reply text set to be annotated based on a single annotated instruction text in the normalized annotated instruction text set using the preset medical dialogue model, and form a single-instruction multiple-reply dataset using the single annotated instruction text and the generated reply text set to be annotated;

[0045] a reply text set screening module, configured to obtain a preset doctor feedback model, perform a difference assessment on the single-command multiple-reply data set using the preset medical feedback model, obtain a difference assessment result of the single-command multiple-reply data set, perform a difference verification on the difference assessment result of the single-command multiple-reply data set, and generate a differentiated reply text set to be annotated based on the verification result or return the result to the reply text set generation module to regenerate the single-command multiple-reply data set;

[0046] A reply text set annotation and verification module is used to annotate the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain a reply text set annotation result; automatically verify the reply text set annotation result according to a preset reply text annotation result verification rule, and re-annotate or generate a reply text set annotation result according to the verification result;

[0047] An annotation result export module is used to export a reply text annotation result file in a specified JSON format based on the annotation results of the reply text set that has passed the automatic verification;

[0048] A cloud computing server, configured to run the instruction text department classification model, the medical dialogue model, and the doctor feedback model, and to perform iterative training of the instruction text department classification model and the doctor feedback model;

[0049] Multiple text annotation terminals are used to provide annotation operation terminals to annotation doctors to perform annotation operations on instruction text sets and reply text sets;

[0050] The web backend server is used to respectively connect and call the instruction text set preprocessing module, the annotation task allocation module, the instruction text annotation and screening module, the reply text set generation module, the reply text set screening module, the reply text set annotation and verification module, the annotation result export module, the cloud computing server and multiple text annotation terminals.

[0051] Furthermore, the instruction text department classification model in the instruction text set preprocessing module is a two-layer pre-trained classification model based on the random forest model.

[0052] 1) A manual classification unit, which is used for the annotation doctor to manually classify the instruction text set to be classified and annotated, and obtain a classified instruction text set to be annotated with a number of texts N as a training text set T;

[0053] 2) A sample random extraction unit is used to randomly extract N samples with replacement from the training text set T, extracting one sample each time, and finally forming a sample set D with N samples. A decision tree is trained using the formed sample set D as the sample at the root node of the decision tree.

[0054] 3) a feature random extraction unit, configured to extract a keyword group that can characterize the department type from a single instruction text sample in the sample set D as a feature of the sample; when each sample has M features, when each node of the decision tree needs to be split, randomly select m features from the M features, and then use the CART algorithm to select one feature from the m features as the split feature of the node; divide the sample set D into two parts based on the split feature, and simultaneously establish the left and right child nodes of the current split node, with the sample set D of the left child node being D1 and the sample set D of the right child node being D2;

[0055] 4) Node splitting unit, which is used to call the feature random extraction unit to recursively split the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node splits, the node is considered a leaf node, and the classification corresponding to the leaf node is the final classification prediction;

[0056] 5) Random forest construction unit, which is used to call the artificial classification unit, sample random extraction unit, feature random extraction unit and decision tree node splitting unit in sequence, build a large number of decision trees in parallel, construct a random forest, and thus complete the model training;

[0057] After the training is completed and the pre-trained classification model is obtained, a new instruction text sample is input into the random forest in the classification model. Each decision tree in the random forest judges and classifies the input instruction text sample respectively. Each decision tree will obtain the classification result of a certain department. The classification model will use the department result with the most categories in the classification results of the decision tree in the random forest as the final department classification result.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] (1) The labeling method and system proposed in the present invention unify all labeling doctors into one platform, forming a complete labeling process. At the same time, an instruction text department classification model based on a random forest model and a doctor feedback model based on a Transformer Encoder structure are added. On the one hand, it can greatly reduce the workload of labeling doctors and greatly improve the labeling efficiency of labeling doctors. On the other hand, the manual labeling results and manual classification results can form training data to train the above models, improve the performance of the algorithm model, and thus obtain a trained algorithm model to feed back to the labeling system, thereby improving the closed-loop efficiency.

[0060] (2) The annotation method and system proposed in the present invention include a complete and reasonable text annotation system that can meet the professionalism and accuracy requirements of the medical field. In addition, the technical solution disclosed in the present invention can accelerate the production speed of annotation results, thereby using the annotation results to form training data for better evaluation and improvement of dialogue generation models in the medical field. This has great practical significance for improving the performance of current dialogue generation models in the medical field.

[0061] (3) The annotation method and system proposed in the present invention include strict rules for filtering instruction text results and verifying reply text annotation results, which can effectively reduce the annotation error rate and improve the annotation accuracy of the annotating doctor;

[0062] (4) The annotation system proposed in the present invention provides a visual operation interface, which is easy to use and very friendly to the annotation doctors. It also provides user management functions. Administrators can use management tools to easily manage the information of the annotated doctors. The operation interface is developed based on the Python streamlit library, which is simple to deploy and has good portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is an overall flow chart of a method for annotating a conversation text representing doctor feedback according to an embodiment of the present invention;

[0064] Figure 2 2. It is a structural diagram of the instruction text department classification model in an embodiment of the present invention;

[0065] Figure 3 is a schematic structural diagram of a doctor feedback model in an embodiment of the present invention;

[0066] Figure 4 4 is an overall structural diagram of a conversation text annotation system for representing doctor feedback in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0068] This embodiment discloses a method for annotating a dialogue text to characterize doctor feedback. Figures 1 to 3 , Figure 1 FIG. 1 is a flow chart illustrating a method for annotating a conversation text representing a doctor's feedback according to an embodiment of the present disclosure. Figure 1 As shown in , the method for annotating a conversation text representing doctor feedback includes the following steps:

[0069] S1. Obtain a set of instruction texts to be classified and labeled, obtain an instruction text department classification model, initialize a labeling task allocation pool, classify the instruction texts to be classified and labeled by department using the instruction text department classification model, obtain a classified set of instruction texts to be labeled and its classification labels, and store the set of instruction texts to be labeled and its classification labels in the labeling task allocation pool.

[0070] The set of instruction texts to be classified and labeled is derived from an Internet hospital or an offline hospital.

[0071] As an embodiment, the classified instruction text set to be annotated is subdivided into primary and secondary categories according to its classification labels. The primary categories are non-medical, internal medicine, surgery, obstetrics and gynecology, pediatrics, orthopedics, ophthalmology, stomatology, otolaryngology and head and neck oncology, dermatology and venereology, andrology, skin beauty, burns, psychiatry, traditional Chinese medicine, integrated traditional Chinese and Western medicine, infectious diseases, rehabilitation medicine, anesthesiology, occupational diseases, nutrition, medical imaging, pathology, nursing, pharmacy, general practice, and others. The secondary categories are further subdivided based on the primary categories. For example, internal medicine is divided into cardiovascular medicine, neurology, gastroenterology, endocrinology, immunology, respiratory medicine, nephrology, hematology, infectious diseases, allergy, general medicine, and hyperbaric oxygen department. Surgery is divided into neurosurgery, functional neurosurgery, cardiovascular surgery, thoracic surgery, plastic surgery, breast surgery, urology, hepatobiliary surgery, anorectal surgery, vascular surgery, general surgery, and gastrointestinal surgery.

[0072] Among them, such as Figure 2As shown in , the instruction text category classification model is a two-layer pre-trained classification model based on the random forest model, wherein the first layer of the pre-trained classification model is used to classify the first-level departments, and the second layer of the pre-trained classification model is used to classify the second-level departments. The specific training steps of the random forest model are as follows:

[0073] 1) Manual classification to obtain a training text set T. The annotating doctor manually classifies the instruction text set to be classified and obtains a classified instruction text set N of texts as the training text set T;

[0074] 2) Randomly sample and train a decision tree. Randomly sample N times with replacement from the training text set T, sampling one sample each time, to form a sample set D with N samples. Use the sample set D to train a decision tree, using it as the sample at the root node of the decision tree.

[0075] 3) Randomly select features as node splitting features to divide the sample set. First, extract keyword groups that can characterize the department type from a single instruction text sample in the sample set D as the sample feature. When each sample has M features, when each node of the decision tree needs to be split, randomly select m features from these M features and satisfy the condition that m is less than M / 10. Then use the CART algorithm to select 1 feature from these m features as the splitting feature of the node. According to the splitting feature, the sample set can be divided into two parts D1 and D2, and the left and right child nodes of the current split node are established at the same time. The sample set D of the left child node is D1, and the sample set D of the right child node is D2.

[0076] The specific approach of the CART algorithm is as follows:

[0077] Assume that in the classification problem, there are K categories and the probability of the kth category is p k , then the expression of Gini index Gini is:

[0078]

[0079] Among them, the larger the Gini index is, the greater the uncertainty in classification is. Conversely, the smaller the Gini index is, the smaller the uncertainty in classification is, and the more thorough the classification of the text is.

[0080] When traversing each feature of the sample, if the sample set D is divided into two parts, D1 and D2, according to feature A, D1 is the sample set that meets feature A, and D2 is the sample set that does not meet feature A, then the Gini index of the sample set D under the condition of feature A is:

[0081]

[0082] Among them, |D| represents the number of elements in sample set D, |D1| represents the number of elements in sample set D1, |D2| represents the number of elements in sample set D2, Gini(D1) represents the Gini index of sample set D1, Gini(D2) represents the Gini index of sample set D2, and Gini(D,A) represents the Gini index of sample set D after segmentation by feature A, which represents the uncertainty of set D after segmentation by feature A.

[0083] Each decision tree in the random forest is based on the CART algorithm, which continuously traverses all possible split features of the feature subset of the decision tree to find the split feature with the smallest Gini index, and divides the data set into two subsets until the Gini index is less than a preset threshold.

[0084] 4) The decision tree repeats step 3) recursively splitting the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node split, then the node is a leaf node, and the classification corresponding to the leaf node is the final classification prediction;

[0085] 5) Follow steps 1) to 4) to build a large number of decision trees in parallel and construct a random forest. The model training is now complete.

[0086] After the training is completed and the pre-trained classification model is obtained, a new instruction text sample is input into the random forest in the classification model. Each decision tree in the random forest judges and classifies the input instruction text sample respectively. Each decision tree will obtain the classification result of a certain department. The classification model will use the department result with the most categories in the classification results of the decision trees in the random forest as the final department classification result.

[0087] S2. The annotation task allocation pool automatically assigns annotation tasks to annotation doctors. The annotation tasks include instruction text annotation tasks and reply text annotation tasks. The reply text annotation tasks are assigned to annotation doctors in corresponding departments based on the classification labels of the corresponding instruction texts. The instruction text annotation tasks include a preset instruction text annotation system and the set of classified instruction texts to be annotated. The reply text annotation tasks include a preset reply text annotation system and the set of reply texts to be annotated.

[0088] S3. If the annotation task assignment pool assigns an instruction text annotation task to the annotation doctor, the annotation doctor annotates the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtains the annotation results of the instruction text set, and selects a standardized annotated instruction text set according to the preset instruction text annotation result screening rules;

[0089] In a specific embodiment, step S3 includes the following steps:

[0090] S301. Acquire an assigned preset instruction text annotation system, wherein the assigned preset instruction text annotation system includes the following indicators:

[0091] The sensitivity of the personally identifiable information in the instruction text, the closed domain of the instruction text, the user's intent in the instruction text, and the harmlessness of the instruction text's content;

[0092] S302: The annotation doctor annotates all indicators of the classified instruction text set according to the assigned preset instruction text annotation system to obtain an instruction text set annotation result.

[0093] As an example, the annotation content may be:

[0094] The indicator of the sensitivity of the personal identity information of the instruction text is marked with "yes" or "no". If the instruction text contains information that can be used to personally identify a person, such as name, company name, contact number, ID number, etc., it is marked as "yes"; if it does not contain personal information, it is marked as "no".

[0095] The indicator of the closed field of the instruction text is marked with "yes" or "no". If no more information than the provided information should be used, the instruction text is a "closed field" and is marked with "yes", otherwise it is marked with "no".

[0096] The indicator of the user intention of the instruction text is marked with "yes" or "no". If it is unclear what the user wants the artificial intelligence system to do based on the instruction text, then the instruction text is unclear in user intention and is marked as "no". Otherwise, the instruction text is clear in user intention and is marked as "yes".

[0097] The indicator of the harmlessness of the content of the instruction text is marked with "yes" or "no". If the instruction text asks about pornography, violence, encouragement of violence, abuse, terrorism, self-harm content, or asks about content that defames (unfair criticism) protected classes, including: nationality or ancestry, gender, age, physical or mental disability, veteran status, genetic information, nationality, etc., then the content of the instruction text is harmful and marked as "yes". Otherwise, the content of the instruction text is harmless and marked as "no".

[0098] S303: Obtain preset instruction text annotation result screening rules, wherein the instruction text annotation result screening rules include:

[0099] The protection of personally identifiable information in the instruction text annotation results, the clarity of user intent in the instruction text annotation results, and the harmlessness of the content of the instruction text annotation results;

[0100] S304: Automatically screen the annotation results of the instruction text set corresponding to the annotating doctor according to the preset instruction text annotation result screening rules to obtain a standardized annotated instruction text set. The automatic screening process is specifically as follows:

[0101] According to the instruction text annotation result screening rules, if the personal identity information sensitivity index in the instruction text annotation result is marked as "yes", then for the protection of personal identity information, the instruction text annotation result needs to be eliminated, otherwise the instruction text annotation result is retained.

[0102] If the user intention indicator in the instruction text annotation result is marked as "no", then for the sake of the clarity of the user intention of the instruction text annotation result, the instruction text annotation result needs to be eliminated, otherwise the instruction text annotation result is retained.

[0103] If the content harmlessness index in the instruction text annotation result is marked as "yes", then for the sake of the harmlessness of the content of the instruction text annotation result, the instruction text annotation result needs to be removed, otherwise the instruction text annotation result is retained.

[0104] S4. Inputting a single labeled instruction text from the standardized labeled instruction text set into the medical dialogue model to generate a set of response texts to be labeled, wherein the single labeled instruction text and the generated set of response texts to be labeled form a single-instruction, multiple-response dataset. Performing a difference evaluation on the single-instruction, multiple-response dataset using a pre-trained physician feedback model to obtain a difference evaluation result.

[0105] The preset medical dialogue model calls an existing pre-trained medical dialogue generation model. The pre-trained medical dialogue generation model is a medical dialogue generation model that is trained and improved by generating training data through the present annotation method. It can be changed to a different medical dialogue generation model according to the usage scenario and is not limited to a specific medical dialogue generation model.

[0106] The doctor feedback model is a model of a Transformer encoder structure trained on a doctor feedback dataset, wherein the doctor feedback dataset is formed based on the annotation results of the reply text set that has passed the verification in the reply text annotation task. The Transformer encoder structure includes an encoder, a linear layer, and a softmax layer, and the encoder includes a feedforward layer and a self-attention layer. The process of performing a difference evaluation on the single-command multiple-reply dataset by the pre-trained doctor feedback model specifically includes the following steps:

[0107] S401: Input a single-command multiple-reply dataset file in JSON format to the doctor feedback model, and extract all reply texts to be annotated from the single-command multiple-reply dataset file in JSON format;

[0108] S402: Input the extracted single reply text into the encoder, which extracts information of the reply text and encodes it into an output vector representation, wherein the output vector includes relationship information between the reply text and other reply texts;

[0109] S403: Input the relationship information into a linear layer to obtain a difference score between the reply text and other reply texts;

[0110] S404: Input the difference score into a softmax layer to obtain a normalized difference value between 0 and 1.

[0111] S405, repeating the above steps S402 to S405, inputting each extracted reply text into the encoder, linear layer and softmax layer in sequence until the difference of all reply texts is obtained;

[0112] S406 , integrating the differences of all reply texts into the single-command multiple-reply dataset file in JSON format input in step S401 , and obtaining a difference evaluation result stored in JSON format.

[0113] The difference evaluation results include the single-command multiple-reply data set and the difference between each reply text in the single-command multiple-reply data set, wherein the difference represents the difference between a certain reply text and other reply texts. The smaller the difference, the more similar the reply text is to the other reply texts.

[0114] S5. Perform a difference check on the difference evaluation results of the single-command multiple-reply data set. If the check fails, return to step S4 until the check passes. If the check passes, obtain a differentiated set of reply texts to be annotated and store them in the annotation task allocation pool.

[0115] The specific steps of the difference verification are as follows:

[0116] 1) Obtain the difference evaluation results of the doctor feedback model and store them in a JSON format file;

[0117] 2) The difference between each reply text read from the JSON file of the difference evaluation result is verified according to the set difference interval to generate the difference verification result of each reply text, wherein the difference interval will be set and adjusted according to the actual application requirements. The principle should be to avoid high duplication between reply texts while avoiding too low correlation between reply texts.

[0118] As an example, in the application, the difference range is set to 0.6 to 0.8. When the difference of a reply text is lower than 0.6, it means that the reply text is too similar to other reply texts and does not meet the requirements of this application; when the difference of a reply text is higher than 0.8, it means that the relevance of the reply text to other reply texts is too low and does not meet the requirements of this application.

[0119] 3) If the difference of each reply text read from the difference evaluation result is within the set difference range, the difference check of each reply text is passed, and a differentiated reply text set to be annotated is generated according to the single-instruction multiple-reply data set in the difference evaluation result; if the difference of a reply text read from the difference evaluation result is outside the set difference range, there is a reply text that fails the difference check, and the reply text that fails the check is discarded, and the instruction text in the single-instruction multiple-reply data set in this difference evaluation result is read and re-input into the medical dialogue model to generate a new reply text, forming a new single-instruction multiple-reply data set, and repeating steps S4 to S5.

[0120] S6. If the annotation task allocation pool assigns a reply text annotation task to the annotation doctor, the annotation doctor annotates the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain an annotation result for the reply text set;

[0121] The reply text annotation system includes:

[0122] Attribute annotation of a single reply text in the differentiated set of reply texts to be annotated, wherein the attributes include an overall evaluation of the reply text, the medical relevance of the reply text, the medical accuracy of the reply text, the medical safety of the reply text, the medical helpfulness of the reply text, and the continuity of the reply text

[0123] As an embodiment, the overall evaluation of the reply text is marked with a score of 1 to 7, where 1 means it is terrible, completely useless, and may cause real harm, and 7 means the output is almost perfect and the annotator can't think of a better way.

[0124] The medical relevance of the response text is marked with "yes" or "no", which is not related to the quality of the response text. For example, if a person needs treatment for a sprained ankle, but the output is a treatment for burns, it will be marked as "no". If the output is related to the sprained ankle treatment, it should be marked as "yes" even if it does not effectively solve the problem.

[0125] The medical accuracy of the response text is marked with "yes" or "no", that is, the output of the model should be consistent with the general understanding of people in the medical field. Taking a sprained ankle as an example, if a solution to burns is given and the solution to burns is medically accurate, then it should be marked as "yes", otherwise it should be marked as "no".

[0126] The medical safety of the response text is marked with a score of 0 to 2, where 0 indicates that it is clearly unsafe, 1 indicates that the output (or input) is slightly unsafe or is considered unsafe by some people, and 2 indicates that the output (or input) is completely unsafe.

[0127] The medical helpfulness of the response text is marked with "yes" or "no". Helpfulness means that the model's output can effectively alleviate or relieve the symptoms mentioned in the user's question. For example, if the model output can effectively alleviate the pain and other discomfort symptoms caused by burns, it will be marked as "yes". However, if the model outputs a solution for a sprained ankle but cannot effectively resolve the burn, it should be marked as "no".

[0128] The continuity of the response text is marked on a scale of 0 to 2, where the output has high continuity if it is a reasonable continuation of the input text, 0 means that the output is not reasonable at all given the input text, 1 means that the output is somewhat specious and it contains one error, and 2 means that the output is perfect.

[0129] Sorting and annotating the differentiated set of reply texts to be annotated, wherein the sorting and annotation indicators include medical safety, medical relevance, medical accuracy, medical helpfulness and medical authenticity, and their importance decreases in order, and the order of all reply texts in the differentiated set of reply texts to be annotated is determined according to the sorting and annotation indicators, and at most only two reply texts are allowed to be in parallel. For example, there are 5 reply texts, namely reply text A, reply text B, reply text C, reply text D, and reply text E. If the order of the 5 reply texts is determined according to the sorting and annotation indicators to be reply text A before reply text B, reply text B before reply text C, reply text C and reply text D in parallel, and reply text D before reply text E, then the sorting and annotation result should be A>B>C=D>E.

[0130] The reply text set annotation result includes: the annotation result of each attribute of each reply text in the differentiated reply text set to be annotated and the order of all reply texts in the differentiated reply text set to be annotated;

[0131] S7: Automatically verify the reply text set annotation results corresponding to the annotating doctor according to the preset reply text annotation result verification rules, and re-annotate or export the reply text set annotation results according to the verification results. Specifically, the following steps are included:

[0132] S701. Automatically verify the reply text annotation result corresponding to the annotating doctor according to the preset reply text annotation result verification rules to obtain an automatic verification result, wherein the preset reply text annotation result verification rules include annotation integrity verification and annotation standardization verification.

[0133] Specifically, the integrity check is to check whether there are any missing annotation indicators that have not been marked during the annotation process of the reply text by the annotating doctor, such as whether there are any attribute annotations of the reply text that have not been marked, and whether there are any sorting annotations of the reply text set to be marked that have not been marked. The annotation standardization check is to check the standardization of the annotation operation of the annotating doctor during the annotation process of the reply text. For example, when annotating the attribute of the reply text, the overall evaluation of the reply text is required to be marked with a score of 1 to 7. If it is marked with other scores, it is not standardized. When annotating the sorting of the reply text set to be marked, it is required that only two reply texts are allowed to be listed in parallel. If there are more than two reply texts listed in parallel order, it is not standardized.

[0134] S702: Determine whether there is a reply text annotation result that fails verification based on the automatic verification result, wherein the automatic verification result includes an annotation integrity verification result and an annotation standardization verification result. If any of the automatic verification results fails, the verification is deemed to have failed.

[0135] S703. When there are reply text annotation results that fail the verification, the reply text annotation results that fail the verification are returned to the corresponding annotation doctor for re-annotation, and the above steps are repeated until all reply text annotation results pass the verification;

[0136] S704. When there are no doctor annotation results that have failed verification, a doctor feedback dataset is formed based on the annotation results of the reply text set that have passed verification, so as to train the doctor feedback model using the doctor feedback dataset, and a reply text annotation result file in a specified JSON format can be exported using the annotation results of the reply text set that has passed verification;

[0137] Among them, the annotation results based on the reply text set will form training data for training the preset medical dialogue model.

[0138] As another embodiment, this embodiment is a system for implementing a method for annotating a conversation text representing a doctor's feedback. Figure 4 , Figure 4 The present invention illustrates a conversation text annotation system for characterizing doctor feedback according to an embodiment of the present disclosure, the system comprising the following modules:

[0139] The instruction text set preprocessing module is used to obtain the instruction text set to be labeled from Internet hospitals and offline hospitals and the instruction text department classification model, and use the instruction text department classification model to classify the instruction text set to be labeled to obtain the classified instruction text set to be labeled and its classification label;

[0140] The classified instruction text set to be annotated is subdivided into first-level categories and second-level categories according to its classification labels.

[0141] The instruction text department classification model is a two-layer pre-trained classification model based on the random forest model, including the following units:

[0142] A manual classification unit, which is used for the annotation doctor to manually classify the set of instruction texts to be classified;

[0143] The sample random extraction unit is used to randomly extract N samples with replacement from the training text set T, extracting one sample each time, and finally forming a sample set D with N samples. A decision tree is trained using the formed sample set D as the sample at the root node of the decision tree.

[0144] A feature random extraction unit is used to extract a keyword group that can characterize the department type from a single instruction text sample in the sample set D as a feature of the sample, and then use the CART algorithm to select a feature from the m features as the split feature of the node. According to the split feature, the sample set D is divided into two parts D1 and D2, and the left and right child nodes of the current split node are established at the same time, with the sample set D of the left child node being D1 and the sample set D of the right child node being D2;

[0145] The node splitting unit is used to call the feature random extraction unit to recursively split the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node splits, then the node is a leaf node, and the classification corresponding to the leaf node is the final classification prediction;

[0146] The random forest construction unit is used to call the artificial classification unit, the sample random extraction unit, the feature random extraction unit and the decision tree node splitting unit in sequence, establish a large number of decision trees in parallel, construct a random forest, and thus complete the model training.

[0147] Annotation task assignment module, used to store and assign instruction text annotation tasks and reply text annotation tasks;

[0148] The labeling task assignment module includes:

[0149] A labeling task storage unit, configured to store data for instruction text labeling tasks and reply text labeling tasks, wherein the instruction text labeling tasks include a preset instruction text labeling system and the classified set of instruction texts to be labeled, and the reply text labeling tasks include a preset reply text labeling system and the set of reply texts to be labeled;

[0150] a marking task assigning unit, configured to assign the stored instruction text marking task or reply text marking task to a marking doctor, wherein when assigning a reply text marking task, the task will be assigned to a marking doctor of a corresponding department according to the classification label;

[0151] An instruction text set annotation and screening module is used to annotate the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtain the instruction text set annotation results, and screen out a standardized annotated instruction text set according to the preset instruction text annotation result screening rules;

[0152] The instruction text set annotation and screening module includes:

[0153] An instruction text set annotation unit is configured to annotate the classified instruction text set to be annotated based on the preset instruction text annotation system, the annotation doctor, and other indicators such as the primary medical discipline, secondary medical discipline, personal identity information sensitivity, closed domain, user intent, and content harmlessness of the instruction text to be annotated, to obtain an instruction text set annotation result;

[0154] The instruction text set annotation result screening unit is used to automatically screen the instruction text set annotation results based on the preset instruction text annotation result screening rules, from multiple indicators such as the protection of personal identity information, the clarity of user intentions and the harmlessness of content, to obtain a standardized annotated instruction text set.

[0155] a reply text set generation module, configured to obtain a preset medical dialogue model, generate a reply text set to be annotated based on a single annotated instruction text in the normalized annotated instruction text set using the preset medical dialogue model, and form a single-instruction multiple-reply dataset using the single annotated instruction text and the generated reply text set to be annotated;

[0156] a reply text set screening module, configured to obtain a preset doctor feedback model, perform a difference assessment on the single-command multiple-reply data set using the preset medical feedback model, obtain a difference assessment result of the single-command multiple-reply data set, perform a difference verification on the difference assessment result of the single-command multiple-reply data set, and generate a differentiated reply text set to be annotated based on the verification result or return the result to the reply text set generation module to regenerate the single-command multiple-reply data set;

[0157] A reply text set annotation and verification module is configured to annotate the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain a reply text set annotation result. The reply text set annotation result is automatically verified according to a preset reply text annotation result verification rule, and re-annotated or generated a reply text set annotation result based on the verification result;

[0158] The reply text set annotation and verification module includes:

[0159] The reply text set annotation unit is used to perform attribute annotation on individual reply texts in the differentiated reply text set to be annotated based on the assigned reply text annotation system from multiple indicators such as overall evaluation of the reply text, medical relevance, medical accuracy, medical safety, medical helpfulness and continuity, and to sort and annotate the differentiated reply text set to be annotated based on multiple indicators such as medical safety, medical relevance, medical accuracy, medical helpfulness and medical authenticity, and a maximum of two reply texts are allowed to be listed in parallel.

[0160] The automatic verification unit is used to automatically verify the reply text set annotation results from the two indicators of annotation completeness and annotation standardization according to the preset reply text annotation result verification rules, and obtain the annotation completeness verification result and the annotation standardization verification result.

[0161] The re-marking unit is used to return the reply text marking result of the failed verification to the corresponding marking doctor for re-marking when the automatic verification fails.

[0162] a labeling result export module, configured to export a reply text labeling result file in a specified JSON format based on the automatically verified reply text set labeling results. The reply text set labeling results will form training data for training the preset medical dialogue model;

[0163] A cloud computing server, configured to run the instruction text department classification model, the medical dialogue model, and the doctor feedback model, and to perform iterative training of the instruction text department classification model and the doctor feedback model, wherein the cloud computing server is a GPU server;

[0164] Multiple text annotation terminals are used to provide annotation operation terminals to annotation doctors to perform annotation operations on instruction text sets and reply text sets;

[0165] The web backend server is used to respectively connect and call the instruction text set preprocessing module, the annotation task allocation module, the instruction text annotation and screening module, the reply text set generation module, the reply text set screening module, the reply text set annotation and verification module, the annotation result export module, the cloud computing server and multiple text annotation terminals to run the conversation text annotation method for representing doctor feedback.

[0166] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for annotating conversation texts to characterize doctor feedback, characterized in that: The following steps are involved: S1. Obtain a set of instruction texts to be classified and labeled, initialize a labeling task allocation pool, classify the set of instruction texts to be classified and labeled by department using an instruction text department classification model, obtain a classified set of instruction texts to be labeled and its classification labels, and store the classified set of instruction texts to be labeled and its classification labels in the labeling task allocation pool; S2. The labeling task allocation pool automatically allocates labeling tasks to labeling doctors, wherein the labeling tasks include instruction text labeling tasks and reply text labeling tasks, wherein the reply text labeling tasks are allocated to labeling doctors in corresponding departments according to the classification labels of the corresponding instruction texts; the instruction text labeling tasks include the instruction text labeling system and the classified instruction text set to be labeled, and the reply text labeling tasks include the reply text labeling system and the reply text set to be labeled; S3. If the annotation task assignment pool assigns an instruction text annotation task to the annotation doctor, the annotation doctor annotates the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtains an annotation result of the instruction text set, and selects a standardized annotated instruction text set according to the instruction text annotation result screening rule; S4. The medical dialogue model generates a set of reply texts to be annotated based on a single annotated instruction text in the standardized annotated instruction text set, wherein the single annotated instruction text and the generated set of reply texts to be annotated form a single-instruction multiple-reply dataset, and performs a difference evaluation on the single-instruction multiple-reply dataset using a pre-trained doctor feedback model to obtain a difference evaluation result. S5. Perform a difference check on the difference evaluation results of the single-command multiple-reply data set. If the check fails, return to step S4 until the check passes. If the check passes, obtain a differentiated set of reply texts to be annotated and store them in the annotation task allocation pool. S6. If the annotation task allocation pool assigns a reply text annotation task to the annotation doctor, the annotation doctor annotates the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain an annotation result for the reply text set; S7. Automatically verify the reply text set annotation results corresponding to the annotating doctor according to the reply text annotation result verification rules, and re-annotate or export the reply text set annotation results according to the verification results.

2. A method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: The set of instruction texts to be classified and labeled originates from an Internet hospital or an offline hospital; the classified set of instruction texts to be labeled is subdivided into primary categories and secondary categories according to its classification labels, and the secondary categories will be further subdivided based on the primary categories.

3. The method for annotating a conversation text representing doctor feedback according to claim 2, characterized in that: The instruction text department classification model is a two-layer pre-trained classification model based on the random forest model, wherein the first layer pre-trained classification model is used to classify the first-level departments, and the second layer pre-trained classification model is used to classify the second-level departments. The training steps of the instruction text department classification model are as follows: 1) Manual classification to obtain a training text set T: The annotating doctor manually classifies the instruction text set to be classified and obtains a classified instruction text set N of texts as the training text set T; 2) Randomly extracting samples and training a decision tree: randomly extracting N samples with replacement from the training text set T, extracting one sample each time, and finally forming a sample set D with N samples. A decision tree is trained using the formed sample set D as the sample at the root node of the decision tree; 3) Randomly select features as node splitting features and divide the sample set: First, extract a keyword group that can characterize the department type from a single instruction text sample in the sample set D as the sample feature. Each sample has M features. When each node of the decision tree needs to be split, randomly select m features from the M features. Then, use the CART algorithm to select one feature from these m features as the splitting feature of the node. Divide the sample set D into two parts based on the splitting feature, and establish the left and right child nodes of the current split node. The sample set D of the left child node is D1, and the sample set D of the right child node is D2. 4) The decision tree repeats step 3) to recursively split the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node split, then the node is a leaf node, and the classification corresponding to the leaf node is the final classification prediction; 5) Following steps 1) to 4), a large number of decision trees are built in parallel to construct a random forest. At this point, the instruction text department classification model training is completed; A new instruction text sample is input into the random forest in the instruction text department classification model. Each decision tree in the random forest judges and classifies the input instruction text sample respectively. Each decision tree will obtain the classification result of a certain department. The department classification model will use the department result with the most department categories in the classification results of the decision trees in the random forest as the final department classification result.

4. The method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: The instruction text annotation system includes the following indicators: the sensitivity of the personal identity information of the instruction text, the closed domain of the instruction text, the user intention of the instruction text, and the harmlessness of the content of the instruction text; The instruction text annotation result screening rules include: protection of personal identity information of the instruction text annotation result, clarity of user intention of the instruction text annotation result, and harmlessness of the content of the instruction text annotation result.

5. The method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: The doctor feedback model is a model of a pre-trained Transformer encoder structure, wherein the Transformer encoder structure includes an encoder, a linear layer and a softmax layer, and the encoder includes a feedforward layer and a self-attention layer; The difference evaluation results include the single-command multiple-reply data set and the difference between each reply text in the single-command multiple-reply data set, wherein the difference represents the difference between a certain reply text and other reply texts. The smaller the difference, the more similar the reply text is to the other reply texts.

6. The method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: In step S5, the specific steps of the difference verification are: 1) Obtaining differential evaluation results of the doctor feedback model; 2) reading the difference of each reply text from the difference evaluation result, performing verification according to a preset difference interval, and generating a difference verification result for each reply text; 3) If the difference of each reply text read from the difference evaluation result is within the preset difference range, the difference check of each reply text is passed, and a differentiated reply text set to be annotated is generated according to the single-instruction multiple-reply data set in the difference evaluation result; if the difference of a reply text read from the difference evaluation result is outside the preset difference range, there is a reply text that fails the difference check, and the reply text that fails the check is discarded, and the instruction text in the single-instruction multiple-reply data set in the difference evaluation result is read, and re-input into the preset medical dialogue model to generate a new reply text, forming a new single-instruction multiple-reply data set, and repeating steps S4 to S5.

7. The method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: The assigned reply text annotation system includes: Annotating attributes of individual reply texts in the differentiated set of reply texts to be annotated, wherein the attributes include an overall evaluation of the reply text, the medical relevance of the reply text, the medical accuracy of the reply text, the medical safety of the reply text, the medical helpfulness of the reply text, and the continuity of the reply text; Sorting and annotating the differentiated set of response texts to be annotated, wherein the sorting and annotation indicators include medical safety, medical relevance, medical accuracy, medical helpfulness, and medical authenticity in descending order of importance, and determining the order of all response texts in the differentiated set of response texts to be annotated based on the sorting and annotation indicators; The reply text set annotation result includes: assigning scores to various attributes of each reply text in the differentiated reply text set to be annotated and the order of all reply texts in the differentiated reply text set to be annotated.

8. The method for annotating a conversation text representing doctor feedback according to claim 1, characterized in that: Step S7 includes the following sub-steps: S701. Automatically verify the annotation results of the reply text set corresponding to the annotating doctor according to a preset reply text annotation result verification rule to obtain an automatic verification result, wherein the preset reply text annotation result verification rule includes a annotation integrity verification and a annotation standardization verification; S702: Determine whether there is a reply text annotation result that fails verification based on the automatic verification result, wherein the automatic verification result includes an annotation integrity verification result and an annotation standardization verification result. If any automatic verification result fails, the verification is deemed to have failed. S703. When there are reply text annotation results that fail the verification, the reply text annotation results that fail the verification are returned to the corresponding annotation doctor for re-annotation, and steps S701 to S703 are repeated until all reply text annotation results pass the verification; S704. When there are no doctor annotation results that fail the verification, a doctor feedback data set is formed based on the annotation results of the reply text set that pass the verification, the doctor feedback model is trained using the doctor feedback data set, and the reply text annotation result file in the specified JSON format can be exported using the annotation results of the reply text set that pass the verification.

9. A system for implementing the method for annotating a conversation text representing doctor feedback as described in claim 1, characterized in that: include: An instruction text set preprocessing module is used to obtain instruction text sets to be annotated from Internet hospitals and offline hospitals and a preset instruction text department classification model, and classify the instruction text sets to be annotated using the preset instruction text department classification model to obtain a classified instruction text set to be annotated and classification labels. The classified instruction text set to be annotated is subdivided into primary and secondary categories according to its classification labels, and the secondary categories are further subdivided based on the primary categories; Annotation task assignment module, used to store and assign instruction text annotation tasks and reply text annotation tasks; An instruction text set annotation and screening module is used to annotate the classified instruction text set to be annotated according to the assigned instruction text annotation system, obtain the instruction text set annotation results, and screen out a standardized annotated instruction text set according to the preset instruction text annotation result screening rules; a reply text set generation module, configured to obtain a preset medical dialogue model, generate a reply text set to be annotated based on a single annotated instruction text in the normalized annotated instruction text set using the preset medical dialogue model, and form a single-instruction multiple-reply dataset using the single annotated instruction text and the generated reply text set to be annotated; a reply text set screening module, configured to obtain a preset doctor feedback model, perform a difference assessment on the single-command multiple-reply data set using the preset medical feedback model, obtain a difference assessment result of the single-command multiple-reply data set, perform a difference verification on the difference assessment result of the single-command multiple-reply data set, and generate a differentiated reply text set to be annotated based on the verification result or return the result to the reply text set generation module to regenerate the single-command multiple-reply data set; A reply text set annotation and verification module is used to annotate the differentiated reply text set to be annotated according to the assigned reply text annotation system to obtain a reply text set annotation result; automatically verify the reply text set annotation result according to a preset reply text annotation result verification rule, and re-annotate or generate a reply text set annotation result according to the verification result; The annotation result export module is used to export the reply text annotation result file in the specified JSON format based on the annotation results of the reply text set that has passed the automatic verification; A cloud computing server, configured to run the instruction text department classification model, the medical dialogue model, and the doctor feedback model, and to perform iterative training of the instruction text department classification model and the doctor feedback model; Multiple text annotation terminals are used to provide annotation operation terminals to annotation doctors to perform annotation operations on instruction text sets and reply text sets; The web backend server is used to respectively connect and call the instruction text set preprocessing module, the annotation task allocation module, the instruction text annotation and screening module, the reply text set generation module, the reply text set screening module, the reply text set annotation and verification module, the annotation result export module, the cloud computing server and multiple text annotation terminals.

10. The conversation text annotation system for characterizing doctor feedback according to claim 9, characterized in that: The instruction text department classification model in the instruction text set preprocessing module is a two-layer pre-trained classification model based on the random forest model. 1) A manual classification unit, which is used for the annotation doctor to manually classify the instruction text set to be classified and annotated, and obtain a classified instruction text set to be annotated with a number of texts N as a training text set T; 2) a sample random extraction unit, configured to randomly extract N samples with replacement from the training text set T, extracting one sample each time, and ultimately forming a sample set D with N samples, and using the formed sample set D to train a decision tree as the sample at the root node of the decision tree; 3) a feature random extraction unit, configured to extract a keyword group that can characterize the department type from a single instruction text sample in the sample set D as a feature of the sample; when each sample has M features, when each node of the decision tree needs to be split, randomly select n features from the M features, and then use the CART algorithm to select one feature from the m features as the split feature of the node; divide the sample set D into two parts based on the split feature, and simultaneously establish the left and right child nodes of the current split node, with the sample set D of the left child node being D1 and the sample set D of the right child node being D2; 4) Node splitting unit, which is used to call the feature random extraction unit to recursively split the left and right child nodes until a leaf node is formed. If the node splitting feature selected by the child node is the same as the feature selected when its parent node splits, the node is considered a leaf node, and the classification corresponding to the leaf node is the final classification prediction; 5) Random forest construction unit, which is used to call the artificial classification unit, sample random extraction unit, feature random extraction unit and decision tree node splitting unit in sequence, build a large number of decision trees in parallel, construct a random forest, and thus complete the model training; After the training is completed and the pre-trained classification model is obtained, a new instruction text sample is input into the random forest in the classification model. Each decision tree in the random forest judges and classifies the input instruction text sample respectively. Each decision tree will obtain the classification result of a certain department. The classification model will use the department result with the most categories in the classification results of the decision trees in the random forest as the final department classification result.

Images