Comprehensive internet medical data cleaning method and system, and electronic equipment

Abstract

The invention relates to a comprehensive internet medical data cleaning method and system and electronic equipment, and belongs to the field of medical data processing. The method comprises the steps that a data cleaning framework is constructed to conduct comprehensive data cleaning on acquired medical data, and a quality filtering module in the data cleaning framework screens and cleans the data based on keywords, text statistical features and measurement features. The data cleaning framework comprises four modules: a format unification module for converting medical data into a unified standard format; the quality filtering module is used for eliminating noise and invalid information in the data; the data deduplication module identifies and deletes duplicate information in the data on a text level and a line level; the privacy removal module recognizes and removes personal identity recognition information. The method improves the quality of Chinese internet medical data, efficiently processes a large amount of medical dialogue data, has the characteristic of high modularization, supports off-line deployment, does not generate extra cost, and effectively avoids the problem of privacy disclosure.

Description

Technical Field

[0001] The present invention relates to a comprehensive Internet medical data cleaning method, system and electronic equipment, belonging to the technical field of medical data processing. Background Art

[0002] Medical big models have attracted widespread attention in recent years due to their wide application value. With the development of medical big models, people have higher and higher requirements for their basic capabilities. At present, large-scale high-quality data can improve the basic capabilities of medical big models better than the improvement brought by algorithm changes. Therefore, the scale and quality of medical data have become the key to building medical big models. Internet medical data, as the most common form of medical data, often contains various types of noise and needs to be cleaned before it can be used as training corpus for medical big models. Current research has proposed a variety of data cleaning methods, which can be summarized as rule-based methods and learning-based methods. However, rule-based methods have the problem of incomplete rule design, while learning-based methods often need to use ChatGPT, word embedding models or quality classification models to clean data, and sometimes manual inspection is required for assistance, which has a high cleaning cost. How to effectively improve the quality of Internet medical data and develop more comprehensive, efficient and low-cost data cleaning methods has become a key issue that needs to be solved in the current field of medical big model research and development.

[0003] Currently, most of the cleaning of Internet medical data is done when building Internet medical data sets and training corpora for medical big models. These works do not show the specific details of data cleaning, do not perform quality analysis on the data, and lack an implementable reference path. In addition, there is currently a lack of comprehensive, effective or low-cost cleaning methods for large-scale Chinese Internet medical data. This means that the current open source Internet medical data sets are likely to contain noise. If these data are directly used as training corpora for medical big models, they may have a certain impact on the performance of medical big models. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a comprehensive Internet medical data cleaning method, system, and electronic device to improve the quality of Chinese Internet medical data. The cleaning method of the present invention is more comprehensive, more efficient, and has lower cost.

[0005] The technical solution of the present invention is: a comprehensive method for cleaning Internet medical data, the method comprising:

[0006] Step 1: Use web crawlers to obtain medical data;

[0007] Step 2: Build a data cleaning framework to perform comprehensive data cleaning on the acquired medical data. The quality filtering module in the data cleaning framework screens and cleans the data based on keywords, text statistical features, and metric features.

[0008] Furthermore, in step 1, the web crawler automatically crawls Chinese Internet medical data from the Internet; Chinese Internet medical data includes online consultation platforms, official websites of medical institutions, health information sharing forums and social media.

[0009] The Chinese Internet medical data of the present invention is data obtained from an online medical (consultation) platform, and the data is mainly obtained by web crawlers. Web crawlers are computer programs that follow pre-set rules. Web crawlers automatically crawl Chinese Internet medical data from the Internet; medical data crawling is a technology involving information security and patient privacy protection. Therefore, before crawling Internet medical data, relevant laws and regulations and ethical standards need to be followed to ensure the legality and morality of the data crawling process. Currently, based on web crawler technology, many research teams have obtained a considerable amount of online medical consultation data and constructed online medical consultation data sets, such as cMedQA2, webMedQA, Huatuo26M, IARNN, ​​CMDD, and MedDialog.

[0010] Furthermore, in step 2, the data cleaning framework includes a format unification module, a quality filtering module, a data deduplication module, and a privacy removal module. Step 2 includes:

[0011] Step 2.1, use the format unification module to process medical data with chaotic formats, and convert the medical data into a unified standard format to achieve standardization of text content; due to differences in user input habits, continuous updating of online medical platform data, and inconsistent data synchronization between the web and mobile terminals of the online medical platform, the collected Internet medical data has format inconsistencies; therefore, the present invention uses a format unification module to process medical data with chaotic formats, and converts the medical data into a unified standard format.

[0012] Step 2.2: Use the quality filtering module to filter and clean the data based on keywords, text statistical features, and metric features to eliminate noise and invalid information in the data to improve the overall quality of the data.

[0013] Step 2.3: Use the data deduplication module to identify and delete duplicate information in the data at the text level and line level to avoid unnecessary interference of duplicate information on subsequent analysis and model training;

[0014] Step 2.4: Use the privacy removal module to identify and remove personal identification information to prevent personal privacy information leakage and protect user privacy.

[0015] The step 2.1 comprises:

[0016] Using the corresponding relationship between the Unicode values ​​of full-width symbols and half-width symbols, convert full-width numbers to half-width numbers. Specifically, build a list of full-width and half-width punctuation marks, traverse all data, and convert half-width punctuation marks to corresponding full-width punctuation marks. During the processing, retain the basic formats such as decimal points and serial numbers;

[0017] Construct a dictionary of traditional and simplified Chinese characters, and traverse all texts in units of characters. If the text contains traditional Chinese characters that appear in the dictionary, convert the traditional Chinese characters into corresponding simplified Chinese characters. After the traversal is completed, the Chinese character format can be unified.

[0018] Furthermore, the Internet medical data extracted from online medical forums contains a large amount of low-quality information, such as noise and irrelevant elements; these elements will have a negative impact on the performance of the medical large language model to a large extent; in this regard, a quality filtering module is designed to remove low-quality data in the medical corpus and improve the overall quality of the corpus. The quality filtering module includes a keyword filtering module, a statistical filtering module, and a metric filtering module;

[0019] The keyword filtering module is used to identify and remove noise and useless elements;

[0020] The statistical filtering module is used to identify and delete abnormal text segments using text statistical features;

[0021] The metric filtering module is used to delete data that is error-free but contains little medical information.

[0022] Furthermore, the step 2.2 comprises:

[0023] Step 2.2.1: There are a large number of obvious noise and useless elements in Internet medical data. The keyword filtering module queries the keyword list based on the constructed keyword list while traversing the original Internet text, locates and deletes the noise and useless elements in the Internet medical data; the noise and useless elements include web links, HTML fields, special symbols, redundant punctuation marks, reply templates, and texts suggesting offline medical treatment;

[0024] Step 2.2.2, the essence of Internet medical data is medical text. The statistical filtering module analyzes the statistical features of the text to discover the noise patterns hidden in the data, sets thresholds for the indicators of the text statistical features, and filters a certain amount of noise in the data; the text statistical features include text length, word frequency distribution, and symbol distribution; the text length and the ratio of punctuation marks to Chinese characters are used as indicators of text statistical features;

[0025] Step 2.2.3. The above two steps can filter out data with obvious errors. However, these two methods are not enough to limit the noise. The metric filtering module deletes the data without errors but with little medical information based on the information entropy index. Specifically, the text information entropy scores of questions and answers from 1-gram to 4-gram in the statistics are counted in units of words and stored in a one-dimensional list. By comparing data with high and low medical information content, an information entropy threshold is set, and question and answer data below this threshold is deleted.

[0026] Furthermore, considering many factors such as user behavior, data collection methods, data migration or integration, website or system errors, etc., data duplication is a common problem in Internet medical data; this data duplication may be reflected in the text of questions and answers, or may involve the repetition of the entire conversation; in addition, studies have found that repeated data in text corpora will reduce the output diversity of language models, and a high proportion of repeated data will cause the model to become unstable during training, thereby affecting the performance of the training model; for this reason, the present invention designs a data deduplication module to further improve data quality, reduce information redundancy and reduce its potential impact on model training; in the data deduplication module, Internet medical data is deduplicated at the text level and the row level respectively;

[0027] The step 2.3 comprises:

[0028] Step 2.3.1: Perform text-level deduplication: Use the character list to calculate the repetition rate of repeated text, and combine the information entropy threshold and the repetition rate threshold to filter out repeated information in the text to reduce text redundancy; the character list is constructed in units of characters, including a parent list and a child list;

[0029] Step 2.3.2, perform row-level deduplication: The deduplication of the entire Internet medical data can be attributed to the deduplication of row data. All the medical data to be processed are unified into rows and stored in a large file; a row hash value deduplication algorithm is designed, which can deduplicate large-scale files by row with smaller memory and faster speed; specifically, the row hash value deduplication algorithm splits the large file into several sub-files, traverses all sub-files, calculates the hash value of each row in the file, and sorts them using the hash value, and deduplicates files with the same hash value; finally, the deduplicated files are sorted and merged to obtain the final deduplicated data file; in the processing process, a multi-threaded method is used to accelerate the data deduplication process.

[0030] Furthermore, one of the main sources of Internet medical data is online medical websites, which contain a large amount of user-generated protected health information (PHI). For example, in the original MedDialog-CN dataset, there is a large amount of information that can identify personal identities, such as the contact information of doctors and patients, and the patient's hospitalization number. If these private information are not processed, directly training the medical dialogue model may cause the medical dialogue model to generate responses containing private information of doctors or patients, thereby increasing the risk of privacy leakage. Therefore, the present invention introduces a privacy removal module, the goal of which is to identify and delete personal identification information from medical dialogue data to protect personal privacy and reduce the risk of privacy leakage. This processing step is crucial to ensuring the security of the use of the medical dialogue model, especially when processing medical data containing sensitive personal information.

[0031] In the step 2.4, a regular expression is constructed based on the user digital features appearing in the medical data, and the medical data with privacy information is located and deleted by using the re.findall and keyword set method when traversing the Internet medical data; the user digital features include mobile phone number, landline number, QQ number, WeChat number, email address, zip code, hospitalization number, outpatient number, medical record number, and medical card number.

[0032] The present invention also provides a comprehensive Internet medical data cleaning system, including: a module for executing the above-mentioned comprehensive Internet medical data cleaning method.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned comprehensive Internet medical data cleaning method when executing the program.

[0034] The present invention also provides a computer program product, including a computer program, which implements the above-mentioned comprehensive Internet medical data cleaning method when executed by a processor.

[0035] To verify the effectiveness of the proposed data cleaning framework, this paper applies this cleaning framework on a large-scale Chinese medical dialogue dataset MedDialog-CN. The noise composition of the dataset before and after data cleaning is analyzed in detail, and the medical dialogue model is trained on the data before and after cleaning. The effectiveness of the data cleaning framework is verified through automatic evaluation, manual evaluation and ablation experiments;

[0036] include:

[0037] (1) Use automatic indicators Perplexity, NIST-2, BLEU-2, BLEU-4, METEOR, Entropy-4, Dist-1 and Dist-2 to evaluate the responses generated by the trained language model; among them, perplexity measures the quality of the generated responses, and the lower the indicator, the better. NIST, BLEU and METEOR measure the similarity between the generated response and the ground-truth by matching on n-grams, and the higher the indicator, the better. Entropy and Dist measure the diversity of the generated responses, and the higher the indicator, the better.

[0038] (2) Evaluators select the better response from a pair of model responses based on three evaluation criteria (relevance, informativeness, and human similarity). To ensure fairness, all model information is anonymized and the interaction position is randomized; under the same evaluation criteria, the model-generated responses are ranked, and better model responses are ranked higher (the model's response is more consistent with the conversation history, contains more medical information, and is more similar to the doctor's expression).

[0039] (3) Ablation experiments were conducted on each data cleaning module to comprehensively evaluate the applicability of each module in the Chinese Internet medical data cleaning framework and its promoting effect.

[0040] The beneficial effects of the present invention are:

[0041] 1. For Chinese Internet medical data, this invention proposes a comprehensive data cleaning framework to address the challenges of high cost, incompleteness and low efficiency of current data cleaning methods; it is currently a rare comprehensive analysis and cleaning method for this type of data. The data cleaning ideas in this invention can be applied to other types of text cleaning processes;

[0042] 2. The data cleaning framework proposed in the present invention has modular characteristics, supports offline deployment, and does not incur additional costs. The present invention divides the common noise in Chinese Internet medical data into four categories, and designs four modules of format unification, quality filtering, data deduplication, and privacy removal according to these four types of noise to clean the original data in detail, so as to obtain cleaner data, which can be applied to text data legally obtained from Internet medical platforms;

[0043] 3. The present invention applies a data cleaning framework to the MedDialog-CN dataset, trains three typical language models on the datasets before and after data cleaning, and conducts automatic and manual evaluation experiments on the responses generated by the models. The experimental results show that the responses generated by the model trained on the cleaned data can contain more medical information, are more relevant to the questions, and are closer to the original answers. This proves that this data cleaning framework improves the performance of the medical model by improving the quality of medical dialogue data;

[0044] 4. A Chinese Internet medical data cleaning method is proposed to improve the quality of Chinese Internet medical data;

[0045] 5. This data cleaning method can efficiently process a large amount of medical conversation data in a short time. It is highly modular, supports offline deployment, does not incur additional costs, and effectively avoids privacy leakage caused by data flowing into the online network. It has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention;

[0047] Figure 2 is an overall flow chart of the quality filtering module in the present invention;

[0048] Figure 3 This is a detailed statistical diagram of the filtered elements in the MedDialog-CN training set. DETAILED DESCRIPTION

[0049] Example 1: Figure 1-Figure 3 As shown, a comprehensive method for cleaning Internet medical data includes:

[0050] Step 1: Use web crawlers to obtain medical data;

[0051] Furthermore, in step 1, Chinese Internet medical data includes online consultation platforms, official websites of medical institutions, health information sharing forums and social media; a comprehensive Internet medical data cleaning method is aimed at data obtained from online medical (consultation) platforms, and its data acquisition method is mainly a web crawler. A web crawler is a computer program that follows pre-set rules. The web crawler automatically grabs Chinese Internet medical data from the Internet; Figure 1 As shown in the figure, medical data crawling is a technology involving information security and patient privacy protection. Therefore, before crawling Internet medical data, relevant laws and regulations and ethical standards need to be followed to ensure the legality and ethics of the data crawling process. Currently, based on web crawler technology, many research teams have obtained a considerable amount of online medical consultation data and constructed online medical consultation data sets, such as cMedQA2, webMedQA, Huatuo26M, IARNN, ​​CMDD, and MedDialog. The present invention applies a data cleaning framework to the MedDialog-CN data set, which comes from the Good Doctor Online Forum and the online inquiry time is from 2010 to 2020. The data set contains 3.4 million conversations between patients and doctors, 11.3 million speeches, 660.2 million tokens, and covers 172 disease specialties. Each conversation starts with the patient describing his or her condition and medical history, including past history, course of disease, and medication.

[0052] Step 2: Build a data cleaning framework to perform comprehensive data cleaning on the acquired medical data. The quality filtering module in the data cleaning framework screens and cleans the data based on keywords, text statistical features, and metric features.

[0053] Furthermore, in step 2, the Internet cleaning framework is as follows: Figure 1 As shown, the data cleaning framework includes a format unification module, a quality filtering module, a data deduplication module, and a privacy removal module. Step 2 includes:

[0054] Step 2.1, use the format unification module to process the medical data with chaotic formats and convert the medical data into a unified standard format to achieve standardization of text content;

[0055] Step 2.2: Use the quality filtering module to filter and clean the data based on keywords, text statistical features, and metric features to eliminate noise and invalid information in the data to improve the overall quality of the data.

[0056] Step 2.3: Use the data deduplication module to identify and delete duplicate information in the data at the text level and line level to avoid unnecessary interference of duplicate information on subsequent analysis and model training;

[0057] Step 2.4: Use the privacy removal module to identify and remove personal identification information to prevent personal privacy information leakage and protect user privacy.

[0058] Furthermore, due to differences in user input habits, continuous updating of online medical platform data, and inconsistent synchronization of data between the web and mobile terminals of the online medical platform, the collected Internet medical data has inconsistencies in format; the present invention designs an algorithm to unify the data format;

[0059] The step 2.1 comprises:

[0060] Using the corresponding relationship between the Unicode values ​​of full-width symbols and half-width symbols, convert full-width numbers to half-width numbers. Specifically, build a list of full-width and half-width punctuation marks, traverse all data, and convert half-width punctuation marks to corresponding full-width punctuation marks. During the processing, retain the basic formats such as decimal points and serial numbers;

[0061] Construct a dictionary of traditional and simplified Chinese characters, and traverse all texts in units of characters. If the text contains traditional Chinese characters that appear in the dictionary, convert the traditional Chinese characters into corresponding simplified Chinese characters. After the traversal is completed, the Chinese character format can be unified.

[0062] Furthermore, the Internet medical data extracted from online medical forums contains a large amount of low-quality information, such as noise and irrelevant elements; these elements will have a negative impact on the performance of the medical large language model to a large extent; in this regard, a quality filtering module is designed to remove low-quality data in the medical corpus and improve the overall quality of the corpus. The overall flow chart of the quality filtering module is shown in the figure. Figure 2 As shown, the quality filtering module includes a keyword filtering module, a statistical filtering module, and a metric filtering module;

[0063] The keyword filtering module is used to identify and remove noise and useless elements;

[0064] The statistical filtering module is used to identify and delete abnormal text segments using text statistical features;

[0065] The metric filtering module is used to delete data that is error-free but contains little medical information.

[0066] Furthermore, the step 2.2 comprises:

[0067] Step 2.2.1, the keyword filtering module queries the keyword list based on the constructed keyword list in the process of traversing the original Internet text, locates and deletes the noise and useless elements in the Internet medical data; there are a large number of obvious noise and useless elements in the Internet medical data, and the noise and useless elements include web links, HTML fields, special symbols, redundant punctuation marks, reply templates, and texts suggesting offline medical treatment;

[0068] Step 2.2.2, the essence of Internet medical data is medical text. The statistical filtering module analyzes the statistical features of the text to discover the noise patterns hidden in the data, sets thresholds for the indicators of the text statistical features, and filters a certain amount of noise in the data; the text statistical features include text length, word frequency distribution, and symbol distribution; the text length and the ratio of punctuation marks to Chinese characters are used as indicators of text statistical features;

[0069] Step 2.2.3: The keyword filtering module and the statistical filtering module can filter out data with obvious errors. However, these two methods are not enough to limit the noise. The metric filtering module deletes data without errors but with little medical information based on the information entropy index. Specifically, the text information entropy scores of questions and answers from 1-gram to 4-gram in the statistics are stored in a one-dimensional list, and the information entropy threshold is set by comparing data with high and low medical information content, and the question and answer data below this threshold are deleted.

[0070] Furthermore, considering many factors such as user behavior, data collection methods, data migration or integration, website or system errors, etc., data duplication is a common problem in Internet medical data; this data duplication may be reflected in the text of questions and answers, or may involve the repetition of the entire conversation; in addition, studies have found that repeated data in text corpora will reduce the output diversity of language models, and a high proportion of repeated data will cause the model to become unstable during the training process, thereby affecting the performance of the training model; for this reason, the present invention designs a data deduplication module to further improve data quality, reduce information redundancy and reduce its potential impact on model training; in the data deduplication module, two algorithms are specifically designed to deduplicate Internet medical data at the text level and the row level respectively;

[0071] The step 2.3 comprises:

[0072] Step 2.3.1: Perform text-level deduplication: Use the character list to calculate the repetition rate of repeated text, and combine the information entropy threshold and the repetition rate threshold to filter out repeated information in the text to reduce text redundancy; the character list is constructed in units of characters, including a parent list and a child list;

[0073] Step 2.3.2, perform row-level deduplication: The deduplication of the entire Internet medical data is attributed to the deduplication of row data, and all the medical data to be processed are unified into rows and stored in a large file; a row hash value deduplication algorithm is designed, which can deduplicate large-scale files by row with smaller memory and faster speed; specifically, the row hash value deduplication algorithm splits the large file into several sub-files, traverses all sub-files, calculates the hash value of each row in the file, and sorts them using the hash value, and deduplicates files with the same hash value; finally, the deduplicated files are sorted and merged to obtain the final deduplicated data file; in the processing process, a multi-threaded method is used to accelerate the data deduplication process.

[0074] Furthermore, one of the main sources of Internet medical data is online medical websites, which contain a large amount of user-generated protected health information (PHI). For example, in the original MedDialog-CN dataset, there is a large amount of information that can identify personal identities, such as the contact information of doctors and patients, and the patient's hospitalization number. If these private information are not processed, directly training the medical dialogue model may cause the medical dialogue model to generate responses containing private information of doctors or patients, thereby increasing the risk of privacy leakage. Therefore, the present invention introduces a privacy removal module, the goal of which is to identify and delete personal identification information from medical dialogue data to protect personal privacy and reduce the risk of privacy leakage. This processing step is crucial to ensure the security of the use of the medical dialogue model, especially when processing medical data containing sensitive personal information.

[0075] In the step 2.4, a regular expression is constructed based on the user digital features appearing in the medical data, and the medical data with privacy information is located and deleted by using the re.findall and keyword set method when traversing the Internet medical data; the user digital features include mobile phone number, landline number, QQ number, WeChat number, email address, zip code, hospitalization number, outpatient number, medical record number, and medical card number.

[0076] Furthermore, a comprehensive Internet medical data cleaning system includes: a module for executing a comprehensive Internet medical data cleaning method.

[0077] Furthermore, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the comprehensive Internet medical data cleaning method when executing the program.

[0078] Furthermore, the present invention also provides a computer program product, including a computer program, which implements the above-mentioned comprehensive Internet medical data cleaning method when executed by a processor.

[0079] To verify the effectiveness of the proposed data cleaning framework, this paper applies this cleaning framework on a large-scale Chinese medical dialogue dataset MedDialog-CN. The noise composition of the dataset before and after data cleaning is analyzed in detail, and the medical dialogue model is trained on the data before and after cleaning. The effectiveness of the data cleaning framework is verified through automatic evaluation, manual evaluation and ablation experiments;

[0080] The responses generated by the trained language model are evaluated using automatic metrics Perplexity, NIST-2, BLEU-2, BLEU-4, METEOR, Entropy-4, Dist-1 and Dist-2; among them, perplexity measures the quality of the generated responses, and the lower the metric, the better. NIST, BLEU and METEOR measure the similarity between the generated response and the ground-truth by matching on n-grams, and the higher the metric, the better. Entropy and Dist measure the diversity of the generated responses, and the higher the metric, the better.

[0081] Table 1 shows the overall statistics of the MedDialog-CN dataset and the cleaned dataset. Compared with the unprocessed original data, the total number of medical conversations in the cleaned dataset decreased by an average of 26.84%, and the number of speeches by doctors and patients decreased by an average of 16.93%. This proves that the MedDialog-CN dataset contains a lot of noise (such as invalid data, redundant data, and data with low medical information).

[0082] Table 1 shows the statistics of the MedDialog-CN dataset before and after data cleaning

[0083]

[0084] Figure 3The following is a statistical diagram of the cleaned elements in the MedDialog-CN training set. First, the quality filtering module removed 88.19% of the data in the training set (including shorter questions and answers (45.89%), answers to offline medical consultations (27.46%), text templates (8.31%), data with low information entropy (3.18%), data with abnormal punctuation and character ratios (2.14%), data containing web links (0.52%), data containing HTML (0.42%), and data containing special symbols (0.27%)). This proves that the quality filtering module can effectively filter out most of the noise in the training set. In addition, we noticed that the amount of data cleaned by the data deduplication module is relatively small (4.73%). This is because the metric filtering submodule in the quality filtering module has removed some duplicate data, and when there is duplicate data inside the text, its information entropy is usually relatively low. The privacy removal module filtered 7.08% of the data in the training set (considering that the quality filtering module may also filter out some data containing personal privacy), which shows that there is a large amount of user privacy information in the MedDialog-CN dataset, emphasizing the importance of data privacy protection.

[0085] Automatic evaluation:

[0086] In the present invention, the automatic indicators Perplexity, NIST-2, BLEU-2, BLEU-4, METEOR, Entropy-4, Dist-1 and Dist-2 are used to evaluate the responses generated by the language model. Among them, perplexity measures the quality of the generated responses, and the lower the indicator, the better. NIST, BLEU and METEOR measure the similarity between the generated response and the ground-truth by matching on n-gram, and the higher the indicator, the better. Entropy and Dist measure the diversity of the generated responses, and the higher the indicator, the better.

[0087] The present invention automatically evaluates the dialogue models Transformer (O), BERT-GPT (O), and DialoGPT (O) trained with the original data set on the original test set. The dialogue models Transformer (C), BERT-GPT (C), and DialoGPT (C) trained with the cleaned data set are evaluated on the cleaned test set. The evaluation results are shown in Table 2.

[0088] Table 2 shows the results of the automatic evaluation.

[0089]

[0090] From the automatic evaluation results, it can be seen that all models trained on the training set after data cleaning have improved in all indicators compared with before, proving that the proposed data cleaning framework can improve the performance of the medical dialogue model. Among them, BERT-GPT (C) has the lowest perplexity, which may be because the BERT-GPT model has been pre-trained on a large number of Chinese corpora. Transformer (C) performs better in NIST-4, BLEU-2, METEOR, and Dist indicators, and DialoGPT (C) performs better in BLEU-4, Entropy-4, and Dist-2 indicators. It proves that these two models can generate responses that are more similar and diverse to Groundtruth.

[0091] Manual evaluation:

[0092] In order to gain a deeper understanding of whether the proposed data cleaning framework can improve the quality of medical dialogue data and enhance the performance of the medical dialogue model, the present invention invited five medical students to manually evaluate the responses of the medical dialogue model. Specifically, the present invention randomly selected 50 single-round dialogues (Single QA) and 50 multi-round dialogues (Multiple QA) from the original MedDialog-CN test set, and asked the evaluators to evaluate the responses generated by the model in single-round and multi-round scenarios respectively.

[0093] Evaluators select the better response from a pair of model responses based on three types of evaluation criteria (relevance, informativeness, and human similarity). To ensure fairness, all model information is anonymized and the interaction position is randomized; under the same evaluation criteria, if more than three of the five evaluators believe that the response generated by a certain model is better, then the response generated by this model under this standard is considered to be better. The responses generated by the models are sorted, and better model responses are ranked higher (the model's response is more consistent with the conversation history, contains more medical information, and is more similar to the doctor's expression). The results of the manual evaluation of the present invention are shown in Table 3:

[0094] Table 3 shows the results of manual evaluation.

[0095]

[0096] From the results in Table 3, we can see that the rankings of the models trained with data after data cleaning are higher than those of the models trained with data before data cleaning, which proves that the proposed data cleaning framework can improve the performance of the medical dialogue model. Among them, BERT-GPT(C) is generally ranked high in relevance and human similarity, proving that the responses it generates are more similar to ground truth. DialoGPT(C) ranks high in informativeness, proving that DialoGPT can generate responses with diverse and rich medical information. Compared with the first two models, although the overall ranking of Transformer(C) is not much different from that of Transformer(O). The main reason may be that the responses generated by Transformer(O) are better and the automatic evaluation indicators are also higher. On the one hand, it is difficult to have a significant improvement in higher performance indicators; on the other hand, compared with the other two medical dialogue models, the data cleaning method may be insensitive to the performance improvement of the Transformer model.

[0097] Ablation experiment:

[0098] In order to comprehensively evaluate the applicability of each module in the Chinese Internet medical data cleaning framework and its promoting effect, an ablation experiment was conducted on each data cleaning module. The MedDialog-CN dataset was cleaned using a single cleaning module, and the cleaned data was used to train the medical dialogue model. In order to distinguish the medical dialogue model trained with the cleaned data of each module, the following representation was introduced:

[0099] The dialogue models trained with the cleaned data using the unified format module are: Transformer(U), BERT-GPT(U), DialoGPT(U)

[0100] The dialogue models trained with the data cleaned by the quality filtering module are represented as: Transformer(Q), BERT-GPT(Q), DialoGPT(Q)

[0101] The dialogue models trained with the data cleaned by the data deduplication module are represented as: Transformer (D), BERT-GPT (D), DialoGPT (D)

[0102] The dialogue models trained with the data cleaned by the privacy removal module are represented as: Transformer (P), BERT-GPT (P), DialoGPT (P)

[0103] Table 4 shows the results of the ablation experiment. It can be seen from Table 4 that each module in the Internet data cleaning framework can bring a certain promotion effect. First, the medical dialogue model trained with the data cleaned by the format unification module has a slight improvement in the automatic evaluation index, which proves that the promotion effect of the format unification model is small. This is because this module can only unify the format of Internet medical data and cannot filter data; secondly, the medical dialogue model trained with the data cleaned by the quality filtering module has a significant improvement in the automatic evaluation index, indicating that the quality filtering module has a large promotion effect and can clean most of the noise in the Internet medical data. This result is also consistent with the results of the data set analysis before and after data cleaning; in addition, the model trained with the data cleaned by the data deduplication module and the privacy removal module has a significant improvement in the automatic evaluation index, and the results of the index improvement are relatively close, indicating that the MedDialog-CN dataset contains a large amount of duplicate data and personal privacy data, which indirectly shows that the quality filtering module can filter out some duplicate data and data containing personal privacy, and once again emphasizes the importance of duplicate data removal and data privacy protection in the data cleaning process.

[0104] Table 4 shows the results of the ablation experiment.

[0105]

[0106] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A comprehensive Internet medical data cleaning method, characterized by: The method comprises: Step 1: Use web crawlers to obtain medical data; Step 2: Build a data cleaning framework to perform comprehensive data cleaning on the acquired medical data. The quality filtering module in the data cleaning framework screens and cleans the data based on keywords, text statistical features, and metric features.

2. A comprehensive Internet medical data cleaning method according to claim 1, characterized in that: In step 1, the web crawler automatically crawls Chinese Internet medical data from the Internet; Chinese Internet medical data includes online consultation platforms, official websites of medical institutions, health information sharing forums and social media.

3. A comprehensive Internet medical data cleaning method according to claim 1, characterized in that: In step 2, the data cleaning framework includes a format unification module, a quality filtering module, a data deduplication module, and a privacy removal module. Step 2 includes: Step 2.1, use the format unification module to process the medical data with chaotic formats and convert the medical data into a unified standard format; Step 2.2: Use the quality filtering module to filter and clean the data based on keywords, text statistical features, and metric features to eliminate noise and invalid information in the data; Step 2.3: Use the data deduplication module to identify and delete duplicate information in the data at the text level and line level; Step 2.4: Use the privacy removal module to identify and remove personally identifiable information.

4. A comprehensive Internet medical data cleaning method according to claim 1, characterized in that: The step 2.1 comprises: Convert full-width numbers to half-width numbers using the corresponding relationship between the Unicode values ​​of full-width symbols and half-width symbols; Build a dictionary of traditional and simplified Chinese characters, traverse all texts in units of characters, and if the text contains traditional Chinese characters that appear in the dictionary, convert the traditional Chinese characters into the corresponding simplified Chinese characters.

5. A comprehensive Internet medical data cleaning method according to claim 1, characterized in that: The quality filtering module includes a keyword filtering module, a statistical filtering module, and a metric filtering module; The keyword filtering module is used to identify and remove noise and useless elements; The statistical filtering module is used to identify and delete abnormal text segments using text statistical features; The metric filtering module is used to delete data that is error-free but contains little medical information.

6. A comprehensive Internet medical data cleaning method according to claim 3, characterized in that: The step 2.2 comprises: Step 2.2.1, the keyword filtering module queries the keyword list based on the constructed keyword list in the process of traversing the original Internet text, locates and deletes the noise and useless elements in the Internet medical data; the noise and useless elements include web links, HTML fields, special symbols, redundant punctuation marks, reply templates, and texts suggesting offline medical treatment; Step 2.2.2, the statistical filtering module analyzes the statistical features of the text to find the noise patterns hidden in the data, sets thresholds for the indicators of the statistical features of the text, and filters the noise; the statistical features of the text include text length, word frequency distribution, and symbol distribution; the text length and the ratio of punctuation marks to Chinese characters are used as indicators of the statistical features of the text; Step 2.2.3, the metric filtering module deletes data that are not wrong but have little medical information content based on the information entropy index; specifically, in units of words, the text information entropy scores of questions and answers from 1-gram to 4-gram in the statistics are stored in a one-dimensional list. By comparing data with high and low medical information content, an information entropy threshold is set, and question and answer data below this threshold are deleted.

7. A comprehensive Internet medical data cleaning method according to claim 3, characterized in that: The step 2.3 comprises: Step 2.3.1: Perform text-level deduplication: Use the character list to calculate the repetition rate of repeated text, and filter out repeated information in the text by combining the information entropy threshold and the repetition rate threshold; Step 2.3.2, perform row-level deduplication: unify all the medical data to be processed into rows and store them in a large file; design a row hash value deduplication algorithm, split the large file into several sub-files, traverse all sub-files, calculate the hash value of each row in the file, and use the hash value to sort them, and deduplicate the files with the same hash value; finally, sort and merge the deduplicated files to obtain the final deduplicated data file; in the processing process, use a multi-threaded method to accelerate the data deduplication process; In the step 2.4, a regular expression is constructed based on the user digital features appearing in the medical data, and the medical data with privacy information is located and deleted by using the re.findall and keyword set method when traversing the Internet medical data; the user digital features include mobile phone number, landline number, QQ number, WeChat number, email address, zip code, hospitalization number, outpatient number, medical record number, and medical card number.

8. A comprehensive Internet medical data cleaning system, characterized by: include: A module for executing a comprehensive Internet medical data cleaning method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements a comprehensive Internet medical data cleaning method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a comprehensive Internet medical data cleaning method as described in any one of claims 1 to 9.

Images