Corpus data collection method and device based on medical question and answer large model and medium

Abstract

The invention discloses a corpus data collection method and device based on a medical question and answer large model and a medium, and relates to the technical field of corpora, and the method comprises the steps: obtaining drug screening data and representative drugs based on the medical question and answer large model; obtaining an easy-to-doubt condition and a standby condition corresponding to the representative drug; acquiring data easy to doubt; data processing is optimized on the basis of the easily suspected collected data; the method is used for solving the problems that in an existing corpus data acquisition method, when almost identical problems exist in medical data but corresponding data are completely inconsistent due to small differences, almost identical problems are classified into one class and partial deduplication is carried out when data processing is carried out only through quality filtering and deduplication; the problems that data recorded in a corpus is incomplete and cannot be effectively collected are solved.

Description

Technical Field

[0001] This invention relates to the field of corpus technology, specifically to a corpus data acquisition method, device, and medium based on a large medical question-and-answer model. Background Technology

[0002] A corpus is a large-scale electronic text library that has been scientifically sampled and processed to store real-world materials used in actual language use. It is widely used in fields such as linguistics, machine translation, and language teaching. Corpus data collection refers to the process of collecting and organizing data for training, validating, and testing language models or other artificial intelligence models in the fields of natural language processing and machine learning. The quality and diversity of corpus collection have a significant impact on model performance.

[0003] Existing methods for corpus data collection typically involve quality filtering and deduplication to reduce repetitive and low-quality content in the corpus. These processes, including quality filtering, deduplication, hate speech detection, and perplexity calculation, improve the model's generalization ability and output reliability. While this improves data quality, it faces challenges when effective duplicate data exists. For example, in medical data, nearly identical issues may appear with minor differences leading to inconsistencies. In such cases, quality filtering and deduplication alone can cause similar issues to be grouped together and partially deduplicated, resulting in incomplete data in the corpus and hindering effective data collection. For instance, patent application CN119474555A discloses a method for constructing a corpus based on a webpage dataset. The proposed method and system utilize a series of operations, including quality filtering, deduplication, hate speech detection, and perplexity calculation, to construct a large-scale, high-quality corpus based on massive web page datasets. This significantly reduces duplicate and low-quality content in the corpus, improving the model's generalization ability and output reliability. Other improvements in corpus data acquisition, typically focusing on the construction of corpora and subcorpora, still fail to address the issue of effective duplicate data. For example, in medical data, nearly identical problems may exist, but minor differences lead to completely inconsistent data. In such cases, quality filtering and deduplication alone can cause nearly identical problems to be grouped together and partially deduplicated, resulting in incomplete data in the corpus and hindering effective data collection. Therefore, it is necessary to improve existing corpus data acquisition methods. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the prior art. By proposing a corpus data acquisition method, device, and medium based on a large medical question-and-answer model, it addresses the problem in existing corpus data acquisition methods where, when valid duplicate data exists (e.g., medical data contains almost identical questions but with completely inconsistent corresponding data due to minor differences), data processing using only quality filtering and deduplication results in almost identical questions being grouped together and partially deduplicated, leading to incomplete data in the corpus and the inability to effectively acquire data.

[0005] To achieve the above objectives, firstly, this application provides a corpus data collection method based on a large-scale medical question-answering model, comprising the following steps:

[0006] Based on the medical question-and-answer big data model, we obtain the question and answer data of users when using the medical question-and-answer big data model; we analyze the question data, filter the question data based on the analysis results, and obtain drug screening data and the representative drug corresponding to each drug screening data;

[0007] The drug screening data is analyzed based on the drug instructions of representative drugs, and the questionable clauses and pending clauses in the drug instructions of the corresponding representative drugs are obtained based on the analysis results; all questionable clauses are screened, and the questionable collection data of each representative drug is obtained based on the screening results and pending clauses.

[0008] During corpus data collection, data processing was optimized based on the collection of easily questionable data representing all drugs.

[0009] Furthermore, based on the medical question-and-answer model, user question and answer data are obtained when using the model; the question data is analyzed, and based on the analysis results, the question data is filtered to obtain drug screening data and representative drugs corresponding to each drug screening data, including:

[0010] Data extraction is performed on the medical question-and-answer model. All questions raised by users in the medical question-and-answer model are recorded and designated as question data. For any question in the question data, the answer sent by the medical question-and-answer model based on the question is designated as the question data answer. All data answers are recorded and designated as answer data.

[0011] The question-based drug screening method was used to analyze all the question data, and based on the analysis results, the drug screening data corresponding to all the question data and the representative drug corresponding to each drug screening data were obtained.

[0012] Furthermore, the question-based drug screening method includes:

[0013] Establish a drug name database. Based on web crawling, extract drug names from the data already learned by the medical question-and-answer model, and store all the extracted drug names into the drug name database.

[0014] For any given query data: perform word segmentation on the query data, and record each Chinese word obtained from the segmentation as the query Chinese word; for any given query Chinese word, compare the query Chinese word with all drug names in the drug name database, and when the query Chinese word is the same as any drug name in the drug name database, record the query Chinese word as the drug Chinese word;

[0015] Retrieve all Chinese words for the corresponding drugs from all the query data.

[0016] Furthermore, the question-based drug screening method also includes:

[0017] For any drug name A in the drug name database: record the question data containing drug name A in the Chinese words of the drug as the drug data to be screened, perform Chinese word recognition processing on the answer data corresponding to the drug data to be screened, and when drug name A is present in the answer data corresponding to the drug data to be screened, record the drug data to be screened as the drug screening data of drug name A.

[0018] The names of drugs for which drug screening data exists in the drug name database are designated as representative drugs, and the drug screening data corresponding to each representative drug is recorded.

[0019] Furthermore, the drug screening data is analyzed based on the drug instructions for representative drugs, and the questionable clauses and pending clauses in the drug instructions for representative drugs are obtained based on the analysis results, including:

[0020] For any representative drug, obtain its instruction leaflet and record it as the representative instruction leaflet; divide the text in the representative instruction leaflet according to the serial number preceding each paragraph and record them sequentially as representative segment DW1 to representative segment DW. n Where n represents the maximum value of the serial number preceding the text in the instruction manual, and for a positive integer c less than or equal to n and greater than or equal to 1, it represents the DW segment. c This represents the text paragraph whose prefix number is 'c' in the instruction manual.

[0021] For any drug screening data corresponding to a representative drug, the text composed of the drug screening data and the corresponding response data is recorded as the drug comparison text. The drug comparison text is then compared with all representative text segments in turn, and the representative text segment with the highest similarity is recorded as the suspicious text segment of the drug screening data. The same representative text segment can be recorded as the suspicious text segment of multiple drug screening data.

[0022] Furthermore, the analysis of drug screening data based on the drug instructions for representative drugs, and the acquisition of questionable clauses and pending clauses from the drug instructions for representative drugs based on the analysis results, also includes:

[0023] Obtain the suspicious text segments of all drug screening data corresponding to the drug, and use the suspicious screening algorithm to obtain the suspicious reference value of the drug. The suspicious screening algorithm includes: Where T is the suspicious reference value, t is the number of drug screening data representing the drug, and R is the reference value. i The similarity between the suspicious text segment of the i-th drug screening data and the drug comparison text of the drug screening data in all drug screening data;

[0024] For any representative passage, Let be the comparison value of the representative text segment, where b is the maximum value among all similarities when the representative text segment is marked as a suspicious text segment, and b is 0 when the representative text segment is not marked as a suspicious text segment; v is the number of representative text segments marked as suspicious text segments.

[0025] Obtain the comparison values ​​of all representative text segments, and record the absolute value of the difference between the comparison value of the representative text segment and the suspicious reference value as the screening value of the representative text segment; record the average value of all screening values ​​as the screening measure value, and record the paragraphs in the representative specification corresponding to the representative text segments with screening values ​​less than the screening measure value as the suspicious clauses in the representative specification, and record the paragraphs in the representative specification corresponding to the representative text segments with screening values ​​greater than or equal to the screening measure value as the pending clauses in the representative specification;

[0026] Obtain all questionable clauses and all pending clauses corresponding to all representative drugs.

[0027] Furthermore, all potentially problematic clauses are screened, and based on the screening results and the clauses to be used, potentially problematic data for each representative drug is collected, including:

[0028] For any erroneous regulation representing a drug, the response data in the drug comparison text corresponding to the erroneous regulation is recorded as the response screening data. The response screening data is compared with the erroneous regulation in terms of text similarity, and the obtained similarity is recorded as the response similarity of the erroneous regulation.

[0029] Obtain the response similarity of all questionable clauses representing a drug, and record the average of all response similarities as the single drug average;

[0030] Obtain the average value of each individual drug for all representative drugs, and record the average of all individual drug average values ​​as the multi-drug average value;

[0031] For any representative drug: when the average value of a single drug is greater than or equal to the average value of multiple drugs, the suspicious clauses among all the suspicious clauses of the representative drug whose response similarity is greater than the average value of a single drug of the representative drug are recorded as the collection clauses.

[0032] When the average value of a single drug representing a drug is less than the average value of multiple drugs, the suspicious regulations with a response similarity greater than the average value of multiple drugs among all the suspicious regulations representing the drug, and the z pending regulations with smaller screening values ​​among the pending regulations representing the drug, are recorded as the collected regulations, where z is the number of suspicious regulations among all the suspicious regulations representing the drug with a response similarity greater than or equal to the average value of a single drug representing the drug and less than the average value of multiple drugs.

[0033] When the representative text corresponding to the collection regulations for representative drugs is recorded as a suspicious text, all the drug screening data corresponding to it are recorded as the suspicious collection data for representative drugs.

[0034] Obtain all suspicious data corresponding to representative drugs.

[0035] Furthermore, during corpus data collection, based on the easily questionable data collected from all representative drugs, data processing was optimized, including:

[0036] During corpus data collection, when any of the user-submitted questions β represent a drug α, question β is compared with all the easily questioned data representing drug α in turn. When any similarity score is greater than E, question β is retained during data processing.

[0037] Secondly, this application provides an electronic device including a processor and a memory, the memory storing computer-readable instructions, which, when executed by the processor, perform the steps in the method described above.

[0038] Thirdly, this application provides a storage medium on which a computer program is stored, which, when executed by a processor, performs the steps of the method described above.

[0039] The beneficial effects of this invention are as follows: This application first acquires user question and answer data when using the medical question and answer model based on the medical question and answer model; analyzes the question data, and filters the question data based on the analysis results to obtain drug screening data and representative drugs corresponding to each drug screening data. The advantage of this is that by acquiring question and answer data and obtaining drug screening data and corresponding representative drugs through data analysis, it is possible to obtain data with analytical value from the large amount of data stored in the medical question and answer model, namely drug screening data and drugs corresponding to data with analytical value, so as to clarify the direction of data analysis in subsequent analysis, ensure effective data analysis, and achieve the purpose of complete and effective collection of data in the corpus.

[0040] This application also analyzes drug screening data based on the drug instructions of representative drugs, and obtains the questionable clauses and pending clauses from the drug instructions corresponding to the representative drugs based on the analysis results; it filters all questionable clauses, and obtains the questionable collection data for each representative drug based on the filtering results and pending clauses; finally, it optimizes the data processing based on the questionable collection data of all representative drugs during corpus data collection. The advantage of this is that by obtaining the questionable clauses and pending clauses and optimizing the data processing based on the questionable collection data, it helps to obtain a variety of effective questions raised by users in medical data and record and save them. This solves the problem that when filtering and deduplicating the data collected from the corpus, filtering or deduplicating question data with slight differences but all of them are valid, the data recorded in the corpus is incomplete, thereby improving the completeness and effectiveness of corpus data collection. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the steps of the method of the present invention;

[0042] Figure 2 This is a flowchart illustrating the steps of the question-based drug screening method of the present invention;

[0043] Figure 3 This is a flowchart illustrating the steps for obtaining the questionable clauses and pending clauses of the present invention.

[0044] Figure 4 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1, please refer to Figure 1 As shown, this application provides a corpus data collection method based on a large medical question-answering model, including the following steps:

[0047] Step S1: Based on the medical question-and-answer big data model, obtain the question data and answer data of users when using the medical question-and-answer big data model; analyze the question data, filter the question data based on the analysis results, and obtain drug screening data and the representative drug corresponding to each drug screening data;

[0048] Step S1 includes: Step S101, extracting data from the medical question-and-answer model, recording all questions raised by users in the medical question-and-answer model and recording them as question data; for any question in the question data, recording the answer sent by the medical question-and-answer model based on the question as the question data answer; recording all data answers and recording them as answer data;

[0049] In practice, due to different ports for sending answers or different users sending answers, there may be multiple answer data corresponding to the same question data.

[0050] Step S102: Analyze all question data using the question-based drug screening method, and obtain the drug screening data corresponding to all question data and the representative drug corresponding to each drug screening data based on the analysis results;

[0051] Please see Figure 2 As shown, the question-and-answer drug screening method includes: step S1021, establishing a drug name database, crawling drug names from the data already learned by the medical question-and-answer big model based on web crawlers, and storing all the crawled drug names into the drug name database;

[0052] In the specific implementation process, when crawling drug names, data can be filtered by web crawlers and accurately captured by AI recognition. The data already learned by the medical question-and-answer model can be processed to obtain the drug names recorded in the medical question-and-answer model, so as to conduct a comprehensive and effective analysis in subsequent analysis.

[0053] Step S1022: For any query data: perform word segmentation on the query data, and record each Chinese word obtained from the segmentation as the query Chinese word; for any query Chinese word, compare the query Chinese word with all drug names in the drug name database, and when the query Chinese word is the same as any drug name in the drug name database, record the query Chinese word as the drug Chinese word;

[0054] Step S1023: Obtain all Chinese words for the corresponding drugs from all query data;

[0055] Step S1024: For any drug name A in the drug name database: record the question data containing drug name A in the Chinese words of the drug as the drug data to be screened, perform Chinese word recognition processing on the answer data corresponding to the drug data to be screened, and when drug name A is present in the answer data corresponding to the drug data to be screened, record the drug data to be screened as the drug screening data of drug name A.

[0056] In the specific implementation process, since there may be a possibility that the answer data and question data in the same group are unrelated, when obtaining drug screening data, data in which both the question data and the corresponding answer data have the same drug name should be selected to ensure the effectiveness of data analysis.

[0057] Step S1025: Record the drug names in the drug name database that have drug screening data as representative drugs, and record the drug screening data corresponding to each representative drug.

[0058] In practice, the same representative drug can correspond to multiple drug screening data.

[0059] Step S2: Analyze the drug screening data based on the drug instructions of representative drugs, and obtain the questionable clauses and pending clauses in the drug instructions of the corresponding representative drugs based on the analysis results; screen all questionable clauses, and obtain the questionable collection data for each representative drug based on the screening results and pending clauses.

[0060] Step S2 includes: Step S201, please refer to... Figure 3 As shown, for any representative drug, obtain the drug instructions for the representative drug and record it as the representative instructions; based on the serial number before each paragraph in the representative instructions, divide the text in the representative instructions into segments and record them sequentially as representative segment DW1 to representative segment DW. n Where n represents the maximum value of the serial number preceding the text in the instruction manual, and for a positive integer c less than or equal to n and greater than or equal to 1, it represents the DW segment. c This represents the text paragraph whose prefix number is 'c' in the instruction manual.

[0061] In this embodiment, it is assumed that all drug instructions use Arabic numerals as the serial numbers before each paragraph. When other symbols or letters are used as serial numbers before each paragraph in actual implementation, the corresponding text can be divided according to the serial numbers before each paragraph in the actual drug instructions to obtain representative text segments. The purpose of obtaining representative text segments in this embodiment is only to reasonably divide the drug instructions and analyze them in segments to prevent the data volume from being too large and resulting in a lot of invalid data in the questionable clauses.

[0062] Step S202: For any drug screening data corresponding to a representative drug, the text composed of the drug screening data and the answer data corresponding to the drug screening data is recorded as the drug comparison text; the drug comparison text is used to compare the similarity of all representative text segments in turn, and the representative text segment with the highest similarity is recorded as the suspicious text segment of the drug screening data. The same representative text segment can be recorded as the suspicious text segment of multiple drug screening data.

[0063] Step S203: Obtain the suspicious text segments representing all drug screening data corresponding to the drug, and use the suspicious screening algorithm to obtain the suspicious reference value representing the drug. The suspicious screening algorithm includes: Where T is the suspicious reference value, t is the number of drug screening data representing the drug, and R is the reference value. i The similarity between the suspicious text segment of the i-th drug screening data and the drug comparison text of the drug screening data in all drug screening data;

[0064] In a specific implementation process, for example, during a data analysis, the number of drug screening data representing a drug is 3, the value of n is 10, and the similarity between the suspicious text segment of each drug screening data and the drug comparison text of the drug screening data is 50%, 80%, and 40%, respectively. Then, through calculation, the suspicious reference value is approximately 1.89. In this embodiment, the acquisition of the suspicious reference value is only to provide data support for the subsequent screening of representative text segments, and the suspicious reference value only describes the parameters related to the suspicious text segments and similarity of all drug screening data corresponding to the drug.

[0065] Step S204: For any representative text segment, Let be the comparison value of the representative text segment, where b is the maximum value among all similarities when the representative text segment is marked as a suspicious text segment, and b is 0 when the representative text segment is not marked as a suspicious text segment; v is the number of representative text segments marked as suspicious text segments.

[0066] In the specific implementation process, for example, in a data processing, when a representative text segment is recorded as a suspicious text segment, the maximum value of all similarities is 80%, the number of representative text segments recorded as suspicious text segments is 3, and t is 3, the suspicious reference value is 1.89. Through calculation, the comparison value of the representative text segment is 0.8, and the screening value is 1.09.

[0067] Step S205: Obtain the comparison values ​​of all representative text segments, and record the absolute value of the difference between the comparison value of the representative text segment and the suspicious reference value as the screening value of the representative text segment; record the average value of all screening values ​​as the screening measure value, and record the paragraph in the representative specification corresponding to the representative text segment whose screening value is less than the screening measure value as the suspicious clause of the representative specification, and record the paragraph in the representative specification corresponding to the representative text segment whose screening value is greater than or equal to the screening measure value as the pending clause of the representative specification;

[0068] In the specific implementation process, the smaller the screening value of representative text segment A, the more it indicates that among all the suspicious text segments corresponding to the representative drug, representative text segment A fits the similarity relationship between the representative drug and the drug comparison text. In other words, the more representative text segment A can be used as data for data comparison of the representative drug in subsequent analysis.

[0069] Step S206: Obtain all questionable clauses and all pending clauses corresponding to all representative drugs;

[0070] Step S207: For any suspicious regulation representing a drug, record the answer data in the drug comparison text corresponding to the suspicious regulation as the answer screening data, compare the text similarity between the answer screening data and the suspicious regulation, and record the obtained similarity as the answer similarity of the suspicious regulation.

[0071] Step S208: Obtain the response similarity of all questionable clauses representing the drug, and record the average of all response similarities as the single drug average.

[0072] In the specific implementation process, for example, during a data processing session, if the similarity of responses to all questionable regulations representing a drug is 70%, 80%, and 85% respectively, then the average value for a single drug can be calculated to be 78.3%.

[0073] Step S209: Obtain the average value of each individual drug corresponding to all representative drugs, and record the average value of all individual drug average values ​​as the multi-drug average value;

[0074] Step S210: For any representative drug: when the average value of a single drug is greater than or equal to the average value of multiple drugs, the suspicious clauses among all the suspicious clauses of the representative drug whose response similarity is greater than the average value of a single drug of the representative drug are recorded as the collection clauses.

[0075] In the specific implementation process, in order to ensure that the amount of data collected by the regulations is sufficient, when the average value of a single drug representing a drug is greater than or equal to the average value of multiple drugs, only the suspicious regulations among all the suspicious regulations of the representative drug whose response similarity is greater than the average value of a single drug representing the drug need to be recorded as the collection regulations; when the average value of a single drug representing a drug is less than the average value of multiple drugs, the data can be supplemented by the pending regulations, that is, the z pending regulations with smaller screening values ​​among the pending regulations of the representative drug are used.

[0076] Step S211: When the average value of a single drug representing a drug is less than the average value of multiple drugs, the suspicious regulations with a response similarity greater than the average value of multiple drugs among all the suspicious regulations representing the drug, and the z pending regulations with smaller screening values ​​among the pending regulations representing the drug, are recorded as collected regulations, where z is the number of suspicious regulations among all the suspicious regulations representing the drug with a response similarity greater than or equal to the average value of a single drug representing the drug and less than the average value of multiple drugs.

[0077] In the specific implementation process, for example, in a data processing, if the number of suspicious clauses with a response similarity greater than or equal to the average value of the single drug representing the drug is 5, and the number of suspicious clauses with a similarity less than the average value of multiple drugs is 4, then the value of z should be less than or equal to 4.

[0078] Step S212: Record all drug screening data corresponding to the representative text segment of the collection regulations representing drugs as suspicious text segments as suspicious collection data representing drugs.

[0079] Step S213: Obtain the suspicious data corresponding to all representative drugs.

[0080] Step S3: Optimize data processing based on the suspicious data collected from all representative drugs during corpus data collection;

[0081] Step S3 includes: during corpus data collection, when any of the user-submitted questions β represent a drug α, the question β is compared with all the suspicious data representing drug α in sequence. When any similarity score is greater than E, the question β is retained during data processing.

[0082] In the specific implementation process, the value of E can be set according to the actual data collection needs. When the actual corpus data collection requires more precise data, the value of E can be appropriately increased. When the actual corpus data collection requires more extensive data, the value of E can be appropriately decreased. In this embodiment, the value of E is set to 60%.

[0083] Example 2, please refer to Figure 4 As shown, Figure 4A schematic diagram of an electronic device is provided, which may include a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The memory stores computer-readable instructions, and the processor can call these instructions. When the processor executes a computer-readable instruction, it performs steps similar to those in the corpus data acquisition method based on a large medical question-and-answer model to achieve the following functions: First, it acquires user question and answer data based on the large medical question-and-answer model; it analyzes the question data, filters it based on the analysis results to obtain drug screening data and representative drugs corresponding to each drug screening data; then, it analyzes the drug screening data based on the drug instructions of the representative drugs, and obtains the questionable clauses and pending clauses from the drug instructions corresponding to the representative drugs; it filters all questionable clauses, and obtains the questionable collection data for each representative drug based on the filtering results and the pending clauses; finally, it optimizes data processing based on the questionable collection data of all representative drugs during corpus data acquisition.

[0084] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] Example 3: This application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the corpus data acquisition method based on the medical question-and-answer model provided by the above methods. This method includes: firstly, acquiring user question data and answer data when using the medical question-and-answer model based on the medical question-and-answer model; analyzing the question data, filtering the question data based on the analysis results to obtain drug screening data and representative drugs corresponding to each drug screening data; then analyzing the drug screening data based on the drug instructions of the representative drugs, and acquiring the questionable clauses and pending clauses in the drug instructions corresponding to the representative drugs based on the analysis results; filtering all questionable clauses, and acquiring questionable collection data for each representative drug based on the filtering results and pending clauses; finally, optimizing data processing based on the questionable collection data of all representative drugs during corpus data acquisition.

[0086] Example 4: This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps of the above-described corpus data acquisition method based on a large medical question-and-answer model to achieve the following functions: First, it acquires user question data and answer data when using the large medical question-and-answer model; it analyzes the question data, filters the question data based on the analysis results, and obtains drug screening data and representative drugs corresponding to each drug screening data; then, it analyzes the drug screening data based on the drug instructions of the representative drugs, and obtains the questionable clauses and pending clauses in the drug instructions corresponding to the representative drugs based on the analysis results; it filters all questionable clauses, and obtains the questionable collection data for each representative drug based on the filtering results and pending clauses; finally, it optimizes the data processing based on the questionable collection data of all representative drugs during corpus data acquisition.

[0087] Based on the above description of the embodiments, the embodiments of the present invention can be provided as methods, systems, or computer program products. Based on this understanding, the technical solutions described above, or the parts that contribute to the prior art, can be embodied in the form of software products. These computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or certain parts of the embodiments.

[0088] In the embodiments provided in this application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces. The indirect coupling or communication connection between systems, modules, and units may be electrical, mechanical, or other forms.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A corpus data collection method based on a large-scale medical question-answering model, characterized in that, Includes the following steps: Based on the medical question-and-answer big data model, we obtain the question and answer data of users when using the medical question-and-answer big data model; we analyze the question data, filter the question data based on the analysis results, and obtain drug screening data and the representative drug corresponding to each drug screening data; The drug screening data is analyzed based on the drug instructions of representative drugs, and the questionable clauses and pending clauses in the drug instructions of representative drugs are obtained based on the analysis results. All potentially problematic clauses are screened, and based on the screening results and the clauses to be used, potentially problematic data for each representative drug is collected. During corpus data collection, data processing was optimized based on the collection of easily questionable data representing all drugs.

2. The corpus data acquisition method based on a large medical question-answering model according to claim 1, characterized in that, Based on the medical question-and-answer big data model, we obtain user question and answer data when using the medical question-and-answer big data model; The query data was analyzed, and based on the analysis results, the query data was filtered to obtain drug screening data and representative drugs corresponding to each drug screening data, including: Data extraction is performed on the medical question-and-answer model. All questions raised by users in the medical question-and-answer model are recorded and designated as question data. For any question in the question data, the answer sent by the medical question-and-answer model based on the question is designated as the question data answer. All data answers are recorded and designated as answer data. The question-based drug screening method was used to analyze all the question data, and based on the analysis results, the drug screening data corresponding to all the question data and the representative drug corresponding to each drug screening data were obtained.

3. The corpus data acquisition method based on a large medical question-answering model according to claim 2, characterized in that, Question-based drug screening methods include: Establish a drug name database. Based on web crawling, extract drug names from the data already learned by the medical question-and-answer model, and store all the extracted drug names into the drug name database. For any given query data: perform word segmentation on the query data, and record each Chinese word obtained from the segmentation as the query Chinese word; for any given query Chinese word, compare the query Chinese word with all drug names in the drug name database, and when the query Chinese word is the same as any drug name in the drug name database, record the query Chinese word as the drug Chinese word; Retrieve all Chinese words for the corresponding drugs from all the query data.

4. The corpus data acquisition method based on a large medical question-answering model according to claim 3, characterized in that, Question-based drug screening methods also include: For any drug name A in the drug name database: record the question data containing drug name A in the Chinese words of the drug as the drug data to be screened, perform Chinese word recognition processing on the answer data corresponding to the drug data to be screened, and when drug name A is present in the answer data corresponding to the drug data to be screened, record the drug data to be screened as the drug screening data of drug name A. The names of drugs for which drug screening data exists in the drug name database are designated as representative drugs, and the drug screening data corresponding to each representative drug is recorded.

5. The corpus data acquisition method based on a large medical question-answering model according to claim 4, characterized in that, The drug screening data was analyzed based on the drug instructions for representative drugs, and the questionable clauses and pending clauses in the drug instructions for representative drugs were obtained based on the analysis results. For any representative drug, obtain its instruction leaflet and record it as the representative instruction leaflet; divide the text in the representative instruction leaflet according to the serial number preceding each paragraph and record them sequentially as representative segment DW1 to representative segment DW. n Where n represents the maximum value of the serial number preceding the text in the instruction manual, and for a positive integer c less than or equal to n and greater than or equal to 1, it represents the DW segment. c This represents the text paragraph whose prefix number is 'c' in the instruction manual. For any drug screening data corresponding to a representative drug, the text composed of the drug screening data and the corresponding response data is recorded as the drug comparison text. The drug comparison text is then compared with all representative text segments in turn, and the representative text segment with the highest similarity is recorded as the suspicious text segment of the drug screening data. The same representative text segment can be recorded as the suspicious text segment of multiple drug screening data.

6. The corpus data acquisition method based on a large medical question-answering model according to claim 1, characterized in that, The drug screening data is analyzed based on the drug instructions of representative drugs, and the questionable clauses and pending clauses in the drug instructions of representative drugs are obtained based on the analysis results. Obtain the suspicious text segments of all drug screening data corresponding to the drug, and use the suspicious screening algorithm to obtain the suspicious reference value of the drug. The suspicious screening algorithm includes: Where T is the suspicious reference value, t is the number of drug screening data representing the drug, and R is the reference value. i The similarity between the suspicious text segment of the i-th drug screening data and the drug comparison text of the drug screening data in all drug screening data; For any representative passage, Let be the comparison value of the representative text segment, where b is the maximum value among all similarities when the representative text segment is marked as a suspicious text segment, and b is 0 when the representative text segment is not marked as a suspicious text segment; v is the number of representative text segments marked as suspicious text segments. Obtain the comparison values ​​of all representative text segments, and record the absolute value of the difference between the comparison value of the representative text segment and the suspicious reference value as the screening value of the representative text segment; record the average value of all screening values ​​as the screening measure value, and record the paragraphs in the representative specification corresponding to the representative text segments with screening values ​​less than the screening measure value as the suspicious clauses in the representative specification, and record the paragraphs in the representative specification corresponding to the representative text segments with screening values ​​greater than or equal to the screening measure value as the pending clauses in the representative specification; Obtain all questionable clauses and all pending clauses corresponding to all representative drugs.

7. The corpus data acquisition method based on a large medical question-answering model according to claim 1, characterized in that, All potentially problematic clauses are screened, and based on the screening results and the clauses to be used, potentially problematic data for each representative drug is collected, including: For any erroneous regulation representing a drug, the response data in the drug comparison text corresponding to the erroneous regulation is recorded as the response screening data. The response screening data is compared with the erroneous regulation in terms of text similarity, and the obtained similarity is recorded as the response similarity of the erroneous regulation. Obtain the response similarity of all questionable clauses representing a drug, and record the average of all response similarities as the single drug average; Obtain the average value of each individual drug for all representative drugs, and record the average of all individual drug average values ​​as the multi-drug average value; For any representative drug: when the average value of a single drug is greater than or equal to the average value of multiple drugs, the suspicious clauses among all the suspicious clauses of the representative drug whose response similarity is greater than the average value of a single drug of the representative drug are recorded as the collection clauses. When the average value of a single drug representing a drug is less than the average value of multiple drugs, the suspicious regulations with a response similarity greater than the average value of multiple drugs among all the suspicious regulations representing the drug, and the z pending regulations with smaller screening values ​​among the pending regulations representing the drug, are recorded as the collected regulations, where z is the number of suspicious regulations among all the suspicious regulations representing the drug with a response similarity greater than or equal to the average value of a single drug representing the drug and less than the average value of multiple drugs. When the representative text corresponding to the collection regulations for representative drugs is recorded as a suspicious text, all the drug screening data corresponding to it are recorded as the suspicious collection data for representative drugs. Obtain all suspicious data corresponding to representative drugs.

8. The corpus data acquisition method based on a large medical question-answering model according to claim 1, characterized in that, During corpus data collection, based on the easily questionable data collected from all representative drugs, data processing was optimized, including: During corpus data collection, if any of the user-submitted questions β represent a drug α, then question β is compared with all the easily suspected data representing drug α in turn. If any similarity score is greater than E, then question β is retained during data processing.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method as described in any one of claims 1-8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-8.

Images