Contrast test identification method and device based on large language model and storage medium

Abstract

The invention relates to a contrast test recognition method and device based on a large language model and a storage medium, and is applied to the technical field of medical information, and the method comprises the steps: carrying out word segmentation processing on obtained medical literature data, and then combining the processed medical literature data with RCT-related cue words to obtain an instruction data set; setting a large language model framework and a corresponding loss function, performing iterative training on the model framework through the instruction data set, inputting to-be-classified medical literature data into a trained RCT recognition large language model, and outputting an RCT recognition result according to a format agreed by cue words; according to the scheme, the cue word is combined with the training data, the training data does not need to be labeled, and the problem that the generalization ability of an existing machine learning model in different medical fields and different research problem data sets is greatly different, and the machine learning model extremely depends on a large number of labeled data sets is solved.

Description

Technical Field

[0001] The present invention relates to the field of medical information technology, and in particular to a control trial identification method, device and storage medium based on a large language model. Background Art

[0002] Systematic reviews (SRs), especially those based on randomized controlled trials (RCTs), are crucial for clinical decision-making because they provide high-quality evidence support. Nevertheless, most systematic reviews require screening of a large amount of literature, a process that is usually completed by human experts and is time-consuming and labor-intensive. According to research statistics, it takes an average of about 67.3 weeks to complete a systematic review. However, when retrieving RCTs from databases, traditional retrieval methods are not efficient because usually only about 7% of the search results are truly related to RCTs. Therefore, accurately identifying these RCTs is a crucial step when conducting a systematic review.

[0003] At present, the main method for automatically identifying RCTs in the industry is the automatic identification method of RCTs based on machine learning algorithms. In order to improve the efficiency of RCT literature identification, researchers have developed machine learning (ML) models to identify RCTs. The purpose of these models is to accelerate the workflow of systematic reviews, but their popularity in practical applications is limited. The main disadvantage is that there are large differences in the generalization ability of machine learning models in different medical fields and different research problem data sets, and they are extremely dependent on a large number of labeled data sets. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a control trial identification method, device and storage medium based on a large language model to solve the problem that the generalization ability of existing machine learning models in different medical fields and different research problem data sets varies greatly, and they are extremely dependent on a large number of labeled data sets.

[0005] According to a first aspect of an embodiment of the present invention, a control experiment identification method based on a large language model is provided, the method comprising:

[0006] Obtain titles and abstracts of multiple medical articles;

[0007] Performing word segmentation processing on the titles and abstracts of the plurality of medical documents;

[0008] Setting prompt words related to RCT for large language model training, combining the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing, to obtain an instruction data set;

[0009] Building a large language model framework, and setting a loss function for the large language model framework;

[0010] Iteratively training the large language model framework using the instruction data set until a preset stop condition is met, thereby obtaining a trained RCT recognition large language model;

[0011] The medical literature data to be classified is obtained, and the medical literature data to be classified is input into the trained RCT recognition large language model after word segmentation processing, and the trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words.

[0012] Preferably,

[0013] Before the word segmentation processing of the titles and abstracts of the plurality of medical documents is performed, the following steps are further included:

[0014] Performing data cleaning on the titles and abstracts of the plurality of medical documents; the data cleaning is used to remove special characters and non-text content in the titles and abstracts of the medical documents;

[0015] The word segmentation processing of the titles and abstracts of the plurality of medical documents further includes: removing stop words;

[0016] The stop word removal is used to remove stop words in the titles and abstracts of medical documents.

[0017] Preferably,

[0018] The loss function adopts the cross entropy loss function, and the expression of the cross entropy loss function is:

[0019]

[0020] In the formula, L(y,y * ) represents the loss value; y represents the true label; y * represents the predicted probability of each category; N represents the total number of samples; y i and i * They represent the true label and predicted probability of the i-th sample respectively.

[0021] Preferably,

[0022] A sigmoid function is set in the large language model framework, and the prediction probability output by the model is mapped to a (0, 1) interval through the sigmoid function.

[0023] Preferably, it also includes:

[0024] Setting an RCT identifier, and setting a classification threshold in the RCT identifier;

[0025] Inputting the predicted probability of the medical literature data to be classified output by the trained RCT recognition large language model into the RCT identifier;

[0026] The RCT identifier outputs a document with a predicted probability greater than or equal to the classification threshold as “RCT document”, and outputs a document with a predicted probability less than the classification threshold as “non-RCT document”.

[0027] Preferably,

[0028] The word segmentation processing of the titles and abstracts of the plurality of medical documents includes:

[0029] Use IKAnalyzer or HanLP word segmentation tools to segment the titles and abstracts of multiple medical documents;

[0030] Convert each word segment into a unique index according to the preset vocabulary, and combine the indexes corresponding to one or more words according to the preset model input data length to obtain the input sequence;

[0031] The beginning of each input sequence is marked with '[CLS]', and the end of each input sequence is marked with '[SEP]'.

[0032] According to a second aspect of an embodiment of the present invention, a control experiment identification device based on a large language model is provided, wherein the device comprises:

[0033] Data acquisition module: used to obtain titles and abstracts of multiple medical documents;

[0034] A word segmentation processing module: used for performing word segmentation processing on the titles and abstracts of the plurality of medical documents;

[0035] Training data acquisition module: used to set prompt words related to RCT for large language model training, and combine the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing to obtain an instruction data set;

[0036] Model framework setting module: used to build a large language model framework and set the loss function for the large language model framework;

[0037] Iterative training module: used for iteratively training the large language model framework through the instruction data set until a preset stop condition is met to obtain a trained RCT recognition large language model;

[0038] Classification and recognition module: used to obtain medical literature data to be classified, and input the medical literature data to be classified into the trained RCT recognition large language model after word segmentation. The trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words.

[0039] According to a third aspect of an embodiment of the present invention, there is provided a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented.

[0040] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0041] The present application obtains an instruction data set by performing word segmentation on the acquired medical literature data and combining it with prompt words related to RCT; a large language model framework and a corresponding loss function are set, the model framework is iteratively trained through the instruction data set, the medical literature data to be classified is input into the trained RCT recognition large language model, and the RCT recognition result is output in the format agreed upon by the prompt words; the scheme of the present application combines the prompt words with the training data without the need to annotate the training data, thereby solving the problem that the generalization ability of existing machine learning models in different medical fields and different research problem data sets varies greatly, and the model is extremely dependent on a large number of annotated data sets.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] Figure 1 is a flow chart of a control experiment identification method based on a large language model according to an exemplary embodiment;

[0045] Figure 2 is a schematic diagram of the architecture of an RCT recognition model according to another exemplary embodiment;

[0046] Figure 3 is a system schematic diagram of a control experiment recognition device based on a large language model according to another exemplary embodiment;

[0047] In the accompanying drawings: 1-data acquisition module, 2-word segmentation processing module, 3-training data acquisition module, 4-model framework setting module, 5-iterative training module, 6-classification and recognition module. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0049] Embodiment 1

[0050] Figure 1 is a flow chart of a control experiment identification method based on a large language model according to an exemplary embodiment. Figure 1 As shown, the method includes:

[0051] S1, obtain titles and abstracts of multiple medical articles;

[0052] S2, performing word segmentation processing on the titles and abstracts of the plurality of medical documents;

[0053] S3, setting prompt words related to RCT for large language model training, combining the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing, to obtain an instruction data set;

[0054] S4, constructing a large language model framework, and setting a loss function for the large language model framework;

[0055] S5, iteratively training the large language model framework through the instruction data set until a preset stop condition is met, thereby obtaining a trained RCT recognition large language model;

[0056] S6, obtaining medical literature data to be classified, performing word segmentation processing on the medical literature data to be classified and inputting the data into the trained RCT recognition large language model, wherein the trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words;

[0057] It is understandable that if Figure 2 As shown, a large number of titles and abstracts of medical literature are obtained, and data cleaning is performed on the obtained titles and abstracts of medical literature to remove useless information, such as special characters, non-text content, etc. By cleaning the data, noise and erroneous information can be reduced, and the data quality and the accuracy and robustness of the model can be improved, ensuring that the model can learn and predict more accurately, specifically:

[0058] The specific steps of data cleaning include:

[0059] Determine the definition scope of special characters: Special characters can include non-ASCII characters, punctuation marks, control characters, etc. Use regular expressions or character sets to traverse and check: Check each data in the title and abstract of medical literature to determine whether it contains special characters, and choose to delete or replace the special characters that are screened out;

[0060] For the titles and abstracts of the medical literature after data cleaning, word segmentation is performed, specifically:

[0061] Input the title and abstract of the medical literature into IKAnalyzer or HanLP word segmentation tools. IKAnalyzer can accurately segment Chinese texts, identify various Chinese words and phrases, and effectively improve the accuracy of text processing. It also supports the recognition of new words and professional terms, and is expanded through custom dictionaries to improve the accuracy of word segmentation. It uses optimized algorithms and data structures, has a fast word segmentation speed, and can meet the needs of large-scale text processing. Custom words can be added according to specific fields and needs, which is of great significance for medical literature where special medical terms are prevalent. HanLP not only provides word segmentation functions, but also includes various natural language processing functions such as part-of-speech tagging, named entity recognition, and dependency syntax analysis, which is suitable for complex natural language processing tasks.

[0062] For the multiple tokens obtained after word segmentation, in order to facilitate model processing, we need to convert the tokens into numbers, so we need to pre-build a dictionary (vocabulary), map each word to a unique index number, and then combine the indexes corresponding to one or more word segmentations to obtain input sequences of uniform length; at the same time, the beginning of each input sequence is marked with '[CLS]', and the end of each input sequence is marked with '[SEP]';

[0063] Set the RCT-related prompt words for large language model training, and then combine the prompt words with the input sequence to obtain the instruction data set;

[0064] A base language model and its size and parameters are selected, and a loss function is determined. This application selects cross entropy as the loss function. The cross entropy loss function is widely used in classification problems, especially binary classification problems, such as logistic regression, classification layers in neural networks, etc., and the judgment of whether a medical document is an RCT document in this application is a typical binary classification problem. It can well measure the difference between the predicted probability distribution and the true probability distribution, especially when processing probability outputs. It is more effective than loss functions such as mean square error. Therefore, the cross entropy loss function is more appropriate, and the expression of the cross entropy loss function is:

[0065]

[0066] In the formula, L(y,y * ) represents the loss value; y represents the true label; y * represents the predicted probability of each category; N represents the total number of samples; y i and i * Represent the true label and predicted probability of the i-th sample respectively;

[0067] The large language model framework uses Transformer. Based on the attention mechanism, Transformer can effectively process data in binary classification tasks and provide accurate prediction results. Through the attention mechanism, it can better understand and process text data, thus performing well in binary classification tasks. Compared with other model frameworks, Transformer can effectively process long-distance dependencies in text, which is very important for feature extraction in binary classification tasks.

[0068] At the same time, the Sigmoid function is used to map the output of the model to the interval (0, 1). This mapping allows the output to be interpreted as probability, that is, the output value can be expressed as the probability of belonging to a certain category. This is particularly important for classification problems because the probability ranges from 0 to 1 and can be intuitively interpreted as the possibility of an event occurring. The Sigmoid function is simple to calculate, smooth, and continuously differentiable, which enables it to effectively update parameters in the back-propagation algorithm.

[0069] In the process of training the model framework, until the loss value of the above-mentioned loss function no longer decreases or reaches the preset number of iterations, the trained RCT recognition large language model is obtained. For new medical literature data, the above-mentioned data cleaning, word segmentation and stop word removal are first performed on it, and then the data is input into the above-mentioned trained RCT recognition large language model. The model output is the probability of an RCT document, and the classification threshold preset in the RCT identifier is generally set to 0.5. The RCT identifier directly outputs the result of whether it is an RCT document.

[0070] In order to facilitate understanding of the above content, this embodiment also provides specific cases for illustration, such as:

[0071] If you are an expert in the field of evidence-based medicine, and you are very good at judging whether a document is RCT-related, refer to the following example and generate the corresponding answer. Please judge whether the following medical document is RCT literature, and return the result (prompt word content) with "yes" or "no" only; (1)

[0073] enter:

[0074] Title: Documentation quality as a surrogate marker of PHTLS course awareness and training effectiveness, part of a prospective longitudinal mixed methods EPPTC trial;

[0075] Abstract: Objective: The care of severely injured patients requires multidisciplinary teamwork. The reduction in the number of accident victims ultimately affects daily tasks and skills. The PHTLS (“Prehospital Trauma Life Support”) course is a two-day course for medical and non-medical rescue service personnel that aims to improve prehospital care of trauma patients worldwide. This study aimed to examine the quality of documentation before and after the PHTLS course as a surrogate endpoint for training effectiveness and awareness. Methods: This was a prospective pre- and post-intervention trial as part of the mixed-methods longitudinal EPPTC (Effect of Paramedic Training on Prehospital Trauma Care) study, evaluating the PHTLS course on subjective and objective changes in participants and actual patient care. The courses outlined the SAMPLE method for interrogating anamnestic information, which is considered relevant to patient safety and includes “allergies”, “medications” and “patient history” (AMP). The course was not focused on documentation. Results: A total of 320 protocols were analyzed before and after the training. The PHTLS course resulted in a significant increase in “AMP” information in the documentation (p<0.001). Subgroup analyses of "allergies" (+47.2%), "medications" (+38.1%), and "medical history" (+27.8%) before and after the PHTLS course showed a significant increase in information content. Conclusions: In summary, we showed that PHTLS training can improve documentation quality, which we used as a surrogate endpoint for learning effectiveness and awareness. In this regard, we demonstrated that participants used some parts of the training in real life, thus indicating that the learning approach of the PHTLS training was effective. However, these results do not indicate whether patient care was changed. Copyright 2017 Haske et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

[0076] Output: No (2)

[0078] enter:

[0079] Title: Student perceptions and effectiveness of an innovative learning tool: the anatomy glove learning system.

[0080] Abstract: A trend in anatomy education has been to develop alternative teaching methods to replace or supplement the cadaver laboratory experience; however, empirical evidence on their effectiveness is often not reported. This study aimed to evaluate the effectiveness of the Anatomical Glove Learning System (AGLS), which enables students to learn the relationship between hand structure and function by drawing hand structures onto a worn glove with imprinted bones. Massage therapy students (n = 73) were divided into two groups and were instructed to draw muscles on either: (1) the glove using the AGLS instructional video (3D group); or (2) paper with palmar / dorsal views of the hand bones during an instructor-led activity (2D group). Confidence measures and knowledge tests were completed before, during, and one week after the learning condition. Confidence in hand anatomy gradually increased in the 3D group (3.2 / 10, 4.7 / 10, and 4.8 / 10), whereas confidence in the 2D group began to decline after one week (3.2 / 10, 4.4 / 10, and 3.9 / 10). After learning, hand anatomy knowledge improved immediately in both groups (P<0.001). Students' perceptions of AGLS were also assessed using a 10pt Likert scale evaluation questionnaire (10=high). Students felt that the AGLS video (mean=8.3+ / -2.0) and glove (mean=8.1+ / -1.8) helped improve their understanding of hand anatomy, and most students preferred AGLS as a learning tool (mean=4.6+ / -2.2). This study provides evidence that AGLS and traditional 2D learning methods are equally effective in improving students' confidence and knowledge of hand anatomy.

[0081] Output: Yes.

[0082] Embodiment 2

[0083] Figure 3 is a system schematic diagram of a control experiment identification device based on a large language model according to another exemplary embodiment, the device comprising:

[0084] Data acquisition module 1: used to obtain titles and abstracts of multiple medical documents;

[0085] Word segmentation processing module 2: used for performing word segmentation processing on the titles and abstracts of the plurality of medical documents;

[0086] Training data acquisition module 3: used to set prompt words related to RCT for large language model training, and combine the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing to obtain an instruction data set;

[0087] Model framework setting module 4: used to build a large language model framework and set a loss function for the large language model framework;

[0088] Iterative training module 5: used to iteratively train the large language model framework through the instruction data set until a preset stop condition is met to obtain a trained RCT recognition large language model;

[0089] Classification and recognition module 6: used to obtain medical literature data to be classified, and input the medical literature data to be classified into the trained RCT recognition large language model after word segmentation processing. The trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words.

[0090] Embodiment three:

[0091] This embodiment provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented;

[0092] It is understandable that the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0093] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0094] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0095] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0096] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0097] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0098] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0099] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0100] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0101] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A control experiment identification method based on a large language model, characterized in that: The method comprises: Obtain titles and abstracts of multiple medical articles; Performing word segmentation processing on the titles and abstracts of the multiple medical documents; Setting prompt words related to RCT for large language model training, combining the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing, to obtain an instruction data set; Building a large language model framework, and setting a loss function for the large language model framework; Iteratively training the large language model framework using the instruction data set until a preset stop condition is met, thereby obtaining a trained RCT recognition large language model; The medical literature data to be classified is obtained, and the medical literature data to be classified is input into the trained RCT recognition large language model after word segmentation processing, and the trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words.

2. The method according to claim 1, characterized in that Before the word segmentation processing of the titles and abstracts of the plurality of medical documents is performed, the following steps are further included: Performing data cleaning on the titles and abstracts of the plurality of medical documents; the data cleaning is used to remove special characters and non-text content in the titles and abstracts of the medical documents; The word segmentation processing of the titles and abstracts of the plurality of medical documents further includes: removing stop words; The stop word removal is used to remove stop words in the titles and abstracts of medical documents.

3. The method according to claim 2, characterized in that The loss function adopts the cross entropy loss function, and the expression of the cross entropy loss function is: In the formula, L(y,y * ) represents the loss value; y represents the true label; y * represents the predicted probability of each category; N represents the total number of samples; y i and i * They represent the true label and predicted probability of the i-th sample respectively.

4. The method according to claim 3, characterized in that A sigmoid function is set in the large language model framework, and the prediction probability output by the model is mapped to a (0, 1) interval through the sigmoid function.

5. The method according to claim 4, characterized in that Also includes: Setting an RCT identifier, and setting a classification threshold in the RCT identifier; Inputting the predicted probability of the medical literature data to be classified output by the trained RCT recognition large language model into the RCT identifier; The RCT identifier outputs a document with a predicted probability greater than or equal to the classification threshold as “RCT document”, and outputs a document with a predicted probability less than the classification threshold as “non-RCT document”.

6. The method according to claim 5, characterized in that The word segmentation processing of the titles and abstracts of the plurality of medical documents includes: Use IKAnalyzer or HanLP word segmentation tools to segment the titles and abstracts of multiple medical documents; Convert each word segment into a unique index according to the preset vocabulary, and combine the indexes corresponding to one or more words according to the preset model input data length to obtain the input sequence; The beginning of each input sequence is marked with '[CLS]', and the end of each input sequence is marked with '[SEP]'.

7. A control test recognition device based on a large language model, characterized in that: The device comprises: Data acquisition module: used to obtain titles and abstracts of multiple medical documents; A word segmentation processing module: used for performing word segmentation processing on the titles and abstracts of the plurality of medical documents; Training data acquisition module: used to set prompt words related to RCT for large language model training, and combine the prompt words with the titles and abstracts of multiple medical documents after word segmentation processing to obtain an instruction data set; Model framework setting module: used to build a large language model framework and set the loss function for the large language model framework; Iterative training module: used for iteratively training the large language model framework through the instruction data set until a preset stop condition is met to obtain a trained RCT recognition large language model; Classification and recognition module: used to obtain medical literature data to be classified, and input the medical literature data to be classified into the trained RCT recognition large language model after word segmentation. The trained RCT recognition large language model outputs the RCT recognition result according to the format agreed upon by the prompt words.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, each step of the control experiment identification method based on a large language model as described in any one of claims 1 to 6 is implemented.

Images