A method and device for screening clinical trial subjects based on a large language model

Abstract

The present invention discloses a method and device for clinical trial subject screening based on a large language model, comprising: obtaining first document data; constructing input text for the large language model using task instructions, the first document data, and a list of screening criteria words for the clinical trial; the large language model obtaining a probability value for each screening criteria word based on the input text; using the correct screening criteria word as an acceptance sample, eliminating the correct screening criteria word, and then sampling a number of other screening criteria words from high to low probability to obtain a rejection sample set; using the acceptance and rejection sample sets to perform preference training on the large language model, and updating the parameter values ​​of the large language model through backpropagation; repeating the above steps to iteratively train the large language model, and using the trained large language model to screen clinical trial subjects. The present invention can improve the accuracy of the model in clinical trial subject screening tasks.

Description

Technical Field

[0001] The present invention belongs to the field of medical clinical research, and in particular relates to a method and device for screening clinical trial subjects based on a large language model. Background Art

[0002] Clinical trials are scientific studies conducted on human volunteers (i.e., subjects) to evaluate the effectiveness, safety, and possible side effects of drugs or treatments. Such trials play a vital role in promoting medical progress and improving human health.

[0003] The subject screening process is completed by comparing the subject's electronic medical record with the clinical trial screening criteria. For example, Chinese Patent Publication No. CN114464327A discloses a method and apparatus for screening subjects for clinical research projects. Chinese Patent Publication No. CN116994765A discloses a method and system for screening subjects.

[0004] In recent years, with the development of large language models, methods based on large language models have gradually achieved significant results in clinical trial subject screening tasks. However, traditional large language model loss functions only use the correct clinical trial screening criteria words as loss calculations, without fully utilizing the clinical trial screening criteria words in the calculations. Summary of the Invention

[0005] The present invention provides a method and device for screening clinical trial subjects based on a large language model. By using negative sampling to construct a pair of acceptance samples and rejection samples for preference learning of the large language model, the model is helped to better distinguish correct and incorrect answers, thereby improving the accuracy of the model in the subject screening task of clinical trials.

[0006] A method for screening clinical trial subjects based on a large language model, comprising:

[0007] Acquiring first document data, wherein the first document data is a text of a clinical electronic medical record;

[0008] Constructing the input text of the large language model from the task instruction, the first document data, and the clinical trial screening standard word list; wherein the task instruction is used to tell the large language model the task type to be generated, and the clinical trial screening standard word list is used to limit the classification range generated by the large language model;

[0009] The large language model predicts the probability matrix based on the input text and obtains the probability value of each screening standard word from the probability matrix;

[0010] The correct screening standard words are taken as the acceptance samples. After eliminating the correct screening standard words, several other screening standard words are sampled from high to low probability values ​​to obtain the rejection sample set, and negative sampling preference is performed;

[0011] Use the set of accepted samples and rejected samples to perform preference training on the large language model, and update the parameter values ​​of the large language model through back propagation;

[0012] Repeat the above steps to iteratively train the large language model, and use the trained large language model to screen clinical trial subjects.

[0013] Furthermore, the first document data and the clinical trial screening criteria word list are pieced together into a string, which is then combined with multiple different task instructions into multiple strings; the multiple strings are processed by Tokenizer to obtain a vector matrix of integer sequences as the input text of the large language model.

[0014] Furthermore, the large language model predicts the probability matrix based on the input text, specifically:

[0015] The vector matrix is ​​input into the transformer architecture of the large language model for semantic information encoding. The Transformer architecture consists of an input embedding layer, an encoding layer, and an output layer. The input embedding layer converts the discrete sequence vector matrix into a continuous and dense vector representation. The encoding layer uses a multi-head self-attention mechanism to capture the long-distance dependencies and contextual semantic information in the input sequence. The output layer is a linear layer that converts the encoded information of the encoding layer into a vocabulary sequence probability matrix.

[0016] Furthermore, the probability value of each screening criterion word is obtained from the probability matrix, specifically:

[0017] Record the index of the clinical trial screening standard word list in the large language model vocabulary to obtain an index list; the matrix vector shape output after the Transformer encoding layer is (L, d), and the matrix vector obtained after the output layer is (L, V) tensor; L is the position of the generated sentence, d is the dimension of the hidden layer, and V is the probability value of the vocabulary at the position of the generated sentence; obtain the corresponding label probability value in the index list vocabulary of the clinical trial screening standard words, thereby obtaining the probability value of the clinical trial screening standard word.

[0018] Furthermore, the first document data is manually annotated with the screening criteria words for the clinical trial to obtain the correct screening criteria words. By feeding the first document data and the correct screening criteria words into the large language model, the large language model generates an inference process, and uses heuristic rules to determine whether the inference process is consistent with the result. The rejection sample set that meets the conclusion is saved, and negative sampling preference data can be obtained.

[0019] Furthermore, when using negative sampling preference data to train the large language model, the parameters of the large language model are updated, allowing the large language model to generate new negative sampling preference data better and more appropriately, and continuously iterate to train the large language model online.

[0020] Furthermore, the large language model is trained with negative sampling preference, and the loss function is as follows:

[0021] L=-∑log(P i )+∑βlog(N j )

[0022] Where, P i N is the probability value generated by the language model for the accepted sample. j Generates a probability value for the large language model that rejects samples after negative sampling, and β is the penalty coefficient for rejecting samples.

[0023] A clinical trial subject screening device based on a large language model includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned clinical trial subject screening method.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention uses negative sampling to generate multiple negative samples to improve the recognition effect of positive samples.

[0026] 2. The present invention is based on the labeled conclusions, and the large model generates the reasoning process of the thought chain. By learning the reasoning process of the thought chain and adding the labeled conclusions, the recognition effect of the model is improved.

[0027] 3. The present invention trains a large model through online learning, and samples the trained large model to generate preference data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a flow chart of a method for screening clinical trial subjects based on a large language model according to an embodiment of the present invention.

[0029] Figure 2 Schematic diagram of input text construction of a large language model in an embodiment of the present invention.

[0030] Figure 3 Schematic diagram of the encoding and result probability value calculation process of a large language model in an embodiment of the present invention.

[0031] Figure 4 Schematic diagram of constructing pairs of accepted samples and rejected samples in an embodiment of the present invention.

[0032] Figure 5 Schematic diagram of the inference process of generating a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0034] like Figure 1 As shown, a method for screening clinical trial subjects based on a large language model includes the following steps:

[0035] Step S110: Acquire first document data, which is the text of clinical electronic medical records. The text of clinical electronic medical records is recognized and used for screening subjects for clinical trials.

[0036] Specifically, descriptions of Chinese electronic medical record text fall under the category of clinical trial screening criteria. As clinical trials become increasingly complex and numerous, large language models are emerging and demonstrating promising results in clinical trial subject recruitment. Clinical trial screening criteria are categorized into eight predefined categories and 44 semantic subcategories. The eight categories are "Health Status," "Treatment or Healthcare," "Diagnosis or Laboratory Test," "Demographic Characteristics," "Ethical Considerations," "Lifestyle Choices," "Data or Patient Source," and "Other." The semantic subcategories of the screening criteria under the category of "Health Status" are "Disease", "Symptoms (patient's feelings)", "Signs (doctor's test)", "Pregnancy-related", "Tumor progression", "Disease stage", "Allergy tolerance", "Organ tissue status", "Life expectancy", and "Oral-related"; the semantic subcategories of the screening criteria under the category of "Treatment or health care" are "Drugs", "Treatment or surgery", "Equipment", and "Nursing"; the semantic subcategories of the screening criteria under the category of "Diagnosis or laboratory test" are "Diagnosis", "Laboratory examination", "Risk assessment", and "Receptor status"; the semantic subcategories of the screening criteria under the category of "Demographic characteristics" are "Age", "Special patient characteristics", "Reading and writing ability", "Gender," "Education," "Residence," and "Race." The semantic subcategories of the screening criteria under the "Ethical Considerations" category include "Informed Consent," "Participation in Other Trials," "Investigator's Decision," "Capacity," "Ethical Review," and "Dependence." The semantic subcategories of the screening criteria under the "Lifestyle Choices" category include "Addictive Behaviors," "Sleep," "Exercise," "Diet," "Alcohol Use," "Sexual Orientation," "Smoking Status," and "Blood Donation." The semantic subcategories of the screening criteria under the "Data or Patient Source" category include "Case Source," "Disability Group," "Healthy Group," and "Data Availability." The semantic subcategories of the screening criteria under the "Other" category include "Sentences Containing Multiple Categories." The first document data uses the example of "patients with implants such as pacemakers, metal stents, plates, and joints that are metal or susceptible to electrical pulses." The corresponding clinical trial screening criteria category is the standard term "Device" under the "Treatment or Healthcare" category.

[0037] Step S120 : After acquiring the first document data, the task instruction and the clinical trial screening standard word list are combined to construct the input text of the large language model.

[0038] Specifically, such as Figure 2As shown, task instructions are used to instruct the large language model to perform the task type. Three different task instruction descriptions are defined here, all of which instruct the large language model to perform the task of clinical trial subject screening. Defining different instruction descriptions allows the large language model to better identify clinical trial subject screening tasks from different descriptive perspectives. Three different task instruction descriptions are defined, including: "Based on the following description of the research subject, determine the type of trial screening criteria:," "Given a piece of information about patient selection, determine the type of trial screening criteria:," and "Read the following text and select the correct type of clinical trial screening criteria:." The clinical trial screening criteria word list is used to limit the classification scope generated by the large language model. It consists of 44 screening criteria semantic subcategories, namely the screening categories of clinical trial screening criteria, defined as follows: "Type options: disease, symptoms (patient experience), signs (doctor's test), pregnancy-related, tumor progression, disease stage, allergy tolerance, organ tissue status, life expectancy, oral-related, medication, treatment or surgery, equipment, nursing, diagnosis, laboratory test, risk assessment, receptor status, age, special patient characteristics, literacy, gender, education, living conditions, race, informed consent, participation in other trials, investigator decision, ability, ethical review, dependency, addictive behavior, sleep, exercise, diet, alcohol use, sexual orientation, smoking status, blood donation, case source, disability group, healthy group, data availability." The three different task instruction descriptions, the first document data, and the clinical trial screening criteria word list can be combined into three sets of large language model input texts.

[0039] One of the large language model input texts is as follows: "Please determine the type of trial screening criteria based on the description of the following research subjects: \nPatients with pacemakers, metal stents, plates, joints, and other metal or electrical pulse-susceptible implants. \nType options: Disease, Symptoms (patient perception), Signs (doctor's examination), Pregnancy-related, Tumor progression, Disease stage, Allergy tolerance, Organ tissue status, Life expectancy, Oral-related, Drugs, Treatment or surgery, Equipment, Nursing, Diagnosis, Laboratory tests, Risk assessment, Receptor status, Age, Special patient characteristics, Literacy, Gender, Education, Living conditions, Race, Informed consent, Participation in other trials, Investigator decision, Ability, Ethical review, Dependence, Addictive behavior, Sleep, Exercise, Diet, Alcohol use, Sexual orientation, Smoking status, Blood donation, Case source, Disability group, Healthy group, Data availability."

[0040] Step S130 , using the text input into the large language model to call the large language model to obtain the large language model prediction probability matrix result, and obtaining the probability value of the screening standard word of the clinical trial from the large language model prediction probability matrix.

[0041] Specifically, such as Figure 3As shown in the following text, "Please determine the type of trial screening criteria based on the description of the following research subjects: Patients with pacemakers, metal stents, plates, joints, and other metal or electrical pulse-susceptible implants. Type options: Disease, Symptoms (patient perception), Signs (doctor's test), Pregnancy-related, Tumor progression, Disease stage, Allergy tolerance, Organ and tissue status, Life expectancy, Oral-related, Medication, Treatment or surgery, Equipment, Nursing, Diagnosis, Laboratory tests, Risk assessment, Receptor status, Age, Special patient characteristics, Literacy, Gender, Education, Living conditions, Race, Informed consent, Participation in other trials, Investigator decision, Ability, Ethical review, Dependency, Addictive behavior, Sleep, Exercise, Diet, Alcohol use, Sexual orientation, Smoking status, Blood donation, Case source, Disability group, Healthy group, Data availability" is the input text for the large language model to perform clinical trial subject screening. To convert it into a dictionary sequence (index) vector matrix that the large language model can accept, the input text data needs to be processed into a vector matrix of integer sequences through Tokenizer.

[0042] Tokenizers are a method used to digitize and serialize text. Because computers cannot directly understand the meaning of language, they typically convert each word (which in Chinese can be a single character or a phrase) into a unique integer. This converts the entire text into a sequence of numbers. This sequence is then further vectorized, and the resulting vector data can be fed into the model for processing.

[0043] Specifically, the input text is converted into the following sequence (index) vector matrix after passing through the Tokenizer: "[102185,25,21,24,92015,11,109391,25,57750,345,35568,99482,25,102113,115839,105748,16,7948,11,108181,16,9754,198,46451,99252,99497,25,16,104480,118945,3344 [7,101924,102113,115839,105748,11,42192,16872,101775,105748,117949,11,...5373,121003,111192,99797,100416,11,99493,99772,116545,99543,38342,72586,20221,1773]", the above vector matrix is ​​input into the transform architecture of the large language model for semantic information encoding.

[0044] Among them, the Transformer architecture is the core structure of the large language model, which consists of three main components: the input embedding layer (embed_tokens), which converts the discrete sequence (index) into a continuous dense vector representation; the Transformer encoder encoder, which uses a multi-head self-attention mechanism to capture the long-distance dependencies and contextual semantic information in the input sequence; the output layer (lm_head): is a linear layer used to generate a vocabulary sequence probability matrix from the encoder encoding information.

[0045] Specifically, the sequence (index) vector matrix "[102185,25,21,24,92015,11,109391,25,57750,345,35568,99482,25,102113,115839,105748,16,7948,11,108181,16,9754,198,46451,99252,99497,25,16,104480,118945,33447,101924,102113,11583 After the input embedding layer (embed_tokens) of the Transformer, the discrete sequence (index) is converted into a continuous dense vector matrix. The shape of the embed_tokens mapping matrix is ​​[sequence_length, token_indexs, 4096], where sequence_length is the sentence length, token_indexs is the dictionary size, and 4096 represents the hidden layer size of the large language model. The result then passes through the Transformer Encoder, which uses a stacked multi-head self-attention mechanism to extract increasingly abstract and complex text features layer by layer. The output matrix is ​​a vector with a shape of [sequence_length, 4096], where sequence_length represents the sentence length and 4096 represents the size of the hidden layer of the large language model. Finally, the output layer (lm_head) converts the encoder information into a word sequence probability matrix. This entire process allows the large language model to generate a probability matrix for the clinical trial's screening criteria for the subject screening task.

[0046] The large language model generates the probability matrix of screening criteria words for clinical trials, which is obtained by the following method: the large language model vocabulary (w1, w2, ..., w k ), where k = V, V is the size of the vocabulary. The words in the clinical trial screening criteria word list can be found in the corresponding position in the large language model vocabulary. The index of the clinical trial screening criteria word list in the vocabulary is recorded to obtain the index list. The output matrix vector shape after the Transformer encoding layer is (L, d), and after passing through the lm_head layer, the matrix vector (L, V) tensor is obtained. L is the position of the generated sentence, and V is the probability value of the vocabulary at the position of the generated sentence. By obtaining the probability value of the corresponding label in the index list vocabulary of the clinical trial screening criteria word, the probability value of the clinical trial screening criteria word can be obtained.

[0047] Step S140 , taking the correct clinical trial screening criteria words as acceptance samples, removing the correct clinical trial screening criteria words, and sampling other clinical trial screening criteria words with high probability as rejection sample sets to perform negative sampling preference training.

[0048] Specifically, there are 44 words in the screening criteria for clinical trials, as follows: "disease, symptoms (patient experience), signs (doctor's test), pregnancy-related, tumor progression, disease stage, allergy tolerance, organ and tissue status, life expectancy, oral-related, medication, treatment or surgery, equipment, nursing, diagnosis, laboratory tests, risk assessment, receptor status, age, special patient characteristics, literacy, gender, education, living conditions, race, informed consent, participation in other trials, researcher decision, ability, ethical review, dependency, addictive behavior, sleep, exercise, diet, alcohol use, sexual orientation, smoking status, blood donation, case source, disabled group, healthy group, data availability, and sentences containing multiple categories." Based on step S130, the probability value of the screening criteria word output of the clinical trial corresponding to the first document data after passing through the large language model can be obtained. Sort the probability values ​​from large to small, select the top 4 clinical trial screening criteria words and put them into the temporary sample set, select the correct clinical trial screening criteria words as the accepted samples, remove the correct clinical trial screening criteria words from the temporary sample set, and select the top 3 sample sets. These sample sets are high-probability classification words, indicating that they are samples that are easily misidentified by the large language model and are very suitable for being rejected sample sets. Figure 4 As shown in the figure, the large language model calculates and filters the clinical trial screening criteria term "device" as an acceptance sample, with a large language model-predicted probability of 0.85. The clinical trial screening criteria terms "drug," "ability," and "race" are rejection samples, with large language model-predicted probability values ​​of 0.91, 0.81, and 0.76, respectively, making them the top three probability values ​​for clinical trial screening criteria terms.

[0049] The loss function formula for defining the preference training of the large language model is as follows:

[0050] L=-∑log(P i )+∑βlog(N j )

[0051] Where, P i N is the probability value generated by the language model for the accepted sample. j Generates a probability value for the large language model that rejects samples after negative sampling. β is the penalty coefficient for rejecting samples, which is set to 2.

[0052] The loss function formula of preference training shows that the loss value is -0.9964181834795334=-math.log(0.85)+2*(math.log(0.91)+math.log(0.81)+math.log(0.76)). Finally, the parameter values ​​of the large language model are updated through backpropagation.

[0053] In step S150, the preference training data is generated by self-improvement of a large language model.

[0054] The details are as follows: Due to the manual annotation of the screening standard words of clinical trials in the electronic medical record text, the screening standard words of the electronic medical record and clinical trials are given to the large language model, such as Figure 5 As shown, the large language model generates the inference process and uses heuristic rules to determine whether the inference process is consistent with the results. The heuristic rules check whether the text contains "sorry" and is relatively short (i.e., less than 50 words). It usually indicates that the LLM is having difficulty responding to the evolved instructions. The secondary inference results are filtered.

[0055] Step S160: During the training of preference data, the parameters of the large language model will be updated during the learning process. θ is used as the parameter of the model, and θ0 is the parameter of the initial model. After one round of iteration, we get θ i Model parameters: Apply the latest training parameters to the self-improving large language model, allowing the large language model to better and more appropriately generate new preferred training data, and then iteratively train the large language model through continuous online training.

[0056] Based on the same inventive principle, an embodiment of the present invention also provides a clinical trial subject screening device based on a large language model, which is characterized in that it includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned clinical trial subject screening method.

[0057] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for screening clinical trial subjects based on a large language model, characterized in that: include: Acquiring first document data, wherein the first document data is a text of a clinical electronic medical record; Constructing the input text of the large language model from the task instruction, the first document data, and the clinical trial screening standard word list; wherein the task instruction is used to tell the large language model the task type to be generated, and the clinical trial screening standard word list is used to limit the classification range generated by the large language model; The large language model predicts the probability matrix based on the input text and obtains the probability value of each screening standard word from the probability matrix; The correct screening standard words are taken as the acceptance samples. After eliminating the correct screening standard words, several other screening standard words are sampled from high to low probability values ​​to obtain the rejection sample set, and negative sampling preference is performed; Use the set of accepted samples and rejected samples to perform preference training on the large language model, and update the parameter values ​​of the large language model through back propagation; Repeat the above steps to iteratively train the large language model, and use the trained large language model to screen clinical trial subjects.

2. The method for screening clinical trial subjects based on a large language model according to claim 1, characterized in that: The first document data and the clinical trial screening criteria word list are concatenated into a string, which is then combined with multiple different task instructions to form multiple strings. The multiple strings are processed through a Tokenizer to obtain a vector matrix of integer sequences as the input text of the large language model.

3. The method for screening clinical trial subjects based on a large language model according to claim 2, characterized in that: The large language model predicts the probability matrix based on the input text, specifically: The vector matrix is ​​input into the transformer architecture of the large language model for semantic information encoding. The Transformer architecture consists of an input embedding layer, an encoding layer, and an output layer. The input embedding layer converts the discrete sequence vector matrix into a continuous and dense vector representation. The encoding layer uses a multi-head self-attention mechanism to capture the long-distance dependencies and contextual semantic information in the input sequence. The output layer is a linear layer that converts the encoded information of the encoding layer into a vocabulary sequence probability matrix.

4. The method for screening clinical trial subjects based on a large language model according to claim 3, characterized in that: Get the probability value of each filter word from the probability matrix, specifically: Record the index of the clinical trial screening standard word list in the large language model vocabulary to obtain an index list; the matrix vector shape output after the Transformer encoding layer is (L, d), and after passing through the output layer, the matrix vector (L, V) tensor is obtained; L is the position of the generated sentence, d is the dimension of the hidden layer, and V is the probability value of the vocabulary at the position of the generated sentence; the probability value of the corresponding label is obtained from the index list vocabulary of the clinical trial screening standard words, thereby obtaining the probability value of the clinical trial screening standard word.

5. The method for screening clinical trial subjects based on a large language model according to claim 1, characterized in that: Manually annotate the first document data with the screening criteria words for the clinical trial to obtain the correct screening criteria words. Then, by feeding the first document data and the correct screening criteria words into the large language model, the large language model generates an inference process and uses heuristic rules to determine whether the inference process is consistent with the result. The rejection sample set that meets the conclusion is saved to obtain negative sampling preference data.

6. The method for screening clinical trial subjects based on a large language model according to claim 1, characterized in that: When training the large language model using negative sampling preference data, the large language model updates its parameters, allowing it to generate new negative sampling preference data better and more appropriately, and continuously iterates to train the large language model online.

7. The method for screening clinical trial subjects based on a large language model according to claim 1, characterized in that: The large language model is trained with negative sampling preference, and the loss function is as follows: L=-∑log(P i )+∑βlog(N j ) Where, P i N is the probability value generated by the language model for the accepted sample. j Generates a probability value for the large language model that rejects samples after negative sampling, and β is the penalty coefficient for rejecting samples.

8. A clinical trial subject screening device based on a large language model, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the clinical trial subject screening method according to any one of claims 1 to 7.

Images