Large medical model pre-training data construction method, system, equipment and medium

Abstract

The invention discloses a medical large model pre-training data construction method, system and device and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a medical text corpus set, analyzing the medical text corpus set into a plurality of medical samples, and calculating a static semantic density score, a model uncertainty score and a dynamic course difficulty score of the medical samples; a comprehensive sampling weight is obtained through fusion according to set weights, and a training batch of the round is obtained through weighted sparse sampling and used for updating model parameters and generating a sample loss value; learning progress feedback is generated through loss value comparison, and the dynamic course difficulty score is updated in a closed loop mode for next-round sampling. According to the method, the storage and calculation cost can be reduced, meanwhile, the precision of medical text description by the model is kept, the engineering landing efficiency of the large medical model is improved, the core problem that the course design lags behind the model capability is solved, the data construction process has the real adaptive capability, and the training process of the large medical model is efficient.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, and medium for constructing pre-training data for a large medical model. Background Technology

[0002] With the rapid development of medical artificial intelligence, large-scale medical models have demonstrated enormous potential in application scenarios. These models require pre-training on massive amounts of medical text to master complex medical knowledge, such as descriptions. However, the medical text corpus is vast and of varying quality. Indiscriminate training on the entire corpus would result in extremely high computational costs and I / O loads, severely limiting the model's training efficiency and deployment speed. Therefore, how to efficiently select high-value samples from the medical corpus during the pre-training phase and dynamically match them to the model's learning progress has become a core technical problem that urgently needs to be solved in the development of large-scale medical models.

[0003] To alleviate the aforementioned problems, existing technologies mainly employ three types of methods: First, random uniform sampling, which reduces the amount of training data by decreasing the sampling ratio; second, static weighted sampling, such as ranking samples by fixed priority based on perplexity or TF-IDF weights; and third, basic course learning, where a static difficulty table is predefined manually and fed into the model in a preset order. These methods reduce training overhead to some extent and attempt to allow the model to learn from easy to difficult.

[0004] However, existing technologies still have significant drawbacks: random sampling and static weighting are not coupled with the model's real-time learning state, leading to repeated training with a large amount of low-value medical text, wasting computational resources; the difficulty of manually preset courses cannot be adjusted according to the model's mastery level, and even after the model has fully absorbed simple samples such as "text descriptions," the system continues to feed similar data, resulting in overtraining; the sampling strategy and course learning are independent of each other, making it difficult to coordinate optimization and achieve sufficient learning. These shortcomings make existing methods inefficient and poorly adaptable in the pre-training of large medical models, failing to meet the dual requirements of training speed and medical knowledge accuracy in practical applications.

[0005] In summary, the current field of medical large-scale model pre-training lacks a data construction scheme that can dynamically adjust sampling strategies and course difficulty based on the model's learning status, leading to wasted training resources and insufficient mastery of medical knowledge. Therefore, there is an urgent need to propose a data construction method that is coupled in real time with the model's learning progress and adaptively evolves, in order to solve the problems of static, fragmented, and inefficient existing technologies. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically the problems of wasted training resources and low learning efficiency caused by the static and fixed sampling strategies and course difficulty in the pre-training of existing large-scale medical models. Specifically, this invention provides a method, system, device, and medium for constructing pre-training data for large-scale medical models, as detailed below: 1) In a first aspect, the present invention provides a method for constructing pre-training data for a large medical model, the specific technical solution of which is as follows: S1: Obtain a medical text corpus and parse it line by line to obtain multiple medical samples; for each medical sample, calculate its static semantic density score, model uncertainty score, and dynamic course difficulty score. S2, based on the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample, the comprehensive sampling weight of the medical sample is obtained by fusion according to the set weight; S3, perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; use the training batch to update the parameters of the large medical model and obtain the loss value of each medical sample in this round; S4 generates learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample. The dynamic course difficulty score is updated based on this learning progress feedback to form a closed-loop control, which is used for the calculation of the sampling weight of the medical sample in the next round.

[0007] The beneficial effects of the medical large-scale model pre-training data construction method provided by this invention are as follows: By integrating static semantic density score, model uncertainty score, and dynamic course difficulty score into a comprehensive sampling weight, sparse sampling is directly targeted at medical samples with high knowledge density that the model has not yet mastered, significantly reducing invalid I / O and improving training speed. Learning progress feedback generated by "current round - previous round loss value" updates the dynamic course difficulty score in real time, replacing manual presets and achieving automatic matching of course difficulty with model learning status. This avoids repeated training on samples already mastered, saving computational resources. The updated dynamic course difficulty score immediately participates in the next round of comprehensive sampling weight calculation, forming a closed loop of "sampling → training → feedback → resampling," allowing the large-scale medical model to encounter more advanced medical terminology within the same epoch, accelerating knowledge accumulation. Weighted sparse sampling only uses about 5% of medical samples to achieve task metrics comparable to full training, reducing storage and computational costs while maintaining the model's accuracy in describing medical texts. This improves the engineering deployment efficiency of large-scale medical models, solving the core problem of "course design lagging behind model capability," and enabling the data construction process to have true adaptive capabilities, thus facilitating the efficient training process of large-scale medical models.

[0008] Based on the above solution, the present invention can be further improved as follows.

[0009] Furthermore, the static semantic density score of each medical sample is obtained by weighted fusion of the entity density and syntactic complexity of the medical sample; where the entity density of the medical sample is the number of entities in the sample divided by the number of sentences, and the syntactic complexity of the medical sample is the average dependency tree depth.

[0010] Furthermore, the model uncertainty score for each medical sample is obtained by calculating the entropy of the predicted probability distribution of that medical sample using the large medical model.

[0011] Furthermore, the steps for updating the dynamic course difficulty score for each medical sample include: The learning progress feedback is multiplied by a first-order coefficient λ, and then added to (1-λ) multiplied by the previous round of dynamic course difficulty score to obtain the current round of dynamic course difficulty score; where λ is set to 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively, to achieve adaptive matching between the difficulty of medical samples and the learning stage of the large medical model.

[0012] Furthermore, the weighted sparse sampling is sorted in descending order of all comprehensive sampling weights, and no more than 5% of the total medical samples are selected to form the training batch for this round, so as to reduce the I / O load of the medical corpus.

[0013] Furthermore, the specific steps for generating learning progress feedback are as follows: ; in, For the i-th medical sample in round Feedback on learning progress For the i-th medical sample, For medical samples In the round The loss value, For medical samples In the previous round The loss value.

[0014] Furthermore, the closed-loop control includes: If the dynamic course difficulty score of the medical sample is higher than the preset high threshold and its model uncertainty score is lower than the preset low threshold, the dynamic course difficulty score of the medical sample will be reduced by the attenuation ratio. If the model uncertainty score of the medical sample continues to be higher than the preset high confusion threshold for a number of rounds, its dynamic course difficulty score will be increased by incremental steps, and the increased value will not exceed the upper limit, thereby correcting the difficulty of the medical sample.

[0015] 2) In a second aspect, the present invention also provides a medical large model pre-training data construction system, the specific technical solution of which is as follows: a first calculation module, a second calculation module, a third calculation module and a control module; The first calculation module is used to acquire a medical text corpus and parse it line by line to obtain multiple medical samples; for each medical sample, its static semantic density score, model uncertainty score and dynamic course difficulty score are calculated respectively. The second calculation module is used to fuse the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample according to a set weight to obtain the comprehensive sampling weight of the medical sample. The third calculation module is used to perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; the training batch is used to update the parameters of the large medical model and obtain the loss value of each medical sample in this round; The control module is used to generate learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample, update the dynamic course difficulty score based on the learning progress feedback, form a closed-loop control, and calculate the sampling weight of the medical sample in the next round.

[0016] Based on the above solution, the present invention can be further improved as follows.

[0017] Furthermore, the static semantic density score of each medical sample is obtained by weighted fusion of the entity density and syntactic complexity of the medical sample; where the entity density of the medical sample is the number of entities in the sample divided by the number of sentences, and the syntactic complexity of the medical sample is the average dependency tree depth.

[0018] Furthermore, the model uncertainty score for each medical sample is obtained by calculating the entropy of the predicted probability distribution of that medical sample using the large medical model.

[0019] Furthermore, the steps for updating the dynamic course difficulty score for each medical sample include: The learning progress feedback is multiplied by a first-order coefficient λ, and then added to (1-λ) multiplied by the previous round of dynamic course difficulty score to obtain the current round of dynamic course difficulty score; where λ is set to 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively, to achieve adaptive matching between the difficulty of medical samples and the learning stage of the large medical model.

[0020] Furthermore, the weighted sparse sampling is sorted in descending order of all comprehensive sampling weights, and no more than 5% of the total medical samples are selected to form the training batch for this round, so as to reduce the I / O load of the medical corpus.

[0021] Furthermore, the specific steps for generating learning progress feedback are as follows: ; in, For the i-th medical sample in round Feedback on learning progress For the i-th medical sample, For medical samples In the round The loss value, For medical samples In the previous round The loss value.

[0022] Furthermore, the closed-loop control includes: If the dynamic course difficulty score of the medical sample is higher than the preset high threshold and its model uncertainty score is lower than the preset low threshold, the dynamic course difficulty score of the medical sample will be reduced by the attenuation ratio. If the model uncertainty score of the medical sample continues to be higher than the preset high confusion threshold for a number of rounds, its dynamic course difficulty score will be increased by incremental steps, and the increased value will not exceed the upper limit, thereby correcting the difficulty of the medical sample.

[0023] 3) In a third aspect, the present invention also provides a computer device, the computer device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement any of the above methods.

[0024] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0025] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0026] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the steps of a method for constructing pre-training data for a large medical model according to an embodiment of the present invention. Figure 2 This is a structural framework diagram of an electronic device according to the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, an embodiment of the present invention provides a method for constructing pre-training data for a large medical model, comprising the following steps: S1. Obtain a medical text corpus and parse it line by line to obtain multiple medical samples; for each medical sample, calculate its static semantic density score, model uncertainty score, and dynamic course difficulty score. S2, based on the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample, the comprehensive sampling weight of the medical sample is obtained by fusion according to the set weight; S3: Perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; use the training batch to update the parameters of the large medical model and obtain the loss value of each medical sample in this round. S4 generates learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample. The dynamic course difficulty score is updated based on this learning progress feedback to form a closed-loop control, which is used for the calculation of the sampling weight of the medical sample in the next round.

[0029] The beneficial effects of the medical large-scale model pre-training data construction method provided by this invention are as follows: By integrating static semantic density score, model uncertainty score, and dynamic course difficulty score into a comprehensive sampling weight, sparse sampling is directly targeted at medical samples with high knowledge density that the model has not yet mastered, significantly reducing invalid I / O and improving training speed. Learning progress feedback generated by "current round - previous round loss value" updates the dynamic course difficulty score in real time, replacing manual presets and achieving automatic matching of course difficulty with model learning status. This avoids repeated training on already mastered samples, saving computational resources. The updated dynamic course difficulty score immediately participates in the next round of comprehensive sampling weight calculation, forming a closed loop of "sampling → training → feedback → resampling," allowing the large-scale medical model to encounter more advanced medical terminology within the same epoch, accelerating knowledge accumulation. Weighted sparse sampling only uses about 5% of medical samples to achieve task metrics comparable to full training, reducing storage and computational costs while maintaining the model's accuracy in describing medical texts. This improves the engineering deployment efficiency of large-scale medical models, solving the core problem of "course design lagging behind model capability," and enabling the data construction process to have true adaptive capabilities, thus facilitating the efficient training process of large-scale medical models.

[0030] It should be noted that, for ease of understanding, the technical terms used in this solution will be explained one by one, and will not be repeated hereafter: Medical text corpus: refers to the collection of raw texts from medical literature, guidelines, descriptions, etc., and serves as the source of pre-training data for the medical big data model. In this embodiment, the medical text corpus is used to train the medical big data model, which is then used to further generate and interpret texts that appear during the medical process, such as generating medical professional promotional slogans based on medical text descriptions.

[0031] Medical sample: The smallest training unit obtained after parsing each sentence in the corpus, typically a complete sentence or paragraph, used for subsequent score calculation and sampling. In this embodiment, it is used... Let i represent the medical sample, where i refers to the i-th medical sample.

[0032] Static semantic density score: derived from the formula The calculation yielded the result. In the formula, Refers to medical samples The static semantic density score, For medical samples The vocabulary set, v is the vocabulary set Any word in the list is used to iterate and calculate the static semantic density score, v∈ Represents a single word in a vocabulary set. Entity density is calculated by dividing the number of entities by the number of sentences. The syntactic complexity is the average dependency tree depth. The static semantic density score reflects the richness of medical information contained in the medical sample itself and does not change with training rounds.

[0033] Entity density: As mentioned above, this refers to the number of medical entities identified in a medical sample divided by the number of sentences.

[0034] Syntactic complexity: As mentioned above, this refers to the average dependency tree depth, which measures the complexity of sentence structure and, together with entity density, constitutes the static semantic density score.

[0035] Model uncertainty score: derived from the formula The entropy value is calculated from the predicted probability distribution output by the large medical model. A higher entropy indicates greater uncertainty in the large medical model regarding that particular medical sample. In the formula, For large medical models of medical samples Predicted as category The probability, This refers to the category index in the output of the large-scale medical model prediction. It is used to iterate through all possible categories to calculate the model uncertainty score (i.e., entropy). The large-scale medical model predicts a medical sample and outputs a probability distribution for each category. Corresponding probability ; Indicates the th in this distribution One category; It is merely a category index variable with no medical meaning, used for entropy calculation.

[0036] Dynamic course difficulty score: Pass This indicates the difficulty level used to characterize the changes in medical samples as the training process progresses. The dynamic course difficulty is calculated based on learning progress feedback, and the update mechanism is as follows: In the formula, The dynamic course difficulty score for round t. For feedback on learning progress, λ is a first-order coefficient. For the previous round ( 1) Dynamic course difficulty score.

[0037] First-order coefficient λ: Momentum decay coefficient, with a value of 0.3 (controlling the smoothness of course score updates). It is set to 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively: the early stage of training refers to the medical large-scale model training epochs of 1-50. , Give higher weight ( The focus is on basic language structure; the mid-term of training refers to the 51-200 training epochs of the large medical model. ,balance With history ( ), handling technical terminology and logical reasoning; late training refers to after 201 training iterations of the large medical model. , Weight The performance was improved to 0.6, which helps to overcome high uncertainty data; thus achieving a match between the smoothness of difficulty updates and the model stage.

[0038] Learning progress feedback: Pass This indicates that, according to the formula The calculation yields the result; where, For the i-th medical sample in round Feedback on learning progress For the i-th medical sample, For medical samples In the round The loss value, For medical samples In the previous round The loss value. Through rounds Compared to the previous round Comparing the loss values ​​of two adjacent rounds quantifies the learning progress of the large medical model on the medical sample.

[0039] Weighted sparse sampling: Sort all samples in descending order according to their comprehensive sampling weights, and select no more than 5% of the total medical samples to form the training batch for this round (e.g., only 50 million samples are sampled when the total data volume is 1 billion) to reduce the I / O load of the medical corpus.

[0040] Comprehensive sampling weight: from the formula The calculation yields the result; where, For medical samples In training rounds The overall sampling weight determines the probability of it being selected; For normalized weight coefficients, satisfying The initial value is set to The final sampling priority value is obtained by fusing the static semantic density score, model uncertainty score, and dynamic course difficulty score.

[0041] Preset high threshold / preset low threshold / preset high confusion threshold refers to several thresholds set according to experiments or needs. It is only used to distinguish between them and has no other meaning.

[0042] In another embodiment of this solution, S1 is specifically implemented as follows: After obtaining the medical text corpus, the original text is first segmented by periods, line breaks, or XML tags, so that each medical sample... To preserve complete semantics, the entity density is obtained by dividing the number of entities by the number of sentences. Simultaneously, a dependency parser is used to generate dependency trees, and the average dependency tree depth across all words is calculated as the syntactic complexity. The entity density is then multiplied by the syntactic complexity and normalized according to the vocabulary size |V| to obtain the static semantic density score. The same medical sample is fed into the current round of large-scale medical models, and the predicted probability distribution output by the softmax function is used. ,according to Calculate the entropy value to obtain the model uncertainty score; retain the dynamic course difficulty score from the previous round. Feedback on the progress of this round of learning We obtain the weighted sum of the first-order coefficients λ. After the update is completed, the dynamic course difficulty score for this round is obtained. The above three steps are performed on each medical sample in turn, which realizes the overall process of "obtaining a medical text corpus and parsing it line by line to obtain multiple medical samples, and calculating the static semantic density score, model uncertainty score and dynamic course difficulty score for each medical sample respectively".

[0043] In another embodiment of this solution, S2 is specifically implemented as follows: Substituting the three scores (static semantic density score, model uncertainty score, and dynamic course difficulty score) of the same medical sample into the comprehensive sampling weight formula, we first keep the sum of the three coefficients at 1, that is, we set the weights to satisfy... Then, select the corresponding values ​​according to the current training phase: for the initial phase The value remains unchanged in the mid-term; however, if the average learning progress feedback is detected to be below 0.05 in the later stage, then... Increase to 0.6 and adjust the remaining coefficients simultaneously; after assigning the coefficients, proceed directly according to the formula: Calculated This refers to the comprehensive sampling weight of the medical sample in this round of training, which is used for subsequent sparse sampling and sorting.

[0044] In another embodiment of this solution, S3 is specifically implemented as follows: The comprehensive sampling weight set of all medical samples A weighted random sampler is used to sort medical samples by weight from highest to lowest, and the top K=5% of these samples form the current training batch to reduce the I / O load of the medical corpus. This training batch is then fed into the large-scale medical model for forward propagation to obtain the predicted distribution, and cross-entropy loss is calculated between this distribution and the ground truth labels. After backpropagation, the parameters of the large-scale medical model are updated, and each medical sample is recorded. The loss value in this round This is used for subsequent learning progress feedback calculations.

[0045] In another embodiment of this solution, S4 is specifically implemented as follows: For each medical sample Take the loss value of this round. Compared with the previous round of loss value From the formula: The calculation yields the following result: For medical samples In the round The loss value, and The meaning is consistent; For medical samples In the previous round The loss value, and The meaning is consistent. The received learning progress feedback will be... Difficulty score compared to the previous round of dynamic courses The dynamic update mechanism for exponential moving average updates is as follows: λ is the momentum decay coefficient, which is taken as 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively.

[0046] New Immediately replace the old value and write it back to the medical sample attribute; this will be directly called during the next round of comprehensive sampling weight calculation. , with static semantic density score and the new model uncertainty score Merge again This completes the closed-loop control process of "loss comparison → progress feedback → difficulty update → resampling".

[0047] Furthermore, the closed-loop control includes: if the dynamic course difficulty score of a medical sample is higher than a preset high threshold and its model uncertainty score is lower than a preset low threshold, then the dynamic course difficulty score of the medical sample is reduced by a decay ratio; if the model uncertainty score of the medical sample remains higher than a preset high confusion threshold for a certain number of rounds, then its dynamic course difficulty score is increased by an incremental step size, and the increased value does not exceed the upper limit, thereby correcting the difficulty of the medical sample. Specifically, after sampling, if the high... Sample Significant decrease ( ),but Attenuation according to the formula: ;like If the value is consistently greater than 1.5 (the model remains perplexed), then... Increase by 0.1: In other words, a difficulty score is given for the updated dynamic course. Execution of the closed-loop determination of the disclosure document: If >0.7 and If the value is less than 0.5, it indicates that the model has grasped the sample, and the value should be immediately adjusted downwards by a decay coefficient of 0.8. = 0.8 If two consecutive rounds If the value is greater than 1.5, it indicates that the model remains perplexed. Therefore, the value is increased by an increment of 0.1 and truncated to the upper limit of 1. = min( (+ 0.1, 1); The adjusted new value is written back to the medical sample attributes and directly participates in the next round of comprehensive sampling weight calculation, realizing an online difficulty correction closed loop without manual intervention.

[0048] Furthermore, the learning progress feedback is used to dynamically adjust the weighting coefficients: when the batch average learning progress feedback is higher than 0.15, the course difficulty level is increased; when it is lower than 0.05, the course difficulty level is decreased, and the weight corresponding to the model uncertainty score is increased simultaneously, to achieve real-time matching between the difficulty of medical samples and the learning speed of the large medical model. Specifically: when a change in model learning progress is detected, the system automatically adjusts the course difficulty. ;in, This represents the number of samples in the current batch. Indicates the current training difficulty level. If (If the learning progress is fast), the difficulty level will be automatically increased; if If the learning progress is slow, the difficulty level should be reduced.

[0049] The breakthrough of this invention lies in the fact that course learning is no longer based on preset rules, but rather on the automatic generation of difficulty curves through model states. The core innovation is upgrading the course learning mechanism from predefined rules to a model-driven dynamic feedback system, forming a closed-loop synergy with dynamic sparse sampling. When the model's learning efficiency decreases on a certain type of sample, the system automatically increases its difficulty level, achieving "learning progress driving course evolution." Compared with existing technologies, this solution addresses the core problem of "course design lagging behind model capabilities," enabling the data construction process to possess true adaptive capabilities and efficiently train large models.

[0050] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, and these situations are also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0051] The present invention also provides a medical large model pre-training data construction system, the specific technical solution of which is as follows: a first calculation module, a second calculation module, a third calculation module and a control module; The first calculation module is used to acquire a medical text corpus and parse it line by line to obtain multiple medical samples; for each medical sample, its static semantic density score, model uncertainty score, and dynamic course difficulty score are calculated respectively. The second calculation module is used to fuse the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample according to the set weights to obtain the comprehensive sampling weight of the medical sample. The third calculation module is used to perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; the training batch is used to update the parameters of the large medical model and obtain the loss value of each medical sample in this round; The control module is used to generate learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample. Based on this learning progress feedback, the dynamic course difficulty score is updated to form a closed-loop control, which is used for the calculation of the sampling weight of the medical sample in the next round.

[0052] Based on the above solution, the present invention can be further improved as follows.

[0053] Furthermore, the static semantic density score of each medical sample is obtained by weighted fusion of the entity density and syntactic complexity of the medical sample; where the entity density of the medical sample is the number of entities in the sample divided by the number of sentences, and the syntactic complexity of the medical sample is the average dependency tree depth.

[0054] Furthermore, the model uncertainty score for each medical sample is obtained by calculating the entropy of the predicted probability distribution of that medical sample using the large medical model.

[0055] Furthermore, the steps for updating the dynamic course difficulty score for each medical sample include: The learning progress feedback is multiplied by a first-order coefficient λ, and then added to (1-λ) multiplied by the previous round of dynamic course difficulty score to obtain the current round of dynamic course difficulty score; where λ is set to 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively, to achieve adaptive matching between the difficulty of medical samples and the learning stage of the large medical model.

[0056] Furthermore, the weighted sparse sampling is sorted in descending order of all comprehensive sampling weights, and no more than 5% of the total medical samples are selected to form the training batch for this round, in order to reduce the I / O load of the medical corpus.

[0057] Furthermore, the specific steps for generating learning progress feedback are as follows: ; in, For the i-th medical sample in round Feedback on learning progress For the i-th medical sample, For medical samples In the round The loss value, For medical samples In the previous round The loss value.

[0058] Furthermore, closed-loop control includes: If the dynamic course difficulty score of the medical sample is higher than the preset high threshold and its model uncertainty score is lower than the preset low threshold, the dynamic course difficulty score of the medical sample will be reduced by the attenuation ratio. If the model uncertainty score of the medical sample continues to be higher than the preset high confusion threshold for a number of rounds, its dynamic course difficulty score will be increased by incremental steps, and the increased value will not exceed the upper limit, thereby correcting the difficulty of the medical sample.

[0059] It should be noted that the beneficial effects of the medical large-scale model pre-training data construction system provided in the above embodiments are the same as those of the medical large-scale model pre-training data construction method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0060] like Figure 2 As shown, an embodiment of the present invention provides a computer device 300, which includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330, which is loaded and executed by the processor 320 to enable the computer device 300 to implement any of the above-described methods. Specifically: The computer device 300 can vary considerably due to differences in configuration or performance. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. The one or more memories 310 store at least one computer program 330, which is loaded and executed by the one or more processors 320 to enable the computer device 300 to implement the medical large-scale model pre-training data construction method provided in the above embodiments. Of course, the computer device 300 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 300 may also include other components for implementing device functions, which will not be elaborated here.

[0061] An embodiment of the present invention provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described methods.

[0062] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0063] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the above-described methods for constructing pre-trained data for a large medical model.

[0064] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0065] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0066] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0067] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for constructing pre-training data for a large medical model, characterized in that, include: S1: Obtain a medical text corpus and parse it line by line to obtain multiple medical samples; For each medical sample, its static semantic density score, model uncertainty score, and dynamic course difficulty score are calculated separately. S2, based on the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample, the comprehensive sampling weight of the medical sample is obtained by fusion according to the set weight; S3, perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; use the training batch to update the parameters of the large medical model and obtain the loss value of each medical sample in this round; S4 generates learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample. The dynamic course difficulty score is updated based on this learning progress feedback to form a closed-loop control, which is used for the calculation of the sampling weight of the medical sample in the next round.

2. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The static semantic density score of each medical sample is obtained by weighted fusion of the entity density and syntactic complexity of the medical sample; where the entity density of the medical sample is the number of entities in the sample divided by the number of sentences, and the syntactic complexity of the medical sample is the average dependency tree depth.

3. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The model uncertainty score for each medical sample is obtained by calculating the entropy of the predicted probability distribution of that medical sample using the large medical model.

4. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The steps for updating the dynamic course difficulty score for each medical sample include: The learning progress feedback is multiplied by a first-order coefficient λ, and then added to (1-λ) multiplied by the previous round of dynamic course difficulty score to obtain the current round of dynamic course difficulty score; where λ is set to 0.5, 0.3, and 0.1 in the early, middle, and late stages of training, respectively, to achieve adaptive matching between the difficulty of medical samples and the learning stage of the large medical model.

5. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The weighted sparse sampling is sorted in descending order of all comprehensive sampling weights, and no more than 5% of the total medical samples are selected to form the training batch for this round, so as to reduce the I / O load of the medical corpus.

6. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The specific steps for generating learning progress feedback are as follows: ； in, For the i-th medical sample in round Feedback on learning progress For the i-th medical sample, For medical samples In the round The loss value, For medical samples In the previous round The loss value.

7. The method for constructing pre-training data for a large medical model according to claim 1, characterized in that, The closed-loop control includes: If the dynamic course difficulty score of the medical sample is higher than the preset high threshold and its model uncertainty score is lower than the preset low threshold, the dynamic course difficulty score of the medical sample will be reduced by the attenuation ratio. If the model uncertainty score of the medical sample continues to be higher than the preset high confusion threshold for a number of rounds, its dynamic course difficulty score will be increased by incremental steps, and the increased value will not exceed the upper limit, thereby correcting the difficulty of the medical sample.

8. A system for constructing pre-training data for a large medical model, characterized in that, include: The system comprises a first calculation module, a second calculation module, a third calculation module, and a control module. The first calculation module is used to acquire a medical text corpus and parse it line by line to obtain multiple medical samples; for each medical sample, its static semantic density score, model uncertainty score and dynamic course difficulty score are calculated respectively. The second calculation module is used to fuse the static semantic density score, model uncertainty score and dynamic course difficulty score of each medical sample according to a set weight to obtain the comprehensive sampling weight of the medical sample. The third calculation module is used to perform weighted sparse sampling on all medical samples according to all comprehensive sampling weights to obtain the training batch for this round; the training batch is used to update the parameters of the large medical model and obtain the loss value of each medical sample in this round; The control module is used to generate learning progress feedback based on the comparison between the current loss value and the previous loss value of each medical sample, update the dynamic course difficulty score based on the learning progress feedback, form a closed-loop control, and calculate the sampling weight of the medical sample in the next round.

9. A computer device, characterized in that, The computer device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the computer device to perform the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to perform the method as claimed in any one of claims 1 to 7.

Images