Gastroscope large model and training method based on course learning and reinforcement learning iteration
By combining cross-modal feature alignment and curriculum-based learning methods with reinforcement learning strategies to optimize algorithms, the problems of cross-modal feature fusion and professional knowledge injection in large gastroscopy models have been solved, improving diagnostic accuracy and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGYUAN ARTIFICIAL INTELLIGENCE IND TECH RES INST
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-29
AI Technical Summary
In cross-modal scenarios, the large-scale gastroscopy model suffers from a mismatch between the dimensions of image features and language features, low accuracy of feature fusion, a lack of systematic domain knowledge injection, and low efficiency in learning professional knowledge, making it difficult to meet the actual clinical diagnostic needs.
Cross-modal feature alignment is achieved by constructing feature dimension alignment units, training is conducted by dividing the learning into increasingly difficult course-like stages, and the model parameters are iteratively updated by combining a multi-dimensional reward evaluation system and reinforcement learning strategies to optimize the algorithm.
It improves the cross-modal fusion accuracy of gastroscopy image features and language features, enhances the model's learning efficiency and mastery of gastroscopy expertise, and improves the diagnostic accuracy and clinical adaptability for difficult cases.
Smart Images
Figure CN122116030A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a large-scale gastroscopy model and training method based on curriculum-based learning and reinforcement learning iteration. Background Technology
[0002] The large-scale gastroscopy model represents a revolutionary intelligent innovation in the field of digestive endoscopy. It is adapted to the entire process of gastroscopy image analysis and clinical diagnosis and treatment, while scientific training methods are the core support for its practical application. The combination of these two aspects brings a key breakthrough to the diagnosis and treatment of digestive tract diseases. This model relies on massive amounts of gastroscopy images and clinical data from multiple centers. Through pre-training and then transfer learning to adapt to clinical tasks, it solves the pain points of traditional models such as data scarcity and poor generalization, significantly improving the accuracy and efficiency of gastroscopy diagnosis, reducing missed diagnoses, and reducing the need for labeled data. This helps primary hospitals popularize accurate diagnosis, and can also automatically generate diagnosis and treatment reports, assist clinical decision-making, provide inclusive technical support for early screening and treatment of gastric cancer, and optimize the allocation of medical resources.
[0003] Currently, the training of large-scale gastroscopy models has the following shortcomings: in cross-modal scenarios, the dimensions of gastroscopy image features and language features do not match, resulting in low accuracy of feature fusion; the injection of domain knowledge lacks systematicity, and the model has low learning efficiency and weak mastery of professional knowledge; the optimization process lacks clinically oriented evaluation criteria, and its diagnostic adaptability to difficult cases is insufficient, making it difficult to meet the actual clinical diagnostic needs. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides a large-scale gastroscopy model and training method based on curriculum-based learning and reinforcement learning iteration.
[0005] According to a first aspect of the present invention, a method for training a large-scale gastroscopy model based on iterative curriculum learning and reinforcement learning is provided, comprising:
[0006] S1. Initialize the visual feature extraction sub-model and the generative language sub-model, and construct a feature dimension alignment unit between them; determine whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, map the image features to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis for cross-modal features.
[0007] S2 divides the training process into multiple course-style learning stages with increasing difficulty. Specifically: determine the current training data type. If it is image-text pair data, perform cross-modal feature fusion training, i.e., the first stage; if it is plain text industry standard data, perform domain knowledge injection training, i.e., the second stage; after the model completes basic knowledge learning, perform concept anchoring based on medical terminology data, i.e., the third stage; and use clinical instruction data to perform instruction-following training of response logic, i.e., the fourth stage.
[0008] S3, Obtain the model after course-based learning as the base; Construct a reward evaluation system containing multi-dimensional indicators; Input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; Determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients;
[0009] S4. Based on the comprehensive reward value, the model parameters are iteratively updated using a strategy optimization algorithm; it is determined whether the iteration process meets the preset convergence condition or round threshold; if not, the next round of sampling and optimization continues until the final large model for gastroscopy diagnosis is obtained.
[0010] Further, in S1, the construction of the feature dimension alignment unit includes:
[0011] Configure a network structure containing two fully connected layers as the feature dimension alignment unit;
[0012] Obtain the 768-dimensional original image feature vector extracted by the visual feature extraction sub-model;
[0013] The 768-dimensional original image feature vector is input into the feature dimension alignment unit;
[0014] Based on the preset word embedding dimension of 4096 of the generative language sub-model, the input feature vector is linearly transformed and nonlinearly activated by the feature dimension alignment unit to determine the target feature vector with an output dimension of 4096.
[0015] Determine whether the target feature vector can be concatenated with the text embedding vector of the generative language sub-model; if it can be concatenated, then the legality of the cross-modal input sequence is established.
[0016] Furthermore, S2 includes:
[0017] When training enters the first stage, the backbone parameters of the visual feature extraction sub-model and the generative language sub-model are frozen, and only the feature dimension alignment unit is set to a trainable state; the prediction error is calculated based on the cross-entropy loss function, and the alignment unit parameters are updated to achieve preliminary fusion;
[0018] When the number of training rounds reaches the preset round threshold and the second stage is entered, the visual feature extraction sub-model is kept in a frozen state; the industry standard text sequence is input into the generative language sub-model; the language sub-model is fine-tuned using the autoregressive language modeling loss function so that the model parameter distribution shifts towards the professional standards of gastroenterology.
[0019] Determine whether the training has entered the third stage; if so, input terminology data containing key points of specific disease diagnosis; simultaneously unfreeze the visual feature extraction sub-model and feature dimension alignment unit, and establish a mapping relationship between visual regions and medical concepts through a terminology matching task;
[0020] Determine whether the training has entered the fourth stage; if so, input an instruction pair consisting of clinical diagnostic instructions and standard response text; calculate the difference between the model output and the standard response based on the instruction matching loss function; determine whether the model has the ability to respond to the clinical diagnostic instructions.
[0021] Furthermore, in S3, the process for determining the comprehensive reward value is as follows:
[0022] The reward components are defined, including diagnostic accuracy rewards, structural compliance rewards, and language quality rewards.
[0023] The average score of the model's output by multiple senior physicians is used as a reward for the diagnostic accuracy.
[0024] The report module output by the model is determined to fully cover the preset schema structure, and a compliance reward for the structure is generated accordingly.
[0025] The fluency of the model's output text and the standardization of its use of technical terms are evaluated, and a language quality reward is generated accordingly.
[0026] The comprehensive reward value is calculated based on the diagnostic accuracy reward, structural compliance reward, and language quality reward.
[0027] Furthermore, in S4, the iterative update of the model parameters includes a first round of reinforcement learning, specifically:
[0028] Input the difficult case samples into the model in batches;
[0029] Based on the comprehensive reward value, the policy gradient is calculated using a group relative policy optimization algorithm;
[0030] A KL divergence penalty coefficient is introduced; the KL divergence value between the current policy distribution and the old policy distribution is determined; if the KL divergence value is greater than a preset divergence threshold, the comprehensive reward value is penalized and corrected using the KL divergence penalty coefficient to prevent catastrophic forgetting or policy collapse during the model update process.
[0031] Furthermore, in S4, the iterative update of the model parameters includes a second round of reinforcement learning, specifically:
[0032] After the first round of reinforcement learning is completed, the hybrid sampling mechanism is activated;
[0033] Reinforcement learning samples and supervised fine-tuning samples are extracted according to a preset ratio;
[0034] The reinforcement learning samples are interleaved and mixed with the supervised fine-tuning samples;
[0035] Determine the type of the sample being processed; if it is a reinforcement learning sample, continue to apply reward-based optimization updates; if it is a supervised fine-tuning sample, apply supervised loss updates based on the standard answer.
[0036] Furthermore, the method also includes:
[0037] Input the image to be tested into the trained model;
[0038] The test task type is determined, and the test task type includes modality recognition, anatomical localization, and disease diagnosis;
[0039] Accuracy metrics for each test task were calculated based on the OmniMedVQA-GI benchmark dataset.
[0040] If the accuracy of each test task is higher than the preset pass line, the model will be deployed to the inference server.
[0041] During the inference phase, it is determined whether the input data is a single endoscopic image or a sequence of multiple images. If it is a single image, feature extraction is performed directly. If it is an image sequence, temporal features are extracted and aggregated, and finally a structured diagnostic report and the corresponding disease probability distribution are output.
[0042] Furthermore, the parameter settings of the feature dimension alignment unit must satisfy the following:
[0043] The feature dimension alignment unit is determined to be a lightweight architecture;
[0044] Its number of parameters is limited to be much smaller than that of the generative language sub-model;
[0045] This ensures that during the cross-modal pre-training phase of S2, when only the feature dimension alignment unit is trained, the projection from the visual feature space to the text feature space can be achieved quickly without destroying the original semantic representation capability of the language sub-model.
[0046] Furthermore, the logic for setting the number of training rounds is as follows:
[0047] The first round of reinforcement learning is preset to have 10 training rounds, and the second round of reinforcement learning is preset to have 8 training rounds.
[0048] At the end of each training round, monitor the change curve of the overall reward value;
[0049] Determine whether the change curve tends to flatten; if the curve has flattened before reaching the preset number of rounds, terminate the current round of training in advance to avoid the model overfitting the training data and preserve the model's generalization reasoning ability.
[0050] According to a second aspect of the present invention, a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration is provided, comprising:
[0051] The alignment module is used to initialize the visual feature extraction sub-model and the generative language sub-model, and to build a feature dimension alignment unit between them; it determines whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, the image features are mapped to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis for cross-modal features.
[0052] The hierarchical module is used to divide the training process into multiple course-style learning stages with increasing difficulty. Specifically, it determines the current training data type. If it is image-text pair data, cross-modal feature fusion training is performed, which is the first stage. If it is plain text industry standard data, domain knowledge injection training is performed, which is the second stage. After the model completes basic knowledge learning, concept anchoring is performed based on medical terminology data, which is the third stage. The fourth stage is instruction-following training of response logic using clinical instruction data.
[0053] The evaluation module is used to obtain the model after course-based learning as a base; construct a reward evaluation system containing multi-dimensional indicators; input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients.
[0054] The optimization module is used to iteratively update the model parameters using a strategy optimization algorithm based on the comprehensive reward value; determine whether the iteration process meets the preset convergence condition or round threshold; if not, continue the next round of sampling and optimization until the final large model for gastroscopy diagnosis is obtained.
[0055] This invention improves the accuracy of cross-modal feature fusion by constructing feature dimension alignment units and completing cross-modal feature dimension mapping to achieve cross-modal alignment of gastroscopy image features and language features. It also improves the model's learning efficiency and mastery of gastroscopy expertise by dividing learning into multi-stage courses with increasing difficulty and conducting targeted training according to type. Finally, it achieves precise model optimization based on clinical diagnostic results by constructing a multi-dimensional reward evaluation system and combining it with reinforcement learning strategies to iteratively update model parameters, thereby improving the diagnostic accuracy and clinical adaptability of the large-scale gastroscopy model for difficult cases.
[0056] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0057] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0058] Figure 1 A flowchart is shown for a gastroscopy large model training method based on curriculum learning and reinforcement learning iteration according to an embodiment of the present invention.
[0059] Figure 2 A block diagram of a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration according to an embodiment of the present invention is shown. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0061] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0062] Figure 1A flowchart of a gastroscopy large model training method based on curriculum-based learning and reinforcement learning iteration according to an embodiment of the present invention is shown. The method includes:
[0063] S1. Initialize the visual feature extraction sub-model and the generative language sub-model, and construct a feature dimension alignment unit between them; determine whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, map the image features to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis for cross-modal features.
[0064] S2 divides the training process into multiple course-style learning stages with increasing difficulty. Specifically: determine the current training data type. If it is image-text pair data, perform cross-modal feature fusion training, i.e., the first stage; if it is plain text industry standard data, perform domain knowledge injection training, i.e., the second stage; after the model completes basic knowledge learning, perform concept anchoring based on medical terminology data, i.e., the third stage; and use clinical instruction data to perform instruction-following training of response logic, i.e., the fourth stage.
[0065] S3, Obtain the model after course-based learning as the base; Construct a reward evaluation system containing multi-dimensional indicators; Input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; Determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients;
[0066] S4. Based on the comprehensive reward value, the model parameters are iteratively updated using a strategy optimization algorithm; it is determined whether the iteration process meets the preset convergence condition or round threshold; if not, the next round of sampling and optimization continues until the final large model for gastroscopy diagnosis is obtained.
[0067] In some embodiments, in S1, constructing the feature dimension alignment unit includes: configuring a network structure containing two fully connected layers as the feature dimension alignment unit; obtaining the 768-dimensional original image feature vector extracted by the visual feature extraction sub-model; inputting the 768-dimensional original image feature vector into the feature dimension alignment unit; based on the preset word embedding dimension of 4096 of the generative language sub-model, performing linear transformation and nonlinear activation on the input feature vector through the feature dimension alignment unit to determine a target feature vector with an output dimension of 4096; determining whether the target feature vector can be concatenated with the text embedding vector of the generative language sub-model; if concatenation is possible, the legality of the cross-modal input sequence is established. According to the embodiments of the present invention, through a two-layer fully connected layer structure, the 768-dimensional image feature vector is mapped to a 4096-dimensional target vector, matching the word embedding dimension of the language sub-model, realizing cross-modal feature dimension alignment, verifying the feasibility of vector concatenation, establishing the legality of the cross-modal input sequence, and laying the foundation for subsequent cross-modal fusion training.
[0068] For example, a network structure with two fully connected layers is configured as the feature dimension alignment unit. The first fully connected layer has 1024 neurons and uses the ReLU activation function, while the second fully connected layer has 4096 neurons and uses the Sigmoid activation function. A 768-dimensional original image feature vector (each element in the vector ranges from -1.0 to 1.0) is obtained from the visual feature extraction sub-model. This 768-dimensional original image feature vector is then input into the feature dimension alignment unit, linearly transformed by the first fully connected layer (weight matrix dimension is 768×1024, bias vector dimension is 1×1024), and then subjected to ReLU. After activation, a 1024-dimensional intermediate feature vector is obtained. After a second fully connected layer linear transformation (weight matrix dimension is 1024×4096, bias vector dimension is 1×4096) and sigmoid activation, a 4096-dimensional target feature vector is output (each element in the vector takes values in the range [0.0, 1.0]). The text embedding vector of the generative language sub-model is 4096-dimensional, and each element takes values in the range [0.0, 1.0]. The target feature vector and the text embedding vector are concatenated in dimensional order to obtain an 8192-dimensional fusion vector. There is no dimensional conflict in the concatenation process, thus establishing the legality of the cross-modal input sequence.
[0069] In some embodiments, S2 includes: when training enters the first stage, freezing the backbone parameters of the visual feature extraction sub-model and the generative language sub-model, and only setting the feature dimension alignment unit to a trainable state; calculating the prediction error based on the cross-entropy loss function, and updating the alignment unit parameters to achieve preliminary fusion; when it is determined that the number of training rounds has reached a preset round threshold and the second stage has been entered, keeping the visual feature extraction sub-model in a frozen state; inputting an industry standard text sequence into the generative language sub-model; fine-tuning the language sub-model using an autoregressive language modeling loss function to shift the model parameter distribution towards the gastroenterology professional standard; determining whether training has entered the third stage; if so, inputting terminology data containing specific disease diagnosis points; simultaneously unfreezing the visual feature extraction sub-model and the feature dimension alignment unit, and establishing a mapping relationship between visual regions and medical concepts through a terminology matching task; determining whether training has entered the fourth stage; if so, inputting an instruction pair consisting of clinical diagnostic instructions and standard response text; calculating the difference between the model output and the standard response based on the instruction matching loss function; and determining whether the model has the ability to respond to the clinical diagnostic instructions. According to an embodiment of the present invention, through phased training, the backbone parameters are first frozen to optimize the alignment unit to achieve preliminary cross-modal fusion, then the language sub-model is fine-tuned to inject professional specifications, and then the relevant modules are unfrozen to establish a mapping between visual and medical concepts. Finally, the command response capability is strengthened, which not only ensures the basic performance of the model, but also accurately injects domain knowledge, thereby improving the diagnostic professionalism and command adaptability.
[0070] For example, in the first stage: 5000 sets of gastroscopy image-pathology report text pairs were selected as training data. The 12-layer convolutional backbone parameters of the visual feature extraction sub-model and the 24-layer Transformer backbone parameters of the generative language sub-model were frozen, and only the two fully connected layers of the feature dimension alignment unit were retained for training. The cross-entropy loss function was adopted, the learning rate was set to 1e-4, the batch size was 32, and the number of training rounds was set to 80 rounds. The loss value was calculated after each round of iteration. When the loss value fluctuation was less than 1e-5 for 10 consecutive rounds, the first stage of training was completed. At this time, the weight matrix of the first fully connected layer of the alignment unit was updated to 768×1024 dimensions and the bias vector was updated to 1×1024 dimensions. The weight matrix of the second fully connected layer was updated to 1024×4096 dimensions and the bias vector was updated to 1×4096 dimensions.
[0071] Phase Two: When the training rounds reach 80, the second phase begins. The 12-layer convolutional backbone parameters of the visual feature extraction sub-model are kept frozen. 10,000 standard text sequences from the gastroenterology industry (each text is 512 characters long) are input. An autoregressive language modeling loss function is used, with a learning rate of 5e-5, a batch size of 16, and 60 training rounds. After each iteration, the deviation between the model parameters and the industry standard data distribution is calculated. When the deviation drops below 0.02, the second phase of fine-tuning is completed.
[0072] Phase 3: When the training rounds reach 140, the third phase begins. Input is medical terminology data containing diagnostic points for 200 specific diseases (50 terms for each disease, each term being 64 characters long). The parameters of the 12-layer convolutional backbone of the visual feature extraction sub-model and the two fully connected layers of the feature dimension alignment unit are unfrozen. The learning rate is set to 3e-5, the batch size to 24, and training is performed through a terminology matching task for 40 rounds. In each round, the similarity between the visual region features and the medical term vectors is calculated. When the average similarity reaches 0.85 or higher, the third phase of training is completed.
[0073] Phase 4: Upon reaching 180 training epochs, the fourth phase begins. 8000 sets of instruction pairs consisting of clinical diagnostic commands and standard response text are input (each command is 128 characters long, and the corresponding standard response is 256 characters long). An instruction matching loss function is used, with a learning rate of 2e-5, a batch size of 32, and 50 training epochs. In each epoch, the character matching error between the model's output response and the standard response is calculated. When the error rate drops below 0.03, the model is deemed capable of responding to clinical diagnostic commands, completing the entire training process.
[0074] In some embodiments, in S3, the process of determining the comprehensive reward value is as follows: setting reward components, which include diagnostic accuracy reward, structural compliance reward, and language quality reward; obtaining the average score of multiple senior physicians on the model output results as the diagnostic accuracy reward; determining whether the report module output by the model fully covers the preset schema structure, and generating the structural compliance reward accordingly; evaluating the fluency of the model output text and the standardization of the use of professional terminology, and generating the language quality reward accordingly; calculating the comprehensive reward value based on the diagnostic accuracy reward, structural compliance reward, and language quality reward. According to the embodiments of the present invention, by using three reward components—diagnostic accuracy, structural compliance, and language quality—combined with physician scores and preset standards to evaluate the model output, the reliability of diagnostic results and the standardization of report structure are ensured, while also ensuring professional and fluent text. This comprehensively quantifies model performance, provides accurate basis for subsequent parameter iteration and optimization, and improves the practicality and professionalism of the model.
[0075] For example, the weighting coefficients for diagnostic accuracy reward, structural compliance reward, and language quality reward are set to 0.5, 0.3, and 0.2, respectively, with each reward component having a maximum score of 10 points. The comprehensive reward value = diagnostic accuracy reward × 0.5 + structural compliance reward × 0.3 + language quality reward × 0.2. Ten senior physicians were selected to independently score the diagnostic results output by the model for a difficult case sample. The scores were 8.5, 8.8, 9.0, 8.7, 8.9, 9.1, 8.6, 8.8, 9.2, and 8.7, respectively. The average score was calculated as (8.5 + 8.8 + 9.0 + 8.7 + 8.9 + 9.1 + 8.6 + 8.7). (8.8 + 9.2 + 8.7) ÷ 10 = 8.83 points, which is the diagnostic accuracy bonus; the preset report schema structure includes four core modules: examination site, lesion description, preliminary diagnosis, and recommended measures. The model output report is judged to fully cover the above four modules without any missing items, and a structural compliance bonus of 10 points is given; the model output text is evaluated using a text fluency scoring tool and a professional terminology standardization checklist. The fluency score is 9.2 points, the professional terminology is used without errors or omissions, and the standardization score is 9.5 points. The average of the two is (9.2 + 9.5) ÷ 2 = 9.35 points, which is the language quality bonus; the final comprehensive bonus value = 8.83 × 0.5 + 10 × 0.3 + 9.35 × 0.2 = 4.415 + 3 + 1.87 = 9.285 points.
[0076] In some embodiments, in S4, the iterative update of model parameters includes a first round of reinforcement learning, specifically: batch inputting the difficult case samples into the model; calculating the policy gradient using a group relative policy optimization algorithm based on the comprehensive reward value; introducing a KL divergence penalty coefficient; determining the KL divergence value between the current policy distribution and the old policy distribution; if the KL divergence value is greater than a preset divergence threshold, then using the KL divergence penalty coefficient to penalize and correct the comprehensive reward value, thereby preventing catastrophic forgetting or policy collapse during model updates. According to embodiments of the present invention, by batch inputting difficult case samples, calculating the policy gradient using a group relative policy optimization algorithm, and introducing a KL divergence penalty mechanism, the comprehensive reward value can be effectively corrected, avoiding catastrophic forgetting or policy collapse during model updates, and ensuring the stability and effectiveness of model parameter iteration.
[0077] For example, 1000 difficult gastroscopy cases are selected (covering 20 types of complex diseases such as early gastric cancer, complex polyps, and rare inflammation; each case includes complete gastroscopy images and clinical history information). These cases are then input into a base model after course-based learning in batches of 64. The average comprehensive reward value for this batch is known to be 9.285. A group relative strategy optimization algorithm is used, with a strategy gradient step size of 0.01. By calculating the deviation between the output strategy of each sample and the optimal strategy of the group, the average strategy gradient value is obtained as 0.032. A KL divergence penalty coefficient λ = 0.15 is introduced, with a preset divergence threshold of 0.08. The KL divergence value between the current strategy distribution and the old strategy distribution is calculated to be 0.11, which is greater than the preset divergence threshold of 0.08. The corrected comprehensive reward value = original comprehensive reward value × (1 - λ × (KL divergence value - preset divergence threshold)), i.e., 9.285 × (1 - 0.15 × (0.11 - ... 0.08))=9.285×(1 - 0.15×0.03)=9.285×0.9955=9.243 points; Based on the corrected comprehensive reward value of 9.243 points, the weights of the convolutional layer of the visual feature extraction sub-model, the parameters of the Transformer layer of the generative language sub-model, and the weights of the fully connected layer of the feature dimension alignment unit are updated. The update magnitude is adjusted according to the policy gradient direction to complete the parameter iteration of the first round of reinforcement learning.
[0078] In some embodiments, in S4, the iterative update of the model parameters includes a second round of reinforcement learning, specifically: after the first round of reinforcement learning ends, a hybrid sampling mechanism is initiated; reinforcement learning samples and supervised fine-tuning samples are extracted according to a preset ratio; the reinforcement learning samples and supervised fine-tuning samples are interleaved and mixed; the type of the currently processed sample is determined; if it is a reinforcement learning sample, reward-based optimization updates are continued; if it is a supervised fine-tuning sample, supervised loss updates based on the standard answer are applied. According to embodiments of the present invention, by interleaving two types of samples through a hybrid sampling mechanism, corresponding optimization methods are applied to different samples, which not only continuously optimizes model performance with the help of the reward mechanism, but also consolidates standard knowledge with supervised fine-tuning, effectively avoiding model overfitting and performance degradation, and improving generalization ability and diagnostic reliability.
[0079] For example, after the first round of reinforcement learning, a mixed sampling mechanism is activated, setting the sampling ratio of reinforcement learning samples to supervised fine-tuning samples to 7:3. 700 samples are drawn from the reinforcement learning sample library of 1000 difficult gastroscopy cases, and 300 samples are drawn from the supervised fine-tuning sample library consisting of 8000 pairs of gastroscopy image-text pairs, industry standard texts, medical terminology, and clinical instruction pairs. The 700 reinforcement learning samples and 300 supervised fine-tuning samples are randomly interleaved and mixed to form a 1000-sample mixed training set. The sample order is generated by a random shuffling algorithm and has no fixed arrangement. Each sample in the mixed training sample set is read sequentially, and its type is determined by its label: if the sample label is reinforcement learning, it is determined to be a reinforcement learning sample, and the comprehensive reward value calculation rule (diagnosis) is followed. Accuracy is rewarded with a weight of 0.5, structural compliance with a weight of 0.3, and language quality with a weight of 0.2. Based on the corrected comprehensive reward value corresponding to the sample, the group relative policy optimization algorithm (policy gradient step size 0.01, KL divergence penalty coefficient 0.15) is used to optimize and update the model parameters. If the sample label is a supervised fine-tuning class, it is determined as a supervised fine-tuning sample. For image-text pairs, the cross-entropy loss function (learning rate 1e-4) is used; for industry standard text samples, the autoregressive language modeling loss function (learning rate 5e-5) is used; for medical terminology samples, the terminology matching loss function (learning rate 3e-5) is used; and for clinical instruction pairs, the instruction matching loss function (learning rate 2e-5) is used. The supervised loss is calculated based on the standard answer corresponding to the sample, and the model parameters are updated accordingly.
[0080] The mixed training sample set is fed into the training at a batch size of 64, and a total of 40 training rounds are conducted. After each round of iteration, the overall loss value of the model is calculated. When the loss value fluctuates less than 1e-5 for 8 consecutive rounds, the parameter iteration update of the second round of reinforcement learning is completed.
[0081] In some embodiments, the method further includes: inputting the image to be tested into the trained model; determining the test task type, which includes modality recognition, anatomical localization, and disease diagnosis; calculating the accuracy index of each test task based on the OmniMedVQA-GI benchmark set; if the accuracy index of each test task is higher than a preset pass line, then deploying the model to the inference server; during the inference stage, determining whether the input data is a single endoscopic image or a multi-frame image sequence; if it is a single image, directly performing feature extraction; if it is an image sequence, extracting and aggregating temporal features, and finally outputting a structured diagnostic report and the corresponding disease probability distribution. According to embodiments of the present invention, by completing modality recognition, anatomical localization, and disease diagnosis test tasks, ensuring accuracy with the OmniMedVQA-GI benchmark set, and deploying the inference service after achieving the target, it can adapt to single / multi-frame endoscopic image input, efficiently extract features, output a structured diagnostic report and disease probability distribution, and improve the efficiency and accuracy of gastroscopy diagnosis.
[0082] In some embodiments, the parameter settings of the feature dimension alignment unit need to satisfy the following: the feature dimension alignment unit is determined to be a lightweight architecture; its parameter count is limited to be much smaller than the parameter count of the generative language sub-model; to ensure that in the cross-modal pre-training stage of S2, when only the feature dimension alignment unit is trained, the projection from the visual feature space to the text feature space can be quickly achieved without destroying the original semantic representation capability of the language sub-model. According to the embodiments of the present invention, by making the feature dimension alignment unit a lightweight architecture with a parameter count much smaller than the generative language sub-model, when only this unit is trained in the cross-modal pre-training stage of S2, the projection from the visual and text feature spaces can be quickly achieved, while avoiding the destruction of the original semantic representation capability of the language sub-model, thus ensuring training efficiency and model performance.
[0083] In some embodiments, the logic for setting the number of training rounds is as follows: the first round of reinforcement learning is preset to 10 rounds, and the second round of reinforcement learning is preset to 8 rounds; at the end of each round of training, the change curve of the comprehensive reward value is monitored; it is determined whether the change curve tends to flatten; if the curve has flattened before reaching the preset number of rounds, the current round of training is terminated in advance to avoid overfitting the model to the training data and to preserve the model's generalization reasoning ability. According to the embodiments of the present invention, by setting the number of training rounds, the number of two rounds of reinforcement learning is preset, and combined with the monitoring of the change curve of the comprehensive reward value, training can be terminated when the curve flattens in advance, which not only ensures that the model learns fully, but also avoids overfitting, effectively preserves the model's generalization reasoning ability, and improves training efficiency and model practicality.
[0084] For example, the first round of reinforcement learning training is preset to 10 rounds, and the second round to 8 rounds. The criterion for judging a flattened curve in the overall reward value is that the absolute value of the difference between the overall reward values in three consecutive rounds is less than 0.02. During the first round of reinforcement learning, the overall reward value is 8.2 in round 1, 8.5 in round 2, 8.6 in round 3, 8.65 in round 4, 8.66 in round 5, 8.67 in round 6, and 8.67 in round 7. The absolute values of the differences in the overall reward values in rounds 5, 6, and 7 are detected to be 0.01, 0.01, and 0.00 respectively, all less than 0.02, indicating that the curve is flattening. The first round of reinforcement learning was terminated prematurely because the preset number of 10 training rounds had not been reached, with an actual training round of 7 rounds. During the second round of reinforcement learning, the comprehensive reward value was 9.0 points in the first round, 9.1 points in the second round, 9.15 points in the third round, 9.18 points in the fourth round, 9.19 points in the fifth round, 9.2 points in the sixth round, 9.2 points in the seventh round, and 9.2 points in the eighth round. When training reached the sixth, seventh, and eighth rounds, the absolute values of the differences in comprehensive reward values were 0.01, 0.00, and 0.00, respectively, which met the smoothness judgment criteria. However, since the preset number of 8 training rounds had already been reached, the second round of reinforcement learning was terminated according to the preset number of rounds, with an actual training round of 8 rounds.
[0085] Figure 2 A block diagram of a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration according to an embodiment of the present invention is shown, including:
[0086] Alignment module 201 is used to initialize the visual feature extraction sub-model and the generative language sub-model, and construct a feature dimension alignment unit between them; determine whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, the image features are mapped to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis of cross-modal features;
[0087] The hierarchical module 202 is used to divide the training process into multiple course-style learning stages with increasing difficulty. Specifically, it determines the current training data type. If it is image-text pair data, cross-modal feature fusion training is performed, i.e., the first stage. If it is plain text industry standard data, domain knowledge injection training is performed, i.e., the second stage. After the model completes basic knowledge learning, concept anchoring is performed based on medical terminology data, i.e., the third stage. The fourth stage is instruction-following training of response logic using clinical instruction data.
[0088] The evaluation module 203 is used to obtain the model after course-based learning as a base; construct a reward evaluation system containing multi-dimensional indicators; input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; and determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients.
[0089] The optimization module 204 is used to iteratively update the model parameters using a strategy optimization algorithm based on the comprehensive reward value; determine whether the iteration process meets the preset convergence condition or round threshold; if not, continue the next round of sampling and optimization until the final large model for gastroscopy diagnosis is obtained.
[0090] It should be understood that the various processes described above can be used to rearrange, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for training a large-scale gastroscopy model based on iterative curriculum learning and reinforcement learning, characterized in that, include: S1. Initialize the visual feature extraction sub-model and the generative language sub-model, and construct a feature dimension alignment unit between them; determine whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, map the image features to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis for cross-modal features. S2 divides the training process into multiple course-style learning stages with increasing difficulty. Specifically: determine the current training data type. If it is image-text pair data, perform cross-modal feature fusion training, i.e., the first stage; if it is plain text industry standard data, perform domain knowledge injection training, i.e., the second stage; after the model completes basic knowledge learning, perform concept anchoring based on medical terminology data, i.e., the third stage; and use clinical instruction data to perform instruction-following training of response logic, i.e., the fourth stage. S3, Obtain the model after course-based learning as the base; Construct a reward evaluation system containing multi-dimensional indicators; Input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; Determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients; S4, based on the comprehensive reward value, the model parameters are iteratively updated using a strategy optimization algorithm; Determine whether the iterative process meets the preset convergence condition or round threshold; if not, continue the next round of sampling and optimization until the final large-scale gastroscopy diagnostic model is obtained.
2. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 1, characterized in that, In S1, the construction of the feature dimension alignment unit includes: Configure a network structure containing two fully connected layers as the feature dimension alignment unit; Obtain the 768-dimensional original image feature vector extracted by the visual feature extraction sub-model; The 768-dimensional original image feature vector is input into the feature dimension alignment unit; Based on the preset word embedding dimension of 4096 of the generative language sub-model, the input feature vector is linearly transformed and nonlinearly activated by the feature dimension alignment unit to determine the target feature vector with an output dimension of 4096. Determine whether the target feature vector can be concatenated with the text embedding vector of the generative language sub-model; if it can be concatenated, then the legality of the cross-modal input sequence is established.
3. The method for training a large gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 2, characterized in that, S2 include: When training enters the first stage, the backbone parameters of the visual feature extraction sub-model and the generative language sub-model are frozen, and only the feature dimension alignment unit is set to a trainable state; the prediction error is calculated based on the cross-entropy loss function, and the alignment unit parameters are updated to achieve preliminary fusion; When the number of training rounds reaches the preset round threshold and the second stage is entered, the visual feature extraction sub-model is kept in a frozen state; the industry standard text sequence is input into the generative language sub-model; the language sub-model is fine-tuned using the autoregressive language modeling loss function so that the model parameter distribution shifts towards the professional standards of gastroenterology. Determine whether the training has entered the third stage; if so, input terminology data containing key points of specific disease diagnosis; simultaneously unfreeze the visual feature extraction sub-model and feature dimension alignment unit, and establish a mapping relationship between visual regions and medical concepts through a terminology matching task; Determine whether the training has entered the fourth stage; if so, input an instruction pair consisting of clinical diagnostic instructions and standard response text; calculate the difference between the model output and the standard response based on the instruction matching loss function; determine whether the model has the ability to respond to the clinical diagnostic instructions.
4. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 3, characterized in that, In S3, the process for determining the comprehensive reward value is as follows: The reward components are defined, including diagnostic accuracy rewards, structural compliance rewards, and language quality rewards. The average score of the model's output by multiple senior physicians is used as a reward for the diagnostic accuracy. The report module output by the model is determined to fully cover the preset schema structure, and a compliance reward for the structure is generated accordingly. The fluency of the model's output text and the standardization of its use of technical terms are evaluated, and a language quality reward is generated accordingly. The comprehensive reward value is calculated based on the diagnostic accuracy reward, structural compliance reward, and language quality reward.
5. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 4, characterized in that, In S4, the iterative update of the model parameters includes the first round of reinforcement learning, specifically: Input the difficult case samples into the model in batches; Based on the comprehensive reward value, the policy gradient is calculated using a group relative policy optimization algorithm; A KL divergence penalty coefficient is introduced; the KL divergence value between the current policy distribution and the old policy distribution is determined; if the KL divergence value is greater than a preset divergence threshold, the comprehensive reward value is penalized and corrected using the KL divergence penalty coefficient to prevent catastrophic forgetting or policy collapse during the model update process.
6. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 5, characterized in that, In S4, the iterative update of the model parameters includes a second round of reinforcement learning, specifically: After the first round of reinforcement learning is completed, the hybrid sampling mechanism is activated; Reinforcement learning samples and supervised fine-tuning samples are extracted according to a preset ratio; The reinforcement learning samples are interleaved and mixed with the supervised fine-tuning samples; Determine the type of sample being processed; If the sample is for reinforcement learning, continue to apply reward-based optimization updates; if the sample is for supervised fine-tuning, apply supervised loss updates based on the standard answer.
7. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 6, characterized in that, The method further includes: Input the image to be tested into the trained model; The test task type is determined, and the test task type includes modality recognition, anatomical localization, and disease diagnosis; Accuracy metrics for each test task were calculated based on the OmniMedVQA-GI benchmark dataset. If the accuracy of each test task is higher than the preset pass line, the model will be deployed to the inference server. During the inference phase, it is determined whether the input data is a single endoscopic image or a sequence of multiple images. If it is a single image, feature extraction is performed directly. If it is an image sequence, temporal features are extracted and aggregated, and finally a structured diagnostic report and the corresponding disease probability distribution are output.
8. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 7, characterized in that, The parameter settings of the feature dimension alignment unit must meet the following requirements: The feature dimension alignment unit is determined to be a lightweight architecture; Its number of parameters is limited to be much smaller than that of the generative language sub-model; This ensures that during the cross-modal pre-training phase of S2, when only the feature dimension alignment unit is trained, the projection from the visual feature space to the text feature space can be achieved quickly without destroying the original semantic representation capability of the language sub-model.
9. The method for training a large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration as described in claim 8, characterized in that, The logic for setting the number of training rounds is as follows: The first round of reinforcement learning is preset to have 10 training rounds, and the second round of reinforcement learning is preset to have 8 training rounds. At the end of each training round, monitor the change curve of the overall reward value; Determine whether the change curve tends to flatten out; If the curve flattens out before the preset number of rounds is reached, the current round of training is terminated early to avoid overfitting the model to the training data and to preserve the model's generalization and inference capabilities.
10. A large-scale gastroscopy model based on curriculum-based learning and reinforcement learning iteration, characterized in that, include: The alignment module is used to initialize the visual feature extraction sub-model and the generative language sub-model, and to build a feature dimension alignment unit between them; it determines whether the image feature dimension output by the visual feature extraction sub-model is consistent with the word embedding dimension of the generative language sub-model; if they are inconsistent, the image features are mapped to the vector space of the word embedding dimension through the feature dimension alignment unit to establish the alignment basis for cross-modal features. The hierarchical module is used to divide the training process into multiple course-style learning stages with increasing difficulty. Specifically, it determines the current training data type. If it is image-text pair data, cross-modal feature fusion training is performed, which is the first stage. If it is plain text industry standard data, domain knowledge injection training is performed, which is the second stage. After the model completes basic knowledge learning, concept anchoring is performed based on medical terminology data, which is the third stage. The fourth stage is instruction-following training of response logic using clinical instruction data. The evaluation module is used to obtain the model after course-based learning as a base; construct a reward evaluation system containing multi-dimensional indicators; input difficult case samples into the model to generate diagnostic results, and calculate the scores of the diagnostic results under each evaluation indicator; determine the comprehensive reward value based on the scores corresponding to each evaluation indicator and their corresponding weight coefficients. The optimization module is used to iteratively update the model parameters based on the comprehensive reward value using a strategy optimization algorithm; Determine whether the iterative process meets the preset convergence condition or round threshold; if not, continue the next round of sampling and optimization until the final large-scale gastroscopy diagnostic model is obtained.