Training method and device for image pre-training model in medical field

Abstract

The embodiment of the invention relates to a training method and device for an image pre-training model in the medical field. The method comprises the following steps: selecting a visual pre-training model as the image pre-training model; setting an oscillogram category set, a waveform-disease category set, an image category set and an anatomical structure category set; selecting a four-class downstream task head model to be in butt joint with the image pre-training model to obtain a five-class training framework; constructing a first data set according to the oscillogram type set and the waveform-disease type set through a big data acquisition mode, and constructing a second data set according to the image type set and the anatomical structure type set; carrying out five-step training according to the two data sets and the five types of training frames in a step-by-step progressive training mode; and determining that the optimization training of the image pre-training model in the specific field of the medical field is finished when the training in the fifth step is finished. According to the invention, the training difficulty of the image pre-training model can be reduced, and the training efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a training method and apparatus for image pre-training models in the medical field. Background Art

[0002] In the field of medical artificial intelligence (AI), single-modal (2D medical waveform or 3D medical image) visual models are typically used for task processing. Each single-modal visual model usually only supports one specific type of waveform or image. Common specific waveform types for 2D medical waveforms include electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG). Common specific image types for 3D medical images include computed tomography (CT), magnetic resonance imaging (MRI), 3D ultrasound imaging (3D US), and positron emission tomography (PET).

[0003] Conventional single-modal task processing mechanisms, unable to provide a shared encoding platform compatible with both 2D / 3D images and supporting multiple waveform and image types, require customizing, training, and maintaining a single-modal task model for each 2D / 3D task. This not only increases model development time and redundant development workload but also raises model maintenance costs. However, if a shared encoding platform could be provided using an image pre-trained model compatible with both 2D / 3D images and supporting multiple waveform and image types, then each new 2D / 3D task requirement would only require designing a simple downstream task header to interface with the image pre-trained model. Training would only require major adjustments to the downstream task header based on the new task's output requirements, and minor adaptation tweaks to the image pre-trained model. This would not only shorten the development cycle and reduce redundant development workload but also improve development flexibility and lower model maintenance costs.

[0004] Currently, many mature pre-trained Large Vision Models (LVMs) are well-compatible with 2D / 3D images, such as MultiResViT, UniFormer, Hybrid CNN-Transformer, nnU-Net, Uni3D, and Uni-MAE. However, most of these pre-trained LVMs have not undergone Specific Domain Tuning (SDT) training for the medical field. In other words, directly using these pre-trained LVMs as image pre-training models for the medical field cannot achieve the expected results of a shared encoding platform. Therefore, how to perform SDT training on image pre-training models for the medical field is the technical problem that this invention aims to solve. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a training method, apparatus, electronic device, and computer-readable storage medium for image pre-training models in the medical field. This invention first selects a type of pre-trained LVMs that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; it then configures waveform type sets for 2D waveforms supported by the visual pre-training model in the medical field, and sets disease type subsets for each type of waveform to obtain a corresponding full waveform-disease type set; and it configures corresponding image type sets and anatomical structure type sets for 3D images supported by the visual pre-training model in the medical field. Next, it selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head) and interfaces them with the image pre-training model to obtain five training frameworks; it then constructs a first dataset based on the waveform type set and waveform-disease type set, and a second dataset based on the image type set and anatomical structure type set through big data collection; it then performs five steps of training based on the two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it confirms that the targeted training of the image pre-training model in the medical field is complete. Through this invention, on the one hand, an image pre-trained model that is compatible with both 2D and 3D images and supports multiple waveforms and image types can be obtained through SDT training in the medical field as a shared encoding platform; on the other hand, the training difficulty can be reduced and the training efficiency improved through a progressive training method; furthermore, it can also simultaneously train four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head), thereby improving the economic benefits of SDT training.

[0006] To achieve the above objectives, a first aspect of the present invention provides a training method for an image pre-training model in the medical field, the method comprising:

[0007] Choose a visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-training model;

[0008] The 2D waveforms and 3D images in the medical field are used as the 2D and 3D images supported by the visual pre-training model; a corresponding image type set is configured for the 2D waveforms, denoted as the corresponding waveform type set; a corresponding diagnostic disease type set is set for each type of waveform in the waveform type set, denoted as the corresponding first disease type set, and all the obtained first disease type sets constitute the corresponding waveform-disease type set; a corresponding image type set and anatomical structure type set are configured for the 3D images, denoted as the corresponding image type set and anatomical structure set;

[0009] Four types of downstream task head models are selected and connected to the image pre-training model through separate and collective connection methods to obtain five corresponding training frameworks; the four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0010] A first dataset is constructed based on the waveform type set and the waveform-disease type set using big data acquisition methods; and a second dataset is constructed based on the image type set and the anatomical structure type set using big data acquisition methods.

[0011] The training is conducted in five steps using a progressive approach, based on two datasets and the five training frameworks. At the end of the fifth step, the pre-trained image model is confirmed to have completed its domain-specific optimization training in the medical field.

[0012] Preferably, the visual pre-trained model is a large visual model compatible with both 2D and 3D modalities and has been pre-trained based on massive image corpora in a general domain; the large visual model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model;

[0013] The waveform types in the waveform type set include at least electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG).

[0014] The waveform-disease category set includes multiple first disease category sets; the disease categories in the first disease category set corresponding to the electrocardiogram include at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block; the disease categories in the first disease category set corresponding to the electroencephalogram include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, and mental abnormalities; the disease categories in the first disease category set corresponding to the electromyogram include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis, Guillain-Barré syndrome, polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve palsy.

[0015] The image types in the image type set include at least computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional ultrasound images, and positron emission tomography (PET) images.

[0016] The set of anatomical structures includes at least a variety of organ types; the organ types include at least the heart, lungs, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, and bladder;

[0017] The first training framework is formed by connecting the output end of the image pre-training model and the input end of the waveform classification task head; the first training framework is used to perform waveform classification prediction processing based on the 2D waveform input by the framework and output the corresponding first prediction vector; wherein, the first prediction vector is composed of multiple first prediction probabilities, each first prediction probability corresponds to a first prediction type; the first prediction type includes unknown waveform types and multiple known waveform types; the known waveform types correspond one-to-one with the waveform types in the waveform type set.

[0018] The second training framework is formed by connecting the output of the image pre-trained model and the input of the waveform disease prediction task head; the second training framework is used to perform disease classification prediction processing based on the 2D waveform image input by the framework and output the corresponding second prediction vector; wherein, the second prediction vector is composed of multiple second prediction probabilities, each second prediction probability corresponds to a second prediction type; the second prediction type includes unknown disease types and multiple known disease types; the known disease types correspond one-to-one with the disease types in the waveform-disease type set;

[0019] The third training framework is formed by connecting the output end of the image pre-training model and the input end of the image target detection task head; the third training framework is used to perform organ target detection and classification recognition processing based on the 3D image map input by the framework and output the corresponding first predicted target set; wherein, the first predicted target set consists of one or more first target detection boxes; the first target detection box includes at least a detection box identifier, a detection box three-dimensional size, a detection box center point coordinate, and a detection box type; the detection box type includes unknown target types and multiple known target types; the known target types correspond one-to-one with the organ types in the anatomical structure type set;

[0020] The fourth training framework is formed by connecting the output end of the image pre-training model and the input end of the image semantic segmentation task head; the fourth training framework is used to perform voxel-level organ semantic segmentation processing based on the 3D image map input by the framework and output the corresponding first predicted segmentation map; wherein, the three-dimensional size of the first predicted segmentation map is consistent with the three-dimensional size of the 3D image map input by the current framework; the voxel-level features of the first predicted segmentation map include at least one semantic type; the semantic type includes background semantics, unknown organ semantics, and multiple known organ semantics, and the known organ semantics correspond one-to-one with the organ types in the anatomical structure type set.

[0021] The fifth training framework consists of the image pre-training model, the feature distribution module, the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head.

[0022] The fifth training framework is used to identify the 2D / 3D task type input by the current framework. If the current 2D / 3D task type is 2D, then waveform classification prediction and disease classification prediction are performed based on the 2D waveform image input by the current framework, and the corresponding first and second prediction vectors are output. If the current 2D / 3D task type is 3D, then organ target detection and classification recognition and organ semantic segmentation are performed based on the 3D image image input by the current framework, and the corresponding first prediction target set and first prediction segmentation map are output. The 2D / 3D task type includes both 2D and 3D types. When the current framework input image is a 2D waveform image, the corresponding 2D / 3D task type is 2D; when the current framework input image is a 3D image, the corresponding 2D / 3D task type is 3D.

[0023] The first dataset includes multiple first data records; each first data record includes a first training waveform, a first label vector, and a second label vector; the first training waveform is a waveform in a medical field; the first label vector consists of multiple first label probabilities, each first label probability corresponding to a first prediction type; the second label vector consists of multiple second label probabilities, each second label probability corresponding to a second prediction type; in both the first and second label vectors, only one label probability is 1, and the remaining label probabilities are 0; the total set of waveform types for all first training waveforms in the first dataset is greater than the set of waveform types; the total number of data records corresponding to each first prediction type in the first dataset is the same; the total number of data records corresponding to each second prediction type in the first dataset is the same.

[0024] The second dataset includes multiple second data records; the second data records include a first training image, a first label target set, and a first label segmentation map; the first training image is a type of 3D image; the first label target set consists of one or more first label detection boxes; the first label detection box includes at least the detection box identifier, the three-dimensional size of the detection box, the coordinates of the detection box center point, and the detection box type; the three-dimensional size of the first label segmentation map is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation map include at least the semantic type; the image type set of all first training image images in the second dataset matches the image type set; the anatomical structure type set of all first training image images in the second dataset is larger than the anatomical structure type set; the total number of data records corresponding to each type of image image in the second dataset is the same, and the total number of detection boxes corresponding to each type of detection box type in all data records corresponding to each type of image image is the same, and the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same.

[0025] Furthermore, in the first training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding first encoded feature tensor and send it to the waveform classification task head; the waveform classification task head is used to perform classification prediction processing based on the first encoded feature tensor and output the corresponding first prediction vector.

[0026] In the second training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding second encoded feature tensor and send it to the waveform disease prediction task head; the waveform disease prediction task head is used to perform classification prediction processing based on the second encoded feature tensor and output the corresponding second prediction vector.

[0027] In the third training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input to the current framework to obtain the corresponding third encoded feature tensor and send it to the image target detection task head; the image target detection task head is used to perform target detection and classification processing based on the third encoded feature tensor and output the corresponding first predicted target set;

[0028] In the fourth training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input by the current framework to obtain the corresponding fourth encoded feature tensor and send it to the image semantic segmentation task head; the image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the fourth encoded feature tensor and output the corresponding first predicted segmentation map.

[0029] The first frame input terminal of the fifth training frame is used to receive the 2D waveform or the 3D image input of the current frame, the second frame input terminal is used to receive the 2D / 3D task type input of the current frame, and the first, second, third and fourth frame output terminals are used to output the corresponding first prediction vector, second prediction vector, first prediction target set and first prediction segmentation map.

[0030] In the fifth training framework, the input end of the image pre-training model is connected to the input end of the first framework, and the output end is connected to the first input end of the feature distribution module; the second input end of the feature distribution module is connected to the input end of the second framework, and the output end is connected to the input ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head, and the image semantic segmentation task head, respectively; the output ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head, and the image semantic segmentation task head are respectively connected to the output ends of the corresponding first, second, third, and fourth frameworks.

[0031] In the fifth training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform or 3D image input to the current framework to obtain the corresponding fifth encoded feature tensor, which is then sent to the feature distribution module. The feature distribution module is used to identify the 2D / 3D task type input to the current framework. If the current 2D / 3D task type is 2D, the fifth encoded feature tensor is sent to the waveform classification task head and the waveform disease prediction task head. If the current 2D / 3D task type is 3D, the fifth encoded feature tensor is sent to the image target detection task head and the image semantic segmentation task head. The waveform classification task head is used to receive... When the fifth encoded feature tensor is received, classification prediction processing is performed based on the current encoded feature tensor, and the corresponding first prediction vector is output. The waveform disease prediction task head is used to perform classification prediction processing based on the current encoded feature tensor and output the corresponding second prediction vector when the fifth encoded feature tensor is received. The image target detection task head is used to perform target detection and classification recognition processing based on the current encoded feature tensor and output the corresponding first predicted target set when the fifth encoded feature tensor is received. The image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the current encoded feature tensor and output the corresponding first predicted segmentation map when the fifth encoded feature tensor is received.

[0032] Preferably, the five-step training based on the two datasets and the five training frameworks specifically includes:

[0033] The first step of training is performed based on the first dataset and the first training framework;

[0034] After the first step of training is completed, the model parameters of the waveform classification task head are fixed; and the second step of training is carried out based on the first dataset and the second training framework.

[0035] After the second training step is completed, the model parameters of the waveform disease prediction task head are fixed; and the third training step is performed based on the second dataset and the third training framework.

[0036] After the third training step is completed, the model parameters of the image target detection task head are fixed; and the fourth training step is performed based on the second dataset and the fourth training framework.

[0037] After the fourth step of training is completed, the model parameters of the image semantic segmentation task head are fixed; and the fifth step of training is carried out based on the first and second datasets and the fifth training framework.

[0038] Furthermore, the first step of training based on the first dataset and the first training framework specifically includes:

[0039] Step 51: Based on a preset first segmentation ratio, the first dataset is randomly divided into two subsets, denoted as the first training set and the first evaluation set; and the combined set of model parameters of the image pre-trained model and the waveform classification task head is used as the corresponding first frame parameter set.

[0040] Wherein, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set satisfies the first segmentation ratio;

[0041] Step 52: Count the total number of records in the first training set to obtain the corresponding total number N1; and count the total number of types of the first prediction type to obtain the corresponding total number C1; and denote the first training waveform and the first label vector of each first data record in the first training set as the corresponding waveform x. a and label vector y a 1 ≤ index a ≤ N1; and each of the waveforms x a The 2D waveform is input into the first training framework for processing, and the first prediction vector output from the current processing is recorded as the corresponding prediction vector. and the label vector y a The probabilities of each of the first labels are denoted as y. a,b and the prediction vector Each of the first predicted probabilities is denoted as the corresponding 1 ≤ index b ≤ C1; and by each of the predicted vectors and the label vector y a Form a corresponding first prediction-label pair; and input the resulting N1 first prediction-label pairs into a preset first model loss function L. M1 The corresponding first loss value is obtained through calculation;

[0042] Wherein, the first model loss function L M1 It is implemented based on the cross-entropy loss function, specifically as follows:

[0043]

[0044] Step 53: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 54; if not, based on the preset first model optimizer, move towards making the first model loss function L... M1 The direction that reaches the minimum value is modulated once for the first frame parameter set, and the process returns to step 52 when the modulation ends.

[0045] The first model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0046] Step 54: Perform a round of traversal on all the first data records in the first evaluation set; during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; input the first training waveform of the current evaluation record as the current 2D waveform into the first training framework for processing, and record the first prediction vector output by the current processing as the corresponding first prediction vector; form a corresponding second prediction-label pair with the first prediction vector and the first label vector of the current evaluation record; and at the end of this round of traversal, estimate the accuracy, precision, recall and F1 score based on all the obtained second prediction-label pairs to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0047] Step 55: Identify whether the first accuracy, first precision, first recall, and first F1 score all satisfy their respective first accuracy range, first precision range, first recall range, and first F1 score range; if not, return to step 52 to continue training; if yes, confirm that the first step of training is complete.

[0048] Furthermore, the second step of training based on the first dataset and the second training framework specifically includes:

[0049] Step 61: Based on the preset second segmentation ratio, the first dataset is randomly divided into two subsets, which are denoted as the corresponding second training set and second evaluation set; and the model parameter set of the image pre-trained model and the waveform disease prediction task head is used as the corresponding second frame parameter set.

[0050] The second training set and the second evaluation set are both composed of multiple first data records; the ratio of the total number of records in the second training set and the second evaluation set satisfies the second segmentation ratio.

[0051] Step 62: Count the total number of records in the second training set to obtain the corresponding total number N2; and count the total number of types of the second prediction type to obtain the corresponding total number C2; and denote the first training waveform and the second label vector of each first data record in the second training set as the corresponding waveform x. c and label vector y c 1 ≤ index c ≤ N2; and each of the waveforms x cThe 2D waveform is input into the second training framework for processing, and the second prediction vector output from the current processing is recorded as the corresponding prediction vector. and the label vector y c The probabilities of each of the second labels are denoted as y. c,d and the prediction vector Each of the second predicted probabilities is denoted as the corresponding 1 ≤ index d ≤ C2; and by each of the predicted vectors and the label vector y c A corresponding third prediction-label pair is formed; and the resulting N2 third prediction-label pairs are input into a preset second model loss function L. M2 The corresponding second loss value is obtained through calculation;

[0052] Wherein, the second model loss function L M2 It is implemented based on the cross-entropy loss function, specifically as follows:

[0053]

[0054] Step 63: Identify whether the second loss value meets the preset second loss value range; if it does, proceed to step 64; if not, based on the preset second model optimizer, move towards making the second model loss function L... M2 The direction that reaches the minimum value is modulated once for the second frame parameter set, and the process returns to step 62 when the current modulation is completed;

[0055] The second model optimizer includes at least the Adam optimizer and the SGD optimizer;

[0056] Step 64: Perform a round of traversal on all the first data records in the second evaluation set; during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; input the first training waveform of the current evaluation record as the current 2D waveform into the second training framework for processing, and record the second prediction vector output by the current processing as the corresponding second prediction vector; and form a corresponding fourth prediction-label pair by the second prediction vector and the second label vector of the current evaluation record; and at the end of this round of traversal, estimate the accuracy, precision, recall and F1 score based on all the obtained fourth prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score;

[0057] Step 65: Identify whether the second accuracy, second precision, second recall, and second F1 score all satisfy their respective second accuracy range, second precision range, second recall range, and second F1 score range; if not, return to step 62 to continue training; if yes, confirm that the second step of training is complete.

[0058] Furthermore, the third training step based on the second dataset and the third training framework specifically includes:

[0059] Step 71: Count the total number of records in the second dataset to obtain the corresponding total number N3; and count the total number of organ types in the anatomical structure type set to obtain the corresponding total number N4; and use the combined set of model parameters of the image pre-training model and the image target detection task head as the corresponding third frame parameter set;

[0060] Step 72: Denote the first training image and the first label target set of each second data record in the second dataset as the corresponding image image x. e and the set of tags G e 1 ≤ index e ≤ N3; and each of the aforementioned image maps x e The 3D image is input into the third training framework for processing, and the first set of predicted targets output in this processing is recorded as the corresponding prediction set.

[0061] Step 73, for each of the said tag sets G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f 1 ≤ index f ≤ N e ; and the prediction set In and each of the aforementioned label boxes e,f The nearest first target detection box is denoted as the corresponding matching box.

[0062] Step 74, set each of the aforementioned label boxes. e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each of the types T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,fThe corresponding size S e,f The center P e,f The type T e,f The vector Z e,f Form the corresponding feature group bt e,f ;

[0063] Wherein, the vector Z e,f Includes N4 one-hot encodings z e,f,g , 1 ≤ index g ≤ N4; the vector Z e,f Only those of type T are mentioned. e,f The corresponding one-hot encoding z e,f,g The value is 1, and all others are 0;

[0064] Step 75, match each of the aforementioned matching boxes. The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. center type and each of the types Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And by each of the aforementioned matching boxes The corresponding size The center The vector Form corresponding feature groups

[0065] Wherein, the vector Includes N4 one-hot codes The vector Only those of the type mentioned The corresponding one-hot encoding The value is 1, and all others are 0;

[0066] Step 76, divide N3×N e For the feature group bt e,f , Substitute the preset third model loss function L M3 The corresponding third loss value is obtained through calculation;

[0067] Wherein, the third model loss function L M3 It is implemented by combining the cross-entropy loss function for classification, the L2 loss function for center position, and the L1 loss function for 3D dimensions, specifically as follows:

[0068]

[0069] Step 77: Identify whether the third loss value meets the preset range of the third loss value; if not, then based on the preset third model optimizer, move towards making the third model loss function L... M3 The direction that reaches the minimum value is modulated once for the third frame parameter set, and the process returns to step 72 when the modulation ends; if the condition is met, the third step of training is confirmed to be over.

[0070] The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0071] Furthermore, the fourth training step based on the second dataset and the fourth training framework specifically includes:

[0072] The combined set of model parameters of the image pre-trained model and the image semantic segmentation task head is used as the corresponding fourth frame parameter set;

[0073] With the goal of improving the feature consistency of multimodal images under the same anatomical structure, a round of self-supervised comparative learning training is performed on the model parameters of the image pre-training model based on the second dataset and the fourth training framework.

[0074] After this round of comparative learning training, with the goal of improving segmentation consistency, a round of supervised model training is carried out on the parameter set of the fourth training framework based on the second dataset and the fourth training framework.

[0075] And confirm the completion of the fourth step of training after the completion of this round of supervised training.

[0076] More preferably, the step of performing a round of self-supervised comparative learning training on the model parameters of the image pre-training model based on the second dataset and the fourth training framework, with the goal of improving the feature consistency of multimodal images under the same anatomical structure, specifically includes:

[0077] Step 91: Record each of the first training image images containing only one type of organ in the second dataset as a corresponding single-organ image image, and count the total number of the single-organ image images to obtain the corresponding total number N5. Then, form a corresponding multi-organ image set X from the N5 single-organ image images. ALL And each of the single organ images is denoted as the corresponding image image x. i , 1 ≤ index i ≤ N5;

[0078] Step 92, and process the multi-organ image set X ALL The total number of organ types was statistically analyzed to obtain the corresponding total number N6; and the multi-organ image set X was then used to obtain the total number of organ types. ALL Multiple single-organ images corresponding to the same type of organ are clustered together to form a corresponding single-organ image set X.j 1 ≤ index j ≤ N6; and for each of the single organ image sets X i The total number of images is counted to obtain the corresponding total number N. 7,j ;

[0079] Step 93, and then each of the aforementioned image maps x i As the corresponding current image; and the single organ image set X to which the current image is located is used. j Let X be the corresponding current positive sample set; and let X be the set of all other single organ images excluding the current positive sample set. j The corresponding current negative sample set is formed; and each of the single organ images in the current positive sample set, excluding the current image image, is taken as the positive sample image corresponding to the current image image. 1≤Positive Sample Index The total number of positive samples corresponding to the current image. The value is the total number N corresponding to the current positive sample set. 7,j The value is reduced by 1; and each of the single organ images in the current negative sample set is taken as the negative sample image corresponding to the current image. 1≤Negative Sample Index The total number of negative samples corresponding to the current image.

[0080] Step 94, transfer each of the image images x i The current 3D image is input into the image pre-training model for feature encoding processing, and the encoded feature tensor output from this processing is used as the corresponding feature tensor y. i ; and each of the aforementioned image maps x i The corresponding positive sample images The corresponding encoded feature tensor is denoted as the corresponding positive sample tensor. and each of the aforementioned image images x i The corresponding negative sample images The corresponding encoded feature tensor is denoted as the corresponding negative sample tensor. and each of the aforementioned feature tensors y i corresponding The positive sample tensor Form the corresponding positive sample set Each of the aforementioned feature tensors y i corresponding The negative sample tensor Form the corresponding negative sample set and each of the aforementioned feature tensors yi The three-dimensional dimension is denoted as the corresponding height H. i Width W i and depth D i and each of the aforementioned feature tensors y i Considered to be by H i ×W i ×D i Individual tensor The resulting 3D feature map, 1 ≤ index h i ≤H i , 1 ≤ index w i ≤W i 1 ≤ index d i ≤D i ;

[0081] Step 95, convert the N5 feature tensors y obtained this time into... i and the corresponding positive and negative sample sets Substitute the preset fourth model loss function L M4 The corresponding fourth loss value is obtained through calculation;

[0082] Wherein, the fourth model loss function L M4 It is implemented by combining the InfoNCE loss function and the non-integer order total variation loss function, specifically as follows:

[0083]

[0084] λ is a preset tradeoff coefficient;

[0085] L infoNCE () represents the InfoNCE loss function; sim() represents a type of similarity function, which includes at least the dot product similarity function, the cosine vector similarity function, and the Euclidean distance similarity function; τ represents a preset temperature coefficient;

[0086] L FOTV () represents the non-integer order total variation loss function; α is a preset non-integer order parameter; For the α-order derivative operator along the three-dimensional direction, specifically: These are the α-order derivative operators along the height, width, and depth directions, respectively;

[0087] Step 96: Identify whether the fourth loss value meets the preset range of the fourth loss value; if not, then based on the preset fourth model optimizer, move towards making the fourth model loss function L... M4 The direction that reaches the minimum value modulates the model parameters of the image pre-training model in one round, and returns to step 94 when the modulation ends; if satisfied, the contrastive learning training is confirmed to be over.

[0088] The fourth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0089] More preferably, the step of performing a supervised model training on the parameter set of the fourth training framework based on the second dataset and the fourth training framework with the goal of improving segmentation consistency specifically includes:

[0090] Step 101: Record the first training image and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e and label segmentation map y e ; and each of the aforementioned image maps x e The current 3D image is input into the fourth training framework for processing, and the first predicted segmentation map output from the current processing is recorded as the corresponding predicted segmentation map.

[0091] Step 102, use the N3 pairs obtained this time to match the labels and the predicted segmentation map y e , Substitute the preset fifth model loss function L M5 The corresponding fifth loss value is obtained through calculation;

[0092] Wherein, the fifth model loss function L M5 Based on Dice loss function L Dice Implementation, specifically:

[0093]

[0094] Step 103: Identify whether the fifth loss value meets the preset range of the fifth loss value; if not, then based on the preset fifth model optimizer, move towards making the fifth model loss function L... M5 The direction that reaches the minimum value is modulated once for the fourth frame parameter set, and the process returns to step 101 when the modulation ends; if the condition is met, the supervised training for this round is confirmed to be over.

[0095] The fifth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0096] Preferably, the fifth training step based on the first and second datasets and the fifth training framework specifically includes:

[0097] Step 111: Calculate the total number of records in the first dataset to obtain the corresponding total number N8; and calculate the total number of types for the first prediction type to obtain the corresponding total number. The total number of types in the second prediction type is then statistically analyzed to obtain the corresponding total. And denote the first training waveform, the first label vector, and the second label vector of each first data record in the first dataset as the corresponding waveform x. m Label vector y 1,m Label vector y 2,m 1 ≤ index m ≤ N8; and for each of the waveforms x m Set a specific 2D training graph type;

[0098] Step 112: Record the first training image, the first label target set, and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e The tag set G e and the label segmentation image y e ; and for each of the aforementioned image maps x e Set a specific training graph type as 3D;

[0099] Step 113, convert each of the waveforms x m The training graph type and its corresponding type are input into the fifth training framework as the current 2D waveform graph and 2D / 3D task type for processing, and the first and second prediction vectors output in this processing are recorded as the corresponding prediction vectors. and prediction vector

[0100] Step 114, the label vector y 1,m The probabilities of each of the first labels are denoted as y. 1,m,s1 and the prediction vector Each of the first predicted probabilities is denoted as the corresponding 1≤index And by each of the predicted vectors and the label vector y 1,m Form a corresponding fifth prediction-label pair; and then use the label vector y 2,m The probabilities of each of the second labels are denoted as y. 2,m,s2 and the prediction vector Each of the second predicted probabilities is denoted as the corresponding 1≤index And by each of the predicted vectors and the label vector y 2,m A corresponding sixth prediction-label pair is formed; and the N8 fifth prediction-label pairs obtained this time form a corresponding prediction-label pair set. The N8 sixth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0101] Step 115, transfer each of the aforementioned image maps x e The training map type and its corresponding type are input into the fifth training framework as the current 3D image map and the 2D / 3D task type for processing, and the first predicted target set and the first predicted segmentation map output in this processing are recorded as the corresponding prediction set. and the predicted segmentation map

[0102] Step 116, for each of the said tag sets G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f ; and the prediction set In and each of the aforementioned label boxes e,f The nearest first target detection box is denoted as the corresponding matching box. and each of the aforementioned label boxes e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f The center P e,f The type T e,f ; and each of the types T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,f The corresponding size S e,f The center P e,f The type T e,f The vector Z e,f The corresponding feature group bt e,f ;

[0103] Step 117, match each of the aforementioned matching boxes The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. The center The type and each of the types This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector. And by each of the aforementioned matching boxes The corresponding size The center The vector The corresponding feature groups are formed

[0104] Step 118, from each of the aforementioned feature groups bt e,f and the corresponding feature groups A corresponding seventh prediction-label pair is formed; and each of the label segmentation maps y e and the corresponding predicted segmentation map Form a corresponding eighth prediction-label pair; and use the N3×N obtained this time. e Each of the seventh prediction-label pairs forms a corresponding prediction-label pair set. The N3 eighth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0105] Step 119, the predicted-label pair set obtained this time... Substitute the preset multi-task loss function L MT The corresponding sixth loss value is obtained through calculation;

[0106] Wherein, the multi-task loss function L MT The 2D task loss function L 2D and the 3D task loss function L 3D Combining these elements, the 2D task loss function L... 2D The corresponding first and second 2D task loss functions Combining these elements, the 3D task loss function L... 3D Based on the corresponding first and second 3D task loss functions Composed of various elements; the multi-task loss function L MT Specifically:

[0107]

[0108] w 2D w 3D These are two preset weight parameters;

[0109] Step 120: Identify whether the sixth loss value meets the preset range of the sixth loss value; if not, then based on the preset sixth model optimizer, move towards making the multi-task loss function L... MT The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 113 when the modulation ends; if satisfied, the fifth step of training is confirmed to be over.

[0110] The sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0111] A second aspect of the present invention provides an apparatus for implementing the training method of the image pre-training model for the medical field described in the first aspect above. The apparatus includes: a model preparation module, a basic data preparation module, a training framework preparation module, a training dataset preparation module, and a five-step training module.

[0112] The model preparation module is used to select a type of visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-trained model.

[0113] The basic data preparation module is used to use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; and to configure a corresponding image type set for the 2D waveforms, denoted as the corresponding waveform type set; and to set a corresponding diagnostic disease type set for each type of waveform in the waveform type set, denoted as the corresponding first disease type set, and to form a corresponding waveform-disease type set by all the obtained first disease type sets; and to configure a corresponding image type set and anatomical structure type set for the 3D images, denoted as the corresponding image type set and anatomical structure set;

[0114] The training framework preparation module is used to select four types of downstream task head models and connect them to the image pre-training model in separate and collective connection methods to obtain five corresponding training frameworks; the four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0115] The training dataset preparation module is used to construct a first dataset based on the waveform type set and the waveform-disease type set through big data acquisition; and to construct a second dataset based on the image type set and the anatomical structure type set through big data acquisition.

[0116] The five-step training module is used to perform five-step training based on two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it is confirmed that the image pre-trained model has completed the domain-specific optimization training in the medical field.

[0117] A third aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0118] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0119] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0120] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0121] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for image pre-training models in the medical field. As described above, this invention first selects a type of pre-trained LVMs that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; it then configures a waveform type set for the 2D waveforms supported by the visual pre-training model in the medical field, and sets disease type subsets for each type of waveform to obtain a corresponding full waveform-disease type set; and it configures corresponding image type sets and anatomical structure type sets for the 3D images supported by the visual pre-training model in the medical field; next, it selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head) and interfaces them with the image pre-training model to obtain five training frameworks; then, it constructs a first dataset based on the waveform type set and waveform-disease type set, and a second dataset based on the image type set and anatomical structure type set through big data collection; then, it performs five steps of training based on the two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it confirms that the targeted training of the image pre-training model in the medical field is complete. In this invention, on the one hand, an image pre-trained model that is compatible with both 2D and 3D images and supports multiple waveforms and image types is obtained through SDT training in the medical field and serves as a shared encoding platform; on the other hand, the overall training difficulty is reduced and the overall training efficiency is improved through a progressive training method; and furthermore, the economic benefits of SDT training are improved by simultaneously training four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head). Attached Figure Description

[0122] Figure 1 This is a schematic diagram of a training method for an image pre-training model in the medical field provided in Embodiment 1 of the present invention;

[0123] Figure 2 This is a schematic diagram of the five types of training frameworks provided in Embodiment 1 of the present invention;

[0124] Figure 3 This is a module structure diagram of a training device for an image pre-training model in the medical field, provided in Embodiment 2 of the present invention;

[0125] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0126] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0127] Embodiment 1 of the present invention provides a training method for an image pre-training model in the medical field, such as... Figure 1 The diagram illustrates a training method for an image pre-training model in the medical field, as provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0128] Step 1: Select a visual pre-training model that can simultaneously support 2D / 3D image feature encoding as the image pre-training model.

[0129] Here, the visual pre-trained model in this embodiment of the invention is a large visual model that is compatible with both 2D and 3D modalities and has been pre-trained based on massive image corpora in a general domain; the large visual model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model.

[0130] Step 2: Use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; configure the corresponding image type set for the 2D waveforms as the corresponding waveform type set; set the corresponding diagnostic disease type set for each type of waveform in the waveform type set as the corresponding first disease type set, and form the corresponding waveform-disease type set by all the obtained first disease type sets; configure the corresponding image type set and anatomical structure type set for the 3D images as the corresponding image type set and anatomical structure type set.

[0131] Here, the waveform types in the waveform type set of this embodiment of the invention include at least electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG). It should be noted that the types of waveforms in the waveform type set can be added or removed based on actual application requirements.

[0132] The waveform-disease category set in this embodiment of the invention includes multiple first disease category sets; wherein, 1) the disease categories in the first disease category set corresponding to electrocardiogram (ECG) include at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block; 2) the disease categories in the first disease category set corresponding to electroencephalogram (EEG) include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, and mental abnormalities; 3) the disease categories in the first disease category set corresponding to electromyography (EMG) include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis (ALS), Guillain-Barré syndrome, polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve palsy. It should be noted that the first disease category sets corresponding to each waveform category in the waveform-disease category set can be added or removed based on actual application needs, and the disease categories in each first disease category set can also be added or removed based on actual application needs.

[0133] The image image types in this embodiment of the invention include at least computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional ultrasound images, and positron emission tomography (PET) images. It should be noted that the types of images in the image image type set can be added or removed based on actual application requirements.

[0134] The anatomical structure category set of this invention includes at least a variety of organ types; the organ types include at least the heart, lungs, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, and bladder. It should be noted that the anatomical structure categories, i.e., organ types, in the anatomical structure category set can be added to or removed based on actual application needs.

[0135] Step 3: Select four types of downstream task head models and connect them with the image pre-training model using separate and collective connection methods to obtain the corresponding five types of training frameworks.

[0136] Here, the four types of downstream task heads in this embodiment of the invention include waveform classification task head, waveform disease prediction task head, image target detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0137] It should be noted that this embodiment of the invention does not limit the specific model structure of the four types of downstream task heads, but only limits the input / output data and corresponding model functions of the four types of downstream task heads. Specifically, the technical descriptions of the five training frameworks below can be used for further understanding.

[0138] 1) First training framework:

[0139] like Figure 2As shown in the schematic diagram of the five types of training frameworks provided in Embodiment 1 of the present invention, the first training framework of the present invention is formed by connecting the output end of the image pre-training model and the input end of the waveform classification task head.

[0140] The first training framework is used to perform waveform classification and prediction processing based on the 2D waveform image input to the framework and output the corresponding first prediction vector. The first prediction vector consists of multiple first prediction probabilities, each corresponding to a first prediction type. The first prediction types include unknown waveform types and multiple known waveform types; the known waveform types here correspond one-to-one with the waveform types in the waveform image type set.

[0141] In the first training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding first encoded feature tensor and send it to the waveform classification task head; the waveform classification task head is used to perform classification prediction processing based on the first encoded feature tensor and output the corresponding first prediction vector.

[0142] 2) Second training framework:

[0143] like Figure 2 As shown, the second training framework of this embodiment of the invention is formed by connecting the output end of the image pre-trained model and the input end of the waveform disease prediction task head.

[0144] The second training framework is used to perform disease classification prediction based on the 2D waveform image input by the framework and output the corresponding second prediction vector. The second prediction vector consists of multiple second prediction probabilities, each corresponding to a second prediction type. The second prediction types include unknown disease types and multiple known disease types; the known disease types here correspond one-to-one with the disease types in the waveform-disease type set.

[0145] In the second training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding second encoded feature tensor and send it to the waveform disease prediction task head; the waveform disease prediction task head is used to perform classification prediction processing based on the second encoded feature tensor and output the corresponding second prediction vector.

[0146] 3) Third training framework:

[0147] like Figure 2 As shown, the third training framework of this embodiment of the invention is formed by connecting the output end of the image pre-training model and the input end of the image target detection task head.

[0148] The third training framework is used to perform organ target detection and classification based on the 3D image input from the framework and output the corresponding first predicted target set. The first predicted target set consists of one or more first target detection boxes; each first target detection box includes at least a detection box identifier, three-dimensional dimensions, center point coordinates, and detection box type; the detection box type includes unknown target types and multiple known target types; the known target types here correspond one-to-one with the organ types in the anatomical structure category set.

[0149] In the third training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input to the current framework to obtain the corresponding third-encoded feature tensor and send it to the image target detection task head; the image target detection task head is used to perform target detection and classification processing based on the third-encoded feature tensor and output the corresponding first predicted target set.

[0150] 4) Fourth training framework:

[0151] like Figure 2 As shown, the fourth training framework of this embodiment of the invention is formed by connecting the output end of the image pre-training model and the input end of the image semantic segmentation task head.

[0152] The fourth training framework is used to perform voxel-level organ semantic segmentation based on the 3D image input to the framework and output the corresponding first predicted segmentation map. The 3D dimensions of the first predicted segmentation map are consistent with the 3D dimensions of the 3D image input to the current framework; the voxel-level features of the first predicted segmentation map include at least one semantic type; this semantic type includes background semantics, unknown organ semantics, and semantics for multiple known organ types, where the known organ semantics correspond one-to-one with the organ types in the anatomical structure category set.

[0153] In the fourth training framework, the image pre-training model is used to perform feature encoding processing on the 3D image map input by the current framework to obtain the corresponding fourth encoded feature tensor and send it to the image semantic segmentation task head; the image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the fourth encoded feature tensor and output the corresponding first predicted segmentation map.

[0154] 5) Fifth training framework:

[0155] like Figure 2 As shown, the fifth training framework of this embodiment consists of an image pre-training model, a feature distribution module, a waveform classification task head, a waveform disease prediction task head, an image target detection task head, and an image semantic segmentation task head.

[0156] The fifth training framework is used to identify the 2D / 3D task type input by the current framework. If the current 2D / 3D task type is 2D, then waveform classification prediction and disease classification prediction are performed based on the 2D waveform image input by the current framework, and the corresponding first and second prediction vectors are output. If the current 2D / 3D task type is 3D, then organ target detection and classification recognition and organ semantic segmentation are performed based on the 3D image image input by the current framework, and the corresponding first predicted target set and first predicted segmentation map are output. The 2D / 3D task type includes both 2D and 3D types. When the current framework input image is a 2D waveform image, the corresponding 2D / 3D task type is 2D; when the current framework input image is a 3D image, the corresponding 2D / 3D task type is 3D.

[0157] The first frame input of the fifth training frame is used to receive the 2D waveform or 3D image of the current frame input, the second frame input is used to receive the 2D / 3D task type of the current frame input, and the first, second, third and fourth frame outputs are used to output the corresponding first prediction vector, second prediction vector, first prediction target set and first prediction segmentation map.

[0158] In the fifth training framework, the input of the image pre-training model is connected to the input of the first framework, and its output is connected to the first input of the feature distribution module; the second input of the feature distribution module is connected to the input of the second framework, and its output is connected to the inputs of the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head, respectively; the outputs of the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head are connected to the outputs of the corresponding first, second, third, and fourth frameworks, respectively.

[0159] In the fifth training framework, the image pre-trained model is used to perform feature encoding processing on the 2D waveform or 3D image input of the current framework to obtain the corresponding fifth-encoded feature tensor, which is then sent to the feature distribution module. The feature distribution module is used to identify the 2D / 3D task type input of the current framework. If the current 2D / 3D task type is 2D, the fifth-encoded feature tensor is sent to the waveform classification task head and the waveform disease prediction task head. If the current 2D / 3D task type is 3D, the fifth-encoded feature tensor is sent to the image target detection task head and the image semantic segmentation task head. The waveform classification task head is used to process the received fifth-encoded feature tensor... When receiving the fifth encoded feature tensor, the task head performs classification prediction processing based on the current encoded feature tensor and outputs the corresponding first prediction vector; when receiving the fifth encoded feature tensor, the task head performs classification prediction processing based on the current encoded feature tensor and outputs the corresponding second prediction vector; when receiving the fifth encoded feature tensor, the task head performs target detection and classification recognition processing based on the current encoded feature tensor and outputs the corresponding first predicted target set; when receiving the fifth encoded feature tensor, the task head performs voxel-level semantic segmentation processing based on the current encoded feature tensor and outputs the corresponding first predicted segmentation map.

[0160] Step 4: Construct the first dataset based on waveform type set and waveform-disease type set using big data acquisition methods; and construct the second dataset based on image type set and anatomical structure type set using big data acquisition methods.

[0161] Here, the first dataset in this embodiment of the invention includes multiple first data records; the first data record includes a first training waveform, a first label vector, and a second label vector; the first training waveform is a waveform in a medical field; the first label vector is composed of multiple first label probabilities, each first label probability corresponding to a first prediction type; the second label vector is composed of multiple second label probabilities, each second label probability corresponding to a second prediction type; in both the first and second label vectors, only one label probability is 1, and the probabilities of the remaining labels are 0.

[0162] It should be noted that the final first dataset should meet the following requirements: 1) The total set of waveform types for all first training waveforms in the first dataset is greater than the total set of waveform types; 2) The total number of data records corresponding to each first prediction type in the first dataset is the same; 3) The total number of data records corresponding to each second prediction type in the first dataset is the same. It should also be noted that if there are difficulties in data collection during the large-scale data acquisition process, and it is impossible to guarantee that the total number of data records corresponding to each first / second prediction type is the same, then additional image generation models can be used to supplement the training waveforms for each prediction type through data augmentation.

[0163] The second dataset in this embodiment of the invention includes multiple second data records; the second data records include a first training image, a first label target set, and a first label segmentation map; the first training image is a type of 3D image; the first label target set consists of one or more first label detection boxes; the first label detection box includes at least a detection box identifier, a detection box three-dimensional size, detection box center point coordinates, and a detection box type; the three-dimensional size of the first label segmentation map is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation map include at least a semantic type.

[0164] It should be noted that the final second dataset should meet the following requirements: 1) The set of image types for all first training image images in the second dataset matches the set of image types; 2) The set of anatomical structure types for all first training image images in the second dataset is greater than the set of anatomical structure types; 3) The total number of data records corresponding to each type of image image in the second dataset is the same, and the total number of detection boxes for each type of detection box in all data records corresponding to each type of image image is the same, and the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same. It should also be noted that if there are difficulties in data collection, and it is impossible to guarantee that the total number of data records for each type of image image is the same, or that the total number of detection boxes for each type of detection box in all data records corresponding to each type of image image is the same, or that the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same, then missing images can be supplemented by using another medical image generation model through data augmentation.

[0165] Step 5: Following a progressive training approach, conduct five training steps based on two datasets and five training frameworks; and at the end of the fifth training step, confirm that the image pre-trained model has completed domain-specific optimization training in the medical field.

[0166] Specifically, it includes: Step 51, which involves five steps of training based on two datasets and five training frameworks, following a progressive training approach;

[0167] Specifically, it includes: Step 511, performing the first training step based on the first dataset and the first training framework;

[0168] Specifically, it includes: step 5111, randomly dividing the first dataset into two subsets based on a preset first segmentation ratio, denoted as the corresponding first training set and first evaluation set; and using the combined set of model parameters of the image pre-trained model and the waveform classification task head as the corresponding first frame parameter set;

[0169] Wherein, the first segmentation ratio is a pre-set ratio parameter, such as 8:2; both the first training set and the first evaluation set consist of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set satisfies the first segmentation ratio;

[0170] Step 5112: Count the total number of records in the first training set to obtain the corresponding total number N1; and count the total number of types of the first prediction type to obtain the corresponding total number C1; and denote the first training waveform and the first label vector of each first data record in the first training set as the corresponding waveform x. a and label vector y a 1 ≤ index a ≤ N1; and each waveform x a The current 2D waveform is input to the first training framework, processed, and the first prediction vector output from this processing is recorded as the corresponding prediction vector. and label vector y a The probabilities of each first label are denoted as y. a,b and predict vector The first prediction probabilities are denoted as the corresponding 1 ≤ index b ≤ C1; and by each prediction vector and label vector y a Form a corresponding first prediction-label pair; and input the resulting N1 first prediction-label pairs into the preset first model loss function L. M1 The corresponding first loss value is obtained through calculation;

[0171] Wherein, the first model loss function L M1 It is implemented based on the cross-entropy loss function, specifically as follows:

[0172]

[0173] Step 5113: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 5114; if not, based on the preset first model optimizer, move towards making the first model loss function L... M1 The direction that reaches the minimum value is modulated once for the first frame parameter set, and the process returns to step 5112 when the current modulation is completed.

[0174] The first loss value range is a pre-set numerical range; the first model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0175] Step 5114: Perform a round of traversal on all first data records in the first evaluation set; during this round of traversal, the first data record currently being traversed is taken as the corresponding current evaluation record; the first training waveform of the current evaluation record is taken as the current 2D waveform and input into the first training framework for processing, and the first prediction vector output by the current processing is recorded as the corresponding first prediction vector; and a corresponding second prediction-label pair is formed by the first prediction vector and the first label vector of the current evaluation record; and at the end of this round of traversal, the accuracy, precision, recall and F1 score are estimated based on all the obtained second prediction-label pairs to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0176] Step 5115: Identify whether the first accuracy, first precision, first recall, and first F1 score all meet their respective first accuracy range, first precision range, first recall range, and first F1 score range; if not, return to step 5112 to continue training; if yes, confirm that the first step of training is complete.

[0177] Here, the first accuracy range, the first precision range, the first recall range, and the first F1 score range are four pre-set numerical ranges;

[0178] Step 512: After the first training step is completed, the model parameters of the waveform classification task head are fixed; and the second training step is carried out based on the first dataset and the second training framework.

[0179] Specifically, this includes: Step 5121, after the first step of training is completed, the model parameters of the waveform classification task head are solidified;

[0180] Step 5122, and perform the second training step based on the first dataset and the second training framework;

[0181] Specifically, it includes: step 51221, randomly dividing the first dataset into two subsets based on a preset second segmentation ratio, which are denoted as the corresponding second training set and second evaluation set; and taking the combined set of model parameters of the image pre-trained model and the waveform disease prediction task head as the corresponding second frame parameter set;

[0182] The second segmentation ratio is a pre-set ratio parameter, such as 8:2; both the second training set and the second evaluation set consist of multiple first data records; the ratio of the total number of records in the second training set and the second evaluation set satisfies the second segmentation ratio.

[0183] Step 51222: Count the total number of records in the second training set to obtain the corresponding total number N2; and count the total number of types of the second prediction type to obtain the corresponding total number C2; and denote the first training waveform and the second label vector of each first data record in the second training set as the corresponding waveform x. c and label vector y c 1 ≤ index c ≤ N2; and each waveform x c The current 2D waveform is input to the second training framework for processing, and the second prediction vector output from this processing is recorded as the corresponding prediction vector. and label vector y c The probabilities of each second label are denoted as the corresponding y. c,d and predict vector The respective second prediction probabilities are denoted as the corresponding 1 ≤ index d ≤ C2; and by each prediction vector and label vector y c Form a corresponding third prediction-label pair; and input the resulting N2 third prediction-label pairs into the preset second model loss function L. M2 The corresponding second loss value is obtained through calculation;

[0184] Among them, the loss function of the second model L M2 It is implemented based on the cross-entropy loss function, specifically as follows:

[0185]

[0186] Step 51223: Identify whether the second loss value meets the preset range of the second loss value; if it does, proceed to step 51224; if it does not, optimize the second model based on the preset second model optimizer to make the second model loss function L... M2 The direction that reaches the minimum value is modulated once for the second frame parameter set, and the process returns to step 51222 when the current modulation is completed;

[0187] The second loss value range is a pre-set numerical range; the second model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0188] Step 51224: Perform a round of traversal on all first data records in the second evaluation set; during this round of traversal, the first data record currently traversed is taken as the corresponding current evaluation record; the first training waveform of the current evaluation record is taken as the current 2D waveform and input into the second training framework for processing, and the second prediction vector output by the current processing is recorded as the corresponding second prediction vector; and the second prediction vector and the second label vector of the current evaluation record form a corresponding fourth prediction-label pair; at the end of this round of traversal, the accuracy, precision, recall and F1 score are estimated based on all the obtained fourth prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score;

[0189] Step 51225: Identify whether the second accuracy, second precision, second recall, and second F1 score all satisfy their respective corresponding ranges for the second accuracy, second precision, second recall, and second F1 score. If not, return to step 51222 to continue training; if yes, confirm that the second training step is complete.

[0190] Among them, the second accuracy range, the second precision range, the second recall range, and the second F1 score range are four pre-set numerical ranges;

[0191] Step 513: After the second training step is completed, the model parameters of the waveform disease prediction task head are fixed; and the third training step is carried out based on the second dataset and the third training framework.

[0192] Specifically, this includes: Step 5131, after the second training step is completed, the model parameters of the waveform disease prediction task head are solidified;

[0193] Step 5132, and perform the third training step based on the second dataset and the third training framework;

[0194] Specifically, it includes:

[0195] Step 51321: Count the total number of records in the second dataset to obtain the corresponding total number N3; and count the total number of organ types in the anatomical structure type set to obtain the corresponding total number N4; and take the combined set of model parameters of the image pre-trained model and the image target detection task head as the corresponding third frame parameter set;

[0196] Step 51322: Denote the first training image and the first label target set of each second data record in the second dataset as the corresponding image image x. e and the set of tags G e 1 ≤ index e ≤ N3; and each image x eThe current 3D image is used as input to the third training framework for processing, and the first set of predicted targets output from this processing is recorded as the corresponding prediction set.

[0197] Step 51323, for each tag set G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f 1 ≤ index f ≤ N e ; and the prediction set In the middle and each label box e,f The nearest first object detection box is denoted as the corresponding matching box.

[0198] Step 51324, change each label box e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each type T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each label box e,f Corresponding size S e,f Center P e,f Type T e,f Vector Z e,f Form the corresponding feature group bt e,f ;

[0199] Wherein, vector Z e,f Includes N4 one-hot encodings z e,f,g 1 ≤ index g ≤ N4; vector Z e,f Only type T is included. e,f The corresponding one-hot encoding z e,f,g The value is 1, and all others are 0;

[0200] Step 51325, match each box The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimensions. center type and each type Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And composed of each matching box Corresponding size center vector Form corresponding feature groups

[0201] Where, vector Includes N4 one-hot codes vector Only the type Corresponding one-hot encoding The value is 1, and all others are 0;

[0202] Step 51326, change N3×N e For feature group bt e,f , Substitute the preset third model loss function L M3 The corresponding third loss value is obtained through calculation;

[0203] Among them, the loss function of the third model L M3 It is implemented by combining the cross-entropy loss function for classification, the L2 loss function for center position, and the L1 loss function for 3D dimensions, specifically as follows:

[0204]

[0205] Step 51327: Identify whether the third loss value meets the preset range of the third loss value; if not, then based on the preset third model optimizer, move towards making the third model loss function L... M3 The direction that reaches the minimum value is modulated once for the third frame parameter set, and the process returns to step 51322 when the modulation ends; if the condition is met, the third step of training is confirmed to be over.

[0206] The third loss value range is a pre-set numerical range; the third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0207] Step 514: After the third training step is completed, the model parameters of the image target detection task head are fixed; and the fourth training step is carried out based on the second dataset and the fourth training framework.

[0208] Specifically, this includes: Step 5141, after the third step of training is completed, the model parameters of the image target detection task head are solidified;

[0209] Step 5142, and perform the fourth training step based on the second dataset and the fourth training framework;

[0210] Specifically, this includes: step 51421, taking the combined set of model parameters of the image pre-trained model and the image semantic segmentation task head as the corresponding fourth frame parameter set;

[0211] Step 51422, and with the goal of improving the feature consistency of multimodal images under the same anatomical structure, a round of self-supervised comparative learning training is performed on the model parameters of the image pre-training model based on the second dataset and the fourth training framework.

[0212] Specifically, this includes: step 514221, where each first training image in the second dataset containing only one type of organ is recorded as a corresponding single-organ image, and the total number of single-organ image images is counted to obtain the corresponding total number N5, and the corresponding multi-organ image set X is composed of N5 single-organ image images. ALL And record the images of each single organ as the corresponding image x. i , 1 ≤ index i ≤ N5;

[0213] Step 514222, and the multi-organ image set X ALL The total number of organ types was statistically analyzed to obtain the corresponding total number N6; and the multi-organ image set X was then analyzed. ALL Multiple single-organ images corresponding to the same organ type are clustered together to form a corresponding single-organ image set X. j 1 ≤ index j ≤ N6; and for each single organ image set X i The total number of images is counted to obtain the corresponding total number N. 7,j ;

[0214] Step 514223, and then divide each image x i As the corresponding current image; and set the single organ image set X to which the current image belongs. j Let X be the corresponding current positive sample set; and let X be the set of all other single-organ images excluding the current positive sample set. j The corresponding current negative sample set is formed; and each single organ image in the current positive sample set, excluding the current image image, is taken as the positive sample image corresponding to the current image image. 1≤Positive Sample Index This represents the total number of positive samples corresponding to the current image. The value is the total number N corresponding to the current positive sample set. 7,j Decrease the value by 1; and use the single organ images in the current negative sample set as the negative sample images corresponding to the current image. 1≤Negative Sample Index This represents the total number of negative samples corresponding to the current image.

[0215] Step 514224, divide each image x i The current 3D image is used as the input image for pre-training model feature encoding, and the encoded feature tensor output from this process is used as the corresponding feature tensor y.i ; and each image x i Corresponding positive sample images The corresponding encoded feature tensor is denoted as the corresponding positive sample tensor. and each image x i Corresponding negative sample images The corresponding encoded feature tensor is denoted as the corresponding negative sample tensor. and each feature tensor y i corresponding positive sample tensors Form the corresponding positive sample set Each feature tensor y i corresponding negative sample tensor Form the corresponding negative sample set and each feature tensor y i The three-dimensional dimension is denoted as the corresponding height H. i Width W i and depth D i and each feature tensor y i Considered to be by H i ×W i ×D i Individual tensor The resulting 3D feature map, 1 ≤ index h i ≤H i , 1 ≤ index w i ≤W i 1 ≤ index d i ≤D i ;

[0216] Step 514225, convert the N5 feature tensors y obtained this time i and its corresponding positive and negative sample sets Substitute the preset fourth model loss function L M4 The corresponding fourth loss value is obtained through calculation;

[0217] Here, the loss function L of the fourth model M4 It is implemented by combining the InfoNCE loss function and the non-integer order total variation (FOTV) loss function, specifically as follows:

[0218]

[0219] Where λ is a preset trade-off coefficient;

[0220] L infoNCE() represents the InfoNCE loss function; sim() represents a type of similarity function, which includes at least the dot product similarity function, cosine vector similarity function, and Euclidean distance similarity function; τ represents the preset temperature coefficient;

[0221] L FOTV () represents the non-integer order total variation loss function; α is a preset non-integer order parameter; For the α-order derivative operator along the three-dimensional direction, specifically: These are the α-order derivative operators along the height, width, and depth directions, respectively;

[0222] Step 514226: Identify whether the fourth loss value meets the preset range of the fourth loss value; if not, then based on the preset fourth model optimizer, move towards making the fourth model loss function L... M4 The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 514224 when the modulation ends; if satisfied, the contrastive learning training is confirmed to be over.

[0223] The fourth loss value range is a pre-set numerical range; the fourth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0224] Step 51423, and after this round of comparative learning training, with the goal of improving segmentation consistency, a round of supervised model training is carried out on the parameter set of the fourth frame based on the second dataset and the fourth training frame.

[0225] Specifically, this includes: step 514231, where the first training image and the first label segmentation image of each second data record in the second dataset are denoted as the corresponding image image x. e and label segmentation map y e ; and each image x e The current 3D image is used as input to the fourth training framework for processing, and the first predicted segmentation map output from this processing is recorded as the corresponding predicted segmentation map.

[0226] Step 514232: Combine the N3 pairs of labels and the predicted segmentation map y obtained this time. e , Substitute the preset fifth model loss function L M5 The corresponding fifth loss value is obtained through calculation;

[0227] Among them, the loss function L of the fifth model M5 Based on Dice loss function L Dice Implementation, specifically:

[0228]

[0229] Step 514233: Identify whether the fifth loss value meets the preset range of the fifth loss value; if not, then based on the preset fifth model optimizer, move towards making the fifth model loss function L... M5 The direction that reaches the minimum value is used to modulate the fourth frame parameter set in one round, and the process returns to step 514231 when the modulation ends; if the condition is met, the supervised training in this round is confirmed to be over.

[0230] The fifth loss value range is a pre-set numerical range; the fifth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0231] Step 51424, and confirm the end of the fourth step of training after the completion of this round of supervised training;

[0232] Step 515: After the fourth step of training is completed, the model parameters of the image semantic segmentation task head are fixed; and the fifth step of training is carried out based on the first and second datasets and the fifth training framework.

[0233] Specifically, this includes: Step 5151, after the fourth step of training is completed, the model parameters of the image semantic segmentation task head are solidified;

[0234] Step 5152, and perform the fifth training step based on the first and second datasets and the fifth training framework;

[0235] Specifically, this includes: step 51521, calculating the total number of records in the first dataset to obtain the corresponding total number N8; and calculating the total number of types in the first prediction type to obtain the corresponding total number. The total number of types in the second prediction type is then statistically analyzed to obtain the corresponding total. And denote the first training waveform, first label vector, and second label vector of each first data record in the first dataset as the corresponding waveform x. m Label vector y 1,m Label vector y 2,m 1 ≤ index m ≤ N8; and for each waveform x m Set a specific 2D training graph type;

[0236] Step 51522: Denote the first training image, the first label target set, and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e Tag set G e and label segmentation map y e ; and for each image x e Set a specific training graph type as 3D;

[0237] Step 51523, extract each waveform xm The corresponding training graph type is used as the input of the current 2D waveform graph and 2D / 3D task type into the fifth training framework for processing, and the first and second prediction vectors output from the current processing are recorded as the corresponding prediction vectors. and prediction vector

[0238] Step 51524, change the label vector y 1,m The probabilities of each first label are denoted as y. 1,m,s1 and predict vector The first prediction probabilities are denoted as the corresponding 1≤index And by each prediction vector and label vector y 1,m Form a corresponding fifth prediction-label pair; and combine the label vector y 2,m The probabilities of each second label are denoted as the corresponding y. 2,m,s2 and predict vector The respective second prediction probabilities are denoted as the corresponding 1≤index And by each prediction vector and label vector y 2,m A corresponding sixth prediction-label pair is formed; and the N8 fifth prediction-label pairs obtained this time form a corresponding prediction-label pair set. The N8 sixth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0239] Step 51525, divide each image x e The corresponding training map type is used as the current 3D image map and 2D / 3D task type input into the fifth training framework for processing, and the first predicted target set and the first predicted segmentation map output from the current processing are recorded as the corresponding prediction sets. and predicted segmentation map

[0240] Step 51526, for each tag set G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f ; and the prediction set In the middle and each label box e,f The nearest first object detection box is denoted as the corresponding matching box. and each label box e,fThe three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each type T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each label box e,f Corresponding size S e,f Center P e,f Type T e,f Vector Z e,f Form the corresponding feature group bt e,f ;

[0241] Step 51527, match each box The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimensions. center type and each type Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And composed of each matching box Corresponding size center vector Form corresponding feature groups

[0242] Step 51528, from each feature group bt e,f and its corresponding feature groups A corresponding seventh prediction-label pair is formed; and each label segmentation map y is used to form a corresponding seventh prediction-label pair. e and its corresponding predicted segmentation map Form a corresponding eighth prediction-label pair; and use the N3×N obtained this time. e Each seventh prediction-label pair forms a corresponding set of prediction-label pairs. The N3 eighth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0243] Step 51529: The obtained prediction-label pair set... Substitute the preset multi-task loss function L MT The corresponding sixth loss value is obtained through calculation;

[0244] Here, the multi-task loss function L in this embodiment of the invention... MT The 2D task loss function L 2D and the 3D task loss function L 3DComposed of various components, including the 2D task loss function L. 2D The corresponding first and second 2D task loss functions Combining these elements, the 3D task loss function L... 3D Based on the corresponding first and second 3D task loss functions Composed of various elements; this multi-task loss function L MT Specifically:

[0245]

[0246] Among them, w 2D w 3D These are two preset weight parameters;

[0247] Step 51530: Identify whether the sixth loss value meets the preset range of the sixth loss value; if not, then based on the preset sixth model optimizer, move towards making the multi-task loss function L... MT The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 51523 when the modulation ends; if satisfied, the fifth step of training is confirmed to be over.

[0248] The sixth loss value range is a pre-set numerical range; the sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0249] Step 52, and at the end of the fifth step of training, confirm that the image pre-trained model has completed the domain-specific optimization training in the medical field.

[0250] Figure 3 This is a module structure diagram of a training device for an image pre-training model in the medical field, provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 3 As shown, the device includes: a model preparation module 201, a basic data preparation module 202, a training framework preparation module 203, a training dataset preparation module 204, and a five-step training module 205.

[0251] The model preparation module 201 is used to select a visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-trained model.

[0252] The basic data preparation module 202 is used to use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; and to configure the corresponding image type set for the 2D waveforms as the corresponding waveform type set; and to set the corresponding diagnostic disease type set for each type of waveform in the waveform type set as the corresponding first disease type set, and to form the corresponding waveform-disease type set by all the obtained first disease type sets; and to configure the corresponding image type set and anatomical structure type set for the 3D images as the corresponding image type set and anatomical structure type set.

[0253] The training framework preparation module 203 is used to select four types of downstream task head models and connect them with the image pre-training model in separate and collective connection methods to obtain the corresponding five types of training frameworks. The four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head. The five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0254] The training dataset preparation module 204 is used to construct the first dataset based on the waveform image type set and the waveform-disease type set through big data acquisition; and to construct the second dataset based on the image image type set and the anatomical structure type set through big data acquisition.

[0255] The five-step training module 205 is used to perform five-step training in a progressive manner based on two datasets and five training frameworks; and at the end of the fifth step of training, it confirms that the image pre-trained model has completed domain-specific optimization training in the medical field.

[0256] The present invention provides a training device for image pre-training models in the medical field, which can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0257] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the model preparation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0258] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0259] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0260] Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 4 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0261] exist Figure 4The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0262] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0263] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0264] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for image pre-training models in the medical field. As described above, this invention first selects a type of pre-trained LVMs that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; it then configures a waveform type set for the 2D waveforms supported by the visual pre-training model in the medical field, and sets disease type subsets for each type of waveform to obtain a corresponding full waveform-disease type set; and it configures corresponding image type sets and anatomical structure type sets for the 3D images supported by the visual pre-training model in the medical field; next, it selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head) and interfaces them with the image pre-training model to obtain five training frameworks; then, it constructs a first dataset based on the waveform type set and waveform-disease type set, and a second dataset based on the image type set and anatomical structure type set through big data collection; then, it performs five steps of training based on the two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it confirms that the targeted training of the image pre-training model in the medical field is complete. In this invention, on the one hand, an image pre-trained model that is compatible with both 2D and 3D images and supports multiple waveforms and image types is obtained through SDT training in the medical field and serves as a shared encoding platform; on the other hand, the overall training difficulty is reduced and the overall training efficiency is improved through a progressive training method; and furthermore, the economic benefits of SDT training are improved by simultaneously training four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head).

[0265] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0266] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for an image pre-training model used in the medical field, characterized in that, The method includes: Choose a visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; The 2D waveforms and 3D images in the medical field are used as the 2D and 3D images supported by the visual pre-training model; a corresponding image type set is configured for the 2D waveforms, denoted as the corresponding waveform type set; a corresponding diagnostic disease type set is set for each type of waveform in the waveform type set, denoted as the corresponding first disease type set, and all the obtained first disease type sets constitute the corresponding waveform-disease type set; a corresponding image type set and anatomical structure type set are configured for the 3D images, denoted as the corresponding image type set and anatomical structure set; Four types of downstream task head models are selected and connected to the image pre-training model through separate and collective connection methods to obtain five corresponding training frameworks; the four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks. A first dataset is constructed based on the waveform type set and the waveform-disease type set using big data acquisition methods; and a second dataset is constructed based on the image type set and the anatomical structure type set using big data acquisition methods. The training is conducted in five steps using a progressive approach, based on two datasets and the five training frameworks. At the end of the fifth step, the pre-trained image model is confirmed to have completed its domain-specific optimization training in the medical field.

2. The training method for an image pre-training model in the medical field according to claim 1, characterized in that, The visual pre-trained model is a type of large visual model compatible with both 2D and 3D modalities and has been pre-trained based on massive image corpora in general domains; the large visual model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model; The waveform types in the waveform type set include at least electrocardiogram (ECG), electroencephalogram (EEG), and electromyogram (EMG). The waveform-disease category set includes multiple first disease category sets; the disease categories in the first disease category set corresponding to the electrocardiogram include at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block; the disease categories in the first disease category set corresponding to the electroencephalogram include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, and mental abnormalities; the disease categories in the first disease category set corresponding to the electromyogram include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis, Guillain-Barré syndrome, polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve palsy. The image types in the image type set include at least computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional ultrasound images, and positron emission tomography (PET) images. The set of anatomical structures includes at least a variety of organ types; the organ types include at least the heart, lungs, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, and bladder; The first training framework is formed by connecting the output end of the image pre-training model and the input end of the waveform classification task head; the first training framework is used to perform waveform classification prediction processing based on the 2D waveform input by the framework and output the corresponding first prediction vector; wherein, the first prediction vector is composed of multiple first prediction probabilities, each first prediction probability corresponds to a first prediction type; the first prediction type includes unknown waveform types and multiple known waveform types; the known waveform types correspond one-to-one with the waveform types in the waveform type set. The second training framework is formed by connecting the output of the image pre-trained model and the input of the waveform disease prediction task head; the second training framework is used to perform disease classification prediction processing based on the 2D waveform image input by the framework and output the corresponding second prediction vector; wherein, the second prediction vector is composed of multiple second prediction probabilities, each second prediction probability corresponds to a second prediction type; the second prediction type includes unknown disease types and multiple known disease types; the known disease types correspond one-to-one with the disease types in the waveform-disease type set; The third training framework is formed by connecting the output end of the image pre-training model and the input end of the image target detection task head; the third training framework is used to perform organ target detection and classification recognition processing based on the 3D image map input by the framework and output the corresponding first predicted target set; wherein, the first predicted target set consists of one or more first target detection boxes; the first target detection box includes at least a detection box identifier, a detection box three-dimensional size, a detection box center point coordinate, and a detection box type; the detection box type includes unknown target types and multiple known target types; the known target types correspond one-to-one with the organ types in the anatomical structure type set; The fourth training framework is formed by connecting the output end of the image pre-training model and the input end of the image semantic segmentation task head; the fourth training framework is used to perform voxel-level organ semantic segmentation processing based on the 3D image map input by the framework and output the corresponding first predicted segmentation map; wherein, the three-dimensional size of the first predicted segmentation map is consistent with the three-dimensional size of the 3D image map input by the current framework; the voxel-level features of the first predicted segmentation map include at least one semantic type; the semantic type includes background semantics, unknown organ semantics, and multiple known organ semantics, and the known organ semantics correspond one-to-one with the organ types in the anatomical structure type set. The fifth training framework consists of the image pre-training model, the feature distribution module, the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head. The fifth training framework is used to identify the 2D / 3D task type input by the current framework. If the current 2D / 3D task type is 2D, then waveform classification prediction and disease classification prediction are performed based on the 2D waveform image input by the current framework, and the corresponding first and second prediction vectors are output. If the current 2D / 3D task type is 3D, then organ target detection and classification recognition and organ semantic segmentation are performed based on the 3D image image input by the current framework, and the corresponding first prediction target set and first prediction segmentation map are output. The 2D / 3D task type includes both 2D and 3D types. When the current framework input image is a 2D waveform image, the corresponding 2D / 3D task type is 2D; when the current framework input image is a 3D image, the corresponding 2D / 3D task type is 3D. The first dataset includes multiple first data records; each first data record includes a first training waveform, a first label vector, and a second label vector; the first training waveform is a waveform in a medical field; the first label vector consists of multiple first label probabilities, each first label probability corresponding to a first prediction type; the second label vector consists of multiple second label probabilities, each second label probability corresponding to a second prediction type; in both the first and second label vectors, only one label probability is 1, and the remaining label probabilities are 0; the total set of waveform types for all first training waveforms in the first dataset is greater than the set of waveform types; the total number of data records corresponding to each first prediction type in the first dataset is the same; the total number of data records corresponding to each second prediction type in the first dataset is the same. The second dataset includes multiple second data records; the second data records include a first training image, a first label target set, and a first label segmentation map; the first training image is a type of 3D image; the first label target set consists of one or more first label detection boxes; the first label detection box includes at least the detection box identifier, the three-dimensional size of the detection box, the coordinates of the detection box center point, and the detection box type; the three-dimensional size of the first label segmentation map is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation map include at least the semantic type; the image type set of all first training image images in the second dataset matches the image type set; the anatomical structure type set of all first training image images in the second dataset is larger than the anatomical structure type set; the total number of data records corresponding to each type of image image in the second dataset is the same, and the total number of detection boxes corresponding to each type of detection box type in all data records corresponding to each type of image image is the same, and the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same.

3. The training method for an image pre-training model in the medical field according to claim 2, characterized in that, In the first training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding first encoded feature tensor and send it to the waveform classification task head; the waveform classification task head is used to perform classification prediction processing based on the first encoded feature tensor and output the corresponding first prediction vector. In the second training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding second encoded feature tensor and send it to the waveform disease prediction task head; the waveform disease prediction task head is used to perform classification prediction processing based on the second encoded feature tensor and output the corresponding second prediction vector. In the third training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input to the current framework to obtain the corresponding third encoded feature tensor and send it to the image target detection task head; the image target detection task head is used to perform target detection and classification processing based on the third encoded feature tensor and output the corresponding first predicted target set; In the fourth training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input by the current framework to obtain the corresponding fourth encoded feature tensor and send it to the image semantic segmentation task head; the image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the fourth encoded feature tensor and output the corresponding first predicted segmentation map. The first frame input terminal of the fifth training frame is used to receive the 2D waveform or the 3D image input of the current frame, the second frame input terminal is used to receive the 2D / 3D task type input of the current frame, and the first, second, third and fourth frame output terminals are used to output the corresponding first prediction vector, second prediction vector, first prediction target set and first prediction segmentation map. In the fifth training framework, the input end of the image pre-training model is connected to the input end of the first framework, and the output end is connected to the first input end of the feature distribution module; the second input end of the feature distribution module is connected to the input end of the second framework, and the output end is connected to the input ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head, and the image semantic segmentation task head, respectively; the output ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head, and the image semantic segmentation task head are respectively connected to the output ends of the corresponding first, second, third, and fourth frameworks. In the fifth training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform or 3D image input to the current framework to obtain the corresponding fifth encoded feature tensor, which is then sent to the feature distribution module. The feature distribution module is used to identify the 2D / 3D task type input to the current framework. If the current 2D / 3D task type is 2D, the fifth encoded feature tensor is sent to the waveform classification task head and the waveform disease prediction task head. If the current 2D / 3D task type is 3D, the fifth encoded feature tensor is sent to the image target detection task head and the image semantic segmentation task head. The waveform classification task head is used to receive... When the fifth encoded feature tensor is received, classification prediction processing is performed based on the current encoded feature tensor, and the corresponding first prediction vector is output. The waveform disease prediction task head is used to perform classification prediction processing based on the current encoded feature tensor and output the corresponding second prediction vector when the fifth encoded feature tensor is received. The image target detection task head is used to perform target detection and classification recognition processing based on the current encoded feature tensor and output the corresponding first predicted target set when the fifth encoded feature tensor is received. The image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the current encoded feature tensor and output the corresponding first predicted segmentation map when the fifth encoded feature tensor is received.

4. The training method for an image pre-training model in the medical field according to claim 2, characterized in that, The five-step training based on the two datasets and the five training frameworks specifically includes: The first step of training is performed based on the first dataset and the first training framework; After the first step of training is completed, the model parameters of the waveform classification task head are fixed; and the second step of training is carried out based on the first dataset and the second training framework. After the second training step is completed, the model parameters of the waveform disease prediction task head are fixed; and the third training step is performed based on the second dataset and the third training framework. After the third training step is completed, the model parameters of the image target detection task head are fixed; and the fourth training step is performed based on the second dataset and the fourth training framework. After the fourth step of training is completed, the model parameters of the image semantic segmentation task head are fixed; and the fifth step of training is carried out based on the first and second datasets and the fifth training framework.

5. The training method for an image pre-training model in the medical field according to claim 4, characterized in that, The first step of training based on the first dataset and the first training framework specifically includes: Step 51: Based on a preset first segmentation ratio, the first dataset is randomly divided into two subsets, denoted as the first training set and the first evaluation set; and the combined set of model parameters of the image pre-trained model and the waveform classification task head is used as the corresponding first frame parameter set. Wherein, both the first training set and the first evaluation set are composed of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set satisfies the first segmentation ratio; Step 52: Count the total number of records in the first training set to obtain the corresponding total number N1; and count the total number of types of the first prediction type to obtain the corresponding total number C1; and denote the first training waveform and the first label vector of each first data record in the first training set as the corresponding waveform x. a and label vector y a 1 ≤ index a ≤ N1; and each of the waveforms x a The 2D waveform is input into the first training framework for processing, and the first prediction vector output from the current processing is recorded as the corresponding prediction vector. and the label vector y a The probabilities of each of the first labels are denoted as y. a,b and the prediction vector Each of the first predicted probabilities is denoted as the corresponding 1 ≤ index b ≤ C1; and by each of the predicted vectors and the label vector y a Form a corresponding first prediction-label pair; and input the resulting N1 first prediction-label pairs into a preset first model loss function L. M1 The corresponding first loss value is obtained through calculation; Wherein, the first model loss function L M1 It is implemented based on the cross-entropy loss function, specifically as follows: Step 53: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 54; if not, based on the preset first model optimizer, move towards making the first model loss function L... M1 The direction that reaches the minimum value is modulated once for the first frame parameter set, and the process returns to step 52 when the modulation ends. The first model optimizer includes at least the Adam optimizer and the SGD optimizer; Step 54: Perform a round of traversal on all the first data records in the first evaluation set; during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; input the first training waveform of the current evaluation record as the current 2D waveform into the first training framework for processing, and record the first prediction vector output by the current processing as the corresponding first prediction vector; form a corresponding second prediction-label pair with the first prediction vector and the first label vector of the current evaluation record; and at the end of this round of traversal, estimate the accuracy, precision, recall and F1 score based on all the obtained second prediction-label pairs to obtain the corresponding first accuracy, first precision, first recall and first F1 score; Step 55: Identify whether the first accuracy, first precision, first recall, and first F1 score all satisfy their respective first accuracy range, first precision range, first recall range, and first F1 score range; if not, return to step 52 to continue training; if yes, confirm that the first step of training is complete.

6. The training method for an image pre-training model in the medical field according to claim 4, characterized in that, The second training step, based on the first dataset and the second training framework, specifically includes: Step 61: Based on the preset second segmentation ratio, the first dataset is randomly divided into two subsets, which are denoted as the corresponding second training set and second evaluation set; and the model parameter set of the image pre-trained model and the waveform disease prediction task head is used as the corresponding second frame parameter set. The second training set and the second evaluation set are both composed of multiple first data records; the ratio of the total number of records in the second training set and the second evaluation set satisfies the second segmentation ratio. Step 62: Count the total number of records in the second training set to obtain the corresponding total number N2; and count the total number of types of the second prediction type to obtain the corresponding total number C2; and denote the first training waveform and the second label vector of each first data record in the second training set as the corresponding waveform x. c and label vector y c 1 ≤ index c ≤ N2; and each of the waveforms x c The 2D waveform is input into the second training framework for processing, and the second prediction vector output from the current processing is recorded as the corresponding prediction vector. and the label vector y c The probabilities of each of the second labels are denoted as y. c,d and the prediction vector Each of the second prediction probabilities is denoted as the corresponding 1 ≤ index d ≤ C2; and by each of the predicted vectors and the label vector y c A corresponding third prediction-label pair is formed; and the resulting N2 third prediction-label pairs are input into a preset second model loss function L. M2 The corresponding second loss value is obtained through calculation; Wherein, the second model loss function L M2 It is implemented based on the cross-entropy loss function, specifically as follows: Step 63: Identify whether the second loss value meets the preset second loss value range; if it does, proceed to step 64; if not, based on the preset second model optimizer, move towards making the second model loss function L... M2 The direction that reaches the minimum value is modulated once for the second frame parameter set, and the process returns to step 62 when the current modulation is completed; The second model optimizer includes at least the Adam optimizer and the SGD optimizer; Step 64: Perform a round of traversal on all the first data records in the second evaluation set; during this round of traversal, take the currently traversed first data record as the corresponding current evaluation record; input the first training waveform of the current evaluation record as the current 2D waveform into the second training framework for processing, and record the second prediction vector output by the current processing as the corresponding second prediction vector; and form a corresponding fourth prediction-label pair by the second prediction vector and the second label vector of the current evaluation record; and at the end of this round of traversal, estimate the accuracy, precision, recall and F1 score based on all the obtained fourth prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score; Step 65: Identify whether the second accuracy, second precision, second recall, and second F1 score all satisfy their respective second accuracy range, second precision range, second recall range, and second F1 score range; if not, return to step 62 to continue training; if yes, confirm that the second step of training is complete.

7. The training method for an image pre-training model in the medical field according to claim 4, characterized in that, The third training step, based on the second dataset and the third training framework, specifically includes: Step 71: Count the total number of records in the second dataset to obtain the corresponding total number N3; and count the total number of organ types in the anatomical structure type set to obtain the corresponding total number N4; and use the combined set of model parameters of the image pre-training model and the image target detection task head as the corresponding third frame parameter set; Step 72: Denote the first training image and the first label target set of each second data record in the second dataset as the corresponding image image x. e and the set of tags G e 1 ≤ index e ≤ N3; and each of the aforementioned image maps x e The 3D image is input into the third training framework for processing, and the first set of predicted targets output in this processing is recorded as the corresponding prediction set. Step 73, for each of the said tag sets G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f 1 ≤ index f ≤ N e ; and the prediction set In and each of the aforementioned label boxes e,f The nearest first target detection box is denoted as the corresponding matching box. Step 74, set each of the aforementioned label boxes. e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each of the types T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,f The corresponding size S e,f The center P e,f The type T e,f The vector Z e,f Form the corresponding feature group bt e,f ; Wherein, the vector Z e,f Includes N4 one-hot codes z e,f,g , 1 ≤ index g ≤ N4; the vector Z e,f Only those of type T are mentioned. e,f The corresponding one-hot encoding z e,f,g The value is 1, and all others are 0; Step 75, match each of the aforementioned matching boxes. The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. center type and each of the types Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And by each of the aforementioned matching boxes The corresponding size The center The vector Form corresponding feature groups Wherein, the vector Includes N4 one-hot codes The vector Only those of the type mentioned The corresponding one-hot encoding The value is 1, and all others are 0; Step 76, divide N3×N e For the feature group bt e,f , Substitute the preset third model loss function L M3 The corresponding third loss value is obtained through calculation; Wherein, the third model loss function L M3 It is implemented by combining the cross-entropy loss function for classification, the L2 loss function for center position, and the L1 loss function for 3D dimensions, specifically: Step 77: Identify whether the third loss value meets the preset range of the third loss value; if not, then based on the preset third model optimizer, move towards making the third model loss function L... M3 The direction that reaches the minimum value is modulated once for the third frame parameter set, and the process returns to step 72 when the modulation ends; if the condition is met, the third step of training is confirmed to be over. The third model optimizer includes at least the Adam optimizer and the SGD optimizer.

8. The training method for an image pre-training model in the medical field according to claim 7, characterized in that, The fourth training step, based on the second dataset and the fourth training framework, specifically includes: The combined set of model parameters of the image pre-trained model and the image semantic segmentation task head is used as the corresponding fourth frame parameter set; With the goal of improving the feature consistency of multimodal images under the same anatomical structure, a round of self-supervised comparative learning training is performed on the model parameters of the image pre-training model based on the second dataset and the fourth training framework. After this round of comparative learning training, with the goal of improving segmentation consistency, a round of supervised model training is carried out on the parameter set of the fourth training framework based on the second dataset and the fourth training framework. And confirm the completion of the fourth step of training after the completion of this round of supervised training.

9. The training method for an image pre-training model in the medical field according to claim 8, characterized in that, The step of performing a round of self-supervised comparative learning training on the model parameters of the image pre-training model based on the second dataset and the fourth training framework, with the goal of improving the feature consistency of multimodal images under the same anatomical structure, specifically includes: Step 91: Record each of the first training image images containing only one type of organ in the second dataset as a corresponding single-organ image image, and count the total number of the single-organ image images to obtain the corresponding total number N5. Then, form a corresponding multi-organ image set X from the N5 single-organ image images. ALL And each of the single organ images is denoted as the corresponding image image x. i , 1 ≤ index i ≤ N5; Step 92, and process the multi-organ image set X ALL The total number of organ types was statistically analyzed to obtain the corresponding total number N6; and the multi-organ image set X was then used to obtain the total number of organ types. ALL Multiple single-organ images corresponding to the same type of organ are clustered together to form a corresponding single-organ image set X. j 1 ≤ index j ≤ N6; and for each of the single organ image sets X i The total number of images is counted to obtain the corresponding total number N. 7,j ; Step 93, and then each of the aforementioned image maps x i As the corresponding current image; and the single organ image set X to which the current image is located is used. j Let X be the corresponding current positive sample set; and let X be the set of all other single organ images excluding the current positive sample set. j The corresponding current negative sample set is formed; and each of the single organ images in the current positive sample set, excluding the current image image, is taken as the positive sample image corresponding to the current image image. 1≤Positive Sample Index The total number of positive samples corresponding to the current image. The value is the total number N corresponding to the current positive sample set. 7,j The value is reduced by 1; and each of the single organ images in the current negative sample set is taken as the negative sample image corresponding to the current image. 1 ≤ negative sample index n(i) ≤ The total number of negative samples corresponding to the current image. Step 94, transfer each of the image images x i The current 3D image is input into the image pre-training model for feature encoding processing, and the encoded feature tensor output from this processing is used as the corresponding feature tensor y. i ; and each of the aforementioned image maps x i The corresponding positive sample images The corresponding encoded feature tensor is denoted as the corresponding positive sample tensor. and each of the aforementioned image images x i The corresponding negative sample images The corresponding encoded feature tensor is denoted as the corresponding negative sample tensor. and each of the aforementioned feature tensors y i corresponding The positive sample tensor Form the corresponding positive sample set Each of the aforementioned feature tensors y i corresponding The negative sample tensor Form the corresponding negative sample set and each of the aforementioned feature tensors y i The three-dimensional dimension is denoted as the corresponding height H. i Width W i and depth D i and each of the aforementioned feature tensors y i Considered to be by H i ×W i ×D i Individual tensor The resulting 3D feature map, 1 ≤ index h i ≤H i , 1 ≤ index w i ≤W i 1 ≤ index d i ≤D i ; Step 95, convert the N5 feature tensors y obtained this time into... i and the corresponding positive and negative sample sets Substitute the preset fourth model loss function L M4 The corresponding fourth loss value is obtained through calculation; Wherein, the fourth model loss function L M4 It is implemented by combining the InfoNCE loss function and the non-integer order total variation loss function, specifically as follows: λ is a preset tradeoff coefficient; L infoNCE () represents the InfoNCE loss function; sim() represents a type of similarity function, which includes at least the dot product similarity function, the cosine vector similarity function, and the Euclidean distance similarity function; τ represents a preset temperature coefficient; L FOTV () represents the non-integer order total variation loss function; α is a preset non-integer order parameter; For the α-order derivative operator along the three-dimensional direction, specifically: These are the α-order derivative operators along the height, width, and depth directions, respectively; Step 96: Identify whether the fourth loss value meets the preset range of the fourth loss value; if not, then based on the preset fourth model optimizer, move towards making the fourth model loss function L... M4 The direction that reaches the minimum value modulates the model parameters of the image pre-training model in one round, and returns to step 94 when the modulation ends; if satisfied, the contrastive learning training is confirmed to be over. The fourth model optimizer includes at least the Adam optimizer and the SGD optimizer.

10. The training method for an image pre-training model in the medical field according to claim 8, characterized in that, The step of performing a supervised model training on the parameter set of the fourth training framework based on the second dataset and the fourth training framework with the goal of improving segmentation consistency specifically includes: Step 101: Record the first training image and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e and label segmentation map y e ; and each of the aforementioned image maps x e The current 3D image is input into the fourth training framework for processing, and the first predicted segmentation map output from the current processing is recorded as the corresponding predicted segmentation map. Step 102, use the N3 pairs obtained this time to match the labels and the predicted segmentation map y e , Substitute the preset fifth model loss function L M5 The corresponding fifth loss value is obtained through calculation; Wherein, the fifth model loss function L M5 Based on the Dice loss function L Dice Implementation, specifically: Step 103: Identify whether the fifth loss value meets the preset range of the fifth loss value; if not, then based on the preset fifth model optimizer, move towards making the fifth model loss function L... M5 The direction that reaches the minimum value is modulated once for the fourth frame parameter set, and the process returns to step 101 when the modulation ends; if the condition is met, the supervised training for this round is confirmed to be over. The fifth model optimizer includes at least the Adam optimizer and the SGD optimizer.

11. The training method for an image pre-training model in the medical field according to claim 10, characterized in that, The fifth training step, based on the first and second datasets and the fifth training framework, specifically includes: Step 111: Calculate the total number of records in the first dataset to obtain the corresponding total number N8; and calculate the total number of types for the first prediction type to obtain the corresponding total number. The total number of types in the second prediction type is then statistically analyzed to obtain the corresponding total number. And denote the first training waveform, the first label vector, and the second label vector of each first data record in the first dataset as the corresponding waveform x. m Label vector y 1,m Label vector y 2,m 1 ≤ index m ≤ N8; and for each of the aforementioned waveforms x m Set a specific 2D training graph type; Step 112: Record the first training image, the first label target set, and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e The tag set G e and the label segmentation image y e ; and for each of the aforementioned image maps x e Set a specific training graph type as 3D; Step 113, convert each of the waveforms x m The training graph type and its corresponding type are input into the fifth training framework as the current 2D waveform graph and 2D / 3D task type for processing, and the first and second prediction vectors output in this processing are recorded as the corresponding prediction vectors. and prediction vector Step 114, the label vector y 1,m The probabilities of each of the first labels are denoted as y. 1,m,s1 and the prediction vector Each of the first predicted probabilities is denoted as the corresponding 1≤index And by each of the predicted vectors and the label vector y 1,m Form a corresponding fifth prediction-label pair; and then use the label vector y 2,m The probabilities of each of the second labels are denoted as y. 2,m,s2 and the prediction vector Each of the second prediction probabilities is denoted as the corresponding 1≤index And by each of the predicted vectors and the label vector y 2,m A corresponding sixth prediction-label pair is formed; and the N8 fifth prediction-label pairs obtained this time form a corresponding prediction-label pair set. The N8 sixth prediction-label pairs obtained in this study form a corresponding prediction-label pair set. Step 115, transfer each of the aforementioned image maps x e The training map type and its corresponding type are input into the fifth training framework as the current 3D image map and the 2D / 3D task type for processing, and the first predicted target set and the first predicted segmentation map output in this processing are recorded as the corresponding prediction set. and the predicted segmentation map Step 116, for each of the said tag sets G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f ; and the prediction set In and each of the aforementioned label boxes e,f The nearest first target detection box is denoted as the corresponding matching box. and each of the aforementioned label boxes e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f The center P e,f The type T e,f ; and each of the types T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,f The corresponding size S e,f The center P e,f The type T e,f The vector Z e,f The corresponding feature group bt e,f ; Step 117, match each of the aforementioned matching boxes The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. The center The type and each of the types This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector. And by each of the aforementioned matching boxes The corresponding size The center The vector The corresponding feature groups are formed Step 118, from each of the aforementioned feature groups bt e,f and the corresponding feature groups A corresponding seventh prediction-label pair is formed; and each of the label segmentation maps y e and the corresponding predicted segmentation map Form a corresponding eighth prediction-label pair; and use the N3×N obtained this time. e Each of the seventh prediction-label pairs forms a corresponding prediction-label pair set. The N3 eighth prediction-label pairs obtained in this study form a corresponding prediction-label pair set. Step 119, the predicted-label pair set obtained this time... Substitute the preset multi-task loss function L MT The corresponding sixth loss value is obtained through calculation; Wherein, the multi-task loss function L MT The 2D task loss function L 2D and the 3D task loss function L 3D Combining these elements, the 2D task loss function L... 2D The corresponding first and second 2D task loss functions The 3D task loss function L is composed of these components. 3D Based on the corresponding first and second 3D task loss functions Composed of various elements; the multi-task loss function L MT Specifically: w 2D w 3D These are two preset weight parameters; Step 120: Identify whether the sixth loss value meets the preset range of the sixth loss value; if not, then based on the preset sixth model optimizer, move towards making the multi-task loss function L... MT The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 113 when the modulation ends; if satisfied, the fifth step of training is confirmed to be over. The sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

12. An apparatus for performing the training method of an image pre-training model for the medical field according to any one of claims 1-11, characterized in that, The device includes: a model preparation module, a basic data preparation module, a training framework preparation module, a training dataset preparation module, and a five-step training module; The model preparation module is used to select a type of visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-trained model. The basic data preparation module is used to use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; and to configure a corresponding image type set for the 2D waveforms, denoted as the corresponding waveform type set; and to set a corresponding diagnostic disease type set for each type of waveform in the waveform type set, denoted as the corresponding first disease type set, and to form a corresponding waveform-disease type set by all the obtained first disease type sets; and to configure a corresponding image type set and anatomical structure type set for the 3D images, denoted as the corresponding image type set and anatomical structure set; The training framework preparation module is used to select four types of downstream task head models and connect them to the image pre-training model in separate and collective connection methods to obtain five corresponding training frameworks; the four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks. The training dataset preparation module is used to construct a first dataset based on the waveform type set and the waveform-disease type set through big data acquisition; and to construct a second dataset based on the image type set and the anatomical structure type set through big data acquisition. The five-step training module is used to perform five-step training based on two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it is confirmed that the image pre-trained model has completed the domain-specific optimization training in the medical field.

13. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-11; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1-11.

Images