A training method and device of an image pre-training model for a medical field

Abstract

Embodiments of the present application relate to a kind of training method and device of image pre-training model for medical field, the method comprises: selecting a kind of visual pre-training model as image pre-training model;Setting waveform chart category set, waveform-disease category set, image chart category set and anatomical structure category set;Select four categories downstream task head model and image pre-training model are connected to obtain five kinds of training framework;According to waveform chart category set and waveform-disease category set, first data set is constructed by big data acquisition mode, according to image chart category set and anatomical structure category set, second data set is constructed;According to two data sets and five kinds of training framework, five-step training is carried out in the training mode of step-by-step progression;And confirm the specific field optimization training of image pre-training model in medical field ends at the end of fifth step training.Through the present application, the training difficulty of image pre-training model can be reduced, and training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a training method and device of an image pre-training model for the medical field. BACKGROUND

[0002] In the medical artificial intelligence (AI) field, a single modality (2D medical waveform graph or 3D medical image graph) visual model is usually used for task processing, and each single modality visual model usually only supports a specific type of waveform or image. The specific waveform type of the 2D medical waveform graph (2D waveform graph for short) mentioned here is commonly electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), etc., and the specific image type of the 3D medical image graph (3D image graph for short) mentioned here is commonly computed tomography (CT) graph, magnetic resonance imaging (MRI) graph, three-dimensional ultrasound imaging (3D US) graph, positron emission tomography (PET) graph, etc.

[0003] The conventional single modality task processing mechanism cannot provide a shared coding platform that can simultaneously support 2D / 3D images and support multiple types of waveforms and multiple types of images, so it is necessary to customize, train and maintain a single modality task model for each type of 2D / 3D task. This not only causes the model development cycle to grow, but also causes the redundant development workload to increase, and also causes the model maintenance cost to increase. If an image pre-training model that can simultaneously support 2D / 3D images and support multiple types of waveforms and multiple types of images is provided as a shared coding platform, then when a new 2D / 3D task requirement is added, only a simple downstream task head needs to be designed to interface with the image pre-training model, and in terms of training, only the downstream task head needs to be primarily debugged based on the output requirements of the new task, and the image pre-training model only needs to be adaptively fine-tuned. In this way, not only can the development cycle be reduced, but also the redundant development workload can be reduced, and the development flexibility can be improved and the model maintenance cost can be reduced.

[0004] At present, there are many mature pre-training visual large models (LVMs) that can be well compatible with 2D / 3D images, such as MultiResViT, UniFormer, Hybrid CNN-Transformer, nnU-Net, Uni3D, Uni-MAE, etc., but most of these pre-training LVMs have not been trained for specific domain tuning (SDT) in the medical field. That is to say, if these pre-training LVMs are directly used as image pre-training models in the medical field, the expected effect of the shared coding platform cannot be achieved. Therefore, how to perform SDT training on the image pre-training model in the medical field is a technical problem to be solved by the present application. SUMMARY

[0005] The present application is aimed at the defects of the prior art and provides a training method, device, electronic equipment and computer readable storage medium for an image pre-training model in the medical field. The present application first selects a type of pre-training LVMs that can simultaneously support 2D / 3D image feature coding as an image pre-training model; configures a waveform type set for the 2D waveform in the medical field supported by the visual pre-training model and sets a disease type subset for each type of waveform to obtain a corresponding full waveform-disease type set, and configures a corresponding image type set and an anatomical structure type set for the 3D image in the medical field supported by the visual pre-training model; then selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image target detection task head, and image semantic segmentation task head) to interface with the image pre-training model to obtain five types of training frameworks; then constructs a first data set according to the waveform type set and the waveform-disease type set and a second data set according to the image type set and the anatomical structure type set through a big data acquisition method; then performs five-step training according to the two data sets and the five types of training frameworks in a step-by-step progressive training manner; and confirms that the directional training of the image pre-training model in the medical field is completed at the end of the fifth step of training. Through the present application, on the one hand, an image pre-training model that is compatible with 2D / 3D images and supports multiple types of waveforms and multiple types of images can be obtained through SDT training in the medical field as a shared coding platform; on the other hand, the training difficulty can be reduced and the training efficiency can be improved through the step-by-step progressive training manner; and on the other hand, the four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image target detection task head, and image semantic segmentation task head) can be simultaneously trained, thereby improving the economic benefits of SDT training.

[0006] To achieve the above-mentioned object, a first aspect of an embodiment of the present application provides a training method for an image pre-training model in the medical field, which comprises:

[0007] A visual pre-training model capable of supporting 2D / 3D image feature encoding is selected as an image pre-training model;

[0008] 2D waveform graphs and 3D image graphs in the medical field are selected as 2D and 3D images supported by the visual pre-training model; a corresponding image type set is configured for the 2D waveform graphs, denoted as a corresponding waveform graph category set; a corresponding disease category set is set for each type of waveform graph in the waveform graph category set, denoted as a corresponding first disease category set, and a waveform-disease category set is formed by all the first disease category sets obtained; a corresponding image type set and an anatomical structure type set are configured for the 3D image graphs, denoted as a corresponding image graph category set and an anatomical structure category set;

[0009] Four types of downstream task head models are selected to be connected to the image pre-training model in a separate or collective manner to obtain five types of training frameworks; the four types of downstream task head models include a waveform classification task head, a waveform disease prediction task head, an image target detection task head, and an image semantic segmentation task head; the five types of training frameworks include first, second, third, fourth, and fifth training frameworks;

[0010] A first data set is constructed according to the waveform graph category set and the waveform-disease category set through a big data collection method; and a second data set is constructed according to the image graph category set and the anatomical structure category set through a big data collection method;

[0011] Five-step training is performed according to the two data sets and the five types of training frameworks in a step-by-step training manner; and at the end of the fifth step of training, it is confirmed that the specific field optimization training of the image pre-training model in the medical field is completed.

[0012] Preferably, the visual pre-training model is a visual large model compatible with 2D and 3D modalities and has been pre-trained based on a large amount of image corpus in a general field; the visual large model at least includes a MultiResViT model, a UniFormer model, a Hybrid CNN-Transformer model, an nnU-Net model, a Uni3D model, a Uni-MAE model, and a MedFMC model.

[0013] The waveform graph category set includes at least electrocardiogram, electroencephalogram, and electromyogram;

[0014] The waveform-disease category set includes a plurality of the first disease category set; the disease categories of the first disease category set corresponding to electrocardiogram at least include atrial fibrillation, premature beat, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, cardiac conduction block; the disease categories of the first disease category set corresponding to electroencephalogram at least include epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, brain trauma, schizophrenia, manic depression, mental abnormality; the disease categories of the first disease category set corresponding to electromyogram at least include myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis, Guillain-Barre syndrome, multiple peripheral neuropathy, oculomotor nerve injury, trochlear nerve injury, facial neuritis;

[0015] The image graph category set at least includes computed tomography, magnetic resonance imaging, three-dimensional ultrasonic imaging, positron emission tomography;

[0016] The anatomical structure category set at least includes a plurality of organ types; the organ types at least include heart, lung, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, urinary bladder;

[0017] The first training framework is connected by the output end of the image pre-training model and the input end of the waveform classification task head; the first training framework is used for waveform graph classification prediction processing according to the 2D waveform graph of the framework input and outputs the corresponding first prediction vector; wherein the first prediction vector is composed of a plurality of first prediction probabilities, each first prediction probability corresponds to a first prediction type; the first prediction type includes unknown waveform type and multiple known waveform types; the known waveform type is one-to-one corresponding to the waveform graph category in the waveform graph category set;

[0018] The second training framework is connected by the output end of the image pre-training model and the input end of the waveform disease prediction task head; the second training framework is used for disease classification prediction processing according to the 2D waveform graph of the framework input and outputs the corresponding second prediction vector; wherein the second prediction vector is composed of a plurality of second prediction probabilities, each second prediction probability corresponds to a second prediction type; the second prediction type includes unknown disease type and multiple known disease types; the known disease type is one-to-one corresponding to the disease category in the waveform-disease category set;

[0019] The third training framework is connected by an output end of the image pre-training model and an input end of the image target detection task head; the third training framework is used for organ target detection and classification identification processing according to the 3D image input by the framework and outputs a corresponding first prediction target set; wherein the first prediction target set is composed of one or more first target detection boxes; the first target detection box at least includes a detection box identifier, a detection box three-dimensional size, a detection box center point coordinate and a detection box type; the detection box type includes an unknown target type and a plurality of known target types; the known target type is one-to-one corresponding to the organ type of the anatomical structure type set;

[0020] The fourth training framework is connected by an output end of the image pre-training model and an input end of the image semantic segmentation task head; the fourth training framework is used for organ semantic segmentation processing at a voxel level according to the 3D image input by the framework and outputs a corresponding first prediction segmentation map; wherein the three-dimensional size of the first prediction segmentation map is consistent with the three-dimensional size of the 3D image input by the current framework; the voxel level feature of the first prediction segmentation map at least includes a semantic type; the semantic type includes a background semantic, an unknown organ semantic and a plurality of known organ semantics, and the known organ semantic is one-to-one corresponding to the organ type of the anatomical structure type set;

[0021] The fifth training framework is composed of the image pre-training model, a feature distribution module, the waveform classification task head, the waveform disease prediction task head, the image target detection task head and the image semantic segmentation task head;

[0022] The fifth training framework is used for identifying the 2D / 3D task type input by the current framework; if the current 2D / 3D task type is a 2D type, then waveform graph classification prediction and disease classification prediction processing are performed according to the 2D waveform graph input by the current framework and the corresponding first and second prediction vectors are output; if the current 2D / 3D task type is a 3D type, then organ target detection and classification identification and organ semantic segmentation processing are performed according to the 3D image input by the current framework and the corresponding first prediction target set and the first prediction segmentation map are output; wherein the 2D / 3D task type includes a 2D type and a 3D type, the 2D / 3D task type corresponding to the current framework input image being a 2D waveform graph is a 2D type, and the 2D / 3D task type corresponding to the current framework input image being a 3D image is a 3D type;

[0023] The first data set includes a plurality of first data records; the first data record includes a first training waveform graph, a first label vector and a second label vector; the first training waveform graph is a waveform graph in a medical field; the first label vector is composed of a plurality of first label probabilities, each first label probability corresponding to a first prediction type; the second label vector is composed of a plurality of second label probabilities, each second label probability corresponding to a second prediction type; only one label probability in the first and second label vectors is 1, and the rest of the label probabilities are 0; the waveform graph category set of all first training waveform graphs in the first data set is greater than the waveform graph category set; the total number of data records corresponding to each first prediction type in the first data set is the same; the total number of data records corresponding to each second prediction type in the first data set is the same;

[0024] The second data set includes a plurality of second data records; the second data record includes a first training image, a first label target set and a first label segmentation graph; the first training image is a 3D image; the first label target set is composed of one or more first label detection boxes; the first label detection box includes at least the detection box identifier, the detection box three-dimensional size, the detection box center point coordinate and the detection box type; the three-dimensional size of the first label segmentation graph is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation graph include at least the semantic type; the image graph category set of all first training images in the second data set matches the image graph category set; the anatomical structure category set of all first training images in the second data set is greater than the anatomical structure category set; the total number of data records corresponding to each image graph category in the second data set is the same, and the total number of detection boxes corresponding to each detection box type in all data records corresponding to each image graph category is the same, and the total number of image segmentation regions corresponding to each known organ semantic in all data records corresponding to each image graph category is the same.

[0025] Further, in the first training framework, the image pre-training model is used for feature encoding processing of the 2D waveform graph input by the current framework to obtain a corresponding first encoding feature tensor, which is sent to the waveform classification task head; the waveform classification task head is used for classification prediction processing according to the first encoding feature tensor and outputs a corresponding first prediction vector;

[0026] In the second training framework, the image pre-training model is configured to perform feature encoding processing on the 2D waveform graph input by the current framework to obtain a corresponding second encoding feature tensor, and send the second encoding feature tensor to the waveform disease prediction task head; the waveform disease prediction task head is configured to perform classification prediction processing according to the second encoding feature tensor and output a corresponding second prediction vector;

[0027] In the third training framework, the image pre-training model is configured to perform feature encoding processing on the 3D image graph input by the current framework to obtain a corresponding third encoding feature tensor, and send the third encoding feature tensor to the image target detection task head; the image target detection task head is configured to perform target detection and classification identification processing according to the third encoding feature tensor and output a corresponding first prediction target set;

[0028] In the fourth training framework, the image pre-training model is configured to perform feature encoding processing on the 3D image graph input by the current framework to obtain a corresponding fourth encoding feature tensor, and send the fourth encoding feature tensor to the image semantic segmentation task head; the image semantic segmentation task head is configured to perform voxel-level semantic segmentation processing according to the fourth encoding feature tensor and output a corresponding first prediction segmentation map;

[0029] The first framework input end of the fifth training framework is configured to receive the 2D waveform graph or the 3D image graph input by the current framework, the second framework input end is configured to receive the 2D / 3D task type input by the current framework, and the first, second, third and fourth framework output ends are configured to output the first prediction vector, the second prediction vector, the first prediction target set and the first prediction segmentation map respectively;

[0030] In the fifth training framework, the input end of the image pre-training model is connected with the first framework input end, and the output end is connected with the first input end of the feature distribution module; the second input end of the feature distribution module is connected with the second framework input end, and the output end is connected with the input ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head and the image semantic segmentation task head respectively; the output ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head and the image semantic segmentation task head are connected with the first, second, third and fourth framework output ends respectively;

[0031] In the fifth training framework, the image pre-training model is configured to perform feature encoding processing on the 2D waveform graph or the 3D image graph input by the current framework to obtain a corresponding fifth encoding feature tensor, and send the fifth encoding feature tensor to the feature distribution module; the feature distribution module is configured to identify the 2D / 3D task type input by the current framework, and if the current 2D / 3D task type is a 2D type, send the fifth encoding feature tensor to the waveform classification task head and the waveform disease prediction task head, and if the current 2D / 3D task type is a 3D type, send the fifth encoding feature tensor to the image target detection task head and the image semantic segmentation task head; the waveform classification task head is configured to, when receiving the fifth encoding feature tensor, perform classification prediction processing according to the current encoding feature tensor and output a corresponding first prediction vector; the waveform disease prediction task head is configured to, when receiving the fifth encoding feature tensor, perform classification prediction processing according to the current encoding feature tensor and output a corresponding second prediction vector; the image target detection task head is configured to, when receiving the fifth encoding feature tensor, perform target detection and classification identification processing according to the current encoding feature tensor and output a corresponding first prediction target set; and the image semantic segmentation task head is configured to, when receiving the fifth encoding feature tensor, perform voxel-level semantic segmentation processing according to the current encoding feature tensor and output a corresponding first prediction segmentation graph.

[0032] Preferably, the five-step training according to the two data sets and the five types of training frameworks specifically includes:

[0033] performing first-step training according to the first data set and the first training framework;

[0034] After the first-step training is completed, the model parameters of the waveform classification task head are solidified, and second-step training is performed according to the first data set and the second training framework;

[0035] After the second-step training is completed, the model parameters of the waveform disease prediction task head are solidified, and third-step training is performed according to the second data set and the third training framework;

[0036] After the third-step training is completed, the model parameters of the image target detection task head are solidified, and fourth-step training is performed according to the second data set and the fourth training framework;

[0037] After the fourth-step training is completed, the model parameters of the image semantic segmentation task head are solidified, and fifth-step training is performed according to the first and second data sets and the fifth training framework.

[0038] Further, the first-step training according to the first data set and the first training framework specifically includes:

[0039] Step 51, based on the preset first split ratio, randomly split the first data set into two sub-data sets, denoted as the corresponding first training set and the first evaluation set; and the model parameter set of the image pre-training model and the waveform classification task head as the corresponding first framework parameter set;

[0040] Wherein, the first training set and the first evaluation set are both composed of a plurality of first data records; the total number ratio of the first training set and the first evaluation set meets the first split ratio;

[0041] Step 52, the total number of records of the first training set is counted to obtain the corresponding total number N1; and the total number of the first prediction type is counted to obtain the corresponding total number C1; and the first training waveform and the first label vector of each first data record of the first training set are denoted as the corresponding waveform x a and label vector y a , 1≤index a≤N1; and each waveform x a is input into the first training framework as the current 2D waveform for processing, and the first prediction vector output by the current processing is denoted as the corresponding prediction vector And each first label probability of the label vector y a is denoted as the corresponding y a,b , and each first prediction probability of the prediction vector is denoted as the corresponding 1≤index b≤C1; and each prediction vector and the label vector y a comprise a corresponding first prediction-label pair; and the N1 first prediction-label pairs obtained are brought into the preset first model loss function L M1 to obtain the corresponding first loss value;

[0042] Wherein, the first model loss function L M1 is realized based on cross-entropy loss function, specifically:

[0043]

[0044] Step 53, identify whether the first loss value meets the preset first loss value range; if it meets, go to step 54; if it does not meet, based on the preset first model optimizer, a round of modulation is performed on the first framework parameter set in the direction of making the first model loss function L M1 reach the minimum value, and return to step 52 at the end of the current round of modulation;

[0045] wherein the first model optimizer comprises at least an Adam optimizer, an SGD optimizer;

[0046] Step 54, one round of traversal is performed on all the first data records of the first evaluation set; and during the round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; the first training waveform of the current evaluation record is taken as a current 2D waveform, which is input into the first training framework for processing, and the first prediction vector output by the processing is taken as a corresponding first prediction vector; a corresponding second prediction-label pair is formed by the first prediction vector and the first label vector of the current evaluation record; and at the end of the round of traversal, the first accuracy, the first precision, the first recall and the first F1 score are estimated according to all the second prediction-label pairs obtained.

[0047] Step 55, whether the first accuracy, the first precision, the first recall and the first F1 score all satisfy the respective corresponding first accuracy range, first precision range, first recall range and first F1 score range is identified; if not, step 52 is returned to continue training; if yes, it is confirmed that the first step of training is ended.

[0048] Further, the second step of training according to the first data set and the second training framework specifically comprises:

[0049] Step 61, the first data set is randomly divided into two sub-data sets according to a preset second segmentation ratio, which are taken as a corresponding second training set and a second evaluation set; and the model parameter set of the image pre-training model and the waveform disease prediction task head is taken as a corresponding second framework parameter set;

[0050] wherein the second training set and the second evaluation set are both composed of a plurality of first data records; the total number ratio of the records of the second training set and the second evaluation set satisfies the second segmentation ratio;

[0051] Step 62, the total number of records of the second training set is counted to obtain a corresponding total number N2; the total number of types of the second prediction type is counted to obtain a corresponding total number C2; and the first training waveform and the second label vector of each first data record of the second training set are taken as a corresponding waveform x c and a label vector y c , 1≤index c≤N2; and each waveform x cthe second prediction vector outputted by processing the current 2D waveform graph in the second training framework is recorded as a corresponding second prediction vector and each second label probability of the label vector y c is recorded as a corresponding y c,d and each second prediction probability of the prediction vector is recorded as a corresponding 1≤index d≤C2; and each prediction vector and the label vector y c comprise a corresponding third prediction-label pair; and the N2 third prediction-label pairs obtained are brought into a preset second model loss function L M2 to obtain a corresponding second loss value;

[0052] wherein the second model loss function L M2 is realized based on a cross-entropy loss function, and specifically:

[0053]

[0054] Step 63: whether the second loss value satisfies a preset second loss value range is identified; if yes, step 64 is turned to; if no, a second model optimizer is used to modulate the second framework parameter set in a direction of making the second model loss function L M2 reach a minimum value, and step 62 is returned at the end of the modulation;

[0055] wherein the second model optimizer at least comprises an Adam optimizer and an SGD optimizer;

[0056] Step 64: all the first data records of the second evaluation set are traversed; and in the traversal process, the first data record currently traversed is taken as a corresponding current evaluation record; the first training waveform graph of the current evaluation record is input into the second training framework to be processed, and the second prediction vector outputted by processing the current 2D waveform graph in the second training framework is recorded as a corresponding second prediction vector; and a corresponding fourth prediction-label pair is formed by the second prediction vector and the second label vector of the current evaluation record; and at the end of the traversal, the second accuracy, the second precision, the second recall and the second F1 score are obtained by estimating the accuracy, the precision, the recall and the F1 score according to all the fourth prediction-label pairs obtained;

[0057] Step 65, whether the second accuracy, the second precision, the second recall and the second F1 score all satisfy the respective corresponding second accuracy range, second precision range, second recall range and second F1 score range are identified; if not, return to step 62 to continue training; if yes, confirm the second step training is completed.

[0058] Further, the third step training according to the second data set and the third training framework, specifically comprising:

[0059] Step 71, the total number of records of the second data set is counted to obtain the corresponding total number N3; and the total number of organ types of the anatomical structure type set is counted to obtain the corresponding total number N4; and the model parameter set of the image pre-training model and the image target detection task head is taken as the corresponding third framework parameter set;

[0060] Step 72, the first training image and the first label target set of each second data record of the second data set are recorded as the corresponding image x e and label set G e , 1≤index e≤N3; and each image x e is input into the third training framework as the current 3D image and the first prediction target set output by the current processing is recorded as the corresponding prediction set

[0061] Step 73, the total number of label detection boxes of each label set G e is counted to obtain the corresponding total number N e ; and each first label detection box of the label set G e is recorded as the corresponding label box box e,f , 1≤index f≤N e ; and the first target detection box in the prediction set closest to each label box box e,f is recorded as the corresponding matching box

[0062] Step 74, the three-dimensional size of the detection box, the center point coordinate of the detection box and the type of the detection box of each label box box e,f are recorded as the corresponding size S e,f , center P e,f , type T e,f ; and each type T e,f is converted into a one-hot encoding vector with a length of N4 and recorded as the corresponding vector Z e,f ; and each label box box e,fcorresponding size S e,f , the center P e,f , the type T e,f , the vector Z e,f corresponding feature group bt e,f ;

[0063] wherein, the vector Z e,f includes N4one-hot encoding z e,f,g , 1≤index g≤N4; only the one-hot encoding z e,f corresponding to the type T e,f in the vector Z e,f,g is 1, and the rest are 0;

[0064] Step 75, record the detection frame three-dimensional size, the detection frame center point coordinate, and the detection frame type of each matching frame as corresponding size center type , and convert each type into a one-hot encoding vector with a length of N4, recorded as corresponding vector , and form a corresponding feature group bt composed of the size , the center , and the vector of each matching frame

[0065] wherein, the vector includes N4one-hot encoding only the one-hot encoding corresponding to the type in the vector is 1, and the rest are 0;

[0066] Step 76, bring N3×N e feature group bt e,f , into a preset third model loss function L M3 to calculate a corresponding third loss value;

[0067] wherein, the third model loss function L M3 is realized by combining a cross-entropy loss function for loss calculation of classification, an L2 loss function for loss calculation of center position, and an L1 loss function for loss calculation of three-dimensional size, specifically:

[0068]

[0069] Step 77, identify whether the third loss value meets the preset third loss value range; if not, based on the preset third model optimizer, adjust the third model loss function L M3 to the minimum value, and return to step 72 at the end of this round of modulation; if yes, confirm that the third step training is completed.

[0070] Wherein, the third model optimizer at least includes Adam optimizer, SGD optimizer.

[0071] Further, the fourth step training according to the second data set and the fourth training framework specifically includes:

[0072] The model parameter set of the image pre-training model and the image semantic segmentation task head is taken as the corresponding fourth framework parameter set;

[0073] And based on the second data set and the fourth training framework, a round of self-supervised contrastive learning training is performed on the model parameters of the image pre-training model to improve the feature consistency of multi-modal images under the same anatomical structure;

[0074] And after the end of this round of contrastive learning training, a round of supervised model training is performed on the fourth framework parameter set based on the second data set and the fourth training framework to improve the segmentation consistency;

[0075] And after the end of this round of supervised training, it is confirmed that the fourth step training is completed.

[0076] Further preferably, the round of self-supervised contrastive learning training based on the second data set and the fourth training framework to improve the feature consistency of multi-modal images under the same anatomical structure is specifically:

[0077] Step 91, record each of the first training image graphs in the second data set containing only one type of organ as the corresponding single-organ image graph, and count the total number of single-organ image graphs to obtain the corresponding total number N5, and the N5 single-organ image graphs form a corresponding multi-organ image set X ALL ; and each of the single-organ image graphs is recorded as the corresponding image graph x i , 1≤index i≤N5;

[0078] Step 92, and count the total number of organ types of the multi-organ image set X ALL to obtain the corresponding total number N6; and the multiple single-organ image graphs corresponding to the same type of organ type in the multi-organ image set X ALL are grouped into a corresponding single-organ image set Xj , 1≤indexj≤N6; and count the total number of images of each of the single-organ image sets X i to obtain the corresponding total number N 7,j ;

[0079] Step 93, and each of the image maps x i as the corresponding current image map; and the single-organ image set X j in which the current image map is located is recorded as the corresponding current positive sample set; and the rest of all the single-organ image sets X j except the current positive sample set constitutes the corresponding current negative sample set; and each of the single-organ image maps in the current positive sample set except the current image map is taken as the positive sample image corresponding to the current image map 1≤positive sample index is the total number of positive samples corresponding to the current image map, the value of which is the value of the total number N 7,j minus 1 corresponding to the current positive sample set; and each of the single-organ image maps in the current negative sample set is taken as the negative sample image corresponding to the current image map 1≤negative sample index is the total number of negative samples corresponding to the current image map,

[0080] Step 94, each of the image maps x i is input into the image pre-training model as the current 3D image map for feature encoding processing, and the encoding feature tensor output by this processing is taken as the corresponding feature tensor y i ; and the corresponding encoding feature tensor of each of the positive sample images corresponding to each of the image maps x i is recorded as the corresponding positive sample tensor and the corresponding encoding feature tensor of each of the negative sample images corresponding to each of the image maps x i is recorded as the corresponding negative sample tensor and the corresponding i positive sample tensor of each of the feature tensors y constitutes the corresponding positive sample set the corresponding i negative sample tensor of each of the feature tensors y constitutes the corresponding negative sample set and each of the feature tensors yi The three-dimensional size of the feature tensor y i is denoted as the corresponding height H i , width W i , and depth D i , and each of the feature tensors y i is regarded as a three-dimensional feature map composed of H i ×W i ×D i voxel tensors, 1≤index h i ≤H i , 1≤index w i ≤W i , and 1≤index d i ≤D i ;

[0081] Step 95, the N5 feature tensors y and the corresponding positive and negative sample sets obtained this time are brought into a preset fourth model loss function L M4 to calculate the corresponding fourth loss value;

[0082] The fourth model loss function L M4 is realized by combining an InfoNCE loss function and a non-integer order total variation loss function, specifically:

[0083]

[0084] λ is a preset trade-off coefficient;

[0085] L infoNCE () is an InfoNCE loss function; sim() is a similarity function, which at least includes a dot product similarity function, a cosine vector similarity function, and an Euclidean distance similarity function; τ is a preset temperature coefficient;

[0086] L FOTV () is a non-integer order total variation loss function; α is a preset non-integer order parameter; is an α-th order derivative operator along the three-dimensional direction, specifically: ,, and are α-th order derivative operators along the height, width, and depth directions, respectively;

[0087] Step 96, whether the fourth loss value meets a preset fourth loss value range is identified; if not, a round of modulation is performed on the model parameters of the image pre-training model based on a preset fourth model optimizer in a direction that makes the fourth model loss function L M4 reach a minimum value, and step 94 is returned at the end of the round of modulation; if yes, it is confirmed that the current round of contrast learning training is completed;

[0088] The fourth model optimizer at least includes an Adam optimizer and an SGD optimizer.

[0089] Further preferably, the supervised model training of the fourth set of framework parameters based on the second data set and the fourth training framework for improving segmentation consistency comprises the following steps.

[0090] In step 101, the first training image and the first label segmentation image of each second data record in the second data set are recorded as the corresponding image x e and label segmentation image y e , and each image x e is input into the fourth training framework as the current 3D image for processing, and the first predicted segmentation image output by the current processing is recorded as the corresponding predicted segmentation image

[0091] In step 102, the obtained N3 label and predicted segmentation image y e , is input into a preset fifth model loss function L M5 to obtain the corresponding fifth loss value.

[0092] The fifth model loss function L M5 is based on a Dice loss function L Dice , and is specifically:

[0093]

[0094] In step 103, it is identified whether the fifth loss value meets a preset fifth loss value range. If not, a preset fifth model optimizer is used to modulate the fourth set of framework parameters in a direction to minimize the fifth model loss function L M5 , and returns to step 101 at the end of the current modulation. If yes, it is confirmed that the current supervised training is completed.

[0095] The fifth model optimizer at least includes an Adam optimizer and an SGD optimizer.

[0096] Preferably, the fifth-step training according to the first and second data sets and the fifth training framework comprises the following steps.

[0097] In step 111, the total number of records in the first data set is counted to obtain the corresponding total number N8, and the total number of types of the first prediction type is counted to obtain the corresponding total number N9. and the total number of the second prediction type is counted to obtain a corresponding total number and the first training waveform graph, the first label vector, the second label vector of each of the first data records of the first data set are recorded as a corresponding waveform graph x m , a label vector y 1,m , a label vector y 2,m 1≤index m≤N8; and for each of the waveform graphs x m a training graph type is set, which is specifically a 2D type;

[0098] Step 112, the first training image graph, the first label target set and the first label segmentation graph of each of the second data records of the second data set are recorded as a corresponding image graph x e , the label set G e and the label segmentation graph y e ; and for each of the image graphs x e a training graph type is set, which is specifically a 3D type;

[0099] Step 113, each of the waveform graphs x m and the corresponding training graph type is taken as the current 2D waveform graph and 2D / 3D task type to input the fifth training framework for processing, and the first and second prediction vectors output by the processing are recorded as corresponding prediction vectors and prediction vector

[0100] Step 114, each of the first label probabilities of the label vector y 1,m is recorded as a corresponding y 1,m,s1 , and each of the first prediction probabilities of the prediction vector is recorded as a corresponding 1≤index and each of the prediction vectors and the label vector y 1,m comprises a corresponding fifth prediction-label pair; and each of the second label probabilities of the label vector y 2,m is recorded as a corresponding y 2,m,s2 , and each of the second prediction probabilities of the prediction vector is recorded as a corresponding 1≤index and each of the prediction vectors and the label vector y 2,m comprises a corresponding sixth prediction-label pair; and the N8 fifth prediction-label pairs obtained at this time comprise a corresponding prediction-label pair set The N8 sixth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0101] Step 115, transfer each of the aforementioned image maps x e The training map type and its corresponding type are input into the fifth training framework as the current 3D image map and the 2D / 3D task type for processing, and the first predicted target set and the first predicted segmentation map output in this processing are recorded as the corresponding prediction set. and the predicted segmentation map

[0102] Step 116, for each of the said tag sets G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f ; and the prediction set In and each of the aforementioned label boxes e,f The nearest first target detection box is denoted as the corresponding matching box. and each of the aforementioned label boxes e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f The center P e,f The type T e,f ; and each of the aforementioned types T e,f This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,f The corresponding size S e,f The center P e,f The type T e,f The vector Z e,f The corresponding feature group bt e,f ;

[0103] Step 117, match each of the aforementioned matching boxes. The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. The center The type and each of the types This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector. And by each of the aforementioned matching boxes The corresponding size the center the vector the feature group corresponding to the composition

[0104] Step 118, by each of the feature group bt e,f and the corresponding feature group composes a corresponding seventh prediction-label pair; and by each of the label segmentation map y e and the corresponding prediction segmentation map composes a corresponding eighth prediction-label pair; and by the N3xN e obtained this time, the seventh prediction-label pair composes a corresponding prediction-label pair set by the N3 eighth prediction-label pairs obtained this time, a corresponding prediction-label pair set is composed

[0105] Step 119, the prediction-label pair set obtained this time is brought into a preset multi-task loss function L MT to obtain a corresponding sixth loss value;

[0106] Among them, the multi-task loss function L MT is composed of a 2D task loss function L 2D and a 3D task loss function L 3D , the 2D task loss function L 2D is composed of a corresponding first and second 2D task loss function , the 3D task loss function L 3D is composed of a corresponding first and second 3D task loss function ; the multi-task loss function L MT is specifically:

[0107]

[0108] w 2D , w 3D are two preset weight parameters;

[0109] Step 120, whether the sixth loss value meets the preset sixth loss value range is identified; if not, based on the preset sixth model optimizer, the model parameters of the image pre-training model are modulated in the direction of making the multi-task loss function L MT reach the minimum value, and returns to step 113 at the end of this round of modulation; if it is satisfied, the fifth step training is confirmed to be completed;

[0110] The sixth model optimizer comprises at least an Adam optimizer and an SGD optimizer.

[0111] The second aspect of the embodiment of the present application provides a device for implementing the training method of the image pre-training model for the medical field in the first aspect, and the device comprises a model preparation module, a basic data preparation module, a training framework preparation module, a training data set preparation module and a five-step training module.

[0112] The model preparation module is configured to select a visual pre-training model capable of supporting 2D / 3D image feature coding as an image pre-training model.

[0113] The basic data preparation module is configured to take 2D waveform graphs and 3D image graphs in the medical field as 2D and 3D images supported by the visual pre-training model, configure a corresponding image type set for the 2D waveform graphs, and take the corresponding image type set as a waveform type set; set a corresponding disease type set for each type of waveform graph in the waveform type set, and take the corresponding disease type set as a first disease type set, and then obtain all the first disease type sets to form a waveform-disease type set; and configure a corresponding image type set and an anatomical structure type set for the 3D image graphs, and take the corresponding image type set and the anatomical structure type set as an image type set and an anatomical structure type set.

[0114] The training framework preparation module is configured to select four types of downstream task head models to be connected to the image pre-training model in a separate docking manner and a collective docking manner to obtain five types of training frameworks; the four types of downstream task head models comprise a waveform classification task head, a waveform disease prediction task head, an image target detection task head and an image semantic segmentation task head; and the five types of training frameworks comprise a first, a second, a third, a fourth and a fifth training framework.

[0115] The training data set preparation module is configured to construct a first data set according to the waveform type set and the waveform-disease type set by using a big data acquisition method; and construct a second data set according to the image type set and the anatomical structure type set by using a big data acquisition method.

[0116] The five-step training module is configured to perform five-step training according to the two data sets and the five types of training frameworks in a step-by-step training manner; and confirm that the specific field optimization training of the image pre-training model in the medical field is completed at the end of the fifth-step training.

[0117] The third aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor and a transceiver.

[0118] The processor is configured to be coupled with the memory, read and execute instructions in the memory, so as to realize the method steps in the first aspect.

[0119] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0120] A fourth aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0121] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for image pre-training models in the medical field. As described above, this invention first selects a type of pre-trained LVMs that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; it then configures a waveform type set for the 2D waveforms supported by the visual pre-training model in the medical field, and sets disease type subsets for each type of waveform to obtain a corresponding full waveform-disease type set; and it configures corresponding image type sets and anatomical structure type sets for the 3D images supported by the visual pre-training model in the medical field; next, it selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head) and interfaces them with the image pre-training model to obtain five training frameworks; then, it constructs a first dataset based on the waveform type set and waveform-disease type set, and a second dataset based on the image type set and anatomical structure type set through big data collection; then, it performs five steps of training based on the two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it confirms that the targeted training of the image pre-training model in the medical field is complete. In this invention, on the one hand, an image pre-trained model that is compatible with both 2D and 3D images and supports multiple waveforms and image types is obtained through SDT training in the medical field and serves as a shared encoding platform; on the other hand, the overall training difficulty is reduced and the overall training efficiency is improved through a progressive training method; and furthermore, the economic benefits of SDT training are improved by simultaneously training four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head). Attached Figure Description

[0122] Figure 1 This is a schematic diagram of a training method for an image pre-training model in the medical field provided in Embodiment 1 of the present invention;

[0123] Figure 2 This is a schematic diagram of the five types of training frameworks provided in Embodiment 1 of the present invention;

[0124] Figure 3 This is a module structure diagram of a training device for an image pre-training model in the medical field, provided in Embodiment 2 of the present invention;

[0125] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0126] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0127] Embodiment 1 of the present invention provides a training method for an image pre-training model in the medical field, such as... Figure 1 The diagram illustrates a training method for an image pre-training model in the medical field, as provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0128] Step 1: Select a visual pre-training model that can simultaneously support 2D / 3D image feature encoding as the image pre-training model.

[0129] Here, the visual pre-trained model in this embodiment of the invention is a large visual model that is compatible with both 2D and 3D modalities and has been pre-trained based on massive image corpora in a general domain; the large visual model includes at least the MultiResViT model, UniFormer model, Hybrid CNN-Transformer model, nnU-Net model, Uni3D model, Uni-MAE model, and MedFMC model.

[0130] Step 2: Use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; configure the corresponding image type set for the 2D waveforms as the corresponding waveform type set; set the corresponding diagnostic disease type set for each type of waveform in the waveform type set as the corresponding first disease type set, and form the corresponding waveform-disease type set by all the obtained first disease type sets; configure the corresponding image type set and anatomical structure type set for the 3D images as the corresponding image type set and anatomical structure type set.

[0131] Here, the waveform types in the waveform type set of this embodiment of the invention include at least electrocardiogram (ECG), electroencephalogram (EEG), and electromyography (EMG). It should be noted that the types of waveforms in the waveform type set can be added or removed based on actual application requirements.

[0132] The waveform-disease category set in this embodiment of the invention includes multiple first disease category sets; wherein, 1) the disease categories in the first disease category set corresponding to electrocardiogram (ECG) include at least atrial fibrillation, premature beats, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, and cardiac conduction block; 2) the disease categories in the first disease category set corresponding to electroencephalogram (EEG) include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, traumatic brain injury, schizophrenia, bipolar disorder, and mental abnormalities; 3) the disease categories in the first disease category set corresponding to electromyography (EMG) include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis (ALS), Guillain-Barré syndrome, polyneuropathy, oculomotor nerve injury, trochlear nerve injury, and facial nerve palsy. It should be noted that the first disease category sets corresponding to each waveform category in the waveform-disease category set can be added or removed based on actual application needs, and the disease categories in each first disease category set can also be added or removed based on actual application needs.

[0133] The image image types in this embodiment of the invention include at least computed tomography (CT) images, magnetic resonance imaging (MRI) images, three-dimensional ultrasound images, and positron emission tomography (PET) images. It should be noted that the types of images in the image image type set can be added or removed based on actual application requirements.

[0134] The anatomical structure category set of this invention includes at least a variety of organ types; the organ types include at least the heart, lungs, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, and bladder. It should be noted that the anatomical structure categories, i.e., organ types, in the anatomical structure category set can be added to or removed based on actual application needs.

[0135] Step 3: Select four types of downstream task head models and connect them with the image pre-training model using separate and collective connection methods to obtain the corresponding five types of training frameworks.

[0136] Here, the four types of downstream task heads in this embodiment of the invention include waveform classification task head, waveform disease prediction task head, image target detection task head, and image semantic segmentation task head; the five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0137] It should be noted that this embodiment of the invention does not limit the specific model structure of the four types of downstream task heads, but only limits the input / output data and corresponding model functions of the four types of downstream task heads. Specifically, the technical descriptions of the five training frameworks below can be used for further understanding.

[0138] 1) First training framework:

[0139] like Figure 2As shown in the schematic diagram of the five types of training frameworks provided in Embodiment 1 of the present invention, the first training framework of the present invention is formed by connecting the output end of the image pre-training model and the input end of the waveform classification task head.

[0140] The first training framework is used to perform waveform classification and prediction processing based on the 2D waveform image input to the framework and output the corresponding first prediction vector. The first prediction vector consists of multiple first prediction probabilities, each corresponding to a first prediction type. The first prediction types include unknown waveform types and multiple known waveform types; the known waveform types here correspond one-to-one with the waveform types in the waveform image type set.

[0141] In the first training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding first encoded feature tensor and send it to the waveform classification task head; the waveform classification task head is used to perform classification prediction processing based on the first encoded feature tensor and output the corresponding first prediction vector.

[0142] 2) Second training framework:

[0143] like Figure 2 As shown, the second training framework of this embodiment of the invention is formed by connecting the output end of the image pre-trained model and the input end of the waveform disease prediction task head.

[0144] The second training framework is used to perform disease classification prediction based on the 2D waveform image input by the framework and output the corresponding second prediction vector. The second prediction vector consists of multiple second prediction probabilities, each corresponding to a second prediction type. The second prediction types include unknown disease types and multiple known disease types; the known disease types here correspond one-to-one with the disease types in the waveform-disease type set.

[0145] In the second training framework, the image pre-training model is used to perform feature encoding processing on the 2D waveform image input by the current framework to obtain the corresponding second encoded feature tensor and send it to the waveform disease prediction task head; the waveform disease prediction task head is used to perform classification prediction processing based on the second encoded feature tensor and output the corresponding second prediction vector.

[0146] 3) Third training framework:

[0147] like Figure 2 As shown, the third training framework of this embodiment of the invention is formed by connecting the output end of the image pre-training model and the input end of the image target detection task head.

[0148] The third training framework is used to perform organ target detection and classification based on the 3D image input from the framework and output the corresponding first predicted target set. The first predicted target set consists of one or more first target detection boxes; each first target detection box includes at least a detection box identifier, three-dimensional dimensions, center point coordinates, and detection box type; the detection box type includes unknown target types and multiple known target types; the known target types here correspond one-to-one with the organ types in the anatomical structure category set.

[0149] In the third training framework, the image pre-training model is used to perform feature encoding processing on the 3D image input to the current framework to obtain the corresponding third-encoded feature tensor and send it to the image target detection task head; the image target detection task head is used to perform target detection and classification processing based on the third-encoded feature tensor and output the corresponding first predicted target set.

[0150] 4) Fourth training framework:

[0151] like Figure 2 As shown, the fourth training framework of this embodiment of the invention is formed by connecting the output end of the image pre-training model and the input end of the image semantic segmentation task head.

[0152] The fourth training framework is used to perform voxel-level organ semantic segmentation based on the 3D image input to the framework and output the corresponding first predicted segmentation map. The 3D dimensions of the first predicted segmentation map are consistent with the 3D dimensions of the 3D image input to the current framework; the voxel-level features of the first predicted segmentation map include at least one semantic type; this semantic type includes background semantics, unknown organ semantics, and semantics for multiple known organ types, where the known organ semantics correspond one-to-one with the organ types in the anatomical structure category set.

[0153] In the fourth training framework, the image pre-training model is used to perform feature encoding processing on the 3D image map input by the current framework to obtain the corresponding fourth encoded feature tensor and send it to the image semantic segmentation task head; the image semantic segmentation task head is used to perform voxel-level semantic segmentation processing based on the fourth encoded feature tensor and output the corresponding first predicted segmentation map.

[0154] 5) Fifth training framework:

[0155] like Figure 2 As shown, the fifth training framework of this embodiment consists of an image pre-training model, a feature distribution module, a waveform classification task head, a waveform disease prediction task head, an image target detection task head, and an image semantic segmentation task head.

[0156] The fifth training framework is used to identify the 2D / 3D task type input by the current framework. If the current 2D / 3D task type is 2D, then waveform classification prediction and disease classification prediction are performed based on the 2D waveform image input by the current framework, and the corresponding first and second prediction vectors are output. If the current 2D / 3D task type is 3D, then organ target detection and classification recognition and organ semantic segmentation are performed based on the 3D image image input by the current framework, and the corresponding first predicted target set and first predicted segmentation map are output. The 2D / 3D task type includes both 2D and 3D types. When the current framework input image is a 2D waveform image, the corresponding 2D / 3D task type is 2D; when the current framework input image is a 3D image, the corresponding 2D / 3D task type is 3D.

[0157] The first frame input of the fifth training frame is used to receive the 2D waveform or 3D image of the current frame input, the second frame input is used to receive the 2D / 3D task type of the current frame input, and the first, second, third and fourth frame outputs are used to output the corresponding first prediction vector, second prediction vector, first prediction target set and first prediction segmentation map.

[0158] In the fifth training framework, the input of the image pre-training model is connected to the input of the first framework, and its output is connected to the first input of the feature distribution module; the second input of the feature distribution module is connected to the input of the second framework, and its output is connected to the inputs of the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head, respectively; the outputs of the waveform classification task head, the waveform disease prediction task head, the image object detection task head, and the image semantic segmentation task head are connected to the outputs of the corresponding first, second, third, and fourth frameworks, respectively.

[0159] In the fifth training framework, the image pre-trained model is used to perform feature encoding processing on the 2D waveform or 3D image input of the current framework to obtain the corresponding fifth-encoded feature tensor, which is then sent to the feature distribution module. The feature distribution module is used to identify the 2D / 3D task type input of the current framework. If the current 2D / 3D task type is 2D, the fifth-encoded feature tensor is sent to the waveform classification task head and the waveform disease prediction task head. If the current 2D / 3D task type is 3D, the fifth-encoded feature tensor is sent to the image target detection task head and the image semantic segmentation task head. The waveform classification task head is used to process the received fifth-encoded feature tensor... When receiving the fifth encoded feature tensor, the task head performs classification prediction processing based on the current encoded feature tensor and outputs the corresponding first prediction vector; when receiving the fifth encoded feature tensor, the task head performs classification prediction processing based on the current encoded feature tensor and outputs the corresponding second prediction vector; when receiving the fifth encoded feature tensor, the task head performs target detection and classification recognition processing based on the current encoded feature tensor and outputs the corresponding first predicted target set; when receiving the fifth encoded feature tensor, the task head performs voxel-level semantic segmentation processing based on the current encoded feature tensor and outputs the corresponding first predicted segmentation map.

[0160] Step 4: Construct the first dataset based on waveform type set and waveform-disease type set using big data acquisition methods; and construct the second dataset based on image type set and anatomical structure type set using big data acquisition methods.

[0161] Here, the first dataset in this embodiment of the invention includes multiple first data records; the first data record includes a first training waveform, a first label vector, and a second label vector; the first training waveform is a waveform in a medical field; the first label vector is composed of multiple first label probabilities, each first label probability corresponding to a first prediction type; the second label vector is composed of multiple second label probabilities, each second label probability corresponding to a second prediction type; in both the first and second label vectors, only one label probability is 1, and the probabilities of the remaining labels are 0.

[0162] It should be noted that the final first dataset should meet the following requirements: 1) The total set of waveform types for all first training waveforms in the first dataset is greater than the total set of waveform types; 2) The total number of data records corresponding to each first prediction type in the first dataset is the same; 3) The total number of data records corresponding to each second prediction type in the first dataset is the same. It should also be noted that if there are difficulties in data collection during the large-scale data acquisition process, and it is impossible to guarantee that the total number of data records corresponding to each first / second prediction type is the same, then additional image generation models can be used to supplement the training waveforms for each prediction type through data augmentation.

[0163] The second dataset in this embodiment of the invention includes multiple second data records; the second data records include a first training image, a first label target set, and a first label segmentation map; the first training image is a type of 3D image; the first label target set consists of one or more first label detection boxes; the first label detection box includes at least a detection box identifier, a detection box three-dimensional size, detection box center point coordinates, and a detection box type; the three-dimensional size of the first label segmentation map is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation map include at least a semantic type.

[0164] It should be noted that the final second dataset should meet the following requirements: 1) The set of image types for all first training image images in the second dataset matches the set of image types; 2) The set of anatomical structure types for all first training image images in the second dataset is greater than the set of anatomical structure types; 3) The total number of data records corresponding to each type of image image in the second dataset is the same, and the total number of detection boxes for each type of detection box in all data records corresponding to each type of image image is the same, and the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same. It should also be noted that if there are difficulties in data collection, and it is impossible to guarantee that the total number of data records for each type of image image is the same, or that the total number of detection boxes for each type of detection box in all data records corresponding to each type of image image is the same, or that the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image image is the same, then missing images can be supplemented by using another medical image generation model through data augmentation.

[0165] Step 5: Following a progressive training approach, conduct five training steps based on two datasets and five training frameworks; and at the end of the fifth training step, confirm that the image pre-trained model has completed domain-specific optimization training in the medical field.

[0166] Specifically, it includes: Step 51, which involves five steps of training based on two datasets and five training frameworks, following a progressive training approach;

[0167] Specifically, it includes: Step 511, performing the first training step based on the first dataset and the first training framework;

[0168] Specifically, it includes: step 5111, randomly dividing the first dataset into two subsets based on a preset first segmentation ratio, denoted as the corresponding first training set and first evaluation set; and using the combined set of model parameters of the image pre-trained model and the waveform classification task head as the corresponding first frame parameter set;

[0169] Wherein, the first segmentation ratio is a pre-set ratio parameter, such as 8:2; both the first training set and the first evaluation set consist of multiple first data records; the ratio of the total number of records in the first training set and the first evaluation set satisfies the first segmentation ratio;

[0170] Step 5112: Count the total number of records in the first training set to obtain the corresponding total number N1; and count the total number of types of the first prediction type to obtain the corresponding total number C1; and denote the first training waveform and the first label vector of each first data record in the first training set as the corresponding waveform x. a and label vector y a 1 ≤ index a ≤ N1; and each waveform x a The current 2D waveform is input to the first training framework, processed, and the first prediction vector output from this processing is recorded as the corresponding prediction vector. and label vector y a The probabilities of each first label are denoted as y. a,b and predict vector The first prediction probabilities are denoted as the corresponding 1 ≤ index b ≤ C1; and by each prediction vector and label vector y a Form a corresponding first prediction-label pair; and input the resulting N1 first prediction-label pairs into the preset first model loss function L. M1 The corresponding first loss value is obtained through calculation;

[0171] Wherein, the first model loss function L M1 It is implemented based on the cross-entropy loss function, specifically as follows:

[0172]

[0173] Step 5113: Identify whether the first loss value meets the preset first loss value range; if it does, proceed to step 5114; if not, based on the preset first model optimizer, move towards making the first model loss function L... M1 The direction that reaches the minimum value is modulated once for the first frame parameter set, and the process returns to step 5112 when the current modulation is completed.

[0174] The first loss value range is a pre-set numerical range; the first model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0175] Step 5114: Perform a round of traversal on all first data records in the first evaluation set; during this round of traversal, the first data record currently being traversed is taken as the corresponding current evaluation record; the first training waveform of the current evaluation record is taken as the current 2D waveform and input into the first training framework for processing, and the first prediction vector output by the current processing is recorded as the corresponding first prediction vector; and a corresponding second prediction-label pair is formed by the first prediction vector and the first label vector of the current evaluation record; and at the end of this round of traversal, the accuracy, precision, recall and F1 score are estimated based on all the obtained second prediction-label pairs to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0176] Step 5115: Identify whether the first accuracy, first precision, first recall, and first F1 score all meet their respective first accuracy range, first precision range, first recall range, and first F1 score range; if not, return to step 5112 to continue training; if yes, confirm that the first step of training is complete.

[0177] Here, the first accuracy range, the first precision range, the first recall range, and the first F1 score range are four pre-set numerical ranges;

[0178] Step 512: After the first training step is completed, the model parameters of the waveform classification task head are fixed; and the second training step is carried out based on the first dataset and the second training framework.

[0179] Specifically, this includes: Step 5121, after the first step of training is completed, the model parameters of the waveform classification task head are solidified;

[0180] Step 5122, and perform the second training step based on the first dataset and the second training framework;

[0181] Specifically, it includes: step 51221, randomly dividing the first dataset into two subsets based on a preset second segmentation ratio, which are denoted as the corresponding second training set and second evaluation set; and taking the combined set of model parameters of the image pre-trained model and the waveform disease prediction task head as the corresponding second frame parameter set;

[0182] The second segmentation ratio is a pre-set ratio parameter, such as 8:2; both the second training set and the second evaluation set consist of multiple first data records; the ratio of the total number of records in the second training set and the second evaluation set satisfies the second segmentation ratio.

[0183] Step 51222: Count the total number of records in the second training set to obtain the corresponding total number N2; and count the total number of types of the second prediction type to obtain the corresponding total number C2; and denote the first training waveform and the second label vector of each first data record in the second training set as the corresponding waveform x. c and label vector y c 1 ≤ index c ≤ N2; and each waveform x c The current 2D waveform is input to the second training framework for processing, and the second prediction vector output from this processing is recorded as the corresponding prediction vector. and label vector y c The probabilities of each second label are denoted as the corresponding y. c,d and predict vector The respective second prediction probabilities are denoted as the corresponding 1 ≤ index d ≤ C2; and by each prediction vector and label vector y c Form a corresponding third prediction-label pair; and input the resulting N2 third prediction-label pairs into the preset second model loss function L. M2 The corresponding second loss value is obtained through calculation;

[0184] Among them, the loss function of the second model L M2 It is implemented based on the cross-entropy loss function, specifically as follows:

[0185]

[0186] Step 51223: Identify whether the second loss value meets the preset range of the second loss value; if it does, proceed to step 51224; if it does not, optimize the second model based on the preset second model optimizer to make the second model loss function L... M2 The direction that reaches the minimum value is modulated once for the second frame parameter set, and the process returns to step 51222 when the current modulation is completed;

[0187] The second loss value range is a pre-set numerical range; the second model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0188] Step 51224: Perform a round of traversal on all first data records in the second evaluation set; during this round of traversal, the first data record currently traversed is taken as the corresponding current evaluation record; the first training waveform of the current evaluation record is taken as the current 2D waveform and input into the second training framework for processing, and the second prediction vector output by the current processing is recorded as the corresponding second prediction vector; and the second prediction vector and the second label vector of the current evaluation record form a corresponding fourth prediction-label pair; at the end of this round of traversal, the accuracy, precision, recall and F1 score are estimated based on all the obtained fourth prediction-label pairs to obtain the corresponding second accuracy, second precision, second recall and second F1 score;

[0189] Step 51225: Identify whether the second accuracy, second precision, second recall, and second F1 score all satisfy their respective corresponding ranges for the second accuracy, second precision, second recall, and second F1 score. If not, return to step 51222 to continue training; if yes, confirm that the second training step is complete.

[0190] Among them, the second accuracy range, the second precision range, the second recall range, and the second F1 score range are four pre-set numerical ranges;

[0191] Step 513: After the second training step is completed, the model parameters of the waveform disease prediction task head are fixed; and the third training step is carried out based on the second dataset and the third training framework.

[0192] Specifically, this includes: Step 5131, after the second training step is completed, the model parameters of the waveform disease prediction task head are solidified;

[0193] Step 5132, and perform the third training step based on the second dataset and the third training framework;

[0194] Specifically, it includes:

[0195] Step 51321: Count the total number of records in the second dataset to obtain the corresponding total number N3; and count the total number of organ types in the anatomical structure type set to obtain the corresponding total number N4; and take the combined set of model parameters of the image pre-trained model and the image target detection task head as the corresponding third frame parameter set;

[0196] Step 51322: Denote the first training image and the first label target set of each second data record in the second dataset as the corresponding image image x. e and the set of tags G e 1 ≤ index e ≤ N3; and each image x eThe current 3D image is used as input to the third training framework for processing, and the first set of predicted targets output from this processing is recorded as the corresponding prediction set.

[0197] Step 51323, for each tag set G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f 1 ≤ index f ≤ N e ; and the prediction set In the middle and each label box e,f The nearest first object detection box is denoted as the corresponding matching box.

[0198] Step 51324, change each label box e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each type T e,f Converting it into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and comprised of various label boxes e,f Corresponding size S e,f Center P e,f Type T e,f Vector Z e,f Form the corresponding feature group bt e,f ;

[0199] Wherein, vector Z e,f Includes N4 one-hot codes z e,f,g 1 ≤ index g ≤ N4; vector Z e,f Only type T is included. e,f The corresponding one-hot encoding z e,f,g The value is 1, and all others are 0;

[0200] Step 51325, match each box The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimensions. center type and each type Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And composed of each matching box Corresponding size center vector Form corresponding feature groups

[0201] Where, vector Includes N4 one-hot codes vector Only the type Corresponding one-hot encoding The value is 1, and all others are 0;

[0202] Step 51326, change N3×N e For feature group bt e,f , Substitute the preset third model loss function L M3 The corresponding third loss value is obtained through calculation;

[0203] Among them, the loss function of the third model L M3 It is implemented by combining the cross-entropy loss function for classification, the L2 loss function for center position, and the L1 loss function for 3D dimensions, specifically:

[0204]

[0205] Step 51327: Identify whether the third loss value meets the preset range of the third loss value; if not, then based on the preset third model optimizer, move towards making the third model loss function L... M3 The direction that reaches the minimum value is modulated once for the third frame parameter set, and the process returns to step 51322 when the modulation ends; if the condition is met, the third step of training is confirmed to be over.

[0206] The third loss value range is a pre-set numerical range; the third model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0207] Step 514: After the third training step is completed, the model parameters of the image target detection task head are fixed; and the fourth training step is carried out based on the second dataset and the fourth training framework.

[0208] Specifically, this includes: Step 5141, after the third step of training is completed, the model parameters of the image target detection task head are solidified;

[0209] Step 5142, and perform the fourth training step based on the second dataset and the fourth training framework;

[0210] Specifically, this includes: step 51421, taking the combined set of model parameters of the image pre-trained model and the image semantic segmentation task head as the corresponding fourth frame parameter set;

[0211] Step 51422, and with the goal of improving the feature consistency of multimodal images under the same anatomical structure, a round of self-supervised comparative learning training is performed on the model parameters of the image pre-training model based on the second dataset and the fourth training framework.

[0212] Specifically, this includes: step 514221, where each first training image in the second dataset containing only one type of organ is recorded as a corresponding single-organ image, and the total number of single-organ image images is counted to obtain the corresponding total number N5, and the corresponding multi-organ image set X is composed of N5 single-organ image images. ALL And record the images of each single organ as the corresponding image x. i , 1 ≤ index i ≤ N5;

[0213] Step 514222, and the multi-organ image set X ALL The total number of organ types was statistically analyzed to obtain the corresponding total number N6; and the multi-organ image set X was then analyzed. ALL Multiple single-organ images corresponding to the same organ type are clustered together to form a corresponding single-organ image set X. j 1 ≤ index j ≤ N6; and for each single organ image set X i The total number of images is counted to obtain the corresponding total number N. 7,j ;

[0214] Step 514223, and then divide each image x i As the corresponding current image; and set the single organ image set X to which the current image belongs. j Let X be the corresponding current positive sample set; and let X be the set of all other single-organ images excluding the current positive sample set. j The corresponding current negative sample set is formed; and each single organ image in the current positive sample set, excluding the current image image, is taken as the positive sample image corresponding to the current image image. 1≤Positive Sample Index This represents the total number of positive samples corresponding to the current image. The value is the total number N corresponding to the current positive sample set. 7,j Decrease the value by 1; and use the single organ images in the current negative sample set as the negative sample images corresponding to the current image. 1≤Negative Sample Index This represents the total number of negative samples corresponding to the current image.

[0215] Step 514224, transfer each image x i The current 3D image is used as the input image for pre-training model feature encoding, and the encoded feature tensor output from this process is used as the corresponding feature tensor y.i ; and each image x i Corresponding positive sample images The corresponding encoded feature tensor is denoted as the corresponding positive sample tensor. and each image x i Corresponding negative sample images The corresponding encoded feature tensor is denoted as the corresponding negative sample tensor. and each feature tensor y i corresponding positive sample tensors Form the corresponding positive sample set Each feature tensor y i corresponding negative sample tensor Form the corresponding negative sample set and each feature tensor y i The three-dimensional dimension is denoted as the corresponding height H. i Width W i and depth D i and each feature tensor y i Considered to be by H i ×W i ×D i Individual tensor The resulting 3D feature map, 1 ≤ index h i ≤H i , 1 ≤ index w i ≤W i 1 ≤ index d i ≤D i ;

[0216] Step 514225, convert the N5 feature tensors y obtained this time i and its corresponding positive and negative sample sets Substitute the preset fourth model loss function L M4 The corresponding fourth loss value is obtained through calculation;

[0217] Here, the loss function L of the fourth model M4 It is implemented by combining the InfoNCE loss function and the non-integer order total variation (FOTV) loss function, specifically as follows:

[0218]

[0219] Where λ is a preset trade-off coefficient;

[0220] L infoNCE() represents the InfoNCE loss function; sim() represents a type of similarity function, which includes at least the dot product similarity function, cosine vector similarity function, and Euclidean distance similarity function; τ represents the preset temperature coefficient;

[0221] L FOTV () represents the non-integer order total variation loss function; α is a preset non-integer order parameter; For the α-order derivative operator along the three-dimensional direction, specifically: These are the α-order derivative operators along the height, width, and depth directions, respectively;

[0222] Step 514226: Identify whether the fourth loss value meets the preset range of the fourth loss value; if not, then based on the preset fourth model optimizer, move towards making the fourth model loss function L... M4 The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 514224 when the modulation ends; if satisfied, the contrastive learning training is confirmed to be over.

[0223] The fourth loss value range is a pre-set numerical range; the fourth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0224] Step 51423, and after this round of comparative learning training, with the goal of improving segmentation consistency, a round of supervised model training is carried out on the parameter set of the fourth frame based on the second dataset and the fourth training frame.

[0225] Specifically, this includes: step 514231, where the first training image and the first label segmentation image of each second data record in the second dataset are denoted as the corresponding image image x. e and label segmentation map y e ; and each image x e The current 3D image is used as input to the fourth training framework for processing, and the first predicted segmentation map output from this processing is recorded as the corresponding predicted segmentation map.

[0226] Step 514232: Combine the N3 pairs of labels and the predicted segmentation map y obtained this time. e , Substitute the preset fifth model loss function L M5 The corresponding fifth loss value is obtained through calculation;

[0227] Among them, the loss function L of the fifth model M5 Based on Dice loss function L Dice Implementation, specifically:

[0228]

[0229] Step 514233: Identify whether the fifth loss value meets the preset range of the fifth loss value; if not, then based on the preset fifth model optimizer, move towards making the fifth model loss function L... M5 The direction that reaches the minimum value is used to modulate the fourth frame parameter set in one round, and the process returns to step 514231 when the modulation ends; if the condition is met, the supervised training in this round is confirmed to be over.

[0230] The fifth loss value range is a pre-set numerical range; the fifth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0231] Step 51424, and confirm the end of the fourth step of training after the completion of this round of supervised training;

[0232] Step 515: After the fourth step of training is completed, the model parameters of the image semantic segmentation task head are fixed; and the fifth step of training is carried out based on the first and second datasets and the fifth training framework.

[0233] Specifically, this includes: Step 5151, after the fourth step of training is completed, the model parameters of the image semantic segmentation task head are solidified;

[0234] Step 5152, and perform the fifth training step based on the first and second datasets and the fifth training framework;

[0235] Specifically, this includes: step 51521, calculating the total number of records in the first dataset to obtain the corresponding total number N8; and calculating the total number of types in the first prediction type to obtain the corresponding total number. The total number of types in the second prediction type is then statistically analyzed to obtain the corresponding total. And denote the first training waveform, first label vector, and second label vector of each first data record in the first dataset as the corresponding waveform x. m Label vector y 1,m Label vector y 2,m 1 ≤ index m ≤ N8; and for each waveform x m Set a specific 2D training graph type;

[0236] Step 51522: Denote the first training image, the first label target set, and the first label segmentation image of each second data record in the second dataset as the corresponding image image x. e Tag set G e and label segmentation map y e ; and for each image x e Set a specific training graph type as 3D;

[0237] Step 51523, extract each waveform xm The corresponding training graph type is used as the input of the current 2D waveform graph and 2D / 3D task type into the fifth training framework for processing, and the first and second prediction vectors output from the current processing are recorded as the corresponding prediction vectors. and prediction vector

[0238] Step 51524, change the label vector y 1,m The probabilities of each first label are denoted as y. 1,m,s1 and predict vector The first prediction probabilities are denoted as the corresponding 1≤index And by each prediction vector and label vector y 1,m Form a corresponding fifth prediction-label pair; and combine the label vector y 2,m The probabilities of each second label are denoted as the corresponding y. 2,m,s2 and predict vector The respective second prediction probabilities are denoted as the corresponding 1≤index And by each prediction vector and label vector y 2,m A corresponding sixth prediction-label pair is formed; and the N8 fifth prediction-label pairs obtained this time form a corresponding prediction-label pair set. The N8 sixth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0239] Step 51525, divide each image x e The corresponding training map type is used as the current 3D image map and 2D / 3D task type input into the fifth training framework for processing, and the first predicted target set and the first predicted segmentation map output from the current processing are recorded as the corresponding prediction sets. and predicted segmentation map

[0240] Step 51526, for each tag set G e The total number N is obtained by counting the total number of label detection boxes. e ; and the tag set G e Each of the first label detection boxes is denoted as the corresponding label box. e,f ; and the prediction set In the middle and each label box e,f The nearest first object detection box is denoted as the corresponding matching box. and each label box e,fThe three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each type T e,f Converting it into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and comprised of various label boxes e,f Corresponding size S e,f Center P e,f Type T e,f Vector Z e,f Form the corresponding feature group bt e,f ;

[0241] Step 51527, match each box The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the detection frame type are denoted as the corresponding dimensions. center type and each type Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. And composed of each matching box Corresponding size center vector Form corresponding feature groups

[0242] Step 51528, from each feature group bt e,f and its corresponding feature groups A corresponding seventh prediction-label pair is formed; and each label segmentation map y is used to form a corresponding seventh prediction-label pair. e and its corresponding predicted segmentation map Form a corresponding eighth prediction-label pair; and use the N3×N obtained this time. e Each seventh prediction-label pair forms a corresponding set of prediction-label pairs. The N3 eighth prediction-label pairs obtained in this study form a corresponding prediction-label pair set.

[0243] Step 51529: The obtained prediction-label pair set... Substitute the preset multi-task loss function L MT The corresponding sixth loss value is obtained through calculation;

[0244] Here, the multi-task loss function L in this embodiment of the invention... MT The 2D task loss function L 2D and the 3D task loss function L 3DComposed of various components, including the 2D task loss function L. 2D The corresponding first and second 2D task loss functions Combining these elements, the 3D task loss function L... 3D Based on the corresponding first and second 3D task loss functions Composed of various elements; this multi-task loss function L MT Specifically:

[0245]

[0246] Among them, w 2D w 3D These are two preset weight parameters;

[0247] Step 51530: Identify whether the sixth loss value meets the preset range of the sixth loss value; if not, then based on the preset sixth model optimizer, move towards making the multi-task loss function L... MT The direction that reaches the minimum value modulates the model parameters of the image pre-trained model in one round, and returns to step 51523 when the modulation ends; if satisfied, the fifth step of training is confirmed to be over.

[0248] The sixth loss value range is a pre-set numerical range; the sixth model optimizer includes at least the Adam optimizer and the SGD optimizer.

[0249] Step 52, and at the end of the fifth step of training, confirm that the image pre-trained model has completed the domain-specific optimization training in the medical field.

[0250] Figure 3 This is a module structure diagram of a training device for an image pre-training model in the medical field, provided in Embodiment 2 of the present invention. This device can be a terminal device or server implementing the aforementioned method embodiments, or it can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiments. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 3 As shown, the device includes: a model preparation module 201, a basic data preparation module 202, a training framework preparation module 203, a training dataset preparation module 204, and a five-step training module 205.

[0251] The model preparation module 201 is used to select a visual pre-trained model that can simultaneously support 2D / 3D image feature encoding as the image pre-trained model.

[0252] The basic data preparation module 202 is used to use 2D waveforms and 3D images from the medical field as 2D and 3D images supported by the visual pre-training model; and to configure the corresponding image type set for the 2D waveforms as the corresponding waveform type set; and to set the corresponding diagnostic disease type set for each type of waveform in the waveform type set as the corresponding first disease type set, and to form the corresponding waveform-disease type set by all the obtained first disease type sets; and to configure the corresponding image type set and anatomical structure type set for the 3D images as the corresponding image type set and anatomical structure type set.

[0253] The training framework preparation module 203 is used to select four types of downstream task head models and connect them with the image pre-training model in separate and collective connection methods to obtain the corresponding five types of training frameworks. The four types of downstream task heads include waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head. The five types of training frameworks include the first, second, third, fourth, and fifth training frameworks.

[0254] The training dataset preparation module 204 is used to construct the first dataset based on the waveform image type set and the waveform-disease type set through big data acquisition; and to construct the second dataset based on the image image type set and the anatomical structure type set through big data acquisition.

[0255] The five-step training module 205 is used to perform five-step training in a progressive manner based on two datasets and five training frameworks; and at the end of the fifth step of training, it confirms that the image pre-trained model has completed domain-specific optimization training in the medical field.

[0256] The present invention provides a training device for image pre-training models in the medical field, which can execute the method steps in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0257] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the model preparation module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0258] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-on-a-Chip (SOC).

[0259] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0260] Figure 4 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 4 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0261] exist Figure 4The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0262] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0263] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0264] This invention provides a training method, apparatus, electronic device, and computer-readable storage medium for image pre-training models in the medical field. As described above, this invention first selects a type of pre-trained LVMs that can simultaneously support 2D / 3D image feature encoding as the image pre-training model; it then configures a waveform type set for the 2D waveforms supported by the visual pre-training model in the medical field, and sets disease type subsets for each type of waveform to obtain a corresponding full waveform-disease type set; and it configures corresponding image type sets and anatomical structure type sets for the 3D images supported by the visual pre-training model in the medical field; next, it selects four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head) and interfaces them with the image pre-training model to obtain five training frameworks; then, it constructs a first dataset based on the waveform type set and waveform-disease type set, and a second dataset based on the image type set and anatomical structure type set through big data collection; then, it performs five steps of training based on the two datasets and the five training frameworks in a progressive manner; and at the end of the fifth step of training, it confirms that the targeted training of the image pre-training model in the medical field is complete. In this invention, on the one hand, an image pre-trained model that is compatible with both 2D and 3D images and supports multiple waveforms and image types is obtained through SDT training in the medical field and serves as a shared encoding platform; on the other hand, the overall training difficulty is reduced and the overall training efficiency is improved through a progressive training method; and furthermore, the economic benefits of SDT training are improved by simultaneously training four types of downstream task head models (waveform classification task head, waveform disease prediction task head, image object detection task head, and image semantic segmentation task head).

[0265] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0266] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training an image pre-training model in a medical field, characterized in that, The method comprises: selecting a visual pre-training model capable of supporting 2D / 3D image feature encoding as an image pre-training model; 2D waveform graphs and 3D image graphs in the medical field are taken as 2D and 3D images supported by the visual pre-training model; a corresponding image type set is configured for the 2D waveform graphs, which is recorded as a corresponding waveform graph category set; a corresponding disease category set is set for each type of waveform graph in the waveform graph category set, which is recorded as a corresponding first disease category set, and a waveform-disease category set is formed by all the first disease category sets; a corresponding image type set and an anatomical structure type set are configured for the 3D image graphs, which are recorded as a corresponding image graph category set and an anatomical structure category set; four types of downstream task head models are selected to be connected to the image pre-training model in a separate connection mode and a collective connection mode to obtain five types of training frameworks; the four types of downstream task head models include a waveform classification task head, a waveform disease prediction task head, an image target detection task head, and an image semantic segmentation task head; the five types of training frameworks include first, second, third, fourth, and fifth training frameworks; a first data set is constructed according to the waveform graph category set and the waveform-disease category set through a big data collection method; and a second data set is constructed according to the image graph category set and the anatomical structure category set through a big data collection method; five-step training is performed according to the two data sets and the five types of training frameworks in a step-by-step training mode; and the specific field optimization training of the image pre-training model in the medical field is confirmed to be completed at the end of the fifth-step training; wherein the five-step training according to the two data sets and the five types of training frameworks specifically comprises: first-step training is performed according to the first data set and the first training framework; after the first-step training is completed, the model parameters of the waveform classification task head are solidified; and second-step training is performed according to the first data set and the second training framework; after the second-step training is completed, the model parameters of the waveform disease prediction task head are solidified; and third-step training is performed according to the second data set and the third training framework; after the third-step training is completed, the model parameters of the image target detection task head are solidified; and fourth-step training is performed according to the second data set and the fourth training framework; after the fourth-step training is completed, the model parameters of the image semantic segmentation task head are solidified; and fifth-step training is performed according to the first and second data sets and the fifth training framework.

2. The training method of the image pre-training model for the medical field according to claim 1, wherein the visual pre-training model is a visual large model compatible with 2D and 3D modalities and has been pre-trained based on a large amount of image corpus in a general field; the visual large model at least includes a MultiResViT model, a UniFormer model, a Hybrid CNN-Transformer model, an nnU-Net model, a Uni3D model, a Uni-MAE model, and a MedFMC model. ​ The waveform graph category set includes at least electrocardiogram, electroencephalogram, electromyogram; The waveform-disease category set includes a plurality of the first disease category set; the disease categories of the first disease category set corresponding to the electrocardiogram include at least atrial fibrillation, premature beat, tachycardia, bradycardia, myocardial infarction, myocardial ischemia, cardiac conduction block; the disease categories of the first disease category set corresponding to the electroencephalogram include at least epilepsy, cerebral hemorrhage, cerebral thrombosis, cerebral infarction, encephalitis, brain tumor, brain trauma, schizophrenia, manic depression, mental abnormality; the disease categories of the first disease category set corresponding to the electromyogram include at least myasthenia gravis, muscular dystrophy, polymyositis, dermatomyositis, amyotrophic lateral sclerosis, Guillain-Barre syndrome, multiple peripheral neuropathy, oculomotor nerve injury, trochlear nerve injury, facial neuritis; The image graph category set includes at least computed tomography, magnetic resonance imaging, three-dimensional ultrasonic imaging, positron emission computed tomography; The anatomical structure category set includes at least a plurality of organ types; the organ types include at least heart, lung, liver, pancreas, stomach, duodenum, jejunum, ileum, cecum, colon, rectum, urinary bladder; The first training framework is connected by an output end of the image pre-training model and an input end of the waveform classification task head; the first training framework is used for waveform graph classification prediction processing according to the 2D waveform graph of the framework input and outputs a corresponding first prediction vector; wherein the first prediction vector is composed of a plurality of first prediction probabilities, each first prediction probability corresponds to a first prediction type; the first prediction type includes unknown waveform type and multiple known waveform types; the known waveform type corresponds to the waveform graph category in the waveform graph category set one by one; The second training framework is connected by an output end of the image pre-training model and an input end of the waveform disease prediction task head; the second training framework is used for disease classification prediction processing according to the 2D waveform graph of the framework input and outputs a corresponding second prediction vector; wherein the second prediction vector is composed of a plurality of second prediction probabilities, each second prediction probability corresponds to a second prediction type; the second prediction type includes unknown disease type and multiple known disease types; the known disease type corresponds to the disease category in the waveform-disease category set one by one; The third training framework is connected by an output end of the image pre-training model and an input end of the image target detection task head; the third training framework is used for organ target detection and classification identification processing according to the 3D image graph of the framework input and outputs a corresponding first prediction target set; wherein the first prediction target set is composed of one or more first target detection boxes; the first target detection box includes at least detection box identification, detection box three-dimensional size, detection box center point coordinates, detection box type; the detection box type includes unknown target type and multiple known target types; the known target type corresponds to the organ type of the anatomical structure category set one by one; The fourth training framework is connected by an output end of the image pre-training model and an input end of the image semantic segmentation task head; the fourth training framework is used for performing organ semantic segmentation processing at a voxel level on the 3D image picture input by the framework and outputting a corresponding first prediction segmentation picture; wherein, a three-dimensional size of the first prediction segmentation picture is consistent with a three-dimensional size of the 3D image picture input by the current framework; voxel-level features of the first prediction segmentation picture at least include a semantic type; the semantic type includes a background semantic, an unknown organ semantic and a plurality of known organ semantics, and the known organ semantics one-to-one correspond to the organ types in the anatomical structure type set; The fifth training framework is composed of the image pre-training model, a feature distribution module, the waveform classification task head, the waveform disease prediction task head, the image object detection task head and the image semantic segmentation task head; The fifth training framework is used for identifying a 2D / 3D task type input by the current framework; if the current 2D / 3D task type is a 2D type, then waveform graph classification prediction and disease classification prediction processing are performed on the 2D waveform graph input by the current framework and corresponding first and second prediction vectors are outputted; if the current 2D / 3D task type is a 3D type, then organ object detection and classification identification and organ semantic segmentation processing are performed on the 3D image picture input by the current framework and corresponding first prediction object set and first prediction segmentation picture are outputted; wherein, the 2D / 3D task type includes a 2D type and a 3D type, the 2D / 3D task type corresponding to the current framework input image being a 2D waveform graph is a 2D type, and the 2D / 3D task type corresponding to the current framework input image being a 3D image picture is a 3D type; The first data set includes a plurality of first data records; the first data record includes a first training waveform graph, a first label vector and a second label vector; the first training waveform graph is a waveform graph in a medical field; the first label vector is composed of a plurality of first label probabilities, each first label probability corresponding to a first prediction type; the second label vector is composed of a plurality of second label probabilities, each second label probability corresponding to a second prediction type; only one label probability in the first and second label vectors is 1 and the rest of the label probabilities are 0; a waveform graph type set of all the first training waveform graphs in the first data set is greater than the waveform graph type set; the total number of data records corresponding to each first prediction type in the first data set is the same; the total number of data records corresponding to each second prediction type in the first data set is the same; The second data set includes a plurality of second data records; the second data records include a first training image, a first label target set, and a first label segmentation map; the first training image is a type of the 3D image; the first label target set is composed of one or more first label detection boxes; the first label detection box at least includes the detection box identifier, the detection box three-dimensional size, the detection box center point coordinate, and the detection box type; the three-dimensional size of the first label segmentation map is consistent with the three-dimensional size of the first training image; the voxel-level features of the first label segmentation map at least include the semantic type; the image type set of all the first training images in the second data set matches the image type set; the anatomical structure type set of all the first training images in the second data set is greater than the anatomical structure type set; the total number of data records corresponding to each type of image type in the second data set is the same, and the total number of detection boxes corresponding to each type of detection box in all data records corresponding to each type of image type is the same, and the total number of image segmentation regions corresponding to each type of known organ semantics in all data records corresponding to each type of image type is the same.

3. The training method of the image pre-training model for the medical field according to claim 2, characterized in that, in the first training framework, the image pre-training model is used for feature encoding processing on the 2D waveform image input by the current framework to obtain a corresponding first encoding feature tensor, which is sent to the waveform classification task head; the waveform classification task head is used for classification prediction processing according to the first encoding feature tensor and outputs a corresponding first prediction vector; in the second training framework, the image pre-training model is used for feature encoding processing on the 2D waveform image input by the current framework to obtain a corresponding second encoding feature tensor, which is sent to the waveform disease prediction task head; the waveform disease prediction task head is used for classification prediction processing according to the second encoding feature tensor and outputs a corresponding second prediction vector; in the third training framework, the image pre-training model is used for feature encoding processing on the 3D image input by the current framework to obtain a corresponding third encoding feature tensor, which is sent to the image target detection task head; the image target detection task head is used for target detection and classification identification processing according to the third encoding feature tensor and outputs a corresponding first prediction target set; in the fourth training framework, the image pre-training model is used for feature encoding processing on the 3D image input by the current framework to obtain a corresponding fourth encoding feature tensor, which is sent to the image semantic segmentation task head; the image semantic segmentation task head is used for voxel-level semantic segmentation processing according to the fourth encoding feature tensor and outputs a corresponding first prediction segmentation map; The first framework input end of the fifth training framework is configured to receive the 2D waveform graph or the 3D image graph of the current framework input, the second framework input end is configured to receive the 2D / 3D task type of the current framework input, and the first, second, third and fourth framework output ends are configured to output the corresponding first prediction vector, second prediction vector, first prediction target set and first prediction segmentation graph; In the fifth training framework, the input end of the image pre-training model is connected with the first framework input end, and the output end is connected with the first input end of the feature distribution module; the second input end of the feature distribution module is connected with the second framework input end, and the output end is connected with the input ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head and the image semantic segmentation task head; the output ends of the waveform classification task head, the waveform disease prediction task head, the image target detection task head and the image semantic segmentation task head are respectively connected with the corresponding first, second, third and fourth framework output ends; In the fifth training framework, the image pre-training model is configured to perform feature encoding processing on the 2D waveform graph or the 3D image graph of the current framework input to obtain the corresponding fifth encoding feature tensor, and send the fifth encoding feature tensor to the feature distribution module; the feature distribution module is configured to identify the 2D / 3D task type of the current framework input, if the current 2D / 3D task type is 2D type, the fifth encoding feature tensor is sent to the waveform classification task head and the waveform disease prediction task head, if the current 2D / 3D task type is 3D type, the fifth encoding feature tensor is sent to the image target detection task head and the image semantic segmentation task head; the waveform classification task head is configured to, when receiving the fifth encoding feature tensor, perform classification prediction processing according to the current encoding feature tensor and output the corresponding first prediction vector; the waveform disease prediction task head is configured to, when receiving the fifth encoding feature tensor, perform classification prediction processing according to the current encoding feature tensor and output the corresponding second prediction vector; the image target detection task head is configured to, when receiving the fifth encoding feature tensor, perform target detection and classification identification processing according to the current encoding feature tensor and output the corresponding first prediction target set; the image semantic segmentation task head is configured to, when receiving the fifth encoding feature tensor, perform voxel-level semantic segmentation processing according to the current encoding feature tensor and output the corresponding first prediction segmentation graph. 4.The training method of an image pre-training model for a medical field according to claim 2, characterized in that, The first step training according to the first data set and the first training framework specifically includes: Step 51, based on the preset first segmentation ratio, the first data set is randomly segmented into two sub-data sets, denoted as a corresponding first training set and a first evaluation set; and the model parameter set of the image pre-training model and the waveform classification task head is taken as a corresponding first framework parameter set; The first training set and the first evaluation set are composed of a plurality of first data records; the total number of records of the first training set and the first evaluation set meets the first split ratio; Step 52, count the total number of records of the first training set to obtain the corresponding total number N1; and count the total number of types of the first prediction type to obtain the corresponding total number C1; and record the first training waveform diagram and the first label vector of each first data record of the first training set as the corresponding waveform diagram x a and the label vector y a , 1≤index a≤N1; and record each waveform diagram x a as the current 2D waveform diagram, input the first training framework for processing, and record the first prediction vector output by the current processing as the corresponding prediction vector ; and record each first label probability of the label vector y a as the corresponding y a,b , and record each first prediction probability of the prediction vector as the corresponding , 1≤index b≤C1; and each prediction vector and the label vector y a comprise a corresponding first prediction-label pair; and the obtained N1 first prediction-label pairs are brought into a preset first model loss function L M1 to obtain the corresponding first loss value; The first model loss function L M1 Based on the cross-entropy loss function implementation, specifically: ； Step 53, identify whether the first loss value meets a preset first loss value range; if yes, go to step 54; if not, based on a preset first model optimizer, adjust the first model loss function L M1 to a minimum value, and return to step 52 at the end of this round of adjustment. The first model optimizer at least includes an Adam optimizer and an SGD optimizer; Step 54, all the first data records of the first evaluation set are traversed for one round; and in this round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; the first training waveform graph of the current evaluation record is taken as the current 2D waveform graph, which is input into the first training framework for processing, and the first prediction vector output by the processing is taken as a corresponding first prediction vector; and a corresponding second prediction-label pair is composed of the second prediction vector and the second label vector of the current evaluation record; and at the end of this round of traversal, the first accuracy, the first precision, the first recall and the first F1 score are obtained by estimating the accuracy, the precision, the recall and the F1 score according to all the second prediction-label pairs obtained; Step 55, whether the first accuracy, the first precision, the first recall and the first F1 score all meet the corresponding first accuracy range, the first precision range, the first recall range and the first F1 score range is identified; if not, return to step 52 for training; if yes, it is confirmed that the first step of training is completed. 5.The training method of an image pre-training model for a medical field according to claim 2, characterized in that, The second step of training according to the first data set and the second training framework specifically includes: Step 61, based on a preset second split ratio, the first data set is randomly divided into two sub-data sets, which are taken as a corresponding second training set and a second evaluation set; and the model parameter set of the image pre-training model and the waveform disease prediction task head is taken as a corresponding second framework parameter set; The second training set and the second evaluation set are composed of a plurality of first data records; the total number of records of the second training set and the second evaluation set meets the second split ratio; Step 62, count the total number of records of the second training set to obtain the corresponding total number N2; count the total number of types of the second prediction type to obtain the corresponding total number C2; and record the first training waveform graph and the second label vector of each first data record of the second training set as the corresponding waveform graph x c and label vector y c , 1≤index c≤N2; and record each of the waveform graphs x c as the current 2D waveform graph, input the second training framework for processing, and record the second prediction vector output by the current processing as the corresponding prediction vector ; and record each of the second label probabilities of the label vector y c as the corresponding y c,d , and record each of the second prediction probabilities of the prediction vector as the corresponding , 1≤index d≤C2; and each of the prediction vector and the label vector y c comprises a corresponding third prediction-label pair; and the N2 third prediction-label pairs obtained are brought into a preset second model loss function L M2 to obtain the corresponding second loss value; The second model loss function L M2 Based on the cross-entropy loss function implementation, specifically: ； Step 63, identify whether the second loss value meets the preset second loss value range; if yes, go to step 64; if not, based on the preset second model optimizer, adjust the second model loss function L M2 to the minimum direction to the second frame parameter set for a round of modulation, and return to step 62 at the end of this round of modulation; The second model optimizer at least includes an Adam optimizer and an SGD optimizer; Step 64, all the first data records of the second evaluation set are traversed for one round; and in this round of traversal, the first data record currently traversed is taken as a corresponding current evaluation record; the first training waveform graph of the current evaluation record is taken as the current 2D waveform graph, which is input into the second training framework for processing, and the second prediction vector output by the processing is taken as a corresponding second prediction vector; and a corresponding fourth prediction-label pair is composed of the second prediction vector and the second label vector of the current evaluation record; and at the end of this round of traversal, the second accuracy, the second precision, the second recall and the second F1 score are obtained by estimating the accuracy, the precision, the recall and the F1 score according to all the fourth prediction-label pairs obtained; Step 65, whether the second accuracy, the second precision, the second recall and the second F1 score meet the corresponding second accuracy range, second precision range, second recall range and second F1 score range respectively is identified; if not, return to step 62 for continuous training; if yes, it is confirmed that the second step training is completed. 6.The training method of an image pre-training model for a medical field according to claim 2, characterized in that, The third step training according to the second data set and the third training framework specifically comprises: Step 71, the total number of records of the second data set is counted to obtain a corresponding total number N3; and the total number of organ types of the anatomical structure type set is counted to obtain a corresponding total number N4; and the model parameter set of the image pre-training model and the image target detection task head is taken as a corresponding third framework parameter set; Step 72, record the first training image and the first label target set of each second data record of the second data set as the corresponding image x e and label set G e , 1≤index e≤N3; and record each image x e as the current 3D image input into the third training framework for processing and record the first prediction target set output by the processing as the corresponding prediction set ; Step 73, count the total number of label detection boxes of each of the label set G e to obtain the corresponding total number N e ; and record each of the first label detection box of the label set G e as the corresponding label box box e,f , 1≤index f≤N e ; and record the first target detection box in the prediction set closest to each of the label box box e,f as the corresponding matching box ; Step 74, set each of the aforementioned label boxes. e,f The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimension S. e,f Center P e,f Type T e,f ; and each of the aforementioned types T e,f Converting it into a one-hot encoded vector of length N4, denoted as the corresponding vector Z. e,f ; and each of the aforementioned label boxes e,f The corresponding size S e,f The center P e,f The type T e,f The vector Z e,f Form the corresponding feature group bt e,f ; wherein the vector Z e,f includes N4one-hot encodings z e,f,g , 1≤index g≤N4; the vector Z e,f only the one-hot encoding z e,f corresponding to the type T e,f,g is 1, and the rest are all 0; Step 75, match each of the aforementioned matching boxes. The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. ,center ,type ; and each of the aforementioned types Convert it into a one-hot encoded vector of length N4, denoted as the corresponding vector. ; and by each of the aforementioned matching boxes The corresponding size The center The vector Form corresponding feature groups ; wherein the vector includes N4 one-hot encodings ; the vector has only the one-hot encoding corresponding to the type of 1, and the rest of 0s. ​ Step 76, the N3xN e For the feature set bt e,f 、 The preset third model loss function L M3 The corresponding third loss value is calculated. The third model loss function L M3 The third model loss function L is composed of a cross-entropy loss function for loss calculation of classification, an L2 loss function for loss calculation of the center position, and an L1 loss function for loss calculation of the three-dimensional size, and is specifically: ； Step 77, identify whether the third loss value meets the preset third loss value range; if not, based on the preset third model optimizer, adjust the third model loss function L M3 to the minimum direction to the third frame parameter set for a round of modulation, and return to step 72 at the end of this round of modulation; if yes, confirm that the third step training is completed; The third model optimizer at least includes an Adam optimizer and an SGD optimizer.

7. The training method of an image pre-training model for a medical field according to claim 6, characterized in that, The fourth step training according to the second data set and the fourth training framework specifically comprises: The model parameter set of the image pre-training model and the image semantic segmentation task head is taken as a corresponding fourth framework parameter set; And a round of self-supervised contrast learning training is performed on the model parameters of the image pre-training model based on the second data set and the fourth training framework, aiming to improve the feature consistency of multi-modal images under the same anatomical structure; And a round of supervised model training is performed on the fourth framework parameter set based on the second data set and the fourth training framework, aiming to improve segmentation consistency; And it is confirmed that the fourth step training is completed after the round of supervised training is completed.

8. The training method of an image pre-training model for a medical field according to claim 7, characterized in that, The round of self-supervised contrast learning training performed on the model parameters of the image pre-training model based on the second data set and the fourth training framework, aiming to improve the feature consistency of multi-modal images under the same anatomical structure, specifically comprises: Step 91, record each of the first training image graphs in the second data set containing only one type of organ as a corresponding single-organ image graph, count the total number of the single-organ image graphs to obtain a corresponding total number N5, and form a corresponding multi-organ image set X composed of N5 single-organ image graphs ALL ; and record each of the single-organ image graphs as a corresponding image graph x i , 1≤index i≤N5; Step 92, and count the total number of organ types N6 corresponding to the total number of the multi-organ image set X ALL ; and group the multiple single-organ image maps corresponding to the same organ type in the multi-organ image set X ALL into a corresponding single-organ image set X j , 1≤index j≤N6; and count the total number of images N i corresponding to each single-organ image set X 7,j ; Step 93, and each of the image x i As the corresponding current image; and the single organ image set X j Recorded as the corresponding current positive sample set; and all the rest of the single organ image set X j Composed of the corresponding current negative sample set; and each of the single organ image in the current positive sample set, except the current image, as the positive sample image corresponding to the current image , 1≤ positive sample index p(i)≤ , The total number of positive samples corresponding to the current image, The value of the total number N 7,j Subtract 1; and each of the single organ image in the current negative sample set as the negative sample image corresponding to the current image , 1≤ negative sample index n(i)≤ , The total number of negative samples corresponding to the current image, ; Step 94, encode each of the image x i As the current 3D image, input the image pre-training model for feature encoding processing, and output the encoding feature tensor of this processing as the corresponding feature tensor y i ; and each of the image x i Corresponding to each of the positive sample image The corresponding encoding feature tensor is recorded as the corresponding positive sample tensor ; and each of the image x i Corresponding to each of the negative sample image The corresponding encoding feature tensor is recorded as the corresponding negative sample tensor ; and each of the feature tensor y i Corresponding to The positive sample tensor Corresponding to the positive sample set , each of the feature tensor y i Corresponding to The negative sample tensor Corresponding to the negative sample set ; and the three-dimensional size of each of the feature tensor y i The corresponding height H i , width W i and depth D i , and each of the feature tensor y i Viewed as a three-dimensional feature map composed of H i ×W i ×D i Voxel tensor , 1≤index h i ≤H i , 1≤index w i ≤W i , 1≤index d i ≤D i ; Step 95, the N5 feature vectors y obtained this time i and the corresponding positive and negative sample sets , into the preset fourth model loss function L M4 to obtain the corresponding fourth loss value; The fourth model loss function L M4 The InfoNCE loss function and the non-integer order total variation loss function are combined, and specifically, the combination is as follows: ， ， ； λ is a preset weighting coefficient; L infoNCE () is an InfoNCE loss function; sim() is a similarity function, which at least includes a dot product similarity function, a cosine vector similarity function, and an Euclidean distance similarity function; and τ is a preset temperature coefficient. L FOTV () is a non-integer order total variation loss function; and a is a preset non-integer order parameter; is an a-order derivative operator along a three-dimensional direction, and specifically: 、 、 are a-order derivative operators along the height, width and depth directions, respectively. Step 96, identifying whether the fourth loss value meets a preset fourth loss value range; if not, based on a preset fourth model optimizer, the model parameters of the image pre-training model are modulated towards a direction of making the fourth model loss function L M4 reach a minimum value, and returning to step 94 at the end of the current modulation; if yes, it is confirmed that the current contrast learning training is ended. The fourth model optimizer at least includes an Adam optimizer and an SGD optimizer. 9.The training method of an image pre-training model for a medical field according to claim 7, characterized in that, The round of supervised model training performed on the fourth framework parameter set based on the second data set and the fourth training framework, aiming to improve segmentation consistency, specifically comprises: Step 101, record the first training image and the first label segmentation map of each second data record of the second data set as the corresponding image x e and label segmentation map y e ; and record each image x e as the current 3D image, input the fourth training framework for processing, and record the first prediction segmentation map output by the processing as the corresponding prediction segmentation map ; Step 102, inputting the obtained N3 this time into the label, the predicted segmentation graph y e 、 into the preset fifth model loss function L M5 to obtain the corresponding fifth loss value; The fifth model loss function L M5 The Dice loss function L Dice The implementation is specifically: ； Step 103, identifying whether the fifth loss value meets a preset fifth loss value range; if not, based on a preset fifth model optimizer, the fifth model loss function L M5 reaches a minimum value, a round of modulation is performed on the fourth framework parameter set, and step 101 is returned at the end of the round of modulation; if yes, it is confirmed that the current round of supervised training is completed; The fifth model optimizer at least includes an Adam optimizer and an SGD optimizer.

10. The training method of an image pre-training model for a medical field according to claim 9, characterized in that, The fifth step training according to the first and second data sets and the fifth training framework specifically comprises: Step 111, count the total number of records of the first data set to obtain the corresponding total number N8; and count the total number of types of the first prediction type to obtain the corresponding total number N9 ; and count the total number of types of the second prediction type to obtain the corresponding total number N10 ; and record the first training waveform diagram, the first label vector, and the second label vector of each of the first data records of the first data set as the corresponding waveform diagram x m , label vector y 1,m , label vector y 2,m , 1≤index m≤N8; and set a training diagram type of a specific 2D type for each of the waveform diagrams x m ; Step 112, record the first training image, the first label target set and the first label segmentation map of each second data record of the second data set as the corresponding image x e , the label set G e and the label segmentation map y e ; and set a training map type, specifically a 3D type, for each image x e ; Step 113, inputting each of the waveform graph x m and its corresponding training graph type as the current 2D waveform graph and 2D / 3D task type into the fifth training framework for processing and recording the first and second prediction vectors output by the processing as corresponding prediction vectors and prediction vector ; Step 114, record each first label probability of the label vector y 1,m as corresponding y 1,m,s1 , and record each first prediction probability of the prediction vector as corresponding , 1≤index s1≤ ; and each prediction vector and the label vector y 1,m form a corresponding fifth prediction-label pair; and record each second label probability of the label vector y 2,m as corresponding y 2,m,s2 , and record each second prediction probability of the prediction vector as corresponding , 1≤index s2≤ ; and each prediction vector and the label vector y 2,m form a corresponding sixth prediction-label pair; and the N8 fifth prediction-label pairs obtained this time form a corresponding prediction-label pair set , and the N8 sixth prediction-label pairs obtained this time form a corresponding prediction-label pair set ; Step 115, inputting each of the image graph x e and the corresponding training graph type as the current 3D image graph and the 2D / 3D task type into the fifth training framework for processing and recording the first prediction target set and the first prediction segmentation graph output by the processing as the corresponding prediction set and the prediction segmentation graph ; Step 116, count the total number of label detection boxes of each of the label set G e to obtain the corresponding total number N e ; and record each of the first label detection box of the label set G e as the corresponding label box box e,f ; and record the first target detection box in the prediction set closest to each of the label box box e,f as the corresponding matching box ; and record the detection box three-dimensional size, the detection box center point coordinates, the detection box type of each of the label box box e,f as the corresponding size S e,f , the center P e,f , the type T e,f ; and convert each of the type T e,f into a one-hot encoding vector with a vector length of N4, and record it as the corresponding vector Z e,f ; and compose the corresponding feature group bt e,f from the size S e,f , the center P e,f , the type T e,f , and the vector Z e,f of each of the label box box e,f . Step 117, match each of the aforementioned matching boxes. The three-dimensional dimensions of the detection frame, the coordinates of the center point of the detection frame, and the type of the detection frame are denoted as the corresponding dimensions. The center The type ; and each of the aforementioned types This is converted into a one-hot encoded vector of length N4, denoted as the corresponding vector. ; and by each of the aforementioned matching boxes The corresponding size The center The vector The corresponding feature groups are formed ; Step 118, each of the feature groups bt e,f and its corresponding feature group constitute a corresponding seventh prediction-label pair; and each of the label segmentation maps y e and its corresponding prediction segmentation map constitute a corresponding eighth prediction-label pair; and the N3 x N e seventh prediction-label pairs obtained this time constitute a corresponding prediction-label pair set the N3 eighth prediction-label pairs obtained this time constitute a corresponding prediction-label pair set ; Step 119, the prediction-label pair set obtained this time is taken into a preset multi-task loss function L 、 、 、 into a preset multi-task loss function L MT to obtain a corresponding sixth loss value; Wherein, the multi-task loss function L MT Combined by the 2D task loss function L 2D And the 3D task loss function L 3D , the 2D task loss function L 2D Combined by the corresponding first and second 2D task loss functions , , the 3D task loss function L 3D Combined by the corresponding first and second 3D task loss functions , ; the multi-task loss function L MT Specifically: ， ， ， ， ， , ； w 2D , w 3D are two preset weight parameters; Step 120, identifying whether the sixth loss value meets a preset sixth loss value range; if not, based on a preset sixth model optimizer, the model parameters of the image pre-training model are modulated in a direction of making the multi-task loss function L MT reach a minimum value, and returning to step 113 at the end of the current modulation; if yes, confirming that the fifth step training is completed. The sixth model optimizer at least includes an Adam optimizer and an SGD optimizer.

11. An apparatus for performing the training method of the image pre-training model for the medical field according to any one of claims 1-10, characterized in that, The device comprises a model preparation module, a basic data preparation module, a training framework preparation module, a training data set preparation module and a five-step training module. The model preparation module is used to select a visual pre-training model capable of supporting 2D / 3D image feature coding as an image pre-training model; The basic data preparation module is configured to take 2D waveform graphs and 3D image graphs in the medical field as 2D and 3D images supported by the visual pre-training model; configure a corresponding image type set for the 2D waveform graphs as a corresponding waveform graph category set; set a corresponding disease category set for each type of waveform graph in the waveform graph category set as a corresponding first disease category set, and form a corresponding waveform-disease category set from all the first disease category sets; and configure a corresponding image type set and an anatomical structure type set for the 3D image graphs as a corresponding image graph category set and an anatomical structure category set; The training framework preparation module is configured to select four types of downstream task head models to be connected to the image pre-training model in a one-by-one connection mode and a collective connection mode to obtain five types of training frameworks; the four types of downstream task head models include a waveform classification task head, a waveform disease prediction task head, an image target detection task head and an image semantic segmentation task head; and the five types of training frameworks include a first, a second, a third, a fourth and a fifth training framework; The training data set preparation module is configured to construct a first data set according to the waveform graph category set and the waveform-disease category set by a big data collection method; and construct a second data set according to the image graph category set and the anatomical structure category set by a big data collection method; The five-step training module is configured to perform five-step training according to the two data sets and the five types of training frameworks in a step-by-step training mode; and confirm that the specific field optimization training of the image pre-training model in the medical field is completed at the end of the fifth step of training.

12. An electronic device, comprising: Comprise: a memory, a processor and a transceiver; The processor is configured to be coupled with the memory, read and execute instructions in the memory to implement the method of any one of claims 1-10; The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transmission and reception.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by a computer, the computer instructions make the computer execute the method of any one of claims 1-10.

Images