Ddh ultrasound image analysis method based on contrast learning pre-training

Abstract

The application discloses a DDH ultrasonic image analysis method based on contrast learning pre-training, first acquires unlabeled DDH ultrasonic image data; constructs a feature extractor G of a neural network, uses an unlabeled data set, and trains the feature extractor of the neural network in a contrast learning mode; then, according to a key anatomical structure to be checked, part of the data is labeled to form a labeled ultrasonic data set; according to a task target, a task processing network T is constructed, the task processing network T is combined with the trained feature extractor G to form a neural network model O; the neural network model O is trained using the labeled ultrasonic data set to realize DDH ultrasonic image analysis. The method pre-trains the feature extraction network using a large amount of unlabeled data, improves the feature extraction capability, and thus improves the performance of the downstream task in the ultrasonic image examination using deep learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical ultrasound image processing, and mainly relates to a DDH ultrasound image analysis method based on contrast learning pre-training. BACKGROUND

[0002] Developmental dysplasia of the hip (DDH) is the most common musculoskeletal disease that seriously endangers the health of infants. The common examination method is to perform ultrasound image examination. Ultrasound image examination is a non-invasive examination method, which has the advantages of low cost, non-destructive, repeatability, etc. Compared with CT (computed tomography) and MRI (magnetic resonance imaging) methods, it is the preferred image analysis method for screening DDH. However, the diagnostic process requires high anatomical knowledge and experience of the physician, and there is a great subjective factor in ultrasound image diagnosis, and the whole diagnostic process is laborious and time-consuming.

[0003] Deep learning can directly process raw ultrasound image data, analyze ultrasound data, extract image features, and then complete various downstream tasks such as lesion / nodule classification, tissue segmentation, and target detection. Using deep learning to complete automatic ultrasound image examination and auxiliary diagnosis can provide relatively stable diagnostic results and reduce the workload of doctors.

[0004] Using deep learning to complete automatic DDH ultrasound image examination and auxiliary diagnosis requires a large amount of standardized and effective data. However, the current public ultrasound data set is often limited in quantity, which is difficult to meet the needs of deep learning. On the other hand, although hospitals have a large amount of historical ultrasound image data, medical ultrasound images cannot be used for deep learning training without standardized data annotation. Therefore, deploying deep learning ultrasound image examination requires high data annotation cost, and it is difficult to fully utilize the large amount of historical ultrasound data. SUMMARY

[0005] The present application is just for the existing DDH ultrasound image examination and auxiliary diagnosis needs a large number of standard, effective data, the data workload is too large and professional, provides a kind of DDH ultrasound image analysis method based on contrast learning pretraining, first, the ultrasound gray scale image obtained in DDH ultrasound diagnosis is preprocessed after data enhancement, and forms a no-label data set;The feature extractor G of neural network is constructed, and the feature extractor of neural network is trained in the manner of contrast learning using no-label data set;Part of the data is labeled according to the key anatomical structure to be examined to form a labeled ultrasound data set;According to the task target, the task processing network T is constructed, the task processing network T and the trained feature extractor G are combined to form a neural network model O;The labeled ultrasound data set is used to train the neural network model O, and the task performance of the neural network model O is tested, and the DDH ultrasound image analysis is realized.The method uses contrast learning to pretrain no-label data, so that the feature extraction network in the process of deep learning ultrasound detection can learn the basic features of corresponding medical anatomical structure from a large number of no-label DDH ultrasound data, alleviate the difficulty of data labeling when using deep learning for DDH ultrasound image analysis, and the small scale of downstream task data set, and then improve the performance of neural network in downstream task.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is: a DDH ultrasound image analysis method based on contrast learning pretraining, comprising the following steps:

[0007] S1: the ultrasound gray scale image obtained in DDH ultrasound diagnosis is preprocessed after data enhancement, and forms a no-label data set;

[0008] S2: the feature extractor G of neural network is constructed, and the feature extractor of neural network is trained in the manner of contrast learning using no-label data set obtained in step S1;

[0009] S3: part of the data is labeled according to the key anatomical structure to be examined to form a labeled ultrasound data set;

[0010] S4: according to the task target, the task processing network T is constructed, the task processing network T and the trained feature extractor G are combined to form a neural network model O;

[0011] S5: the labeled ultrasound data set labeled in step S3 is used to train the neural network model O, and the task performance of the neural network model O is tested, and the DDH ultrasound image analysis is realized.

[0012] As an improvement of the present application, the data set after data enhancement in step S1 is a gray scale picture with picture resolution of 224*224, single channel, pixel value range of 0-255.

[0013] As an improvement of the present application, the feature extractor G of the neural network constructed in step S2 takes the images in the data set as input and outputs a feature vector

[0014]

[0015] wherein, x i is any one image in the data set, i is the data sequence number, and the value range is (1, n).

[0016] As another improvement of the present application, the training process of the neural network in step S2 is specifically contrastive learning, and the process is as follows:

[0017] The input image x i in the data set is respectively subjected to pixel-level data enhancement in different probabilities of brightness, random affine transformation, random contrast, and random scaling to obtain enhanced image data pairs y i1 ,y i2 .

[0018] y i1 ,y i2 are respectively input into the feature extraction networks G1 and G2 to obtain corresponding feature vectors wherein, the feature extraction networks G1 and G2 have the same structure as G, and the parameters are θ1 and θ2, respectively, and are randomly initialized; a multi-layer perception P is constructed, the parameters are randomly initialized, and is used to transform into , which has the same format as .

[0019] In each training round, the network parameters θ1 and θ2 are updated alternately according to the following formula, and after several rounds of training, the learned network parameters η are obtained.

[0020]

[0021]

[0022] wherein, t represents the current training round, G2(θ2 t-1 ,y i1 ) represents that the G2 network uses the network parameters θ2 t-1 of the previous training round to accept the input y i1 and obtain the output, and L is a cosine similarity loss function.

[0023] As another improvement of the present application, the loss function L is:

[0024]

[0025] wherein, Output the feature vector for building multi-layer perception P, Output the feature vector for network G.

[0026] As another improvement of the application, the task processing network T of step S4 is implemented by multiple convolution layers, activation functions and pooling layers alternately or is composed of fully connected layers.

[0027] As another improvement of the application, the model training in step S5 is specifically: the labeled ultrasound data set and the corresponding data set labeled in step S3 are combined into a total data set, and are divided into a training set, a validation set and a test set, the training set is used, the parameters of the feature extraction network G are fine-tuned based on the back propagation mode, and the parameters of each layer of the task processing network T are fitted; the validation set is used to preliminarily evaluate the discrimination ability of the current model O, and the generalization ability of the model is verified to determine whether to stop training; the test set is used to evaluate the final generalization ability of the model O.

[0028] Compared with the prior art, the application has the beneficial effects: the application discloses a DDH ultrasound image analysis method based on contrast learning pre-training, uses contrast learning to pre-train unlabeled data, so that the feature extraction network in the deep learning ultrasound detection process can learn the basic features of the corresponding medical anatomical structure from a large amount of unlabeled DDH ultrasound data, reduces the need for a large amount of data labeling in the analysis of DDH ultrasound images using deep learning technology, reduces the workload of doctors for data labeling, and can fully utilize a large amount of historical accumulated ultrasound data to improve the utilization rate of data; the contrast learning process in the application can guide the model to learn high-level semantic information through unsupervised learning of a large amount of unlabeled data, and improve the detection accuracy of the target detection network. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The step flow chart of the DDH ultrasound image analysis method based on contrast learning pre-training of the application;

[0030] Figure 2 The schematic diagram of each round of parameter updating in the contrast learning pre-training process of step S2 of the method of the application;

[0031] Figure 3 The schematic diagram of the key medical landmark point extraction of the DDH ultrasound image using the method of the application in embodiment 2 of the application. DETAILED DESCRIPTION

[0032] The application will be further illustrated in combination with the drawings and the specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and are not used to limit the scope of the application.

[0033] Embodiment 1

[0034] A DDH ultrasound image analysis method based on contrast learning pre-training, as shown in the formula: Figure 1

[0035] S1: After data enhancement preprocessing of the ultrasound gray scale images obtained in the DDH ultrasound diagnosis, a unlabeled data set is formed;

[0036] S11: Collecting ultrasound data;

[0037] Collecting ultrasound gray scale images generated in the DDH ultrasound diagnosis process from the ultrasound examination database of the hospital, and requiring clear images. Forming a data set A.

[0038] S12: Processing ultrasound data to form a unlabeled data set;

[0039] Performing padding or cropping operation on each ultrasound gray scale image in the data set A respectively; wherein the padding refers to filling fixed pixel values around the image, and the cropping refers to deleting the pixels around the image to change the image size, ensuring that the resolution of the processed picture is 224*224, single channel, and the pixel value range is 0-255 grayscale picture. Finally, data set B is formed:

[0040] B={x1,x2…xn} n}

[0041] Wherein x is the preprocessed original ultrasound image, which is in the form of a 1*224*224 tensor; n is a natural number greater than 1, representing the data set data volume.

[0042] S2: Constructing a feature extractor G of neural network, using the unlabeled data set obtained in step S1 to train the feature extractor of neural network in a contrast learning manner;

[0043] S21: Constructing the feature extraction part of neural network:

[0044] The feature extraction part G needs to accept single channel image data with size of 1*224*224, and output feature vector When using the images in the data set as input, the process can be represented as follows:

[0045]

[0046] Wherein, x i Any picture in the data set, i is the data serial number, the value range is (1, n); the feature extraction network G can be realized by alternately using multiple convolution layers, activation functions and pooling layers, or directly using the general feature extraction network in the industry.

[0047] ​S22: pre-training the feature extraction network G using contrastive learning

[0048] The input picture is enhanced twice, and the enhanced pictures are input into the feature extraction network with the same structure and independent parameter calculation; the constructed multi-layer perception needs to make the feature vectors before and after the perception conversion have the same format, and the feature vectors are obtained through the multi-layer perception; in each training round, the feature extraction network parameters are updated alternately through self-supervised contrastive learning to obtain the final pre-training network.

[0049] Each picture x i in the data set is input into the feature extraction network G to train its parameters. Wherein i is the data serial number, and the value range is (1, n).

[0050] In each training round:

[0051] First, the picture x i is pixel-level data enhanced with different probabilities of brightness, random affine transformation, random contrast, and random scaling to obtain enhanced image data pairs y i1 ,y i2 .

[0052] Then, y i1 ,y i2 are input into the feature extraction networks G1 and G2 respectively to obtain the corresponding feature vectors Wherein, the feature extraction networks G1 and G2 have the same structure as G, and the parameters are θ1 and θ2 respectively, which are randomly initialized. A multi-layer perception P is constructed and randomly initialized to convert into and with the same format.

[0053] Finally, the network parameters θ1 and θ2 are updated alternately in each training round as shown in the following formula, and after several rounds of training, the learned network parameters η are obtained as shown in the following formula. Figure 2

[0054]

[0055]

[0056] Wherein, t represents the current training round, G2(θ2 t-1 ,y i1 ) means that the G2 network uses the network parameters θ2 t-1 of the previous training round to accept the input y i1 to obtain the output, and L is the cosine similarity loss function:

[0057] ​

[0058] in, To construct the feature vector output by the multilayer sensing P. The feature vector output by network G.

[0059] S3: Label some data according to the inspection content to form labeled ultrasound dataset C of dataset B;

[0060] C = {l1, l2, ... l} m}

[0061] Among them, l i For x i The corresponding labels (1≤i≤m≤n).

[0062] S4: Construct a task processing network T based on the task objective, and combine the task processing network T with the trained feature extractor G to form a neural network model O;

[0063] A task processing network T is constructed based on the inspection of the examination site. Downstream tasks include key anatomical structure detection, standard plane evaluation, and Graf angle calculation. Depending on the downstream task objectives, T can be implemented using multiple convolutional layers, activation functions, and pooling layers alternately, or it can consist of fully connected layers. The task processing network T is combined with a feature extractor G to form a neural network model O. The parameters of the feature extractor in model O are the pre-trained parameters η from step S22, while the parameters of the task processing network are randomly initialized. A corresponding loss function L is selected based on the task. t .

[0064] S5: Train the neural network model O using the labeled ultrasound dataset annotated in step S3, and test the task performance of the neural network model O to achieve DDH ultrasound image analysis.

[0065] After step S3 is completed, the generated label set C and the corresponding dataset {x1, x2, ... x} will be used. m The dataset D is composed of a training set, a validation set, and a test set, which are then input into the neural network model O for training.

[0066] The specific training process is as follows: using the training set, the parameters of the feature extraction network G are fine-tuned based on backpropagation, and the parameters of each layer of the task processing network T are fitted; using the validation set, the discrimination ability of the current model O is initially evaluated, and the generalization ability of the model is verified to determine whether to stop training; using the test set, the final generalization ability of model O is evaluated.

[0067] Example 2

[0068] This embodiment takes the detection of key medical landmarks in DDH ultrasound images as an example.Figure 3 The specific steps of the method are as follows:

[0069] S1: Collect 600 DDH ultrasound data and preprocess them to ensure that the processed images have a resolution of 224*224, are single-channel, and have a pixel value range of 0-255 grayscale images. Form an unlabeled data set B:

[0070] B={x1,x2…x 600};

[0071] S2: Construct and train the feature extractor part of the neural network using the unlabeled data set in a contrast learning manner;

[0072] S21: Use the industry-standard network Resnet30 as the feature extraction network G;

[0073] S22: Pre-train the feature extraction network G using contrast learning. In each training round:

[0074] First, the image x i is subjected to pixel-level data augmentation with different probabilities of brightness, affine transformation, contrast, and scaling to obtain the enhanced image data pair y i1 ,y i2 .

[0075] Then, y i1 ,y i2 are input into the feature extraction networks G1 and G2, respectively, to obtain the corresponding feature vectors where the feature extraction networks G1 and G2 have the same structure as G, with parameters θ1 and θ2, respectively, and are randomly initialized; a multi-layer perception P is constructed, with randomly initialized parameters, to convert into and have the same format.

[0076] Finally, update the network parameters θ1 and θ2 alternately in each training round as follows: after several rounds of training, the learned feature extraction network parameters η are obtained.

[0077]

[0078]

[0079] where t represents the current training round, G2(θ2 t-1 ,y i1 ) means that the G2 network uses the network parameters θ2 t-1 from the previous training round to accept input y i1 and obtain output L, and L is the cosine similarity loss function.

[0080] S3: Pixel-level labeling is performed on 8 key anatomical structures of 200 DDH ultrasound images to form a label set C of data set B, as shown in Figure 3

[0081] The eight key anatomical structures are: 1-bony part of the acetabular roof; 2-synovial fold; 3- junction of cartilage and bone; 4- joint capsule; 5- acetabular roof composed of hyaline cartilage; 6- acetabular labrum; 7- femoral head; 8- bony edge;

[0082] C = {l1, l2…l 200};

[0083] S4: According to the detection task of the above eight key anatomical structures, a task processing network T is constructed, and according to the task requirement of semantic segmentation of key anatomical structures, T is an up-sampling network, which can restore the feature vector extracted by G to a semantic segmentation result with a size of 8*224*224. The task processing network T and the feature extractor G are combined to form a neural network model O, wherein the feature extractor parameter adopts the pre-training parameter η in S22, and the task processing network parameter is randomly initialized. Cross entropy is selected as the loss function of the downstream task.

[0084] S5: The labeled ultrasound data set labeled in step S3 is used to train the neural network model O, and the task performance of the neural network model O is tested to realize DDH ultrasound image analysis.

[0085] The label set C generated after the completion of step S3 and the corresponding data set {x1, x2…x 200} are combined to form a data set D, which is divided into a training set, a validation set and a test set according to a ratio of 6:2:2, and is input into the neural network model O for training.

[0086] The training set is used to fine-tune the parameters of the feature extraction network G based on the Backpropagation method, and the parameters of each layer of the task processing network T are fitted; the validation set is used to preliminarily evaluate the discrimination ability of the current model O, and the generalization ability of the model is verified to determine whether to continue training; the test set is used to evaluate the final generalization ability of the model O.

[0087] The method uses unlabeled DDH ultrasound image data set to train the feature extraction network through the contrast learning method, uses the pre-training network parameter to assist the training of the deep learning network for ultrasound image analysis, thereby improving the detection precision and data utilization, alleviating the problem of difficult data labeling and small scale of downstream task data set when using deep learning for DDH ultrasound image analysis, improving the performance of the neural network in the downstream task, and the method is widely applicable and has good effect. ​​

[0088] It should be noted that the above merely illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, and these improvements and refinements all fall within the protection scope of the claims of the present application.

Claims

1. A DDH ultrasound image analysis method based on contrast learning pre-training, characterized in that, The method comprises the following steps: S1: After data enhancement preprocessing of the ultrasound gray-scale images obtained in the DDH ultrasonic diagnosis, a no-label data set is formed; S2: A feature extractor G of a neural network is constructed, and the no-label data set obtained in step S1 is used to train the feature extractor of the neural network in a contrast learning manner; the training process of the neural network in the contrast learning manner is specifically as follows: Input pictures in a dataset Pixel-level data augmentation is performed on the input pictures in the dataset with different probabilities of brightness, random affine transformation, random contrast, and random scaling, respectively, to obtain enhanced image data pairs ; Will input feature extraction network respectively, get the corresponding feature vector ; wherein, the feature extraction network structure and the same, the parameters are , random initialization; construct multilayer perception , parameter random initialization, used to transform equivalent to , and format updating network parameters in each round of training is performed alternately After several rounds of training, the learned network parameters are obtained ; ； ； wherein, denotes the current training epoch, denotes the network uses the network parameters of the previous training epoch accepts an input yields an output, is a cosine similarity loss function; S3: Part of the data is labeled according to the key anatomical structure to be checked, and a labeled ultrasound data set is formed; S4: A task processing network T is constructed according to a task target, the task processing network T is combined with the trained feature extractor G, and a neural network model O is formed; S5: The labeled ultrasound data set labeled in step S3 is used to train the neural network model O, and the task performance of the neural network model O is tested, so that DDH ultrasonic image analysis is realized.

2. The DDH ultrasound image analysis method based on contrastive learning pre-training of claim 1, wherein: The data set after data enhancement in the step S1 is a gray image with a picture resolution of , single channel, and pixel value range of .

3. The DDH ultrasound image analysis method based on contrastive learning pre-training of claim 2, wherein: The feature extractor G of the neural network constructed in the step S2 outputs a feature vector as an output from an image in the data set as an input : ； wherein, any one picture in the data set, is a data sequence number, and the value range is .

4. The DDH ultrasound image analysis method based on contrastive learning pre-training of claim 3, wherein: The loss function : ； wherein to build multi-layer perception outputted feature vector, for the network outputted feature vector.

5. The DDH ultrasound image analysis method based on contrastive learning pre-training of claim 3, wherein: The task processing network T of step S4 is alternatively implemented by a plurality of convolutional layers, activation functions and pooling layers or is composed of fully connected layers.

6. The DDH ultrasound image analysis method based on contrastive learning pre-training of claim 3, wherein: The model training in the step S5 is specifically: the labeled ultrasound data set and the corresponding data set labeled in the step S3 are combined into a total data set, and are divided into a training set, a verification set and a test set, the training set is used, parameters of the feature extraction network are fine-tuned based on a back propagation manner, parameters of the task processing network are fitted; the verification set is used to preliminarily evaluate a discrimination ability of the current model , verify a generalization ability of the model, and determine whether to stop the continuous training. Evaluate the model using the test set Final generalization capability.

Images