Three-dimensional ultrasonic image classification model training method and device

Abstract

The invention provides a three-dimensional ultrasonic image classification model training method and device, and the method comprises the steps: selecting a plurality of target two-dimensional ultrasonic training images from target three-dimensional ultrasonic training images, and determining mask images corresponding to the target two-dimensional ultrasonic training images, and marking classification results of the mask images in a plurality of feature dimensions; performing feature extraction on each target two-dimensional ultrasonic training image and the mask image by using a feature extraction model to obtain target focus features; performing feature classification on the target focus features by using the classifier under each feature dimension to obtain a corresponding classification result under each feature dimension, and determining a loss function value under each feature dimension to determine an overall loss value; and on the basis of the overall loss value, updating model parameters in the three-dimensional ultrasonic image classification model to obtain an updated target three-dimensional ultrasonic image classification model. According to the method, the accuracy and efficiency of classifying the lesions in the three-dimensional ultrasonic image by the three-dimensional ultrasonic image classification model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of ultrasonic image processing, and in particular, to a method and device for training a three-dimensional ultrasonic image classification model. Background Art

[0002] Compared with X-ray examination of lesions, ultrasonic examination has higher sensitivity to tumors of lesions. Among them, handheld ultrasonic examination has higher requirements for operators, and the repeatability of ultrasonic imaging results is not high. However, collecting three-dimensional automatic ultrasonic images of lesions has great advantages. It increases coronal plane imaging and does not rely too much on operators. However, the efficiency of classifying lesions in three-dimensional ultrasonic images is relatively low.

[0003] Currently, the application of deep learning has brought great assistance to the medical field. In the training method of a deep learning classification model for the benign and malignant of lesions based on ultrasonic images, for example, the existing deep learning classification model for the benign and malignant of breast cancer based on ultrasonic images is judged and trained based on two-dimensional ultrasonic images. However, when using Automated Breast Ultrasound (ABUS) for the benign and malignant examination of tumors, the accuracy and efficiency of the existing deep learning classification model in classifying lesions in such three-dimensional ultrasonic images fail to meet the application requirements. Therefore, how to improve the classification performance of the deep learning classification model for the benign and malignant of lesions based on ultrasonic images for three-dimensional ultrasonic images through training is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present application is to provide a method and device for training a three-dimensional ultrasonic image classification model. By selecting multiple two-dimensional ultrasonic training images in the middle layer of the three-dimensional ultrasonic training image corresponding to the target lesion, and determining the mask image of each two-dimensional ultrasonic training image, extracting features from the two-dimensional ultrasonic training image and its mask image to obtain the target lesion features of the two-dimensional ultrasonic training image, classifying the target lesion features under multiple feature dimensions including benign and malignant dimensions, shape dimensions, and edge dimensions, etc., to obtain a classification result, and based on the labeled classification result and the classification result of the two-dimensional ultrasonic training image under multiple feature dimensions, determining an overall loss value and iteratively updating and optimizing the model parameters in the three-dimensional ultrasonic image classification model to obtain the trained target three-dimensional ultrasonic image classification model, improving the accuracy and efficiency of the three-dimensional ultrasonic image classification model in classifying the target lesion in the three-dimensional ultrasonic image.

[0005] The embodiment of the present application provides a method for training a three-dimensional ultrasonic image classification model. The three-dimensional ultrasonic image classification model includes a feature extraction model and classifiers under multiple feature dimensions. The training method includes:

[0006] Obtain the target three-dimensional ultrasound training images corresponding to the target lesions in the preset training dataset. Select multiple target two-dimensional ultrasound training images from the middle layer of the target three-dimensional ultrasound training images, and respectively determine the mask images corresponding to each target two-dimensional ultrasound training image and the labeled classification results of the target lesions corresponding to each target two-dimensional ultrasound training image in multiple feature dimensions; wherein, the feature dimensions at least include the benign / malignant dimension, the shape dimension, and the edge dimension;

[0007] Use the feature extraction model to extract features from each target two-dimensional ultrasound training image and the corresponding mask image, and obtain the target lesion features corresponding to each target two-dimensional ultrasound training image;

[0008] For the multiple feature dimensions corresponding to the target lesions, use the classifier under each feature dimension to perform corresponding feature classification processing on the target lesion features, and obtain the classification results corresponding to each target two-dimensional ultrasound training image under each feature dimension;

[0009] Based on the labeled classification results and the classification results, determine the loss function value of each target two-dimensional ultrasound training image under each feature dimension, and determine the sum of the loss function values under each feature dimension as the overall loss value of each target two-dimensional ultrasound training image;

[0010] Based on the overall loss value, update and optimize the model parameters in the three-dimensional ultrasound image classification model, and use other three-dimensional ultrasound training images in the preset training dataset to iteratively update and optimize the model parameters until the preset training conditions are met, so as to obtain the updated target three-dimensional ultrasound image classification model.

[0011] Further, the step of using the feature extraction model to extract features from each target two-dimensional ultrasound training image and the corresponding mask image, and obtaining the target lesion features corresponding to each target two-dimensional ultrasound training image includes:

[0012] Perform data augmentation processing on each target two-dimensional ultrasound training image and the corresponding mask image;

[0013] Use the feature extraction model to extract features from each target two-dimensional ultrasound training image and the corresponding mask image after data augmentation processing, and obtain the lesion features corresponding to each target two-dimensional ultrasound training image;

[0014] Convert the dimension of the lesion features to obtain the target lesion features represented as one-dimensional vectors corresponding to each target two-dimensional ultrasound training image.

[0015] Further, the feature extraction model includes a fusion feature extraction model and a lesion feature extraction model;

[0016] The step of using the feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation to obtain the lesion features corresponding to each of the target two-dimensional ultrasound training images includes:

[0017] Using the fusion feature extraction model to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0018] Using the lesion feature extraction model to perform multiple convolutional and window hierarchical processes on the target fusion features to extract features from the target fusion features, so as to obtain the lesion features corresponding to each of the target two-dimensional ultrasound training images.

[0019] Further, the step of using the fusion feature extraction model to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images includes:

[0020] Using the first separable convolution and the second separable convolution in the preset fusion feature extraction model to sequentially extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images to obtain the first fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0021] Using the third separable convolution in the fusion feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images to obtain the second fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0022] Fusing and connecting the first fusion features and the second fusion features to obtain third fusion features, and using the convolutional layer in the fusion feature extraction model to extract features from the third fusion features to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images.

[0023] Further, for the multiple feature dimensions corresponding to the target lesion, using each classifier under each feature dimension to perform corresponding feature classification processing on the target lesion features to obtain the classification results corresponding to each of the target two-dimensional ultrasound training images under each feature dimension, includes:

[0024] Using the first classifier under the benign and malignant dimension to perform benign and malignant classification processing on the target lesion features, and obtaining the benign and malignant classification results corresponding to each target two-dimensional ultrasound training image;

[0025] Using the preset second classifier under the shape dimension to perform shape classification processing on the target lesion features, and obtaining the shape classification results corresponding to each target two-dimensional ultrasound training image;

[0026] Using the preset third classifier under the edge dimension to perform edge classification processing on the target lesion features, and obtaining the edge classification results corresponding to each target two-dimensional ultrasound training image.

[0027] The embodiment of the present application further provides a training device for a three-dimensional ultrasound image classification model. The three-dimensional ultrasound image classification model includes a feature extraction model and classifiers under multiple feature dimensions. The training device includes:

[0028] An image processing module, configured to obtain a target three-dimensional ultrasound training image corresponding to a target lesion in a preset training dataset, select multiple target two-dimensional ultrasound training images in the middle layer of the target three-dimensional ultrasound training image, and respectively determine a mask image corresponding to each target two-dimensional ultrasound training image and the labeled classification results of the target lesion corresponding to each target two-dimensional ultrasound training image under multiple feature dimensions; wherein, the feature dimensions at least include a benign and malignant dimension, a shape dimension, and an edge dimension;

[0029] A feature extraction module, configured to use the feature extraction model to extract features from each target two-dimensional ultrasound training image and the corresponding mask image, and obtain the target lesion features corresponding to each target two-dimensional ultrasound training image;

[0030] A feature classification module, configured to perform corresponding feature classification processing on the target lesion features by using the classifier under each feature dimension for the multiple feature dimensions corresponding to the target lesion, and obtain the classification results corresponding to each target two-dimensional ultrasound training image under each feature dimension;

[0031] A loss calculation module, configured to determine the loss function value of each target two-dimensional ultrasound training image under each feature dimension based on the labeled classification results and the classification results, and determine the sum of the loss function values under each feature dimension as the overall loss value of each target two-dimensional ultrasound training image;

[0032] A model update module, configured to update and optimize the model parameters in the three-dimensional ultrasound image classification model based on the overall loss value, and iteratively update and optimize the model parameters by using other three-dimensional ultrasound training images in a preset training dataset until a preset training condition is met, so as to obtain an updated target three-dimensional ultrasound image classification model.

[0033] Further, when the feature extraction module is configured to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images by using the feature extraction model to obtain target lesion features corresponding to each of the target two-dimensional ultrasound training images, the feature extraction module is configured to:

[0034] Perform data augmentation processing on each of the target two-dimensional ultrasound training images and the corresponding mask images;

[0035] Extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing by using the feature extraction model to obtain lesion features corresponding to each of the target two-dimensional ultrasound training images;

[0036] Convert the dimension of the lesion features to obtain target lesion features corresponding to each of the target two-dimensional ultrasound training images, which are represented as one-dimensional vectors.

[0037] Further, the feature extraction model includes a fusion feature extraction model and a lesion feature extraction model;

[0038] When the feature extraction module is configured to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing by using the feature extraction model to obtain lesion features corresponding to each of the target two-dimensional ultrasound training images, the feature extraction module is configured to:

[0039] Extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing by using the fusion feature extraction model to obtain target fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0040] Perform multiple convolution and window layering processes on the target fusion features by using the lesion feature extraction model to extract features from the target fusion features, so as to obtain lesion features corresponding to each of the target two-dimensional ultrasound training images.

[0041] Further, when the feature extraction module is configured to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing by using the fusion feature extraction model to obtain target fusion features corresponding to each of the target two-dimensional ultrasound training images, the feature extraction module is configured to:

[0042] Use the first separable convolution and the second separable convolution in the preset fusion feature extraction model to sequentially extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, and obtain the first fusion feature corresponding to each of the target two-dimensional ultrasound training images;

[0043] Use the third separable convolution in the fusion feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, and obtain the second fusion feature corresponding to each of the target two-dimensional ultrasound training images;

[0044] Fuse and connect the first fusion feature and the second fusion feature to obtain a third fusion feature, and use the convolutional layer in the fusion feature extraction model to extract features from the third fusion feature to obtain the target fusion feature corresponding to each of the target two-dimensional ultrasound training images.

[0045] Further, when the feature classification module is used to perform corresponding feature classification processing on the target lesion features for multiple feature dimensions corresponding to the target lesion, and obtain the classification results corresponding to each of the target two-dimensional ultrasound training images in each of the feature dimensions, the feature classification module is used for:

[0046] Use the first classifier in the benign and malignant dimension to perform benign and malignant classification processing on the target lesion features, and obtain the benign and malignant classification results corresponding to each of the target two-dimensional ultrasound training images;

[0047] Use the preset second classifier in the shape dimension to perform shape classification processing on the target lesion features, and obtain the shape classification results corresponding to each of the target two-dimensional ultrasound training images;

[0048] Use the preset third classifier in the edge dimension to perform edge classification processing on the target lesion features, and obtain the edge classification results corresponding to each of the target two-dimensional ultrasound training images.

[0049] The embodiment of the present application also provides a classification method for three-dimensional ultrasound images, and the classification method includes:

[0050] Obtain a target three-dimensional ultrasound image from the three-dimensional ultrasound images collected for the target lesion, select multiple target two-dimensional ultrasound images from the middle layer of the target three-dimensional ultrasound image, and determine the target mask image corresponding to each of the target two-dimensional ultrasound images;

[0051] Input each of the target two-dimensional ultrasound images and the corresponding target mask images into a pre-trained target three-dimensional ultrasound image classification model to obtain the benign / malignant classification results of each of the target two-dimensional ultrasound images output by the target three-dimensional ultrasound image classification model;

[0052] Perform weighted average processing on the benign / malignant classification results of each of the target two-dimensional ultrasound images to obtain the target benign / malignant classification result of the target three-dimensional ultrasound image;

[0053] Among them, the target three-dimensional ultrasound image classification model is obtained by training with the training method of the above three-dimensional ultrasound image classification model.

[0054] The embodiment of the present application also provides a classification device for three-dimensional ultrasound images. The classification device includes:

[0055] An image selection module, configured to obtain a target three-dimensional ultrasound image from the three-dimensional ultrasound images collected for a target lesion, select multiple target two-dimensional ultrasound images from the middle layer of the target three-dimensional ultrasound image, and determine the target mask image corresponding to each of the target two-dimensional ultrasound images;

[0056] A model classification module, configured to input each of the target two-dimensional ultrasound images and the corresponding target mask images into a pre-trained target three-dimensional ultrasound image classification model to obtain the benign / malignant classification results of each of the target two-dimensional ultrasound images output by the target three-dimensional ultrasound image classification model;

[0057] A result processing module, configured to perform weighted average processing on the benign / malignant classification results of each of the target two-dimensional ultrasound images to obtain the target benign / malignant classification result of the target three-dimensional ultrasound image.

[0058] The embodiment of the present application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the above method are executed.

[0059] The embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the above method are executed.

[0060] The training method and device for a three-dimensional ultrasound image classification model provided by an embodiment of the present application. The three-dimensional ultrasound image classification model includes a feature extraction model and classifiers in multiple feature dimensions. The classification method includes: obtaining a target three-dimensional ultrasound training image corresponding to a target lesion in a preset training dataset, selecting multiple target two-dimensional ultrasound training images in the middle layer of the target three-dimensional ultrasound training image, and respectively determining a mask image corresponding to each target two-dimensional ultrasound training image and the labeled classification results of the target lesion corresponding to each target two-dimensional ultrasound training image in multiple feature dimensions; wherein, the feature dimensions at least include a benign / malignant dimension, a shape dimension, and an edge dimension; using the feature extraction model to extract features from each target two-dimensional ultrasound training image and the corresponding mask image to obtain the target lesion features corresponding to each target two-dimensional ultrasound training image; for the multiple feature dimensions corresponding to the target lesion, using the classifier in each feature dimension to perform corresponding feature classification processing on the target lesion features to obtain the classification results corresponding to each target two-dimensional ultrasound training image in each feature dimension; based on the labeled classification results and the classification results, determining the loss function value of each target two-dimensional ultrasound training image in each feature dimension, and determining the sum of the loss function values in each feature dimension as the overall loss value of each target two-dimensional ultrasound training image; based on the overall loss value, updating and optimizing the model parameters in the three-dimensional ultrasound image classification model, and using other three-dimensional ultrasound training images in the preset training dataset to iteratively update and optimize the model parameters until the preset training conditions are met to obtain an updated target three-dimensional ultrasound image classification model.

[0061] Compared with the method in the prior art where the deep learning classification model for the benign / malignant of breast cancer based on ultrasound images judges and trains based on two-dimensional ultrasound images, by selecting multiple two-dimensional ultrasound training images in the middle layer of the three-dimensional ultrasound training image corresponding to the target lesion, determining the mask image of each two-dimensional ultrasound training image, extracting features from the two-dimensional ultrasound training image and its mask image to obtain the target lesion features of the two-dimensional ultrasound training image, classifying the target lesion features in multiple feature dimensions including the benign / malignant dimension, the shape dimension, and the edge dimension, obtaining the classification results, and based on the labeled classification results and the classification results of the two-dimensional ultrasound training image in multiple feature dimensions, determining the overall loss value and iteratively updating and optimizing the model parameters in the three-dimensional ultrasound image classification model to obtain the trained target three-dimensional ultrasound image classification model, the accuracy and efficiency of the three-dimensional ultrasound image classification model for classifying the target lesion in the three-dimensional ultrasound image are improved.

[0062] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically gives preferred embodiments and cooperates with the attached drawings for detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] To more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0064] Figure 1 It is a flowchart of a method for training a three-dimensional ultrasound image classification model provided by an embodiment of the present application;

[0065] Figure 2 It is a schematic flowchart of a feature extraction model provided by an embodiment of the present application for extracting target lesion features;

[0066] Figure 3 It is a schematic flowchart of a fusion feature extraction model provided by an embodiment of the present application for feature extraction and fusion;

[0067] Figure 4 It is a model processing flowchart of a three-dimensional ultrasound image classification model provided by an embodiment of the present application;

[0068] Figure 5 It is a schematic structural diagram of a training device for a three-dimensional ultrasound image classification model provided by an embodiment of the present application;

[0069] Figure 6 It is a flowchart of a classification method for a three-dimensional ultrasound image provided by an embodiment of the present application;

[0070] Figure 7 It is a schematic structural diagram of a classification device for a three-dimensional ultrasound image provided by an embodiment of the present application;

[0071] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part rather than all of the embodiments of this application. Components of the embodiments of this application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application that is required to be protected, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without creative efforts belongs to the scope of protection of this application.

[0073] Through research, it is found that currently, the application of deep learning has brought great assistance to the medical field. In the training method of a deep learning classification model for the benign and malignant lesions based on ultrasound images, for example, the existing deep learning classification model for the benign and malignant breast cancer based on ultrasound images is judged and trained based on two-dimensional ultrasound images. However, when using Automated Breast Ultrasound (ABUS) for the benign and malignant examination of tumors, the accuracy and efficiency of the existing deep learning classification model in classifying lesions in such three-dimensional ultrasound images fail to meet the application requirements. Therefore, how to improve the classification performance of the deep learning classification model for the benign and malignant lesions based on ultrasound images for three-dimensional ultrasound images through training is an urgently needed problem to be solved.

[0074] Based on this, the embodiments of this application provide a training method for a three-dimensional ultrasound image classification model. By selecting multiple two-dimensional ultrasound training images from the intermediate layer of the three-dimensional ultrasound training image corresponding to the target lesion, and determining the mask image of each two-dimensional ultrasound training image, feature extraction is performed on the two-dimensional ultrasound training image and its mask image to obtain the target lesion features of the two-dimensional ultrasound training image. Classification is performed on the target lesion features under multiple feature dimensions including the benign and malignant dimension, shape dimension, and edge dimension, etc., to obtain the classification result. And based on the labeled classification result and the classification result of the two-dimensional ultrasound training image under multiple feature dimensions, the overall loss value is determined and the model parameters in the three-dimensional ultrasound image classification model are iteratively updated and optimized to obtain the trained target three-dimensional ultrasound image classification model, improving the accuracy and efficiency of the three-dimensional ultrasound image classification model in classifying the target lesion in the three-dimensional ultrasound image.

[0075] Please refer to Figure 1 , Figure 1 which is a flowchart of a training method for a three-dimensional ultrasound image classification model provided by the embodiments of this application. As Figure 1As shown in , the training method of the three-dimensional ultrasound image classification model provided by the embodiments of the present application, the three-dimensional ultrasound image classification model includes a feature extraction model and classifiers under multiple feature dimensions, and the training method includes:

[0076] S100. Obtain target three-dimensional ultrasound training images corresponding to the target lesion in a preset training dataset, select multiple target two-dimensional ultrasound training images in the middle layer of the target three-dimensional ultrasound training images, and respectively determine the mask image corresponding to each target two-dimensional ultrasound training image and the labeled classification results of the target lesion corresponding to each target two-dimensional ultrasound training image under multiple feature dimensions.

[0077] Among them, the feature dimensions at least include a benign and malignant dimension, a shape dimension, and an edge dimension.

[0078] Here, when using the three-dimensional ultrasound image classification model to classify and judge the benign and malignant of the target lesion, by strengthening the attention of the three-dimensional ultrasound image classification model to the classification results of the shape and edge of the target lesion, it can help the target lesion to better perform the benign and malignant classification judgment and improve the benign and malignant classification performance of the three-dimensional ultrasound image classification model for the target lesion.

[0079] Among them, the feature dimensions may also include other feature dimensions, for example, a size dimension, etc., which are not limited in the present application.

[0080] It should be noted that the target lesion described in the embodiments of the present application refers to the lesion tumor corresponding to the diseased part of the patient; multiple three-dimensional ultrasound images can be collected for the target lesion by using an automatic ultrasound detection device, and then a preset training dataset can be constructed based on the multiple three-dimensional ultrasound images; among them, the preset training dataset includes multiple three-dimensional ultrasound training images corresponding to the target lesion.

[0081] Further, based on the training requirements of the model and the imaging situation of the three-dimensional ultrasound training images, a target three-dimensional ultrasound training image is determined from the multiple three-dimensional ultrasound training images.

[0082] Exemplarily, when the target lesion is a breast tumor, the three-dimensional ultrasound training image may include an Automated Breast Ultrasound (ABUS).

[0083] In the embodiments of the present application, the target three-dimensional ultrasound training image is composed of multiple two-dimensional ultrasound training images, and multiple target two-dimensional ultrasound training images corresponding to the middle layer are selected from the multiple two-dimensional ultrasound training images to perform classification processing on the multiple target two-dimensional ultrasound training images for the target lesion.

[0084] For example, when the target three-dimensional ultrasound training image includes 10 layers, the 5th, 6th, and 7th layers can be selected as the middle layers of the target three-dimensional ultrasound training image, and a two-dimensional ultrasound training image can be selected from each middle layer as the target two-dimensional ultrasound training image, that is, a target two-dimensional ultrasound training image is respectively selected from the three middle layers of the target three-dimensional ultrasound training image.

[0085] In this step, after multiple target two-dimensional ultrasound training images are selected from the middle layers of the target three-dimensional ultrasound training image corresponding to the target lesion, each target two-dimensional ultrasound training image is subjected to binarization processing to obtain a mask image corresponding to each target two-dimensional ultrasound training image.

[0086] Among them, the mask image (Mask image) refers to a binary grayscale image or a grayscale image. Each pixel value of the mask image is used to indicate whether a certain position in the target two-dimensional ultrasound image belongs to the target lesion. The pixel value of the background of the target lesion is set to 1, and the pixel value of the contour of the target lesion is set to 0.

[0087] In the embodiments of the present application, a preset binarization processing method can be used to determine the mask image corresponding to each target two-dimensional ultrasound training image. Exemplarily, the binarization processing method may include, but is not limited to, the global threshold method, the adaptive threshold method, the annotation method, the Otsu method, etc.

[0088] It should be noted that the method for determining the mask image corresponding to each target two-dimensional ultrasound training image is not limited to binarization processing, and can also be determined by means such as manual annotation and model processing.

[0089] Here, the classification performance of the three-dimensional ultrasound training image and the two-dimensional ultrasound training image is compared through a preset classification model to determine the ultrasound image with better performance when classifying the ultrasound image of the target lesion.

[0090] Specifically, the preset classification model is used to respectively perform classification tests on the three-dimensional ultrasound images and two-dimensional ultrasound images in the preset test dataset, and the classification performance results of the classification model when using the three-dimensional ultrasound images and two-dimensional ultrasound images as inputs are respectively obtained.

[0091] Exemplarily, the preset test dataset may include 72 three-dimensional ultrasound images corresponding to 63 lesion samples, 216 two-dimensional ultrasound images are selected from the middle layers of the three-dimensional ultrasound images, and the preset classification model (for example, the Resnet50 model) is used to respectively perform benign and malignant classification of deep learning on the three-dimensional ultrasound images and two-dimensional ultrasound images.

[0092] As for the performance comparison results corresponding to the above examples, as shown in the following table, it can be seen that when using the preset classification model to perform benign and malignant classification on the lesion samples through deep learning, the AUC performance of the classification model based on two-dimensional ultrasound images is higher than that of the classification model based on three-dimensional ultrasound images.

[0093]

[0094] Furthermore, since each layer of each three-dimensional ultrasound training image in the preset training dataset includes the contour corresponding to the target lesion, and each three-dimensional ultrasound training image is also labeled with the benign and malignant classification information, shape classification information, and edge classification information corresponding to the target lesion, that is, the labeled classification results of the target lesion corresponding to each target two-dimensional ultrasound training image can be determined under multiple feature dimensions.

[0095] Among them, the labeled classification results under the multiple feature dimensions may include benign and malignant classification information, shape classification information, and edge classification information. The benign and malignant classification information includes benign and malignant; the shape classification information includes irregular and regular; the edge classification information includes unbounded and bounded.

[0096] S200. Use the feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, and obtain the target lesion features corresponding to each of the target two-dimensional ultrasound training images.

[0097] In the embodiments of the present application, when the feature extraction model extracts features from each target two-dimensional ultrasound training image and its corresponding mask image, it is necessary to first perform data augmentation processing on each target two-dimensional ultrasound training image and the corresponding mask image; then, extract features from each target two-dimensional ultrasound training image and the corresponding mask image after the data augmentation processing to obtain lesion features; finally, through the dimension conversion of the lesion features, obtain the target lesion features corresponding to each target two-dimensional ultrasound image.

[0098] Here, the feature extraction model includes a fusion feature extraction model and a lesion feature extraction model.

[0099] Among them, the fusion feature extraction model is used to extract and fuse features from each target two-dimensional ultrasound training image and the corresponding mask image to obtain the target fusion features corresponding to each target two-dimensional ultrasound training image; the lesion feature extraction model is used to perform further convolution and window layering processing on the target fusion features to obtain the lesion features corresponding to each target two-dimensional ultrasound training image.

[0100] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a feature extraction model provided by the embodiments of the present application for extracting target lesion features.

[0101] As an example of a feature extraction model extracting target lesion features, as Figure 2 shown in , first, after performing data augmentation on each target two-dimensional ultrasound training image and the corresponding mask image, they are input into the fusion feature extraction model for feature extraction and fusion to obtain target fusion features (with a size of H×W×96); then, the target fusion features are input into the lesion feature extraction model, and the target fusion features are subjected to segmentation mapping labeling, convolution, and window layering processing for multiple feature extraction cycles to extract features from the target fusion features to obtain lesion features (with a size of H×W / 32×32, 8 channels); finally, the dimension of the lesion features is transformed to obtain the target lesion features represented as a one-dimensional vector.

[0102] Here, although the target two-dimensional ultrasound training image and the mask image of the target lesion are related, the target two-dimensional ultrasound training image and the mask image are two independent images. Therefore, when performing feature extraction on the images, they should be processed separately to better extract the low-level features of the images.

[0103] In an implementation manner of the present application, in specific implementation, step S200 may include:

[0104] S210. Perform data augmentation on each of the target two-dimensional ultrasound training images and the corresponding mask images.

[0105] In the embodiments of the present application, the purpose of performing data augmentation on the mask image corresponding to the target two-dimensional ultrasound training image is to increase the richness and generality of the data to strengthen the intensity of the model for deep learning.

[0106] In an implementation manner of the present application, in specific implementation, step S210 may include:

[0107] S211. Respectively perform random flipping, data transposition, random brightness and contrast adjustment, and normalization on each of the target two-dimensional ultrasound training images and the corresponding mask images, and perform standardization processing on each of the target two-dimensional ultrasound training images.

[0108] In this step, the random flipping includes but is not limited to random vertical flipping, random horizontal flipping, random 90-degree rotation, and random application of affine transformation, etc.; the data transposition includes transposing the input data by exchanging rows and columns; the random brightness and contrast adjustment means randomly adjusting the brightness and contrast of the image respectively.

[0109] Furthermore, perform normalization on each of the target two-dimensional ultrasound training images and the corresponding mask images to scale the numerical features of each mask image to a preset range.

[0110] Further, perform normalization processing on each target two-dimensional ultrasound training image to scale the numerical features of each target two-dimensional ultrasound training image to a preset range and present a standard normal distribution.

[0111] S212. Based on preset channel dimension parameters, adjust the channel dimensions of each processed target two-dimensional ultrasound training image and the corresponding mask image, so that the shapes and dimension size parameters of each target two-dimensional ultrasound training image and the corresponding mask image are the target shapes and target size parameters corresponding to the channel dimension parameters.

[0112] In the embodiment of the present application, the target shape is generally set to the shape where the channel shape is at the forefront of the channel dimension ("channel_first"). When adjusting the channel dimensions of each processed target two-dimensional ultrasound training image and the corresponding mask image, based on the "original_channel_dim" information extracted from the preset metadata dictionary, determine the channel dimensions of the target two-dimensional ultrasound training image and the corresponding mask image, and move this channel dimension to the forefront position to ensure that the channel is in the first position in the channel shape, that is, the target shape is "number of channels, [spatial dimension 1, spatial dimension 2,...]".

[0113] Among them, the target size parameter (Resize) is generally set to "(224, 224)", and the dimension size parameter refers to resizing the dimension size parameters of the target two-dimensional ultrasound training image and the corresponding mask image to 224 pixels × 224 pixels, respectively representing the length and width of the image.

[0114] S220. Use the feature extraction model to extract features from each processed target two-dimensional ultrasound training image and the corresponding mask image to obtain the lesion features corresponding to each target two-dimensional ultrasound training image.

[0115] In this step, use the fusion feature extraction model to extract and fuse features from each target two-dimensional ultrasound training image and the corresponding mask image after data augmentation processing to obtain the target fusion features corresponding to each target two-dimensional ultrasound training image; then, use the lesion feature extraction model to perform multiple convolutional and window hierarchical processing on the target fusion features to extract features from the target fusion features and obtain the lesion features corresponding to each target two-dimensional ultrasound training image.

[0116] In an implementation manner of the present application, in specific implementation, step S220 may include:

[0117] S221. Use the fusion feature extraction model to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation, to obtain the target fusion feature corresponding to each of the target two-dimensional ultrasound training images.

[0118] In the embodiments of the present application, the fusion feature extraction model can better extract and fuse the features of the mask in the target two-dimensional ultrasound training image and its corresponding mask image, so as to improve the performance of the three-dimensional ultrasound image classification model for classifying target lesions.

[0119] In an implementable manner of the present application, the fusion feature extraction model may select a residual model (ResMask Fusion). As a feature extraction model in the deep learning classification model, the residual model can improve the gradient flow of the deep model and alleviate the problem of gradient disappearance, so as to better extract and fuse the features of the mask in the target two-dimensional ultrasound training image and the corresponding mask image, and improve the performance of the three-dimensional ultrasound image classification model for classification processing.

[0120] Please refer to Figure 3 , Figure 3 , which is a schematic flowchart of feature extraction and fusion by a fusion feature extraction model provided by the embodiments of the present application.

[0121] As Figure 3 shown in, first, use the first separable convolution (3×3) and the second separable convolution (3×3) in the fusion feature extraction model to perform two feature extractions on the target two-dimensional ultrasound training image and the corresponding mask image to obtain the first fusion feature; then, use the third separable convolution (1×1) in the fusion feature extraction model to perform feature extraction on the target two-dimensional ultrasound training image and the corresponding mask image to obtain the second fusion feature; finally, connect and fuse the first fusion feature and the second fusion feature to obtain the third fusion feature, and use the convolutional layer in the fusion feature extraction model to perform feature extraction on the third fusion feature to obtain the target fusion feature; wherein, the size of each fusion feature is H×W×96.

[0122] In an embodiment of the present application, in specific implementation, step S221 may include:

[0123] S2211. Use the first separable convolution and the second separable convolution in the preset fusion feature extraction model to sequentially perform feature extraction on each of the target two-dimensional ultrasound training images and the corresponding mask images, to obtain the first fusion feature corresponding to each of the target two-dimensional ultrasound training images.

[0124] S2212. Use the third separable convolution in the fusion feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, and obtain the second fusion feature corresponding to each of the target two-dimensional ultrasound training images.

[0125] In the embodiments of the present application, the separable convolution (Depthwise Convolution) means that a K×K filter is applied to each input channel separately, that is, each channel has its own convolution kernel. This means that if there are C input channels, then there will be C independent filters, and each filter only acts on the corresponding one input channel, and the result is to obtain C output channels. The first separable convolution, the second separable convolution, and the third separable convolution are used to perform convolution independently on each channel (depth-wise).

[0126] S2213. Fuse and connect the first fusion feature and the second fusion feature to obtain a third fusion feature, and use the convolutional layer in the fusion feature extraction model to extract features from the third fusion feature to obtain the target fusion feature corresponding to each of the target two-dimensional ultrasound training images.

[0127] In this step, the first fusion feature and the second fusion feature obtained by feature extraction are fused and connected to obtain a third fusion feature.

[0128] Further, use the convolutional layer in the fusion feature extraction model to extract features from the third fusion feature to obtain the target fusion feature corresponding to each target two-dimensional ultrasound training image.

[0129] S222. Use the lesion feature extraction model to perform multiple convolutional and window hierarchical processing on the target fusion feature to extract features from the target fusion feature, and obtain the lesion feature corresponding to each of the target two-dimensional ultrasound training images.

[0130] In the embodiments of the present application, the lesion feature extraction model may include a hierarchical vision model.

[0131] In an implementable manner of the present application, the lesion feature extraction model may select a "SwinTransformer V2" model and use it as the backbone for classifying target lesions in three-dimensional ultrasound images.

[0132] Among them, the "Swin Transformer V2" model may include an optimized window partitioning strategy to make the information interaction between adjacent windows smoother. For example, by adopting a more complex window pattern or dynamically adjusting the window size to adapt to different input resolutions. In addition, the "Swin Transformer V2" model removes the use of relative position biases and instead adopts absolute position encoding or other forms of position information embedding methods, which helps reduce computational overhead and improve training stability.

[0133] Here, the lesion feature extraction model can be tested to determine a lesion feature extraction model with better performance when classifying ultrasonic images of target lesions.

[0134] Specifically, using a preset test data set, different types of lesion feature extraction models are respectively subjected to classification tests, and the classification performance results of each lesion feature extraction model are obtained.

[0135] Exemplarily, the preset test data set may include 216 two-dimensional ultrasonic test images corresponding to 63 lesion samples and their corresponding mask images. The Vit_l_32 model, Efficientnet_v2_s model, Densenet121 model, Resnet50 model, and Swin Transformer V2 model are respectively used to perform benign and malignant classification of deep learning on the two-dimensional ultrasonic test images and their corresponding mask images.

[0136] The above example corresponds to a performance test comparison result as shown in the following table. It can be seen that when using the lesion feature extraction model to perform benign and malignant classification of deep learning on lesion samples, using the combined data of the two-dimensional ultrasonic test images and their corresponding mask images as the model input and using different lesion feature extraction models for performance testing, it can be seen that the performance effect based on the "Swin Transformer V2" model is the best.

[0137]

[0138] In an implementation manner of the present application, in specific implementation, step S222 may include:

[0139] S2221. Linearly embed the target fusion feature by using the lesion feature extraction model to map the target fusion feature into the dimension space corresponding to the lesion feature extraction model, and perform convolution and window hierarchical processing on the linearly embedded target fusion feature to obtain the to-be-processed lesion feature.

[0140] In the embodiments of the present application, Linear Embedding is generally used to map high-dimensional input data into a lower-dimensional space while preserving as much of the original information as possible. This process can be achieved through matrix multiplication and is therefore called "linear". In Linear Embedding, each input feature is transformed into a new representation, that is, the target fusion feature is mapped to the corresponding dimensional space of the hierarchical visual model.

[0141] Furthermore, when the lesion feature extraction model performs convolution and window hierarchical processing on the target fusion feature, it specifically may include: using a hierarchical structure to gradually reduce the spatial resolution of the target fusion feature while increasing the number of channels; calculating the self-attention of the target fusion feature using a preset window to ensure information exchange between adjacent windows, and shifting the window positions in odd layers so that each window can share information in a partial area with its surrounding windows. By restricting the self-attention calculation to the local window, self-attention processing with linear complexity is achieved. When the resolution decreases, different windows in the previous layer can be merged into a larger window in the next layer to gradually introduce a larger spatial range.

[0142] S222. Perform convolution and window hierarchical processing on the to-be-processed lesion features for multiple feature extraction cycles to obtain the lesion features corresponding to each target two-dimensional ultrasound image; wherein, in each feature extraction cycle, perform linear projection and encoding classification processing on the to-be-processed lesion features in sequence until the number of executions of the feature extraction cycle meets a preset value.

[0143] In this step, perform convolution and window hierarchical processing on the to-be-processed lesion features for multiple feature extraction cycles to obtain the lesion features corresponding to each target two-dimensional ultrasound image.

[0144] Among them, in each feature extraction cycle, perform linear projection and encoding classification processing on the to-be-processed lesion features in sequence until the number of executions of the feature extraction cycle meets a preset value.

[0145] Here, the preset value corresponds to the set number of executions of the feature extraction cycle, and can be specifically calibrated according to actual classification requirements and information of the target lesion.

[0146] Specifically, in each feature extraction cycle, first, the lesion features to be processed are segmented into several non-overlapping small patches; then, each patch is flattened into a one-dimensional vector and mapped to a vector with a fixed dimension through a linear transformation, that is, each patch is encoded; after that, in order to retain the spatial information in the target two-dimensional ultrasound image, before combining the embeddings of all patches into a sequence, a position encoding is added to each patch; finally, the patch sequence with position encoding is embedded and further subjected to convolution and window hierarchical processing.

[0147] In this way, through the lesion feature extraction model to perform feature extraction processing on the lesion features to be processed for multiple feature extraction cycles, the lesion features corresponding to each target two-dimensional ultrasound training image can be obtained.

[0148] S230. Convert the dimension of the lesion features to obtain the target lesion features corresponding to each target two-dimensional ultrasound training image, which are represented as one-dimensional vectors.

[0149] In this step, the data of the lesion features is flattened into a long vector. Specifically, the height, width, and number of channels of the lesion features are combined into a single dimension while keeping the batch size unchanged, so as to obtain the target lesion features corresponding to each target two-dimensional ultrasound training image, which are represented as one-dimensional vectors, by performing dimension conversion (Flatten) on the lesion features.

[0150] For example, assume that the lesion features are a tensor with a shape of (32, 64, 64, 3), that is, it represents 32 samples, and each sample is an RGB image of 64x64 pixels. Then, after performing dimension conversion processing on the lesion features, the obtained target lesion features are a 2D tensor with a shape of (32, 64×64×3).

[0151] S300. For multiple feature dimensions corresponding to the target lesion, use the classifier under each feature dimension to perform corresponding feature classification processing on the target lesion features, and obtain the classification results corresponding to each target two-dimensional ultrasound training image under each feature dimension.

[0152] In the embodiment of the present application, based on the feature extraction model (fusion feature extraction model and lesion feature extraction model) and the preset classifier under each feature dimension, a three-dimensional ultrasound image classification model for classifying the target lesion in the three-dimensional ultrasound image can be determined. As a deep learning classification model, the three-dimensional ultrasound image classification model can select the target two-dimensional ultrasound image in the three-dimensional ultrasound image, perform feature extraction and classification prediction on the target two-dimensional ultrasound image, and obtain the classification result of the target lesion in the three-dimensional ultrasound image.

[0153] Among them, the three-dimensional ultrasound image classification model adopts a multi-task learning framework. Multi-task learning can improve the generalization ability of the three-dimensional ultrasound image classification model among different tasks by simultaneously processing multiple classification tasks through the classifiers under each feature dimension. While the three-dimensional ultrasound image classification model learns the benign and malignant nature of the target lesion, it enhances the algorithm's attention to the shape features and edge features by learning the shape features and edge features of the target lesion, thereby improving the ability of the three-dimensional ultrasound image classification model to judge the benign and malignant nature of the lesion.

[0154] Specifically, the classifiers under each feature dimension are used to complete benign and malignant classification, shape classification, and edge classification respectively; among them, shape classification and edge classification can improve the performance of the three-dimensional ultrasound image classification model in classifying the benign and malignant nature of the target lesion.

[0155] Here, ablation tests can be performed on the fusion feature extraction model, the lesion feature extraction model, and the classifiers under each feature dimension, and then the model performance in various cases where the fusion feature extraction model and the classifiers under each feature dimension are used or not when the lesion feature extraction model is used as the backbone part can be determined.

[0156] As an example of performing ablation tests on the fusion feature extraction model, the lesion feature extraction model, and the classifiers under each feature dimension, the model performance in the case of determining the existence or non-existence of the fusion feature extraction model and the classifiers under each feature dimension when the lesion feature extraction model is used as the backbone part is shown in the following table.

[0157]

[0158] It can be seen that when the combined data of the two-dimensional ultrasound test images and their corresponding mask images in the preset test dataset is used as the model input, the three-dimensional ultrasound image classification model can achieve better results with the help of the classifiers under multiple feature dimensions. On this basis, the use of the fusion feature extraction model can further improve the performance of the deep learning classification model.

[0159] In an implementation manner of the present application, in specific implementation, step S300 may include:

[0160] S310. Use the first classifier under the benign and malignant dimension to perform benign and malignant classification processing on the target lesion features, and obtain the first benign and malignant classification result corresponding to each target two-dimensional ultrasound training image;

[0161] S320. Use the preset second classifier under the shape dimension to perform shape classification processing on the target lesion features, and obtain the first shape classification result corresponding to each target two-dimensional ultrasound training image;

[0162] S330. Use the third classifier preset under the edge dimension to perform edge classification processing on the target lesion features, and obtain the first edge classification result corresponding to each of the target two-dimensional ultrasound training images.

[0163] In the embodiments of the present application, since both shape and edge are classification labels with relatively strong subjectivity, and a lesion may have multiple features, therefore, the first benign and malignant classification result, the first shape classification result, and the first edge classification result are all binary classification results, that is, the classifier under each feature dimension is a binary classifier.

[0164] Here, the first benign and malignant classification result may include being benign and malignant; the first shape classification result may include irregular and regular, and the first edge classification result may include unbounded and bounded.

[0165] S400. Based on the labeled classification result and the classification result, determine the loss function value of each target two-dimensional ultrasound training image under each feature dimension, and determine the sum of the loss function values under each feature dimension as the overall loss value of each target two-dimensional ultrasound training image.

[0166] In the embodiments of the present application, when training a three-dimensional ultrasound image classification model, hyperparameters such as a loss function, an optimizer, and a learning rate adjustment mechanism need to be set.

[0167] Exemplarily, use "AdamW" as the optimizer and "CosineAnnealingWarmRestarts" as the learning rate adjustment mechanism. For example, in the optimizer, set "learning rate" to 1e-4 and "weight_decay" to 1e-3; in the learning rate adjustment mechanism, set "T_0" to 6 and "T_mult" to 2.

[0168] In this step, based on the labeled classification result and the classification result, use the loss function of the classifier under each feature dimension to calculate the loss function value of each target two-dimensional ultrasound training image under each feature dimension.

[0169] Here, "CrossEntropyLoss" can be used as the loss function of the classifier under each feature dimension. In multi-classification problems, CrossEntropyLoss combines the activation function (Softmax) and the negative log-likelihood loss (Negative Log Likelihood, NLL).

[0170] Specifically, the activation function (Softmax) converts the input raw scores (logits) into a probability distribution. For each sample, the Softmax function calculates the probability for each feature dimension, ensuring that these probability values are between (0, 1) and sum to 1.

[0171] Among them, the expression of the activation function is as follows.

[0172]

[0173] Among them, x i represents the raw score of the i-th class output by the model; Softmax(x i ) represents the probability for each feature dimension.

[0174] After obtaining the probability distribution for each feature dimension, the negative log-likelihood loss is used to measure the difference between the predicted probability distribution and the true label (annotated classification result).

[0175]

[0176] Among them, x is the output vector processed by the activation function; class is the index of the target class for each feature dimension.

[0177] Furthermore, the sum of the loss function values for each feature dimension is determined as the overall loss value of each target two-dimensional ultrasound training image.

[0178] In this step, during the training process of the three-dimensional ultrasound image classification model using the loss function, the loss functions for the benign and malignant classification tasks, shape classification tasks, and edge classification tasks all adopt "CrossEntropyLoss", and the sum of these three loss values is used as the total loss value for model training.

[0179] In the embodiment of the present application, the expression of the overall loss value of each target two-dimensional ultrasound training image is as follows.

[0180] Loss = loss shape + loss benignormalignant + loss boundary .

[0181] Among them, Loss represents the overall loss value of each target two-dimensional ultrasound training image; loss shape represents the loss function value corresponding to the classifier in the shape dimension; loss benignormalignant represents the loss function value corresponding to the classifier in the benign and malignant dimension; loss boundary represents the loss function value corresponding to the classifier in the edge dimension.

[0182] Based on the overall loss value, update and optimize the model parameters in the three-dimensional ultrasound image classification model, and use other three-dimensional ultrasound training images in the preset training dataset to iteratively update and optimize the model parameters until the preset training conditions are met, so as to obtain the updated target three-dimensional ultrasound image classification model.

[0183] In the embodiments of the present application, the classification results in the shape dimension and the edge dimension represent an auxiliary result. This auxiliary result is obtained after the classification result in the benign / malignant dimension is obtained for each target two-dimensional ultrasound training image. The classification results in the shape dimension and the edge dimension can assist the three-dimensional image classification model in improving the classification performance when obtaining a more accurate classification result in the benign / malignant dimension.

[0184] In this way, based on the loss function values of each target two-dimensional ultrasound training image in the shape dimension and the edge dimension, the model parameters updated by the three-dimensional image classification model in the benign / malignant dimension continuously pay attention to the influence of the shape dimension and the edge dimension, so as to obtain a more accurate target classification result.

[0185] In this step, the calculated overall loss value can be used for backpropagation to update and optimize the model parameters in the three-dimensional ultrasound image classification model, so that the three-dimensional ultrasound image classification model learns a richer feature representation. Furthermore, based on the model parameters updated by backpropagation, the classification results and samples in the edge dimension and the shape dimension are deduced to obtain a more accurate classification result in the benign / malignant dimension.

[0186] In the embodiments of the present application, the preset training conditions include that the number of iterative updates reaches the preset number of iterations and / or the overall loss value shows convergence.

[0187] In this way, the calculated total loss value can be used for backpropagation to update the model parameters, so that the model learns a richer feature representation. Furthermore, based on the model parameters updated by backpropagation, the calculation expression of the attention function is deduced to obtain the coefficients in the calculation expression corresponding to the attention function.

[0188] Please refer to Figure 4 , Figure 4 which is a model processing flow chart of a three-dimensional ultrasound image classification model provided by the embodiments of the present application.

[0189] As Figure 4As shown in the figure, first, multiple target two-dimensional ultrasound training images are selected from the middle layer of the target three-dimensional ultrasound training image, and their corresponding mask images are determined; then, feature extraction is performed on the target two-dimensional ultrasound training images and their corresponding mask images respectively, as well as classification in dimensions of benign / malignant, shape, and edge, to obtain classification results for each feature dimension, and further determine the loss function value for each feature dimension to determine the overall loss value; finally, based on the overall loss value determined by the loss function values for each feature dimension, the model parameters in the three-dimensional ultrasound image classification model are updated and optimized to obtain an updated target three-dimensional ultrasound image classification model.

[0190] The training method for the three-dimensional ultrasound image classification model provided by the embodiments of the present application selects multiple two-dimensional ultrasound training images from the middle layer of the three-dimensional ultrasound training image corresponding to the target lesion, determines the mask image for each two-dimensional ultrasound training image, performs feature extraction on the two-dimensional ultrasound training image and its mask image to obtain the target lesion features of the two-dimensional ultrasound training image, classifies the target lesion features in multiple feature dimensions including the benign / malignant dimension, shape dimension, and edge dimension, etc., to obtain classification results, and based on the labeled classification results and classification results of the two-dimensional ultrasound training image in multiple feature dimensions, determines the overall loss value and iteratively updates and optimizes the model parameters in the three-dimensional ultrasound image classification model to obtain a trained target three-dimensional ultrasound image classification model, improving the accuracy and efficiency of the three-dimensional ultrasound image classification model in classifying the target lesion in the three-dimensional ultrasound image.

[0191] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a training device for a three-dimensional ultrasound image classification model provided by the embodiments of the present application. As Figure 5 shown in the figure, the three-dimensional ultrasound image classification model includes a feature extraction model and classifiers in multiple feature dimensions. The training device 500 includes:

[0192] An image processing module 510, configured to obtain a target three-dimensional ultrasound training image corresponding to a target lesion from a preset training dataset, select multiple target two-dimensional ultrasound training images from the middle layer of the target three-dimensional ultrasound training image, and respectively determine the mask image corresponding to each target two-dimensional ultrasound training image and the labeled classification results of the target lesion corresponding to each target two-dimensional ultrasound training image in multiple feature dimensions; where the feature dimensions at least include the benign / malignant dimension, shape dimension, and edge dimension;

[0193] A feature extraction module 520, configured to perform feature extraction on each target two-dimensional ultrasound training image and the corresponding mask image by using the feature extraction model to obtain the target lesion features corresponding to each target two-dimensional ultrasound training image;

[0194] A feature classification module 530, configured to perform corresponding feature classification processing on the target lesion features for multiple feature dimensions corresponding to the target lesion by using the classifier under each feature dimension, so as to obtain classification results corresponding to each target two-dimensional ultrasound training image under each feature dimension;

[0195] A loss calculation module 540, configured to determine a loss function value of each target two-dimensional ultrasound training image under each feature dimension based on the labeled classification result and the classification result, and determine the sum of the loss function values under each feature dimension as the overall loss value of each target two-dimensional ultrasound training image;

[0196] A model update module 550, configured to update and optimize the model parameters in the three-dimensional ultrasound image classification model based on the overall loss value, and iteratively update and optimize the model parameters by using other three-dimensional ultrasound training images in a preset training dataset until a preset training condition is met, so as to obtain an updated target three-dimensional ultrasound image classification model.

[0197] Further, when the feature extraction module 520 is used to extract features of each target two-dimensional ultrasound training image and the corresponding mask image by using the feature extraction model to obtain target lesion features corresponding to each target two-dimensional ultrasound training image, the feature extraction module 520 is configured to:

[0198] Perform data augmentation processing on each target two-dimensional ultrasound training image and the corresponding mask image;

[0199] Extract features of each target two-dimensional ultrasound training image and the corresponding mask image after data augmentation processing by using the feature extraction model to obtain lesion features corresponding to each target two-dimensional ultrasound training image;

[0200] Convert the dimension of the lesion features to obtain target lesion features corresponding to each target two-dimensional ultrasound training image, which are represented as one-dimensional vectors.

[0201] Further, the feature extraction model includes a fusion feature extraction model and a lesion feature extraction model;

[0202] When the feature extraction module 520 is used to extract features of each target two-dimensional ultrasound training image and the corresponding mask image after data augmentation processing by using the feature extraction model to obtain lesion features corresponding to each target two-dimensional ultrasound training image, the feature extraction module 520 is configured to:

[0203] Use the fusion feature extraction model to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing, to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0204] Use the lesion feature extraction model to perform multiple convolution and window layering processes on the target fusion features, to extract features from the target fusion features, and obtain the lesion features corresponding to each of the target two-dimensional ultrasound training images.

[0205] Further, when the feature extraction module 520 is used to use the fusion feature extraction model to extract and fuse features from each of the target two-dimensional ultrasound training images and the corresponding mask images after data augmentation processing, to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images, the feature extraction module 520 is used for:

[0206] Use the first separable convolution and the second separable convolution in the preset fusion feature extraction model to sequentially extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, to obtain the first fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0207] Use the third separable convolution in the fusion feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask images, to obtain the second fusion features corresponding to each of the target two-dimensional ultrasound training images;

[0208] Fusion-connect the first fusion features and the second fusion features to obtain third fusion features, and use the convolutional layer in the fusion feature extraction model to extract features from the third fusion features, to obtain the target fusion features corresponding to each of the target two-dimensional ultrasound training images.

[0209] Further, when the feature classification module 530 is used to perform corresponding feature classification processing on the target lesion features for multiple feature dimensions corresponding to the target lesion, to obtain the classification results corresponding to each of the target two-dimensional ultrasound training images in each of the feature dimensions, the feature classification module 530 is used for:

[0210] Use the first classifier in the benign and malignant dimension to perform benign and malignant classification processing on the target lesion features, to obtain the benign and malignant classification results corresponding to each of the target two-dimensional ultrasound training images;

[0211] Use the preset second classifier in the shape dimension to perform shape classification processing on the target lesion features, to obtain the shape classification results corresponding to each of the target two-dimensional ultrasound training images;

[0212] Perform edge classification processing on the target lesion features using a preset third classifier under the edge dimension to obtain an edge classification result corresponding to each of the target two-dimensional ultrasound training images.

[0213] The training device for the three-dimensional ultrasound image classification model provided by the embodiments of the present application selects multiple two-dimensional ultrasound training images from the intermediate layers of the three-dimensional ultrasound training images corresponding to the target lesion, determines the mask image of each two-dimensional ultrasound training image, extracts features from the two-dimensional ultrasound training image and its mask image to obtain the target lesion features of the two-dimensional ultrasound training image, classifies the target lesion features under multiple feature dimensions including the benign / malignant dimension, shape dimension, and edge dimension, etc., to obtain classification results, and based on the labeled classification results and classification results of the two-dimensional ultrasound training image under multiple feature dimensions, determines the overall loss value and iteratively updates and optimizes the model parameters in the three-dimensional ultrasound image classification model to obtain the trained target three-dimensional ultrasound image classification model, improving the accuracy and efficiency of the three-dimensional ultrasound image classification model in classifying the target lesion in the three-dimensional ultrasound image.

[0214] Please refer to Figure 6 , Figure 6 which is a flowchart of a classification method for three-dimensional ultrasound images provided by the embodiments of the present application. As Figure 6 shown in

[0215] S601. Obtain a target three-dimensional ultrasound image from the three-dimensional ultrasound images collected for the target lesion, select multiple target two-dimensional ultrasound images from the intermediate layer of the target three-dimensional ultrasound image, and determine the target mask image corresponding to each of the target two-dimensional ultrasound images.

[0216] In the embodiments of the present application, the description of step S601 can refer to the description in step S100 and can achieve the same technical effect, which will not be elaborated here.

[0217] S602. Input each of the target two-dimensional ultrasound images and the corresponding target mask images into a pre-trained target three-dimensional ultrasound image classification model to obtain a benign / malignant classification result of each of the target two-dimensional ultrasound images output by the target three-dimensional ultrasound image classification model.

[0218] Among them, the target three-dimensional ultrasound image classification model is trained by the training method of the three-dimensional ultrasound image classification model in the method embodiment as shown above Figure 1 and obtained.

[0219] In the embodiments of the present application, the processing of each target two-dimensional ultrasound image by the target three-dimensional ultrasound image classification model in step S602 can refer to the descriptions in steps S200 to S300 and can achieve the same technical effects, which will not be elaborated here.

[0220] It should be noted that when the target three-dimensional ultrasound image classification model classifies each target two-dimensional ultrasound image, it will only output the benign and malignant classification results of each target two-dimensional ultrasound image based on the learned features of each target two-dimensional ultrasound image in the shape dimension and the edge dimension.

[0221] S603. Perform weighted average processing on the benign and malignant classification results of each target two-dimensional ultrasound image to obtain the target benign and malignant classification result of the target three-dimensional ultrasound image.

[0222] In this step, perform weighted average calculation on the benign and malignant classification results of each target two-dimensional ultrasound image to obtain the weighted average calculation result, and determine this weighted average calculation result as the target benign and malignant classification result of the target three-dimensional ultrasound image.

[0223] The classification method of three-dimensional ultrasound images provided by the embodiments of the present application selects multiple two-dimensional ultrasound images in the middle layer of the three-dimensional ultrasound image collected for the target lesion, determines the mask image of each two-dimensional ultrasound image, and uses the target three-dimensional ultrasound image classification model to extract features from the two-dimensional ultrasound image and its mask image to obtain the target lesion features of the two-dimensional ultrasound image, and classifies the target lesion features in multiple feature dimensions including the benign and malignant dimension, the shape dimension, and the edge dimension, etc., to determine the target benign and malignant classification result of the target three-dimensional ultrasound image, improving the accuracy and efficiency of classifying and judging the target lesion.

[0224] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a classification device for three-dimensional ultrasound images provided by the embodiments of the present application. As Figure 7 shown in, the classification device 700 includes:

[0225] An image selection module 710, configured to obtain a target three-dimensional ultrasound image from the three-dimensional ultrasound image collected for the target lesion, select multiple target two-dimensional ultrasound images in the middle layer of the target three-dimensional ultrasound image, and determine the target mask image corresponding to each target two-dimensional ultrasound image;

[0226] A model classification module 720, configured to input each target two-dimensional ultrasound image and the corresponding target mask image into a pre-trained target three-dimensional ultrasound image classification model to obtain the benign and malignant classification results of each target two-dimensional ultrasound image output by the target three-dimensional ultrasound image classification model;

[0227] The result processing module 730 is configured to perform weighted average processing on the benign and malignant classification results of each of the target two-dimensional ultrasound images to obtain the target benign and malignant classification result of the target three-dimensional ultrasound image.

[0228] The three-dimensional ultrasound image classification device provided by the embodiments of the present application selects multiple two-dimensional ultrasound images from the intermediate layer of the three-dimensional ultrasound image collected for the target lesion, determines the mask image of each two-dimensional ultrasound image, and uses the target three-dimensional ultrasound image classification model to extract features from the two-dimensional ultrasound image and its mask image to obtain the target lesion features of the two-dimensional ultrasound image, classifies the target lesion features under multiple feature dimensions including the benign and malignant dimension, the shape dimension, and the edge dimension, etc., and determines the target benign and malignant classification result of the target three-dimensional ultrasound image, improving the accuracy and efficiency of classifying and judging the target lesion.

[0229] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an electronic device provided by the embodiments of the present application. As Figure 8 shown in

[0230] the electronic device 800 includes a processor 810, a memory 820, and a bus 830. Figure 1 and Figure 6 When the electronic device 800 runs, the processor 810 communicates with the memory 820 through the bus 830. When the machine-readable instructions stored in the memory 820 are executed by the processor 810, the steps in the method embodiments as described above

[0231] The embodiments of the present application further provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps in the method embodiments as described above Figure 1 and Figure 6 can be executed. For the specific implementation manners, reference may be made to the method embodiments, which will not be elaborated herein.

[0232] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be elaborated herein.

[0233] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0234] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0236] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0237] Finally, it should be noted that the above-described embodiments are only specific implementation manners of the present application, used to illustrate the technical solutions of the present application, rather than limiting it. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present application can still modify the technical solutions recorded in the foregoing embodiments or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A training method for a three-dimensional ultrasound image classification model, characterized in that: The three-dimensional ultrasound image classification model includes a feature extraction model and classifiers under multiple feature dimensions, and the training method includes: A target three-dimensional ultrasound training image corresponding to a target lesion is obtained in a preset training data set, a plurality of target two-dimensional ultrasound training images are selected in the middle layer of the target three-dimensional ultrasound training image, and a mask image corresponding to each of the target two-dimensional ultrasound training images and a labeling classification result of the target lesion corresponding to each of the target two-dimensional ultrasound training images under multiple feature dimensions are determined respectively; wherein the feature dimensions include at least a benign or malignant dimension, a shape dimension, and an edge dimension; Using the feature extraction model, extracting features from each of the target two-dimensional ultrasound training images and the corresponding mask image to obtain target lesion features corresponding to each of the target two-dimensional ultrasound training images; For the multiple feature dimensions corresponding to the target lesion, the classifier under each feature dimension is used to perform corresponding feature classification processing on the target lesion feature, so as to obtain the classification result corresponding to each target two-dimensional ultrasound training image under each feature dimension; Based on the labeled classification result and the classification result, determining the loss function value of each target two-dimensional ultrasound training image in each feature dimension, and determining the sum of the loss function values ​​in each feature dimension as the overall loss value of each target two-dimensional ultrasound training image; Based on the overall loss value, the model parameters in the three-dimensional ultrasound image classification model are updated and optimized, and the model parameters are iteratively updated and optimized using other three-dimensional ultrasound training images in a preset training data set until the preset training conditions are met to obtain an updated target three-dimensional ultrasound image classification model.

2. The method according to claim 1, characterized in that The step of using the feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask image to obtain target lesion features corresponding to each of the target two-dimensional ultrasound training images includes: Performing data enhancement processing on each of the target two-dimensional ultrasound training images and the corresponding mask image; Using the feature extraction model, feature extraction is performed on each of the target two-dimensional ultrasound training images and the corresponding mask image after data enhancement processing, so as to obtain the lesion feature corresponding to each of the target two-dimensional ultrasound training images; The lesion features are dimensionally transformed to obtain target lesion features expressed as a one-dimensional vector corresponding to each target two-dimensional ultrasound training image.

3. The method according to claim 2, characterized in that The feature extraction model includes a fusion feature extraction model and a lesion feature extraction model; The step of using the feature extraction model to extract features from each of the target two-dimensional ultrasound training images and the corresponding mask image after data enhancement processing to obtain lesion features corresponding to each of the target two-dimensional ultrasound training images includes: Using the fusion feature extraction model, extracting and fusing features of each of the target two-dimensional ultrasound training images and the corresponding mask image after data enhancement processing, to obtain a target fusion feature corresponding to each of the target two-dimensional ultrasound training images; The target fusion feature is subjected to multiple convolution and window layering processes using the lesion feature extraction model to extract features from the target fusion feature, thereby obtaining lesion features corresponding to each of the target two-dimensional ultrasound training images.

4. The method according to claim 3, characterized in that The method of extracting and fusing features of each target two-dimensional ultrasound training image and the corresponding mask image after data enhancement processing by using the fusion feature extraction model to obtain a target fusion feature corresponding to each target two-dimensional ultrasound training image includes: Using the first separable convolution and the second separable convolution in the preset fusion feature extraction model, feature extraction is performed on each of the target two-dimensional ultrasound training images and the corresponding mask image in turn to obtain a first fusion feature corresponding to each of the target two-dimensional ultrasound training images; Using the third separable convolution in the fusion feature extraction model to perform feature extraction on each of the target two-dimensional ultrasound training images and the corresponding mask image, to obtain a second fusion feature corresponding to each of the target two-dimensional ultrasound training images; The first fusion feature and the second fusion feature are fused and connected to obtain a third fusion feature, and the convolution layer in the fusion feature extraction model is used to extract the third fusion feature to obtain a target fusion feature corresponding to each target two-dimensional ultrasound training image.

5. The method according to claim 1, characterized in that The method of performing corresponding feature classification processing on the target lesion features by using the classifier under each feature dimension for the multiple feature dimensions corresponding to the target lesion to obtain the classification result corresponding to each target two-dimensional ultrasound training image under each feature dimension includes: Using the first classifier under the benign and malignant dimension to perform benign and malignant classification processing on the target lesion features, and obtaining a benign and malignant classification result corresponding to each of the target two-dimensional ultrasound training images; Using a second classifier preset under the shape dimension to perform shape classification processing on the target lesion feature, to obtain a shape classification result corresponding to each of the target two-dimensional ultrasound training images; The target lesion features are subjected to edge classification processing using a third classifier preset under the edge dimension to obtain edge classification results corresponding to each of the target two-dimensional ultrasound training images.

6. A training device for a three-dimensional ultrasound image classification model, characterized in that: The three-dimensional ultrasound image classification model includes a feature extraction model and classifiers under multiple feature dimensions, and the training device includes: An image processing module, used to obtain a target three-dimensional ultrasound training image corresponding to a target lesion in a preset training data set, select multiple target two-dimensional ultrasound training images in the middle layer of the target three-dimensional ultrasound training image, and respectively determine a mask image corresponding to each of the target two-dimensional ultrasound training images and a labeling classification result of the target lesion corresponding to each of the target two-dimensional ultrasound training images under multiple feature dimensions; wherein the feature dimensions at least include a benign or malignant dimension, a shape dimension, and an edge dimension; A feature extraction module, used to perform feature extraction on each of the target two-dimensional ultrasound training images and the corresponding mask image using the feature extraction model to obtain target lesion features corresponding to each of the target two-dimensional ultrasound training images; A feature classification module is used to perform corresponding feature classification processing on the target lesion features using the classifier under each feature dimension for the multiple feature dimensions corresponding to the target lesion, so as to obtain the classification result corresponding to each target two-dimensional ultrasound training image under each feature dimension; A loss calculation module, used to determine the loss function value of each target two-dimensional ultrasound training image in each feature dimension based on the labeled classification result and the classification result, and determine the sum of the loss function values ​​in each feature dimension as the overall loss value of each target two-dimensional ultrasound training image; A model updating module is used to update and optimize the model parameters in the three-dimensional ultrasound image classification model based on the overall loss value, and iteratively update and optimize the model parameters using other three-dimensional ultrasound training images in a preset training data set until the preset training conditions are met to obtain an updated target three-dimensional ultrasound image classification model.

7. A classification method for three-dimensional ultrasound images, characterized in that: The classification method includes: Acquire a target three-dimensional ultrasound image from the three-dimensional ultrasound image collected for the target lesion, select a plurality of target two-dimensional ultrasound images from the middle layer of the target three-dimensional ultrasound image, and determine a target mask image corresponding to each of the target two-dimensional ultrasound images; Inputting each of the target two-dimensional ultrasound images and the corresponding target mask image into a pre-trained target three-dimensional ultrasound image classification model to obtain a benign or malignant classification result of each of the target two-dimensional ultrasound images output by the target three-dimensional ultrasound image classification model; Performing weighted average processing on the benign and malignant classification results of each target two-dimensional ultrasound image to obtain the target benign and malignant classification results of the target three-dimensional ultrasound image; Wherein, the target three-dimensional ultrasound image classification model is obtained by training using the three-dimensional ultrasound image classification model training method described in any one of claims 1 to 6.

8. A three-dimensional ultrasound image classification device, characterized in that: The classification device comprises: An image selection module, used to obtain a target three-dimensional ultrasound image from the three-dimensional ultrasound image collected for the target lesion, select multiple target two-dimensional ultrasound images from the middle layer of the target three-dimensional ultrasound image, and determine a target mask image corresponding to each of the target two-dimensional ultrasound images; A model classification module, used for inputting each of the target two-dimensional ultrasound images and the corresponding target mask image into a pre-trained target three-dimensional ultrasound image classification model, and obtaining a benign or malignant classification result of each of the target two-dimensional ultrasound images output by the target three-dimensional ultrasound image classification model; The result processing module is used to perform weighted average processing on the benign and malignant classification result of each target two-dimensional ultrasound image to obtain the target benign and malignant classification result of the target three-dimensional ultrasound image.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of any method as claimed in claim 1 to 5 or 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 or 7 are executed.

Images