Prediction method of neoadjuvant therapy efficacy of breast cancer based on ultrasound images

Abstract

The present invention discloses a method for predicting the efficacy of neoadjuvant treatment of breast cancer based on ultrasound images, and relates to the technical field of medical image processing. The naming rules of the present invention when constructing a data set can realize effective management and traceability of the data set, and the image cropping method adopted can enable the neural network to obtain more information when fully displaying feature extraction. The classification network structure constructed by the present invention, the feature extraction part is based on the spatial and channel attention module, the convolution attention module, and the depth of the separable convolution to realize effective control feature extraction, and then the two are flattened and spliced, and then the output of the classification network is directly obtained by the convolution operation through the dropout layer, and the efficacy probability of the patient's neoadjuvant treatment is realized. Lightweight network feature extraction and prediction. Based on the patient's multimodal ultrasound data, the present invention extracts the image features of the control images before and during neoadjuvant treatment based on the twin network construction, and predicts the efficacy probability of the patient's neoadjuvant treatment based on the extracted image features, which effectively reduces the misjudgment rate in the neoadjuvant treatment process caused by the strong subjectivity of doctors in the current clinical process and the low quality of medical images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for predicting the efficacy of neoadjuvant therapy (NAT) for breast cancer based on ultrasound images. Background Art

[0002] Feature extraction and image classification based on medical images are common applications of deep learning in the medical field. The advantage of deep learning in this application is that it does not require artificial design or construction of features. The network model can autonomously learn target features through the iterative process of forward propagation and back propagation. At the same time, deep learning also has a strong learning ability for complex data representation in medical images. Lesion recognition and detection is an important task of deep learning in the medical field. The solution to this task makes manual segmentation of lesions no longer require a lot of manpower and material resources. Fang et al. combined YOLOv3 with DMRF-CNN and proposed the YOLOv3-DMRF (Dense Multi-receptive Fields) model. The model is characterized by its multi-scale detection layer, which uses multi-receptive field convolution to detect edge and texture features and can identify thyroid nodules of different sizes. In addition to lesion detection, the more successful application of deep learning in the medical field is the extraction of image features and image classification based on this. With the help of the powerful computing and learning capabilities of computers, it is possible to objectively judge the implicit information in medical images, to a certain extent avoid misdiagnosis and missed diagnosis by doctors, and help doctors formulate medical strategies more reasonably. Specifically, in terms of efficacy prediction, the application of deep learning includes prediction of tumor remission, prediction of patient survival, prediction of lesion recurrence or cancer cell metastasis, etc. Qu et al. trained a deep learning model based on DCE-MRI images at multiple time points before and after treatment, and verified that multiple time nodes can improve the model's ability to evaluate the response to lesion treatment. Byra et al. used Siamese networks to quantitatively characterize the characteristics of breast cancer ultrasound images during neoadjuvant therapy. The final model's AUC (Area Under Curve) value was 0.847. Their research proposed the feasibility of using neural network transfer learning in ultrasound images to predict the efficacy of neoadjuvant therapy. However, the current misjudgment rate of the prediction of adjuvant therapy efficacy of low-quality medical images is high and needs to be further reduced. Summary of the invention

[0003] The purpose of the present invention is to propose a method for predicting the efficacy of neoadjuvant therapy for breast cancer based on ultrasound images, so as to solve the problem of high misjudgment rate in the neoadjuvant therapy process caused by strong subjectivity of doctors in the current clinical process and low quality of medical images.

[0004] The technical solution adopted by the present invention is:

[0005] A method for predicting the efficacy of neoadjuvant therapy for breast cancer based on ultrasound images, the method comprising the following steps:

[0006] Step 1, constructing a data set based on ultrasound images of the patient before and during neoadjuvant therapy at different treatment courses, wherein the ultrasound image data in the data set includes two-dimensional ultrasound images (Two-dimensionalUltrasound) and contrast-enhanced ultrasound images (Contrast-enhanced Ultrasound) of the same patient;

[0007] Step 2: construct a classification network for neoadjuvant therapy efficacy prediction based on the twin network;

[0008] The classification network includes two feature extraction branches with the same network structure and a classifier. One feature extraction branch is used to extract the image features of the ultrasound image corresponding to the patient before neoadjuvant treatment; the other feature extraction branch is used to extract the image features of the ultrasound image of the patient during neoadjuvant treatment; the classifier is used to flatten and splice the feature maps output by the two feature extraction branches, and then output the efficacy probability of the patient's neoadjuvant treatment based on the classification layer. The value range of the probability is a number between 0 and 1. The smaller the value, the better the efficacy of the patient.

[0009] Dividing the data set into sub-data sets corresponding to different treatment courses based on the treatment courses, and dividing the sub-data sets into a first sub-data set and a second sub-data set based on the ultrasound image type of each sub-data set, corresponding to the two-dimensional ultrasound image and the ultrasound contrast image, respectively;

[0010] A classification network is set for each treatment course. Based on the data set corresponding to the treatment course, the classification network is trained on the first and second sub-data sets corresponding to the treatment course in order from the last treatment course to the first treatment course, so as to obtain two classification models for each treatment course.

[0011] During training, the network parameters of the classification network of the first treatment course to be trained can be randomly initialized, or the pre-trained network parameters can be used; the initial values ​​of the network parameters of the classification network of the next treatment course to be trained are the trained network parameters of the classification network of the previous treatment course;

[0012] Step 3, obtaining an ultrasound image pair of the patient to be predicted before and during neoadjuvant therapy, the ultrasound image pair including an image pair based on a two-dimensional ultrasound image and an image pair based on an ultrasound contrast-enhanced image;

[0013] Performing lesion detection on the ultrasound image pair, extracting the lesion area image and performing image preprocessing to obtain a lesion image pair that matches the input of the classification model;

[0014] The lesion image pair is then input into the classification model of the corresponding ultrasound image type of the corresponding treatment course, and the efficacy probabilities output by the two classification models are fused to obtain the neoadjuvant therapy efficacy prediction result of the current patient to be predicted.

[0015] Furthermore, in step 1, the naming fields of the image names of the ultrasound images in the data set include: patient number, treatment course, image type and image number for distinguishing ultrasound contrast images from two-dimensional ultrasound images.

[0016] Furthermore, the image processing used to obtain image pairs for input into the classification network during training includes:

[0017] The lesion area in the image is manually annotated and reviewed by experts to obtain the lesion annotation frame;

[0018] Based on the method of expanding the lesion annotation box outward by a specified number of pixels, the lesion area image is cropped out from the ultrasound image, and then the image size is normalized to obtain the image of the image pair input into the classification network.

[0019] Furthermore, when performing image size normalization, the blank area is filled with black.

[0020] Furthermore, step 1 also includes data enhancement processing of the data set, and the processing method is: sending the ultrasound video to the YOLOv8 network for video recognition, performing frame splitting operations on video segments with confidence levels higher than a specified value (preferably 0.9), and expanding the data set based on the obtained video frame images.

[0021] Furthermore, the feature extraction branch is a stacked structure of several bottleneck structures, wherein the main branch of the bottleneck structure includes, in sequence, a convolution layer with a 1x1 convolution kernel, a batch normalization layer, an activation function layer, a depthwise separable convolution layer with a 3x3 convolution kernel, a batch normalization layer, an activation function layer, a 1x1 convolution layer, a batch normalization layer, an activation function layer, a spatial and channel attention module, and a convolution attention module, and there is a short connection between the input of the bottleneck structure and the output of the spatial and channel attention module.

[0022] Furthermore, the classification layer of the classifier includes a dropout layer, a 1x1 convolution layer, an activation function layer, a 1x1 convolution layer, and a Softmax function layer in sequence.

[0023] Furthermore, the treatment courses were divided into four categories: before treatment, two courses of treatment, four courses of treatment, and six courses of treatment, and corresponding classification models were set for the latter three categories of treatment courses.

[0024] The technical solution provided by the present invention brings at least the following beneficial effects:

[0025] The present invention is based on the patient's multimodal ultrasound data and a twin network to extract image features of control images before and during neoadjuvant therapy, and predicts the probability of the efficacy of the patient's neoadjuvant therapy based on the extracted image features, effectively reducing the misjudgment rate in the neoadjuvant therapy process caused by the strong subjectivity of doctors and the low quality of medical images in the current clinical process.

[0026] The naming rules used in the present invention when constructing a data set can achieve effective management and traceability of the data set, and the image cropping method used can enable the neural network to obtain more information when fully extracting features.

[0027] The classification network structure constructed by the present invention realizes effective control feature extraction based on the spatial and channel attention module, the convolutional attention module, and the depth-separable convolution in the feature extraction part. After flattening and splicing the two, overfitting is prevented by the dropout layer. Then, the output of the classification network is directly obtained by the convolution operation with a convolution kernel of 1x1, and the efficacy probability of the patient's neoadjuvant treatment is realized, thereby realizing lightweight network feature extraction and prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A schematic diagram of the structure of a classification model used in an embodiment of the present invention.

[0030] Figure 2 This is a process diagram for obtaining the prediction result of the efficacy of neoadjuvant therapy for breast cancer based on the trained classification model in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0032] The embodiment of the present invention provides a method for predicting the efficacy of neoadjuvant treatment of breast cancer based on ultrasound images. On the basis of deep learning, a method for predicting the efficacy of neoadjuvant treatment of breast cancer based on ultrasound images is implemented. The method first needs to preprocess and archive ultrasound images of different courses before and during neoadjuvant treatment to construct a data set, and the data set includes two-dimensional ultrasound and ultrasound contrast imaging images of the same patient; then, a classification network for predicting the efficacy of neoadjuvant treatment is constructed based on a twin network, and then classification models for different courses are trained on the two-dimensional ultrasound data set and the ultrasound contrast imaging image data set respectively based on a set training strategy, and the classification model is used to output a probability representing that the patient has a poor efficacy; when the output efficacy probability is less than a preset threshold, it means that the patient has a better efficacy, that is, the smaller the output NAT efficacy probability is, the better NAT treatment efficacy the patient has; the prediction results of the two types of classification models corresponding to the patient in the current course are integrated to obtain the final NAT efficacy prediction result.

[0033] In one embodiment, constructing a data set specifically includes the following steps:

[0034] (1) The ultrasound image source data were obtained from the relevant departments of a provincial people's hospital. A team of medical experts collected ultrasound images of breast cancer patients before and during neoadjuvant therapy from 2018 to 2023, including conventional two-dimensional ultrasound and ultrasound contrast imaging images.

[0035] (2) The images in the ultrasound image source data were preliminarily screened to obtain the NATUS (NAT Ultrasound) dataset, i.e., the neoadjuvant therapy ultrasound image dataset. In order to train the YOLO network, the NATUS dataset was manually annotated.

[0036] (3) In order to obtain the data sets NUTD (two-dimensional NAT ultrasound data set) and NUCE (NAT ultrasound contrast data set) that can be used for classification networks.

[0037] Preferably, the NATUS dataset production process includes:

[0038] (1) Manual image screening. The original image data collected retrospectively from the hospital contains some invalid data, such as the presence of puncture needle images in pictures or videos, or the collection of axillary lymph nodes during the ultrasound contrast video recording process, as well as images that are too blurry or are not the ultrasound images required by this embodiment. These data cannot be used to train the model and need to be manually deleted to avoid subsequent network training.

[0039] (2) Naming and numbering. First, a unique four-digit number was generated for each patient, and the correspondence between these numbers and patients was saved in an Excel file. Next, all ultrasound images were renamed, and the total length of the new image name was 7 digits. Among them, the first 4 digits are the patient number to which the image belongs; the fifth digit is the treatment course to which the image belongs. This digit is 0 for the image before treatment, 1 for the image after two treatment courses, and so on; the sixth digit indicates the type of image. This digit is 1 for the image that is an ultrasound contrast image, and 2 for the image that is a two-dimensional ultrasound image; the last digit is separated from the first 6 digits by an underscore, indicating the image number of this patient, this treatment course, and this ultrasound type.

[0040] Take "017442\_8.JPG" as an example. The patient number of this image is 0174. It is a two-dimensional ultrasound image of this patient after 4 courses of treatment. The last digit 8 indicates that this image is the eighth image under the above conditions. The work of naming and numbering is to effectively manage and trace the data set, which is of great practicality.

[0041] (3) Data labeling. This embodiment uses the labeling tool labelimg to perform data labeling. After setting the image directory and label saving directory, manually select the lesion and mark the category as the ultrasound image category, that is, the lesion in the ultrasound contrast image is marked as 1, and the lesion in the two-dimensional ultrasound is marked as 2. The reason for distinguishing the two is that there are some images that include both ultrasound contrast and two-dimensional ultrasound. If they are labeled uniformly during labeling, it will make it difficult to distinguish the source of the data set later. Here, we still follow the convention of the numbering stage, with 1 representing ultrasound contrast and 2 representing two-dimensional ultrasound.

[0042] (4) Annotation review. After the data annotation work is completed, all images and label files will be sent back to the hospital's ultrasound department, where professional doctors will review the annotations one by one to ensure the accuracy of the annotations. After the review is completed, the doctor's modified annotations will overwrite the original annotations, and the annotations that the doctor believes are misjudged will be deleted.

[0043] Preferably, the NUCE and NUTD dataset production process includes:

[0044] (1) Image cropping. The original ultrasound image contains a lot of irrelevant information such as the machine model and examination date. Tissues and blood vessels that are not related to the disease analysis also occupy most of the area in the image. Therefore, before the image is sent to the classification network, it needs to be cropped. It should be noted that features such as the clarity of the boundary of the tissue near the tumor can also assist in judging the changes in the patient's condition. Therefore, in order to enable the classification network to obtain as much information and extract features as possible, the cropping is not performed completely according to the label file in NATUS, but the four coordinates are expanded outward by 5 pixels.

[0045] (2) Image preprocessing. The cropped images have different sizes. To unify the image sizes, this embodiment first resizes all images to a longer side of 256 pixels while keeping the aspect ratio unchanged. The resized images are then expanded to 256*256 pixels, and the blank areas are filled with black. This step ensures the consistency of the sizes of all images, providing a consistent benchmark for subsequent image processing and sharing.

[0046] (3) Dataset enhancement. In the deep learning related research based on medical images, the problem of small data set or imbalance of positive and negative samples is usually faced, which may lead to overfitting of deep learning network models. In order to meet this challenge, related research will enhance the data set. The data set enhancement of this embodiment is mainly reflected in two aspects. First, in order to increase the number of samples in the data set, this embodiment adopts the strategy of sending ultrasound video to the YOLOv8 network for video recognition, and performs frame splitting operation on the video when the confidence level is higher than 0.9 to achieve the purpose of expanding the data set, and the images obtained after frame splitting are returned to the doctor for review. Secondly, some patients have fewer ultrasound images after certain courses of treatment. In order to solve this problem, this embodiment flips the images of these patients horizontally or vertically to double the number of images. These enhancement strategies help to increase the diversity and richness of the data set to improve the generalization ability of the model and alleviate the overfitting problem.

[0047] (4) Image archiving. In the subsequent Siamese Network classification process, it is necessary to organize the images of the same patient after different courses of treatment and send them to the network for processing. To achieve this goal, this embodiment archives the images according to two levels: patients and courses of treatment. Specifically, for each patient, create a folder and create four subfolders 0, 2, 4, and 6 in it, corresponding to different numbers of courses of treatment. Then, all images are stored in the subfolders corresponding to the courses of treatment for the corresponding patients in sequence. Information such as patient number, number of courses of treatment, and image type has been reflected in the naming of the NATUS dataset, so the naming in the NATUS dataset will also be used in the NUTD and NUCE datasets.

[0048] In one embodiment, the classification network for lesion detection and image classification based on the twin network is specifically constructed as follows:

[0049] In this embodiment, the classification network uses MobileNet as the backbone network, and replaces the SE (Squeeze-and-Excitation) module with the scSE (spatial and channel squeeze & excitation block) attention module on its basis, and adds the CBAM (Convolutional Block Attention Module) attention mechanism after the start (StartStage) and middle (Middle Stage) of the MobileNet network respectively. In addition, the non-global pooling operation in MobileNet is replaced by soft pooling, and finally the classification network SiaMNet (Siamese-MobileNet) of this embodiment is obtained, as shown in FIG. Figure 1 Said. That is, in this embodiment, the classification network includes two twin networks based on MobileNet, which are used to extract features of the input data (control images before and during neoadjuvant therapy), that is, one MobileNet is used to extract the image features of the ultrasound data corresponding to the neoadjuvant therapy before; the other MobileNet is used to extract the image features of the ultrasound data during the neoadjuvant therapy (corresponding to the course of treatment); wherein, the network structure of each backbone network is a stacked structure of several bneck (Bottleneck) structures, wherein the main branch of the bneck structure includes a 1x1 convolution block, a 3x3 depth-separable convolution block (DWConv), a 1x1 convolution block, a spatial and channel attention module (scSE block) and a convolution attention module (CBAM block) in sequence, and there is a short connection (Shortcut) between the input of the bneck structure and the output of the scSE block, that is, the output features of the scSE block are added to the input of the bneck structure and then used as the input of the CBAM block. It should be noted that in this embodiment, the convolution block refers to a convolution layer, a batch normalization layer and an activation function layer in sequence.

[0050] In the classifier part, SiaMNet deleted the original average pooling layer and fully connected layer of MobileNet, flattened and concatenated the features (FeaturesA and FeaturesB) extracted by the two twin networks, and passed the concatenated feature vector (Features) through two 1x1 convolutions to finally output the prediction results of the network. Among them, the two 1x1 convolutions refer to the convolution operation based on the convolution layer and the activation function, and the activation function of the last 1x1 convolution is the Softmax function, which is used to output the efficacy probability of the patient's neoadjuvant treatment. The smaller the value, the better the efficacy. In addition, before the feature vector is sent to the 1x1 convolution, an additional dropout layer needs to be added to prevent overfitting. The final output result of the network is a number p normalized to between 0 and 1, which represents the probability that the patient has a poor efficacy. When p<0.2, it is believed that the patient is more likely to have a good efficacy, and the larger p is, the greater the probability that the patient's neoadjuvant treatment efficacy is poor.

[0051] In one embodiment, the training strategy for the constructed classification network is as follows:

[0052] In the embodiment of the present invention, a classification model based on SiaMNet is set for different courses of treatment, so as to obtain classification models for different courses of treatment. In this embodiment, corresponding classification models are set for courses 2, 4, and 6, which are respectively referred to as SiaM-2, SiaM-4, and SiaM-6. Then the constructed classification model is trained based on the constructed data set. In addition, due to the large difference between two-dimensional ultrasound and ultrasound contrast imaging images, they will also be trained separately during training to ensure that the network can fully learn the characteristics of each type of image. That is, after the training is completed, each course of treatment corresponds to two classification models, one is a classification model based on the NUCE data set (i.e., a classification model based on ultrasound contrast imaging), and the other is a classification model based on the NUTD data set (i.e., a classification model based on two-dimensional ultrasound). Then, when obtaining the efficacy prediction results corresponding to different courses of treatment, the final efficacy prediction results of each course of treatment can be obtained based on the weighted fusion results of the two classification models.

[0053] In addition, when reading in data, on the basis of random reading, it is ensured that the network can traverse each image pair (control images before neoadjuvant therapy and during treatment) during the reading process. Although this strategy requires longer training time, it can ensure that the network extracts features from all images, thereby improving the network's feature extraction ability. In the process of loading the data set, the image rotation and random cropping methods are added. Each time the image is read in, the image size is cropped from 256×256 to 224×224, and a random number of 90° rotations and upside-down flips are performed. During the training process, the SGD (Stochastic gradient descent) optimizer and the cosine annealing learning rate adjustment strategy are used. When freezing, the initial learning rate is set to 0.0001 to improve the learning efficiency of the fully connected layer, and in the fine-tuning stage, the learning rate is set to 0.00001 to reduce the risk of overfitting. During training, the prediction results output by the model are compared and evaluated with the doctor's gold standard, that is, the model parameters are tuned based on the deviation between the prediction results and the doctor's gold standard. After training is completed, it can also be tested on the test set. The model prediction performance can be evaluated by calculating indicators such as accuracy, specificity, and sensitivity based on the results output by the model on the test set.

[0054] For different network models, this embodiment first trains the SiaM-6 model, because the model uses the images of the last course of treatment for classification, and it is easier to classify the extracted features, which is more suitable for the transition from the natural image field to the ultrasound image field. In the training process of SiaM-6, the idea of ​​transfer learning is adopted, and the weights of MobileNet pre-trained on ImageNet are used as the initial weights of the backbone, and the weight parameters of the 1×1 convolution layer and the scSE and CBAM attention mechanisms are randomly initialized. In the training process of SiaM-4 and SiaM-2, the model corresponding to the latter course of treatment is used as the initial weight for migration, that is, the weights trained by SiaM-6 are used as the initial weights of SiaM-4, and then SiaM-4 is fine-tuned on the data set of the corresponding course of treatment; while SiaM-2 uses the weights trained by SiaM-4 as the initial weights and fine-tunes on the corresponding data set. This training strategy enables SiaMNet to inherit and retain the feature extraction capability of ultrasound images.

[0055] The platform environment of this embodiment is: Linux operating system version Ubuntu 18.10LTS, GPU model NvidiaGeForce RTX 2080Ti, video memory size 12GB, CPU model Intel i9-9900K, memory size 64GB. The programming language is Python 3.19.16, the deep learning framework is Pytorch, the torch version is 1.13.1, and the torchvision version is 0.14.1. Based on the Linux system, the construction and training of the convolutional neural network are realized based on pytorch, and other scripts are written in python, and the process commands under Linux are performed using shell language to realize the connection of the entire step.

[0056] See also Figure 2 , the specific processing process of obtaining the prediction results of the efficacy of neoadjuvant therapy for breast cancer based on the trained classification model includes:

[0057] S1, open the image file of the patient to be predicted;

[0058] S2, image legitimacy detection, i.e. checking whether the image data in the currently read image file is required to be input into the classification model for the implementation of the present invention; if the detection fails, re-reading;

[0059] S3, performing lesion detection on the image data in the image file, and then performing lesion area preprocessing, that is, marking and reviewing the lesion area of ​​the image data; then cropping the corresponding lesion area image and performing corresponding image preprocessing, such as image size normalization, image enhancement, etc.;

[0060] Step S4, reading the lesion area image after preprocessing of the lesion area,

[0061] Detect the image type. If it is ultrasound contrast imaging, execute step S5; if it is two-dimensional ultrasound, execute step S6;

[0062] Step S5, inputting the lesion area image into the ultrasound contrast-enhanced classification model corresponding to the current course of treatment, and obtaining the efficacy prediction result corresponding to the current course of treatment, which is the approximate mean of all lesion area images in the current batch;

[0063] Step S6, inputting the lesion area image into the two-dimensional ultrasound-based classification model corresponding to the current course of treatment, and obtaining the efficacy prediction result corresponding to the current course of treatment, wherein the efficacy prediction result is the approximate average value of all lesion area images of the current batch;

[0064] Step S7, performing weighted fusion on the approximate means obtained in steps S5 and S6 (the corresponding weight is a preset value) to output the final prediction result of the patient to be predicted.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0066] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for predicting the efficacy of neoadjuvant therapy for breast cancer based on ultrasound images, characterized in that: The following steps are involved: Step 1, constructing a data set based on ultrasound images of the patient at different courses before and during neoadjuvant therapy, wherein the ultrasound image data in the data set includes two-dimensional ultrasound images and ultrasound contrast images of the same patient; Step 2: construct a classification network for neoadjuvant therapy efficacy prediction based on the twin network; The classification network includes two feature extraction branches with the same network structure and a classifier. One feature extraction branch is used to extract the image features of the ultrasound image corresponding to the patient before neoadjuvant treatment; the other feature extraction branch is used to extract the image features of the ultrasound image of the patient during neoadjuvant treatment; the classifier is used to flatten and splice the feature maps output by the two feature extraction branches, and then output the efficacy probability of the patient's neoadjuvant treatment based on the classification layer. The value range of the probability is a number between 0 and 1. The smaller the value, the better the efficacy of the patient. Dividing the data set into sub-data sets corresponding to different treatment courses based on the treatment courses, and dividing the sub-data sets into a first sub-data set and a second sub-data set based on the ultrasound image type of each sub-data set, corresponding to the two-dimensional ultrasound image and the ultrasound contrast image, respectively; A classification network is set for each treatment course. Based on the data set corresponding to the treatment course, the classification network is trained on the first and second sub-data sets corresponding to the treatment course in order from the last treatment course to the first treatment course, so as to obtain two classification models for each treatment course. During training, the network parameters of the classification network of the first treatment course to be trained can be randomly initialized, or the pre-trained network parameters can be used; the initial values ​​of the network parameters of the classification network of the next treatment course to be trained are the trained network parameters of the classification network of the previous treatment course; Step 3, obtaining an ultrasound image pair of the patient to be predicted before and during neoadjuvant therapy, the ultrasound image pair including an image pair based on a two-dimensional ultrasound image and an image pair based on an ultrasound contrast-enhanced image; Performing lesion detection on the ultrasound image pair, extracting the lesion area image and performing image preprocessing to obtain a lesion image pair that matches the input of the classification model; The lesion image pair is then input into the classification model of the corresponding ultrasound image type of the corresponding treatment course, and the efficacy probabilities output by the two classification models are fused to obtain the neoadjuvant therapy efficacy prediction result of the current patient to be predicted.

2. The method according to claim 1, characterized in that In step 1, the naming fields of the image names of the ultrasound images in the data set include: patient number, treatment course, image type and image number for distinguishing ultrasound contrast images from two-dimensional ultrasound images.

3. The method according to claim 2, characterized in that In step 1, the patient number, treatment course, image type, and image number in the image name are set as follows: The patient number is represented by a number of digits; A single digit is used after the patient number to indicate the course of treatment; A 1-digit number is used to represent the type of image after the corresponding treatment course; The image number is represented by a 1- to 2-digit number or a combination of a 2- to 3-digit number and a special symbol after the image type.

4. The method according to claim 1, characterized in that The image processing used to obtain image pairs for input to the classification network during training includes: The lesion area in the image is manually annotated and reviewed by experts to obtain the lesion annotation frame; Based on the method of expanding the lesion annotation box outward by a specified number of pixels, the lesion area image is cropped out from the ultrasound image, and then the image size is normalized to obtain the image of the image pair input into the classification network.

5. The method according to claim 4, characterized in that When normalizing the image size, fill the blank areas with black.

6. The method according to claim 1, characterized in that Step 1 also includes data enhancement processing of the data set, and the processing method is as follows: the ultrasound video is sent to the YOLOv8 network for video recognition, the video segments with confidence higher than the specified value are deframed, and the data set is expanded based on the obtained video frame images.

7. The method according to claim 1, characterized in that The feature extraction branch is a stacked structure of several bottleneck structures, wherein the main branch of the bottleneck structure includes a convolution layer with a 1x1 convolution kernel, a batch normalization layer, an activation function layer, a depthwise separable convolution layer with a 3x3 convolution kernel, a batch normalization layer, an activation function layer, a 1x1 convolution layer, a batch normalization layer, an activation function layer, a spatial and channel attention module, and a convolution attention module, and there is a short connection between the input of the bottleneck structure and the output of the spatial and channel attention module.

8. The method according to claim 1, characterized in that The classification layers of the classifier include dropout layer, 1x1 convolution layer, activation function layer, 1x1 convolution layer and Softmax function layer in sequence.

9. The method according to any one of claims 1 to 8, characterized in that The treatment courses are divided into four categories: before treatment, two courses of treatment, four courses of treatment and six courses of treatment, and corresponding classification models are set for the latter three categories of treatment courses.

Images