Breast cancer grading analysis system and method based on artificial intelligence, terminal and medium

Abstract

The invention provides a breast cancer grading analysis system and method based on artificial intelligence, a terminal and a medium. The system comprises an ultrasonic image module, a molybdenum target image module and a clinical information module. The ultrasonic image module and the molybdenum target image module can be used independently or in a combined mode. The independent use of the ultrasonic image module can be deployed in the ultrasonic department for use, and the independent use of the molybdenum target image module can be deployed in the radiology department for use, and is suitable for scenes with partial image missing. The mixing module (mixed with the clinical information module or without the clinical information module) can be deployed for use in breast surgery. Under the condition of independent use or mixed use, the system can output two decisions, namely a fine-level pathological layer and a Coarse-level image BI-RADS layer. The method has the technical effects of being high in generalization ability, high in modularization and clinical adaptability, high in accuracy, high in interpretability, high in grading decision-making ability and the like.

Description

Technical Field

[0001] The present application relates to the field of smart medical technology, in particular to the fields of medical image analysis, artificial intelligence and computer science technology. Background Art

[0002] In 2020, breast cancer officially surpassed lung cancer to become the world's number one cancer. Moreover, the incidence rate of breast cancer in China ranks first in the world, twice the global average. Early detection and accurate diagnosis are of great significance for treatment and prognosis. Mammography screening is currently the simplest and most reliable means of diagnosing breast diseases, but it has poor penetration into dense breast tissue. It is generally not recommended for women under 40 years old who have no clear high-risk factors for breast cancer or no abnormalities found in clinical physical examinations.

[0003] Ultrasound imaging, as a non-invasive, low-dose, radiation-free detection technology, has become an important supplementary means of early screening for breast cancer and a common tool for clinical diagnosis. However, due to the complex anatomical structure of breast tissue, which involves multiple imaging modalities and multiview, a single imaging method often fails to produce accurate diagnostic results.

[0004] Therefore, how to design BI-RADS Altas that meets the clinical breast examination standard, use multimodal and multi-angle ultrasound imaging (B-ultrasound, Doppler, elastic imaging, transverse and longitudinal orthogonal sections) and standard perspective mammography image data (CC and MLO perspectives) to improve the accuracy and reliability of breast cancer grading diagnosis, and further guide clinical surgery and treatment plans has always been a research hotspot in the field of medical imaging.

[0005] Traditional breast cancer diagnosis mainly relies on the experience and skills of doctors, but this method has problems such as strong subjectivity and low accuracy. Common AI-based breast cancer diagnosis technologies include machine learning and convolutional neural networks. These technologies aim to extract features from breast cancer images and classify and predict them through training algorithms. However, due to the complex features of breast cancer images and the existence of many different pathological types and molecular subtypes, traditional deep learning models often find it difficult to achieve accurate diagnostic results. With the development of computer technology and artificial intelligence, more and more studies have begun to explore the impact of multimodal information fusion on breast cancer diagnosis. In this context, AI-based breast cancer diagnosis technology has been continuously improved and developed, becoming one of the current research hotspots in the field of breast cancer diagnosis. Summary of the invention

[0006] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide an artificial intelligence-based breast cancer grading analysis system, method, terminal and medium to solve the technical problem that traditional deep learning models often find it difficult to achieve accurate diagnostic results.

[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an artificial intelligence-based breast cancer grading analysis system, comprising an ultrasound image module, a molybdenum target image module and a clinical information module; wherein: the ultrasound image module and the molybdenum target image module are mixed and used together to form a first combination module, and both are respectively based on their own multi-view image training neural network models to output predicted probability values ​​of breast cancer grading multiple classifications; the clinical information module is mixed and used with either or both of the ultrasound image module and the molybdenum target image module to form a second combination module, and the most recently updated neural network model weights in the ultrasound image module and the molybdenum target image module are used as the pre-trained model weights of the second combination module.

[0008] In some embodiments of the first aspect of the present application, the structure of the neural network model used in the ultrasound image module and the molybdenum target image module includes a feature extraction module and a feature fusion module.

[0009] In some embodiments of the first aspect of the present application, the feature extraction module includes: a first convolutional neural network, a squeeze and excitation module, a second convolutional neural network, a self-attention mechanism module, and a gradient-weighted class activation heat map module; wherein: the first convolutional neural network inputs a first-perspective image and a second-perspective image with shared weights; the output of the first convolutional neural network is connected to the input of the squeeze and excitation module; the output of the squeeze and excitation module is connected to the input of the second convolutional neural network; the output of the second convolutional neural network is connected to the input of the self-attention mechanism module; the gradient-weighted class activation heat map module draws and outputs a corresponding heat map based on the output result of the self-attention mechanism module.

[0010] In some embodiments of the first aspect of the present application, the first convolutional neural network inputs two images of different perspectives under the same modality, which are a first perspective image and a second perspective image; the first perspective image and the second perspective image are respectively one and the other of a cranio-caudal breast mammography image and a medial-lateral oblique breast mammography image, or one and the other of a transverse breast ultrasound image and a longitudinal breast ultrasound image; and the first perspective image and the second perspective image share weights.

[0011] In some embodiments of the first aspect of the present application, the self-attention mechanism module calculates the relative importance of each position in the input image compared to all other positions, generates a set of weighted representations accordingly, and uses these weighted representations to perform weighted summation on the input features, introducing global information to capture the spatial relationships and contextual information between objects in the image.

[0012] In some embodiments of the first aspect of the present application, the self-attention mechanism module uses a gradient-weighted class activation heat map to draw a heat map.

[0013] In some embodiments of the first aspect of the present application, the self-attention mechanism module uses the gradient-weighted class activation heat map to draw the heat map in the following manner:

[0014] For the target classification category y, generate the corresponding weight for each position (i, j) on the input image To indicate the importance of image position (i, j) to category y; weight The calculation methods include:

[0015] Among them, k represents the number of layers, y represents the category; A k Represents the features after global average pooling; Represents the feature map A after global average pooling k The weight coefficient related to the target classification category y; Represents feature map A k The partial derivative of the input image channel c at position (i, j);

[0016] The heat map is calculated by weighting Multiplying it with the original feature map and performing the ReLU function operation yields:

[0017] Among them, Grad-CAM (c) (i, j) represents the heat map; represents the weight of the image position (i, j); Represents the activation value of the feature map corresponding to the y-th category in the k-th layer at position (i, j).

[0018] In some embodiments of the first aspect of the present application, the feature fusion module includes a modal attention module, a fully connected layer module, and an interpretability module; the output of the modal attention module is respectively connected to the fully connected layer module and the interpretability module.

[0019] In some embodiments of the first aspect of the present application, the modality attention module is used to perform attention weighted processing on the multi-modality output by the feature extraction module, and extract the correlation relationship between them by learning the correlation of each modality.

[0020] In some embodiments of the first aspect of the present application, after the feature fusion module performs attention weighting through the modal attention module, it directly passes through the fully connected layer module to obtain the final classification result and probability, that is, it directly outputs the predicted probability value of breast cancer grading multi-classification; or, after performing attention weighting through the modal attention module, it uses the interpretability module to learn prototypes from each modality and extract corresponding interpretable features; the interpretability module learns representative prototypes from each modality, and performs similarity comparison during the reasoning process to obtain interpretable features based on domain knowledge.

[0021] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an artificial intelligence-based breast cancer grading analysis method, including: using an ultrasound image module and a molybdenum target image module to form a first combination module, and both of them train neural network models based on their respective multi-view images to output predicted probability values ​​of breast cancer grading multiple classifications; or, using a clinical information module and either or both of the ultrasound image module and the molybdenum target image module to form a second combination module, and using the most recently updated neural network model weights in the ultrasound image module and the molybdenum target image module as the pre-trained model weights of the second combination module.

[0022] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the artificial intelligence-based breast cancer grading analysis method is implemented.

[0023] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the computer device performs the artificial intelligence-based breast cancer grading analysis method.

[0024] As described above, the artificial intelligence-based breast cancer grading analysis system, method, terminal and medium of the present application have the following beneficial effects:

[0025] (1) Strong generalization ability: The technical solution of the present invention has been prospectively externally verified in 4 centers and more than 5 different machines. The AI ​​system can perform a more comprehensive and accurate analysis of breast cancer and has strong adaptability to new samples.

[0026] (2) Modularity and high clinical applicability: The present invention uses an AI system with three modules to train a dedicated breast cancer grading diagnosis model, and combines ultrasound images and mammography images for use alone or in combination. It not only takes into account the factors of individual differences among patients, but also provides a variety of different diagnostic methods to choose from, improving diagnostic accuracy and reliability, thereby better ensuring accuracy and universality. In addition, even when not used in combination, the AI ​​system of the present invention can also analyze separate ultrasound images or mammography images, so that even if a certain image is missing, the AI ​​system can still be used.

[0027] (3) High accuracy: The ultrasound or molybdenum target image module of the present invention alone surpasses the radiologist (i.e., the clinical image BI-RADS grading task). When used in combination, it can achieve cross-level operations and reach the level of biopsy pathology, providing strong support for biopsy methods and surgical plans.

[0028] (4) High interpretability: The present invention can provide doctors with diagnostic evidence (in two ways: post-hoc attention map or heatmap, and prototypes of domain knowledge), thereby enhancing doctors’ diagnostic confidence and enabling better auxiliary diagnosis.

[0029] (5) Strong grading decision-making ability: The present invention can not only provide clinical BI-RADS grading requirements, but also provide judgment of pathological levels to guide clinicians in selecting surgical plans and treatment methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A Shown is a structural schematic diagram of a breast cancer grading analysis system based on artificial intelligence in one embodiment of the present application.

[0031] Figure 1B Shown is a schematic diagram of the structure and application of an artificial intelligence-based breast cancer grading analysis system in one embodiment of the present application.

[0032] Figure 2 Shown is a structural schematic diagram of a feature extraction module in one embodiment of the present application.

[0033] Figure 3A Shown is a structural schematic diagram of the extrusion and excitation module (SENet) in one embodiment of the present application.

[0034] Figure 3B Shown is a schematic diagram of the execution flow of the extrusion and excitation module (SENet) in one embodiment of the present application.

[0035] Figure 4 Shown is a structural schematic diagram of a feature fusion module in one embodiment of the present application.

[0036] Figure 5 Shown is a flow chart of a breast cancer grading analysis method based on artificial intelligence in one embodiment of the present application.

[0037] Figure 6 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0039] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0040] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0041] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0042] To solve the problems in the above background technology, the present invention provides an artificial intelligence-based breast cancer grading diagnosis system, method, medium and terminal, which adopts the technical solution of an artificial intelligence-based breast cancer diagnosis center. The diagnosis center can collect data from various hospitals and use the free combination of the following three submodules: ultrasound image module, mammography image module and clinical information module to train a dedicated breast cancer grading diagnosis model to help doctors make better decisions.

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution in the embodiments of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the invention.

[0044] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0045] <1> BIRADS classification: refers to the BI-RADS classification standard, which is a breast imaging reporting and data system created and recommended by the American College of Radiology, used to evaluate breast changes in imaging examinations.

[0046] <2> CC (The Craniocaudal Projection) position: The craniocaudal position in mammography. A good CC position mammography image shows the breast in the center of the image, the nipple tangent position, a small part of the pectoralis major muscle, the inner breast tissue should be included in the image, and the outer breast tissue should be included in the image as much as possible.

[0047] <3> MLO (The Mediolateral Oblique Projection) position: The medial and lateral oblique position in the mammography projection position. A good MLO position mammography image shows that the breast is pushed forward and upward, the breast substance is fully expanded, the pectoralis major muscle is visible and relatively loose, the lower edge reaches the level of the nipple, the nipple is in the tangent position, part of the abdominal wall is included in the film, but separated from the lower breast, and most of the breast substance is shown in the film. The outline of the nipple can be seen at the outer edge of the breast tissue; the fat tissue behind the breast is well displayed, and the breast has no wrinkles.

[0048] <4> Three modalities of ultrasound:

[0049] B-mode ultrasound (B-ultrasound): constructs the structural information of tissues through ultrasonic reflection signals, which is a grayscale structure image based on sound intensity.

[0050] Color Doppler ultrasound: It uses the Doppler principle to detect blood flow in blood vessels within tissues. It is a dynamic blood flow function test. The image is color-coded to distinguish whether the blood flow direction is toward or away from the ultrasound transducer, and then determine the condition of the arteries and veins.

[0051] Elastography: It is an emerging ultrasonic diagnostic technology in recent years. It provides quantitative and qualitative mechanical parameters of tissues through the tissue deformation caused by stimulation, replacing the traditional palpation method and making up for the shortcomings of medical ultrasonic imaging modality.

[0052] <5> Orthogonal ultrasound (longitudinal and transverse views): find the longest axial section of the mass and then rotate 90 degrees orthogonally to acquire the second section. This method ensures the detection of the most comprehensive information about the mass.

[0053] <6> Batch Normalization: BatchNorm for short. The function of Batch normalization is to normalize the input features of each layer. Normalizing the input features can accelerate the learning of neural networks.

[0054] The embodiments of the present invention provide a breast cancer grading analysis method based on artificial intelligence, a system for a breast cancer grading analysis method based on artificial intelligence, and a storage medium storing an executable program for implementing the breast cancer grading analysis method based on artificial intelligence. As for the implementation of the breast cancer grading analysis method based on artificial intelligence, the embodiments of the present invention will illustrate an exemplary implementation scenario of breast cancer grading analysis based on artificial intelligence.

[0055] like Figure 1AAs shown, a schematic diagram of the structure of a breast cancer grading analysis system based on artificial intelligence in an embodiment of the present invention is shown. The breast cancer grading analysis system 100 based on artificial intelligence in an embodiment of the present invention at least includes: an ultrasound image module 101, a molybdenum target image module 102 and a clinical information module 103; wherein the ultrasound image module 101 and the molybdenum target image module 102 are used alone or in combination to form a first combination module, and both are respectively based on their own multi-view image training neural network models to output the predicted probability value of breast cancer grading multiple classifications; the clinical information module 103 is used in combination with either or both of the ultrasound image module 101 and the molybdenum target image module 102 to form a second combination module, and the latest updated neural network model weights in the ultrasound image module 101 and the molybdenum target image module 102 are used as the pre-trained model weights of the second combination module.

[0056] It is worth noting that the system provided by the embodiment of the present invention can use each module individually or in combination according to different actual scene requirements. Each module can be used individually for scenes where some images are missing. For example, a separate ultrasound image module can be deployed in the ultrasound department, a separate molybdenum target image module can be deployed in the radiology department, and a mixed module (mixed with a clinical information module or without a clinical information module) can be deployed in breast surgery. Finally, the system outputs two decisions, namely, the pathology level (fine-level) and the image BI-RADS level (coarse-level).

[0057] Breast cancer grading refers to the BIRADS grading. The BIRADS grading determines the nature of the tumor based on the size of the tumor, whether the boundary is clear, and the blood flow around the tumor in color Doppler ultrasound, molybdenum target, and MRI examinations. It can be divided into 7 levels from 0 to 6. Different levels represent different degrees of tumor lesions, as shown below:

[0058] Level 0: It means that breast disease classification cannot be performed after ultrasound examination, and other imaging techniques are needed to assist in judgment, such as breast mammography and magnetic resonance imaging. In most cases, it is proved that there is almost no breast lesion.

[0059] Level 1: No abnormalities were found in the ultrasound examination. This generally indicates that the breast is normal and there is no lesion. No treatment is required and regular examinations are sufficient.

[0060] Grade 2: It indicates breast lesions, but they are mainly considered to be benign lesions, most of which are hyperplasia, and malignancy can be basically ruled out.

[0061] Level 3: Considered to be a benign lesion, no special treatment is required, but careful observation is recommended. It is recommended to check the mass every 3-6 months. It is usually a fibroadenoma, multiple cysts, etc.

[0062] Level 4: A lump is found in the breast, indicating a suspected malignant lesion in the breast. It is necessary to combine pathological examination to determine whether there are cancerous cells. If there are, it indicates a malignant tumor. Ultrasound and mammography images also subdivide level 4, including three subtypes: 4a, 4b, and 4c.

[0063] Level 5: Indicates that the breast lump has a high possibility of deterioration. In most cases, it is due to breast cancer. It is recommended to go to the hospital for surgery as soon as possible for pathological examination and diagnosis.

[0064] Level 6: Indicates that the patient has been diagnosed with breast cancer through pathological examination and needs to be controlled through surgery, chemotherapy, radiotherapy, targeted therapy and other methods.

[0065] In summary, the different grade classifications of BI-RADS are mainly used to judge the benign or malignant possibility of a tumor. Most of the lesions at grade 3 and below are benign and do not require medication or surgical treatment. Just pay attention to regular check-ups. At grade 4 and above, malignant lesions need to be considered and relevant treatments need to be carried out to control the disease.

[0066] For ease of understanding, Figure 1B The structure and application diagram of an artificial intelligence-based breast cancer grading analysis system in an embodiment of the present invention are shown. By combining the ultrasound image module, the molybdenum target image module and the clinical information module, a data set is obtained from the hospital database, and the trained model is deployed in the hospital application environment. The system provided in the embodiment of the present invention can provide doctors with diagnostic evidence (two methods, post-hoc attention maps or heat maps, prototypes of domain knowledge) to enhance the doctor's diagnostic confidence so as to better perform auxiliary diagnosis; it can also achieve graded diagnosis to reach the pathological level and imaging level.

[0067] Specifically, the second combined module formed by the mixed use of the ultrasound image module 101, the molybdenum target image module 102 and the clinical information module 103 includes the following:

[0068] Combination module 1) A combination module formed by mixing the clinical information module 103 and the ultrasound image module 101;

[0069] Combination module 2) a combination module formed by mixing the clinical information module 103 and the molybdenum target image module 102;

[0070] Combination module 3) A combination module formed by mixing the clinical information module 103 with the ultrasound image module 101 and the molybdenum target image module 102.

[0071] It is worth noting that the pre-trained model weights have learned a wealth of information about molybdenum targets and ultrasound images. By using this prior knowledge in the combined module tasks, the generalization ability of the model can be effectively improved. Compared with training a model from scratch by randomly initializing the model weights, using a pre-trained model can save a lot of time and resources. By making appropriate adjustments and fine-tuning on the target task, an efficient medical image processing model can be quickly constructed. In addition, the clinical information module 103 includes the patient's basic information, medication information, diagnosis information, treatment plan information, etc.

[0072] In the embodiment of the present invention, the structure of the neural network model used in the ultrasound image module 101 and the molybdenum target image module 102 includes a feature extraction module (View-Speicific) and a feature fusion module (Cross-Modality). The following will explain these two modules respectively.

[0073] The structure of the feature extraction module (View-Specific) is as follows Figure 2 As shown, it includes a first convolutional neural network (CNNblock), a squeeze and excitation module (SENet), a second convolutional neural network (CNN block), a self-attention mechanism module (Self Attention Block), and a gradient-weighted class activation heat map module (Grad-CAM); the first convolutional neural network (CNN block) inputs a first-perspective image and a second-perspective image with shared weights; the output of the first convolutional neural network is connected to the input of the squeeze and excitation module (SENet); the output of the squeeze and excitation module (SENet) is connected to the input of the second convolutional neural network (CNN block); the output of the second convolutional neural network (CNN block) is connected to the input of the self-attention mechanism module (Self Attention Block); the gradient-weighted class activation heat map module (Grad-CAM) draws the corresponding heat map according to the output result of the self-attention mechanism module (Self Attention Block) and outputs it.

[0074] The first convolutional neural network (CNN block) inputs two images of different perspectives under the same modality, which are a first perspective image and a second perspective image; the first perspective image and the second perspective image are one and the other of a craniocaudal mammography image (The Craniocaudal Projection, CC) and a mediolateral oblique mammography image (The Mediolateral Oblique Projection, MLO), or one and the other of a transverse breast ultrasound image and a longitudinal breast ultrasound image; and the first perspective image and the second perspective image share weights.

[0075] It should be noted that sharing weights between two images of the same modality with different perspectives can effectively reduce the amount of calculation. By sharing weights, weight templates can be provided to reduce repeated calculations. The concept of weight sharing means that some local information appears multiple times and is reused globally during model construction. In convolutional neural networks, when the CNN model recognizes an image, the convolution kernel is required to scan the entire image to find the characteristics of each edge. However, some images (such as symmetrical images) must have some repetitive shapes. These repetitive structures mean that the convolution kernel will have some similarities in weights when calculating weights. These similarities are globally shareable. By clustering these shareable weight coefficients and letting them share some weights in a categorical manner, model compression can be achieved.

[0076] The first convolutional neural network (CNN block) and the second convolutional neural network (CNN block) have the same structure, both consisting of a BatchNorm layer, a convolutional layer, and a ReLU activation function, which are used to extract image features.

[0077] Among them: BatchNorm layer is an algorithm that is needed in deep networks to accelerate neural network training, convergence speed and stability; since deep neural networks are mainly used to learn the distribution of training data and achieve good generalization effects on test sets, but if the data input in each batch has different distributions, it will obviously bring difficulties to the training of the network, so BatchNorm layer is needed for normalization. The purpose of the convolution layer is to extract the features of the image, which is the core part of image recognition; the image is a matrix composed of pixels in the computer, and the operation of doing inner product between the image and the filter matrix is ​​convolution; the convolution layer uses the convolution kernel to extract the features of the image. The activation function is mainly used to provide the nonlinear modeling ability of the network. Since the convolution operation is linear, it is not necessarily linearly separable for actual samples. Therefore, nonlinear factors are introduced through the activation function to solve problems that cannot be solved by linear models. The activation functions that can be used in the embodiments of the present invention include but are not limited to Sigmoid function, tanh function, ReLU function, etc., which are not limited to this.

[0078] The squeeze and excitation module (SENet) is used to learn the channel correlation between features and improve the perception and expression ability of the convolutional neural network for the most useful features, thereby enhancing the convolutional neural network's ability to extract the most useful features. It should be understood that SENet's full name is Squeeze-and-Excitation Networks, which consists of a compression module (Squeeze) and an excitation module (Excitation). Specifically, the compression module Squeeze operation is used to reduce the spatial dimension of each channel, and the excitation module Excitation operation is used to convert the obtained vector into a weighted feature map so that the response of each channel is assigned to more useful features.

[0079] The structure and execution process of the squeeze and excitation module (SENet) are as follows Figure 3A and 3B As shown:

[0080] Step a): The two-dimensional feature (H×W) of the single-channel feature vector is compressed into a real number through the global average pooling layer, so that the size of the feature map changes from (H, W, C) to (1, 1, C), and the global feature of each channel is obtained. The specific formula is as follows:

[0081]

[0082] Among them, Z C Represents a real number compressed by global average pooling on a single channel; F sq represents the global average pooling operation; u c is the single-channel feature vector of the input; H represents the height; W represents the width.

[0083] Step b): The excitation module performs a dimensionality reduction operation on the compressed feature map through the first fully connected layer; after the dimensionality reduction operation, a nonlinear operation between layers is performed through the ReLU function; after the nonlinear operation, the feature map is restored to the dimension before dimensionality reduction through the second fully connected layer; after the dimensionality is restored, the feature map of the restored dimension is mapped to between 0 and 1 by the Sigmoid function to obtain a channel weight with the same number as the number of restored dimensions.

[0084] The feature map with a size of 1×1×C obtained by compression is reduced in dimension through a fully connected layer; the nonlinear relationship between layers is increased through the ReLU function; the feature map is restored to a dimension of 1×1×C through a fully connected layer; and the sigmoid function is used to map the restored feature map to between 0 and 1 to obtain 1×1×C channel weights. The specific formula is as follows:

[0085] s=F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z)); Formula (2)

[0086] Among them, s represents the final output channel weight value; F ex represents the excitation operation consisting of two connection layers, a ReLU activation function and a Sigmoid function; δ represents the ReLU activation function operation; σ represents the Sigmoid function operation; W1 and W2 represent the fully connected layer 1 and the fully connected layer 2; z represents the real number generated by the compression operation.

[0087] Step c): Multiply the 1×1×C channel weights by the two-dimensional matrix of the channel corresponding to the original feature map to obtain the compressed and excited feature map. The specific formula is as follows:

[0088]

[0089] in, is the feature map of the final output after compression and excitation; C represents the 1×1×C channel weights generated by the two operations of compression and excitation; u C The feature vector representing the original feature map (H, W, C).

[0090] Combination Figure 3B The process in the figure is as follows: the feature map size of the original image is C*H*W, which is compressed to size C*1*1 after global average pooling. Then, the excitation module reduces the dimension of the compressed feature map of size C*1*1 through the first fully connected layer FC to size After the dimensionality reduction operation, the ReLU function is used to perform nonlinear operations between layers (not shown in the figure); after the nonlinear operation, the feature map is restored to the size C*1*1 through the second fully connected layer FC; then the feature map of size C*1*1 is mapped to between 0 and 1 by the Sigmoid function, thereby obtaining C*1*1 weights.

[0091] The output structure of the squeeze and excitation module (SENet) is convolved by the second convolutional neural network, and the convolution calculation result is input to the self-attention mechanism module (Self Attention Block). The self-attention mechanism module (Self Attention Block) can calculate the relative importance of each position in the input image compared to all other positions (importance is expressed by weight, the higher the weight, the more important it is), generate a set of weighted representations, and use these weighted representations to perform weighted summation on the input features, introducing global information, which enables the model to better capture the spatial relationship and contextual information between objects in the image, thereby improving classification performance. More preferably, the two perspectives in the feature extraction module (View-Specific) share the model weights, extract various image features respectively, and finally splice them into the final perspective features. For example, the various image features extracted include but are not limited to images corresponding to two perspectives of the ultrasound module, two perspectives of the molybdenum target module, and two perspectives of the ultrasound molybdenum target module.

[0092] In some examples, gradient-weighted class activation mapping (Grad-CAM) is used to draw heat maps to visualize the input regions on which the deep learning model's decisions in classification tasks are based, which is used to help understand the internal mechanisms of the neural network and optimize the performance of the model.

[0093] For ease of description, the gradient-weighted class activation heat map (Grad-CAM) is referred to as Grad-CAM below. Commonly used deep learning networks (such as convolutional neural networks CNN) are generally considered to be black boxes with poor interpretability. Therefore, the embodiment of the present invention uses Grad-CAM to draw a heat map to explain the deep learning network. It should also be noted that although ordinary class activation heat map CAM can also draw a heat map, ordinary CAM needs to modify the network structure and retrain to draw a heat map, which not only destroys the original neural network structure, but also faces the problem of computational complexity caused by retraining. The Grad-CAM used in the embodiment of the present invention avoids the problems brought by ordinary CAM and is more universal, so there is no need to modify the network structure or retrain the neural network.

[0094] Specifically, for the target classification category y, Grad-CAM generates a corresponding weight for each position (i, j) on the input image It is used to indicate the importance of image position (i, j) to category y. It can be calculated based on the features after global average pooling and the gradient of the corresponding layer. The specific calculation formula is shown as follows:

[0095]

[0096] Among them, k represents the number of layers, y represents the category; A k Represents the features after global average pooling; Represents the feature map A after global average pooling k The weight coefficient related to the target classification category y; Represents feature map A k The partial derivative of the input image channel c at position (i, j). Finally, the Grad-CAM heat map is obtained by multiplying the weights and the original feature map and performing a ReLU operation as shown below:

[0097]

[0098] in, Represents the original feature, that is, the activation value of the feature map corresponding to the y-th category in the k-th layer at position (i, j).

[0099] The feature fusion module (Cross-Modality) is used to process the feature fusion between different modal images and generate the final output classification result. Its structure is as follows: Figure 4As shown, it includes a modality attention module (Modality Attention Block), a fully connected layer module, and an interpretability module; the output of the modality attention module (Modality Attention Block) is respectively connected to the fully connected layer module and the interpretability module.

[0100] The modality attention block is used to perform attention weighting processing on the multi-modal output of the view-speicific feature extraction module. The modality attention block can effectively process multi-modal input data (such as images or text, etc.) and capture the connection between them by learning the correlation of each modality, thereby improving the classification performance.

[0101] Furthermore, the Modality Attention Block can also adjust the importance of modal features based on the attention mechanism, so that the model can pay more attention to modal features with higher values. For example, higher weights are given to modal features with higher values, while lower weights are given to modal features with lower values.

[0102] In some examples, after the feature fusion module (Cross-Modality) performs attention weighting through the Modality Attention Block (Modality Attention Block), it directly passes through the fully connected layer module to obtain the final classification result and probability, that is, directly outputs the predicted probability value of breast cancer grading multi-classification.

[0103] In some examples, the feature fusion module (Cross-Modality) uses the interpretability module to learn prototypes from each modality and extract corresponding interpretable features after the modality attention module (Modality Attention Block) performs attention weighting. Specifically, the interpretability module learns representative cases, namely prototypes, from each modality, performs similarity comparisons during the reasoning process, and obtains interpretable features based on domain knowledge, thereby forming a better human-computer interaction method so that radiologists can more easily understand the source of evidence for the model to make classification judgments.

[0104] It should be understood that in deep learning, prototypes refer to clustering the input data into multiple clusters using a clustering algorithm, and then calculating the centroid (center point) of each cluster, that is, the prototype. These prototypes are generally considered to be typical points representing different categories in the data set and can be used for classification and other tasks.

[0105] In some examples, the specific calculation process and method of the interpretability module include: for a multimodal image dataset Convolutional Neural Network CNNf m (·), the extracted image features are The interpretability module generates a set of prototypes The similarity score for each prototype is calculated as follows:

[0106]

[0107] in, Represents the jth prototype; Represents the extracted feature graph. After obtaining the similarity score set, it is input into the fully connected layer to obtain the final classification result and probability.

[0108] In an embodiment of the present invention, after the classification results are output by the fully connected layer module and the interpretability module, a bottom-up graded diagnosis result from pathological evaluation to imaging evaluation is provided based on the inference algorithm decision model; wherein the pathological evaluation can be further divided into the biopsy level and the treatment level.

[0109] It is understandable that the decision-making model of the inference algorithm is a tree structure in the computer field. For example, breast disease is the root node, the information at the imaging level is the intermediate node, and the information at the pathology level is the leaf node, thus constructing a tree structure. Inference algorithms are divided into two categories: one is exact inference, and the other is approximate inference.

[0110] In this embodiment of the present invention, in order to alleviate the problem of data imbalance, a label smoothing cross entropy loss function is used, which is defined as follows:

[0111]

[0112] Among them, N represents the number of training samples, K represents the number of categories, and y i is the true label of the i-th sample, one-hot(y i ) means to convert y i Convert to one-hot vector, p i It is the probability distribution of the network prediction label, α represents the smoothing factor, which is used to control the degree of smoothing, and j represents the index of the category, with a total of K categories.

[0113] Preferably, adding a penalty term for label smoothing during training can make the model focus more on learning of minority categories, thereby improving the classification performance of the model. Exemplarily, in order to reduce model overfitting, L2 regularization is used, weight decay is 0.1, batch size is 32, and learning rate is 0.0001.

[0114] In order to verify the prediction effect of the neural network model in the embodiment of the present invention, the embodiment of the present invention uses three indicators to evaluate the model effect, namely, accuracy ACC (Accuracy), Matthews Correlation Coefficient MCC (Matthews Correlation Coefficient), and area under the receiver operating characteristic curve ROC (Receiver Operating Characteristic curve) AUC (Area Under Curve).

[0115] The ultrasound image module has been internally and externally validated in four centers on three instruments. At the pathological level, the MCC was 0.391 and the ACC was 0.538, which have exceeded those of radiologists (four doctors with more than 10 years of experience, with an average MCC of 0.257 and an average ACC of 0.410). At the imaging level, the AUC was 0.921, reaching the level of radiologists (under the three BI-RADS modes 4a+, 4b+, and 4c+, the results of ultrasound doctors were close to the ROC curve, and AI was no worse than that of ultrasound doctors).

[0116] The mammography image module has been internally and externally validated in two centers, two public data sets, and a total of three instruments. At the pathological level, the MCC is 0.266, and the ACC is 0.421 (for five doctors with more than 10 years of experience, the average MCC level is 0.137, and the average ACC level is 0.317, which has exceeded that of radiologists). At the imaging level, the AUC is 0.617, reaching the level of radiologists (under the three BI-RADS modes 4a+, 4b+, and 4c+, the mammography doctors' results are all below the ROC curve, indicating that AI is superior to the doctors' personal experience and judgment).

[0117] The combined module formed by the mixed use of ultrasound image module and molybdenum target image module has been internally and externally verified on 2 centers, 2 molybdenum targets and 2 ultrasound instruments. The AUC at the imaging level is 0.936 (it cannot be directly compared with doctors because few doctors can understand both ultrasound and molybdenum targets). Therefore, the comparison is mainly focused on biopsy pathology. The MCC of the mixed model is 0.425 and the ACC is 0.585, which is close to the quantified biopsy pathology level.

[0118] It should be noted that: the artificial intelligence-based breast cancer grading analysis device provided in the above embodiment only uses the division of the above program modules as an example when performing artificial intelligence-based breast cancer grading analysis. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the artificial intelligence-based breast cancer grading analysis device provided in the above embodiment and the artificial intelligence-based breast cancer grading analysis method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0119] like Figure 5 As shown, a schematic diagram of the process of a breast cancer grading analysis method based on artificial intelligence in an embodiment of the present invention is shown. The breast cancer grading analysis method in an embodiment of the present invention includes the following steps:

[0120] Step S51: using the ultrasound image module and the molybdenum target image module alone or in combination to form a first combined module, and training a neural network model based on their respective multi-view images to output predicted probability values ​​of breast cancer grading and multi-classification.

[0121] Step S52: Use the clinical information module in combination with either or both of the ultrasound image module and the molybdenum target image module to form a second combined module, and use the most recently updated neural network model weights in the ultrasound image module and the molybdenum target image module as the pre-trained model weights of the second combined module.

[0122] It should be noted that the implementation method of the breast cancer grading analysis method based on artificial intelligence in the embodiment of the present invention is similar to the implementation method of the breast cancer grading analysis system based on artificial intelligence mentioned above, so it will not be described in detail.

[0123] The artificial intelligence-based breast cancer grading analysis method provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the artificial intelligence-based breast cancer grading analysis terminal, please refer to Figure 6, is an optional hardware structure diagram of an artificial intelligence-based breast cancer grading analysis terminal 600 provided in an embodiment of the present invention. The terminal 600 can be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The artificial intelligence-based breast cancer grading analysis terminal 600 includes: at least one processor 601, a memory 602, at least one network interface 604 and a user interface 606. The various components in the device are coupled together through a bus system 605. It can be understood that the bus system 605 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 605 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 6 In the specification, various buses are labeled as bus systems.

[0124] The user interface 606 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0125] It is understood that the memory 602 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0126] The memory 602 in the embodiment of the present invention is used to store various categories of data to support the operation of the artificial intelligence-based breast cancer grading analysis terminal 600. Examples of these data include: any executable program used to operate on the artificial intelligence-based breast cancer grading analysis terminal 600, such as an operating system 6021 and an application 6022; the operating system 6021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 6022 can include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The artificial intelligence-based breast cancer grading analysis method provided in the embodiment of the present invention can be included in the application 6022.

[0127] The method disclosed in the above embodiment of the present invention can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 601 or the instruction in the form of software. The above processor 601 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 601 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 601 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0128] In an exemplary embodiment, the artificial intelligence-based breast cancer grading analysis terminal 600 can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), and complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0129] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0130] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0131] In summary, the present application provides a breast cancer grading analysis method, device, terminal and medium based on artificial intelligence, which has strong generalization ability. The technical solution of the present invention has been prospectively externally verified in 4 centers and more than 5 different machines. The AI ​​system can perform a more comprehensive and accurate analysis of breast cancer, and its performance exceeds that of molybdenum target and ultrasound images. Modularity and high clinical applicability: The present invention uses an AI system with three modules to train a dedicated breast cancer grading diagnosis model, and combines ultrasound images and molybdenum target images separately or in combination. It not only takes into account the factors of individual differences of patients, but also provides a variety of different diagnostic methods to choose from, improves diagnostic accuracy and reliability, and thus better guarantees accuracy and universality. High accuracy: The ultrasound or molybdenum target image module of the present invention alone surpasses radiologists, and combined use can achieve leapfrogging and reach the level of biopsy pathology. High interpretability: The present invention can provide diagnostic basis to doctors (two methods, post-hoc attention map or heatmap, prototypes type of domain knowledge), enhance doctors' diagnostic confidence, so as to better perform auxiliary diagnosis. Strong grading decision-making ability: The present invention can not only provide clinical BI-RADS grading requirements, but also provide judgment of pathological levels, guiding clinicians to choose surgical plans and treatment methods. Therefore, the present application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0132] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A breast cancer grading analysis system based on artificial intelligence, characterized in that: It includes ultrasound image module, mammography image module and clinical information module; among which: The ultrasound image module and the molybdenum target image module are used alone or in combination to form a first combined module, and both are respectively based on their own multi-view images to train neural network models to output predicted probability values ​​for breast cancer grading and multi-classification; the clinical information module is used in combination with either or both of the ultrasound image module and the molybdenum target image module to form a second combined module, and the most recently updated neural network model weights in the ultrasound image module and the molybdenum target image module are used as pre-trained model weights of the second combined module.

2. The artificial intelligence-based breast cancer grading analysis system according to claim 1, characterized in that: The structure of the neural network model used in the ultrasound image module and the molybdenum target image module includes a feature extraction module and a feature fusion module.

3. The artificial intelligence-based breast cancer grading analysis system according to claim 2, characterized in that: The feature extraction module includes: a first convolutional neural network, a squeeze and excitation module, a second convolutional neural network, a self-attention mechanism module, and a gradient-weighted class activation heat map module; wherein: The first convolutional neural network inputs a first-perspective image and a second-perspective image with shared weights; the output of the first convolutional neural network is connected to the input of the squeeze and excitation module; the output of the squeeze and excitation module is connected to the input of the second convolutional neural network; the output of the second convolutional neural network is connected to the input of the self-attention mechanism module; the gradient-weighted class activation heat map module draws and outputs the corresponding heat map according to the output result of the self-attention mechanism module.

4. The artificial intelligence-based breast cancer grading analysis system according to claim 3, characterized in that: The first convolutional neural network inputs two images of different perspectives under the same modality, which are a first perspective image and a second perspective image; the first perspective image and the second perspective image are respectively one and the other of a cranio-caudal breast mammography image and a medial-lateral oblique breast mammography image, or one and the other of a transverse breast ultrasound image and a longitudinal breast ultrasound image; and the first perspective image and the second perspective image share weights.

5. The artificial intelligence-based breast cancer grading analysis system according to claim 1, characterized in that: The self-attention mechanism module calculates the relative importance of each position in the input image with respect to all other positions, generates a set of weighted representations accordingly, and uses these weighted representations to perform weighted summation on the input features, introducing global information to capture the spatial relationships and contextual information between objects in the image.

6. The artificial intelligence-based breast cancer grading analysis system according to claim 1, characterized in that: The self-attention mechanism module uses gradient-weighted class activation heatmaps to draw heatmaps.

7. The artificial intelligence-based breast cancer grading analysis system according to claim 6, characterized in that: The self-attention mechanism module uses the gradient-weighted class activation heat map to draw the heat map in the following ways: For the target classification category y, generate the corresponding weight for each position (i, j) on the input image To indicate the importance of image position (i, j) to category y; weight The calculation methods include: Among them, k represents the number of layers, y represents the category; A k Represents the features after global average pooling; Represents the feature map A after global average pooling k The weight coefficient related to the target classification category y; Represents feature map A k The partial derivative of the input image channel c at position (i, j); The heat map is calculated by weighting Multiplying it with the original feature map and performing the ReLU function operation yields: Among them, Grad-CAM (c) (i, j) represents the heat map; represents the weight of the image position (i, j); Represents the activation value of the feature map corresponding to the y-th category in the k-th layer at position (i, j).

8. The artificial intelligence-based breast cancer grading analysis system according to claim 1, characterized in that: The feature fusion module includes a modality attention module, a fully connected layer module, and an interpretability module; the output of the modality attention module is connected to the fully connected layer module and the interpretability module respectively.

9. The artificial intelligence-based breast cancer grading analysis system according to claim 8, characterized in that: The modality attention module is used to perform attention weighting processing on the multi-modality output by the feature extraction module, and extract the correlation relationship between them by learning the correlation of each modality.

10. The artificial intelligence-based breast cancer grading analysis system according to claim 9, characterized in that: After the feature fusion module performs attention weighting through the modality attention module, the feature fusion module directly obtains the final classification result and probability through the fully connected layer module, that is, directly outputs the predicted probability value of breast cancer grading multi-classification; or, after the modality attention module performs attention weighting, the interpretability module is used to learn the prototype from each modality and extract the corresponding interpretable features; The interpretability module learns representative prototypes from each modality and performs similarity comparison during reasoning to obtain interpretable features based on domain knowledge.

11. A breast cancer grading analysis method based on artificial intelligence, characterized in that: include: The ultrasound image module and the molybdenum target image module are used alone or in combination to form a first combined module, and the two are respectively used to train a neural network model based on their respective multi-view images to output a predicted probability value of breast cancer grading and multi-classification; or, The clinical information module is used in combination with either or both of the ultrasound image module and the molybdenum target image module to form a second combined module, and the most recently updated neural network model weights in the ultrasound image module and the molybdenum target image module are used as the pre-trained model weights of the second combined module.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the artificial intelligence-based breast cancer grading analysis method according to claim 12 is implemented.

13. A computer device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the computer device executes the artificial intelligence-based breast cancer grading analysis method as claimed in claim 12.

Images