System for detecting whether breast cancer is benign or malignant based on multiplle modalities

Abstract

Disclosed is a system for detecting whether the breast cancer is benign or malignant based on multiple modalities, which comprises: an ultrasonomics model configured to extract breast tumor position information and breast tumor morphological features on the basis of an acquired grayscale breast ultrasound image, perform global interactive fusion on the breast tumor position information and breast tumor morphological feature information, and convert a Transformer encoder-based feature output into an output vector of the corresponding classification dimensions by means of a fully connected layer; a metabonomics model configured to obtain a metabolic fingerprint spectrogram and analyze the metabolic fingerprint spectrogram on the basis of a metabolic database and a predetermined significantly differential metabolite to obtain abnormal metabolic pathways, signal transduction pathways, and related biochemical reactions associated with breast cancer malignancy; and a detection module configured to perform joint modeling, based on the breast tumor position information, the breast tumor morphological features, and a peak value of the significantly differential metabolite of the breast cancer malignancy, to output a prediction result about whether the breast cancer is benign or malignant.

Description

A breast cancer benignity and malignancy detection system based on multi-modalities TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical artificial intelligence, and particularly relates to a breast cancer benignity and malignancy detection system based on multi-modalities. BACKGROUND

[0002] Breast puncture biopsy is a traditional method for detecting breast cancer benignity and malignancy in clinical practice. However, breast puncture biopsy can cause complications such as upper limb activity limitation and sensory disturbance.

[0003] Ultrasound examination is widely used in breast cancer screening and preoperative evaluation due to its convenience and non-radiation. However, the efficiency of conventional ultrasound examination in diagnosing breast cancer benignity and malignancy is low, with an AUC value of only 0.585-0.719, which cannot meet the clinical diagnostic requirements.

[0004] The occurrence and progression of tumors are closely related to changes in metabolites. During the malignant transformation of breast cancer, the metabolism of tumor cells undergoes a series of profound reorganization, which plays a key role in disease progression and exhibits dependence on specific metabolic pathways. Metabolomics has the technical characteristics of high sensitivity, high resolution, and high throughput, as well as the advantages of non-invasiveness, convenience, and others, and is widely used in tumor marker screening, disease diagnosis, new drug research and development, and other fields. TECHNICAL PROBLEM

[0005] Currently, there is no research on breast cancer benignity and malignancy based on non-invasive ultrasound images and metabolomics in the field. TECHNICAL SOLUTION

[0006] To solve the above problems, the present application provides a breast cancer benignity and malignancy detection system based on multi-modalities, which integrates metabolomics and ultrasoundomics organically, and constructs an artificial intelligence diagnostic tool with high diagnostic efficiency, precision, and minimally invasive through metabolomics analysis, significant difference molecule modeling, and image group nodule position information and morphological information representation. Furthermore, the internal relationship between ultrasound and metabolomics is explored, providing new strategies and theoretical basis for precise diagnosis of breast cancer benignity and malignancy.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows: a breast cancer benignity and malignancy detection system based on multi-modalities, comprising:

[0008] The ultrasonic omics model extracts breast tumor position information and breast tumor morphological feature information based on the obtained breast grayscale ultrasound images of breast cancer patients, and converts the feature output based on the Transformer encoder into an output vector with a corresponding classification number dimension through a full connection layer.

[0009] The metabolomics model obtains the metabolic fingerprint spectrum of the breast cancer patient, and analyzes the metabolic fingerprint spectrum based on the metabolic database and the pre-determined significantly different metabolic molecules to obtain the abnormal metabolic pathways, signaling pathways and related biochemical reactions of breast cancer malignancy.

[0010] The detection module jointly models the peak value of the extracted breast tumor position information, breast tumor morphological features and breast cancer malignant significant difference molecules of the current breast cancer patient to output the prediction result of the current breast cancer patient.

[0011] Preferably, the ultrasonic omics model based on the obtained breast grayscale ultrasound images of breast cancer patients extracts breast tumor position information and breast tumor morphological feature extraction specifically includes:

[0012] For the ultrasound image, the cutting tool is used to cut off the irrelevant text information in the image, and the obtained ultrasound image data is standardized and normalized to reduce the interference factors of high gray scale image spots, blurred boundary, low signal-to-noise ratio, low contrast and uneven intensity.

[0013] The breast tumor position information is extracted, and first, the foreground feature is enhanced by foreground feature optimization, attention mechanism algorithm and context semantic association module:

[0014] For the i-th feature map, relative to the input image has The output stride of the pixel is obtained by transverse connection of the top and bottom paths to obtain the feature map set , The transverse connection realized by the learnable convolution layer, Indicates the nearest neighbor up-sampling of 2 times the scale factor.

[0015] ,

[0016] ,

[0017] For by scale-aware projection function transformed feature map, for learnable parameters.

[0018] scene embedding vector is represented as follows:

[0019] ,

[0020] is a projection function, the output space is , variable is a point-wise inner product similarity estimate:

[0021] ,

[0022] ,

[0023] The re-encoder has learnable parameters , The term included is used to weight the re-encoded feature map, and finally obtain the foreground feature map enhanced by foreground-background relationship :

[0024] .

[0025] Preferably, breast tumor position information extraction is performed, and further a homogeneous positioning method is used to realize standardized registration and automatic quantitative statistics of tumor position information of the whole population, a differential homeomorphism method is used to realize flexible registration under a unified template, and a Lagrange equation is introduced for optimization:

[0026] Suppose there are a picture to be registered and a target registration image , and the registration calculation formula is:

[0027] ,

[0028] Considering that the ultrasound tumor features have relatively low contrast compared to the whole picture, the window sliding multi-head self-attention module W-MSA and the shifted multi-head self-attention module SW-MSA are introduced into the traditional natural language model architecture to realize efficient detection of position information:

[0029] ,

[0030] and respectively represent the first The output features of the multi-head self-attention module W-MSA or the shifted multi-head self-attention module SW-MSA in the module and the multi-layer perception layer module.

[0031] Preferably, the morphological features of the breast tumor are extracted, including:

[0032] Extracting internal texture features: using a local information enhancement algorithm to realize extraction of fine-grained features including texture, calcification and uniformity, and a probability value For:

[0033]

[0034] is a parameter matrix, The probability distribution is normalized, and the loss function is minimized by optimizing the matrix and the convolution kernel parameters, so as to obtain the optimal solution of local feature recognition, and the loss function is defined as follows:

[0035] ,

[0036] ,

[0037] is the number of input images, is an indicative function;

[0038] Extracting echogenic features: using gray scale contrast, histogram equalization and other graphic processing methods to extract the imageological echo difference features inside and outside the lesion;

[0039] Extracting tumor size: the tumor size is calculated according to the length and width data of the segmented tumor.

[0040] Preferably, the ultrasound images are cut and preprocessed: using a cutting tool to cut off irrelevant text and other information in the picture; in order to expand the sample size, geometric methods and singular value decomposition methods are used for data augmentation.

[0041] Preferably, the ultrasound group model further comprises a fusion module for globally interacting and fusing the breast tumor position information and the breast tumor morphological feature information, and the input breast tumor position information and the breast tumor morphological feature information are combined into a feature matrix , which is divided into a fixed-size block sequence , the image blocks are mapped into embedding vectors based on linear projection , a classification label is introduced, and combined into a standard input form of the Transformer, and the position embedding ​Information, which transforms image patches into ordered one-dimensional embeddings and introduces self-attention mechanism:

[0042] ,

[0043] ,

[0044] , , denote query, key and value matrices respectively, denote input sequence vector length, multi-head attention mechanism performs self-attention computation on input sequence, concatenates multiple outputs to get projection, and computes final output vector;

[0045] Multi-head attention computation process is:

[0046] ,

[0047] where:

[0048] ,

[0049] ,

[0050] Finally, the feature output of Transformer encoder is converted into classification vector corresponding to conversion site:

[0051] ,

[0052] When response vector is whether malignant breast cancer occurs, functional independent variable is position information, which is a square integrable function on bounded closed interval , dimensional scalar independent variable is morphological feature, is a dimensional sub-vector of , let be the conditional mean of , and the generalized linear model is described by the following formula:

[0053]

[0054] and are known functions, and the slope function in position information depends on​ the value of the position information and the morphological feature There is an interaction effect, which is expressed as follows:

[0055] ,

[0056] wherein is an unknown second-order derivative continuous function, is a single-index weight vector, and are unknown coefficient functions, respectively representing the main effect of the position information and the interaction effect of the two kinds of information; the morphological feature is independently extracted by a morphological extraction module; the position information main effect is independently extracted by a position extraction module; and the interaction effect information of the two is independently extracted by a fusion module.

[0057] Preferably, the metabolic molecules that determine the significant differences between the benign and malignant breast cancer patient groups specifically include: through data analysis software, the serum metabolomics spectrum peaks of breast cancer patients are studied to find characteristic peaks with significant differences, and through the differences in their metabolomics, significant difference metabolic molecules are found.

[0058] Preferably, the metabolomics model analyzes the metabolic fingerprint based on a metabolic database and pre-determined significant difference metabolic molecules to obtain abnormal metabolic pathways of malignant breast cancer, specifically including: based on the significant difference metabolic molecules, corresponding heat maps and volcano plots are drawn for the metabolic fingerprint, up-regulation factors and down-regulation factors are determined, and then through metabolic database comparison, abnormal metabolic pathways, signal transduction pathways and related biochemical reactions of malignant breast cancer are determined; the significant difference molecular peak value is input into a fully connected neural network for feature fusion to obtain a metabolomics representation.

[0059] Preferably, the ultrasound image group characteristics and the metabolomics characteristics are assigned weights according to an attention mechanism, and after weighting, are input into a fully connected network for classification prediction of benign and malignant breast cancer, and the weights are automatically adjusted during the training process;

[0060] The detection module also analyzes the attention weights of the screened ultrasound imageomics characteristics and metabolomics characteristics to determine the contribution degree of the ultrasound image group and the metabolomics group in the individual breast cancer benign and malignant prediction results.

[0061] Preferably, the metabolic fingerprint of the breast cancer patient is obtained by peripheral blood collection, using a BD 6ml coagulation tube to collect 1 tube of fasting venous blood; after blood sample collection, gently invert and mix immediately, and pay attention to avoid hemolysis; the collected blood is left at room temperature for about 30 minutes, and all blood sample separation is completed within 4 hours; centrifuge at 3000-3500 rpm for 10-15 minutes on a horizontal centrifuge; use a pipette to sequentially aspirate the upper serum, and dispense into pre-labeled nuclease-free cryotubes, 1000ul per tube, and store in a -80°C freezer to avoid repeated freezing and thawing;

[0062] For metabolomics testing: 0.5 ul of serum sample is placed on a polished steel target plate MTP 384, and after air drying, 1 ul of three-dimensional nanomaterial mass spectrometry matrix is added for ionization; an autoflex max time-of-flight tandem mass spectrometer instrument is used for sample detection, a smartbeam-II laser is used in a reflection mode at 355nm with a laser frequency of 1000Hz to collect mass spectrometry spectra of 100-5000Da; each sample is randomly measured 25 times at 20 different grating points. Advantages

[0063] The present application has the following advantages and positive effects compared with the prior art due to the use of the above technical solutions:

[0064] The technical solution of the present embodiment extracts the position information of the breast tumor and the morphological characteristics of the breast tumor from the ultrasound image of the patient through the ultrasoundomics model, obtains the metabolic fingerprint of the patient through the metabolomics model, and then analyzes and detects the extracted information through the detection module. Since the present technical solution extracts the final decisive information and histological characteristics of the breast cancer, which includes the internal and external environment, the position, morphology and metabolism of the tumor are jointly analyzed and detected, the final decisive information and histological characteristics of the breast cancer, which includes the internal and external environment, are extracted, and the macro-micro, tumor local-global, biochemical-physical index three aspects are organically complementary. Through metabolic group analysis, significant difference molecular modeling and image group nodule position information and morphological information representation, a high-diagnostic-performance, precise and minimally-invasive artificial intelligence diagnostic tool is constructed, the internal relationship between ultrasound and metabolomics is revealed, a new strategy and theoretical basis for precise diagnosis of breast cancer is provided, and the development of ultrasound imaging and metabolomics technology from diagnosis to individualized diagnosis and treatment is promoted. BRIEF DESCRIPTION OF DRAWINGS

[0065] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings, in which:

[0066] Fig. 1 is a flow chart of the construction of the breast cancer benign and malignant detection system based on the multimodal of the present application;

[0067] FIG. 2 is an example of a benign breast cancer gray-scale ultrasound image;

[0068] FIG. 3 is an example of a malignant breast cancer gray-scale ultrasound image. Embodiments of the present application

[0069] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present application will be more apparent from the following description and claims. It should be noted that the accompanying drawings are very simplified and all use non-precise ratios, only for the purpose of facilitating and clarifying the purpose of explaining the embodiments of the present application.

[0070] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between the components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly.

[0071] In recent years, scholars at home and abroad have analyzed the metabolite profile characteristics of breast cancer patients through metabolomics technology, screened out valuable potential markers, and opened up new ways for early diagnosis and individualized treatment of breast cancer. Embodiments

[0072] The present embodiment provides a breast cancer benignity and malignancy detection system based on multi-modal, comprising:

[0073] An ultrasoundomics model extracts breast tumor position information and breast tumor morphological features based on the obtained breast gray-scale ultrasound images of breast cancer patients, globally interacts and fuses the breast tumor position information and the breast tumor morphological feature information, and converts the feature output based on the Transformer encoder into an output vector of corresponding classification dimension through a fully connected layer; the breast tumor position information is independently extracted by a position extraction module in the ultrasoundomics model, and the breast tumor morphological features are independently extracted by a morphological extraction module in the ultrasoundomics model;

[0074] A metabolomics model obtains a metabolic fingerprint spectrum of a breast cancer patient, analyzes the metabolic fingerprint spectrum based on a metabolic database and a pre-determined significantly different metabolic molecule to obtain abnormal metabolic pathways, signal transduction pathways and related biochemical reactions of breast cancer malignancy;

[0075] A detection module jointly models and outputs a prediction result of the benignity and malignancy of the current breast cancer patient based on the extracted breast tumor position information, breast tumor morphological features and peak values of breast cancer malignancy significantly different molecules of the current breast cancer patient.

[0076] Fig. 1 shows a flow chart of a multi-modal based breast cancer benign and malignant detection system, the embodiment extracts the breast tumor location information and breast tumor morphological features of the patient's ultrasound image through the ultrasound omics model, obtains the patient's metabolic fingerprint through the metabolomics model, and then analyzes and detects the above extracted information through the detection module. Since the technical solution jointly analyzes and detects the position, shape and metabolic information of the tumor, it extracts the final decisive information and histological features that determine the benign and malignant of breast cancer, including the internal and external environment, and organically complements from the macro-micro, tumor local-global, biochemical-physical index three aspects. Through metabolic group analysis, significant difference molecule modeling and image group node position information and morphological information representation, a high-diagnostic-performance, precise and minimally invasive artificial intelligence diagnostic tool is constructed, the internal relationship between ultrasound and metabolomics is revealed, and new strategies and theoretical basis are provided for the precise diagnosis of benign and malignant breast cancer. It is helpful to promote the development of ultrasound imaging and metabolomics technology from diagnosis to individualized diagnosis and treatment.

[0077] Fig. 2 is an example of a benign breast cancer gray-scale ultrasound, and Fig. 3 is an example of a malignant breast cancer gray-scale ultrasound.

[0078] Further, the ultrasound omics model based on the obtained breast gray-scale ultrasound image of the breast cancer patient performs breast tumor location information extraction and breast tumor morphological feature extraction, which specifically includes:

[0079] For the ultrasound image, the irrelevant text information in the image is cut off using the cutting tool, and the obtained ultrasound image data is standardized and normalized to reduce the interference factors of high gray-scale image spots, blurred boundaries, low signal-to-noise ratio, low contrast and uneven intensity;

[0080] The breast tumor location information is extracted, and first, the foreground features are enhanced through foreground feature optimization, attention mechanism algorithm and context semantic association module:

[0081]

[0082] For the i-th feature map, relative to the input image has The output stride of the pixel is obtained through the horizontal connection of the top and bottom paths to obtain the feature map set , The horizontal connection is realized by a learnable convolutional layer, Indicates the nearest neighbor up-sampling of 2 times the scale factor;

[0083] ,

[0084] ,

[0085] For by scale-aware projection function converted feature map, For learnable parameters;

[0086] scene embedding vector is represented as follows:

[0087] is a projection function, and the output space is , and the variable is a point-wise inner product similarity estimate:

[0088] ,

[0089] ,

[0090] The re-encoder has learnable parameters , The term included is used to weight the re-encoded feature map, and finally obtain the foreground feature map enhanced by foreground-background relationship :

[0091] .

[0092] The technical scheme of the embodiment combines the class imbalance problem of the tumor region (foreground) and the overall (background) region of the breast tumor ultrasound, designs foreground feature optimization, attention mechanism algorithm, and context semantic correlation module to enhance the target of the foreground feature, and can more accurately and effectively identify the tumor region.

[0093] Specifically, for the acquisition of ultrasound images: a 10-15MHz linear probe is used, and the imaging parameters such as depth and gain are optimized according to the image condition of each patient; the patient is supine on the examination bed, and the breast, axillary fossa and supraclavicular fossa are fully exposed; radial scanning is performed with the nipple as the center to determine the position of the lesion; after the patient receives full breast ultrasound examination, the entire scanning process is stored in the ultrasound diagnostic instrument. For ultrasound image cutting and preprocessing: use the cutting tool to cut off irrelevant text information in the picture; in order to expand the sample size, geometric method and singular value decomposition method are used for data augmentation.

[0094] Furthermore, breast tumor position information extraction is performed, and further homogeneous positioning method is used to realize standardized registration and automatic quantitative statistics of tumor position information of the whole population, to realize flexible registration under a unified template using the differential homeomorphism method, and to introduce Lagrange equation for optimization:

[0095] There are pictures to be registered and target registration image , the registration calculation formula is:

[0096] ,

[0097] Considering that ultrasound tumor features have low contrast compared to the entire image, we introduced the window sliding multi-head self-attention module (W-MSA) and the shifted multi-head self-attention module (SW-MSA) into the traditional natural language model architecture to achieve efficient detection of location information:

[0098]

[0099] and Respectively expressed in The output features of the multi-head self-attention module W-MSA or the shifted multi-head self-attention module SW-MSA in the modules and the multi-layer perception layer module.

[0100] Ultrasound features of breast tumors vary in scale and resolution, creating challenges for large-scale statistical analysis of location information. This embodiment designs a homogeneous localization method to achieve standardized registration and automated quantitative statistics of tumor location information across the entire population.

[0101] Furthermore, the extraction of breast tumor morphological features includes:

[0102] Extract internal texture features: Use local information enhancement algorithm to extract fine-grained features including texture, calcification, and uniformity, and probability value for:

[0103] ,

[0104] is the parameter matrix, Normalize the probability distribution and optimize Matrix and convolution kernel parameters make the loss function Minimize, thus obtaining the optimal solution for local feature recognition. The loss function is defined as follows:

[0105]

[0106] ,

[0107] is the number of input images, is an indicative function;

[0108] Extraction of acoustic features: using gray scale contrast, histogram equalization and other graphics processing methods to extract the echo difference characteristics of the image inside and outside the lesion;

[0109] Extraction of tumor size: according to the length and width data of the segmented tumor.

[0110] Combined with literature knowledge and Chinese breast imaging reporting and data system (BI-RADS), the breast tumor features are effectively extracted, including: internal information of the tumor (cystic and solid, calcification, echo, uniformity), edge information (edge sharpness, capsule integrity, shadow degree), tumor size (aspect ratio, diameter) and the like. This embodiment gives a specific means for extracting the morphological features of breast tumors.

[0111] Preferably, the ultrasound omics model further comprises a fusion module for globally interacting and fusing breast tumor location information and breast tumor morphological feature information, and the input breast tumor location information and breast tumor morphological feature information are combined into a feature matrix , divided into a sequence of fixed-size blocks , based on linear projection The image block is mapped into an embedding vector, a classification label is introduced , combined into the standard input form of Transformer, position embedding information, the image block is converted into an ordered one-dimensional embedding, and a self-attention mechanism is introduced:

[0112] ,

[0113] ,

[0114] , , , query, key and value three matrices respectively, represents the input sequence , the vector length in the input sequence , the multi-head attention mechanism performs times of self-attention calculation, and a plurality of outputs are spliced to obtain projection, and the final output vector is calculated.

[0115] The multi-head attention calculation process is:

[0116] ,

[0117] Among them:

[0118] ,

[0119] ,

[0120] The final feature output of the Transformer encoder The classification vector corresponding to the transformation position:

[0121] ,

[0122] When the response vector is whether the breast cancer is malignant, the functional independent variable is the position information, and the bounded closed interval is a square integrable function, the dimension scalar independent variable is the morphological feature, is a one -dimensional sub-vector of , let be the conditional mean of , and the generalized linear model is described by the following formula:

[0123] ,

[0124] and are known functions, and the slope function in the position information depends on the value of , which allows the position information to interact with the morphological feature , which is represented as follows:

[0125] ,

[0126] where is an unknown second-order derivative continuous function, is a single index weight vector, and are unknown coefficient functions, representing the main effect of the position information and the interaction effect of the two types of information, respectively; the morphological feature is independently extracted by the morphological extraction module; the position information main effect is independently extracted by the position extraction module; and the interaction effect information of the two is independently extracted by the fusion module.

[0127] The embodiment provides a specific implementation mode for globally fusing breast tumor ultrasound features by using a natural language model architecture to realize joint prediction of benign and malignant breast cancer, globally fusing multi-modal features, and thus enabling joint prediction.

[0128] Preferably, determining the metabolic molecules with significant differences between benign and malignant breast cancer patient groups specifically includes: studying the serum metabolomics spectrum peaks of breast cancer patients through data analysis software, finding characteristic peaks with significant differences, and finding the metabolic molecules with significant differences among them through the differences in their metabolomics.

[0129] More specifically, the metabolomics model analyzes the metabolic fingerprint spectrum based on the metabolic database and pre-determined significantly different metabolic molecules to obtain abnormal metabolic pathways of malignant breast cancer, specifically including: drawing corresponding heat maps and volcano maps based on the significantly different metabolic molecules for the metabolic fingerprint spectrum, determining up-regulation factors and down-regulation factors, and then determining the abnormal metabolic pathways, signal transduction pathways and related biochemical reactions of malignant breast cancer through comparison with the metabolic database; inputting the significantly different molecule peaks into a fully connected neural network for feature fusion to obtain a metabolome representation.

[0130] Preferably, the ultrasound imaging group features and the metabolomics group features are weighted according to the attention mechanism, and the weighted features are input into the fully connected network for classification prediction of benign and malignant breast cancer, and the weights are automatically adjusted during the training process;

[0131] The detection module also analyzes the attention weights of the screened ultrasound imaging features and metabolomics features to determine the contribution of the ultrasound imaging group and the metabolomics group to the prediction results of individual breast cancer benign or malignant.

[0132] By continuously and automatically adjusting the weights during the training process, the final model prediction accuracy is made higher.

[0133] More specifically, the metabolic fingerprint of breast cancer patients is obtained by peripheral blood sampling. One tube of fasting venous blood is collected using a BD 6ml coagulant tube. The blood sample is immediately mixed by gentle inversion, and care is taken to avoid hemolysis. The collected blood is allowed to stand at room temperature for approximately 30 minutes, and all blood samples are separated within 4 hours. The blood is centrifuged at 3000-3500 rpm for 10-15 minutes. The upper serum layer is pipetted sequentially and dispensed into pre-labeled nuclease-free cryopreservation tubes, 1000µl per tube, and stored in a -80°C freezer to avoid repeated freezing and thawing.

[0134] For metabolomics analysis, 0.5 µl of serum sample was placed on a polished steel target plate (MTP 384). After air drying, 1 µl of a 3D nanomaterial mass spectrometry matrix was added for ionization. Samples were analyzed using an Autoflex Max time-of-flight tandem mass spectrometer. Mass spectra were collected from 100 to 5000 Da using a SmartBeam-II laser operating at 355 nm and a laser frequency of 1000 Hz in reflectance mode. Each sample was randomly measured 25 times at 20 different raster points.

[0135] The technical scheme of the present application is to analyze the correlation between the expression times of the screened ultrasonic imageomics features and the metabolomics features, which can also effectively evaluate the correlation degree of the features, explore the internal correlation between the ultrasonic heterogeneous image features and the metabolic small molecules / paths, and systematically study the molecular biology basis of the ultrasonic heterogeneous image features, so as to clarify the microscopic biological mechanism of the imageomics heterogeneous image features. The internal relationship between ultrasonic-metabolomics is revealed, which provides a new strategy and theoretical basis for the accurate diagnosis of breast cancer, and helps to promote the development of ultrasonic image and metabolomics technology from diagnosis to individualized diagnosis and treatment.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific implementation of the system and device described above can refer to the corresponding process in the foregoing method embodiments.

[0137] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above-described embodiments. Even if various changes are made to the present application, as long as the changes belong to the scope of the claims of the present application and equivalent technologies, they still fall within the protection scope of the present application.

Claims

1. A multi-modal based breast cancer benign-malignant detection system, characterized in that, Comprise: An ultrasound omics model based on the acquired breast gray-scale ultrasound images of breast cancer patients to extract breast tumor location information and breast tumor morphological feature information, and to fuse the breast tumor location information and the breast tumor morphological feature information globally, and to convert the feature output based on the Transformer encoder into an output vector of corresponding classification dimension through a fully connected layer; the breast tumor location information is independently extracted by a location extraction module in the ultrasound omics model, and the breast tumor morphological feature is independently extracted by a morphological extraction module in the ultrasound omics model; A metabolomics model acquires the metabolic fingerprint spectrum of the breast cancer patient, and analyzes the metabolic fingerprint spectrum based on the metabolic database and the pre-determined significantly different metabolic molecules to obtain the abnormal metabolic pathways, signaling pathways and related biochemical reactions of breast cancer malignancy; A detection module based on the extracted breast tumor location information, breast tumor morphological feature and peak value of breast cancer malignant significant difference molecules of the current breast cancer patient to jointly model and output the benign and malignant prediction results of the current breast cancer patient.

2. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 1, wherein, The ultrasound omics model based on the acquired breast gray-scale ultrasound images of breast cancer patients to extract breast tumor location information and breast tumor morphological feature information specifically comprises: For the ultrasound image, use the cutting tool to cut off the irrelevant text information in the image, and standardize and normalize the acquired ultrasound image data to reduce the interference factors of high gray-scale image spots, fuzzy boundary, low signal-to-noise ratio, low contrast and uneven intensity; Extracting breast tumor location information, first enhancing the target features by foreground feature optimization, attention mechanism algorithm and context semantic association module: ， For the i-th feature map, has a relative position to the input image Output stride of the pixels, which gives the feature map collection through horizontal concatenation of the top and bottom paths ， for the lateral connections implemented for the learnable convolutional layers, Indicates the nearest neighbor up-sampling of 2 times the scale factor; ， ， For By scale-aware projection function the converted feature map, For The learnable parameters of the convolutional layer are denoted as: Scene embedding vector Indicated as follows: ， For the projection function, the output space is , variable It is a point-by-point inner product similarity estimate: ， ， The re-encoder has learnable parameters ， comprising Item for weighted re-encoding of feature maps, resulting in foreground feature maps with enhanced foreground-background relationship ： 。 3. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 2, wherein, Extracting breast tumor location information, further using the homogeneous positioning method to realize the standardization registration and automatic quantitative statistics of the tumor location information of the whole population, using the differential homeomorphism method to realize the flexible registration under the unified template, and introducing the Lagrange equation for optimization: Picture to be registered and target registration image The registration calculation formula is: ， Considering that the contrast of the ultrasound tumor features is relatively low compared to the whole picture, the window sliding multi-head self-attention module W-MSA and the shifted multi-head self-attention module SW-MSA are introduced into the traditional natural language model architecture to realize efficient detection of location information: ， and respectively, in the first The output features of the multi-head self-attention module W-MSA or the shifted multi-head self-attention module SW-MSA in the module and the multi-layer perceptron layer module.

4. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 2, wherein, Extracting breast tumor morphological features, including: Extracting internal texture features: Using local information enhancement algorithm to realize the extraction of fine granularity features including texture, calcification, uniformity, and probability value As follows: ， is a parameter matrix, The probability distribution normalization processing is performed by optimizing Matrix and convolution kernel parameters make the loss function The minimum, so as to obtain the optimal solution of local feature recognition, and the loss function is defined as follows: ， ， for the input image number, It is an indicative function; Extracting echogenic features: using gray-scale contrast, histogram equalization and other graphics processing methods to extract the echogenic feature difference inside and outside the lesion; Extracting tumor size: calculating the tumor size according to the length and width data of the segmented tumor.

5. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 2, wherein, Cutting and preprocessing of ultrasound images: using the cutting tool to cut off the irrelevant information such as text in the picture; in order to expand the sample size, the geometric method and singular value decomposition method are used for data augmentation.

6. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 1, wherein, The ultrasonic omics model further comprises a fusion module for globally interacting and fusing breast tumor position information and breast tumor morphological feature information, input breast tumor position information and breast tumor morphological feature information are composed into a feature matrix , divided into a fixed-size block sequence , image blocks are mapped into embedding vectors based on linear projection , a classification label is introduced , combined into a standard input form of a Transformer, position embedding information, image blocks are converted into ordered one-dimensional embeddings, and a self-attention mechanism is introduced: ， ， 、 、 Let Q, K and V denote the query, key and value matrices, respectively, represents an input sequence The length of the vector, the multi-head attention mechanism performs Second self-attention calculation, multiple outputs are spliced ​​together to obtain Projection, calculate the final output vector; The multi-head attention calculation process is: ， Wherein: ， ， Finally, the feature output of Transformer encoder The classification vector corresponding to the conversion position: ， When the response vector For whether or not malignant breast cancer occurs, functional independent variables For position information, a bounded closed interval a square-integrable function on X, dimensional scalar argument for morphological features, is one of Let V be a vector subspace of Rn, let is The conditional mean of is described by the following formula: ， and is a known function, the slope function in the position information Dependent on of the value of the position information With morphological features There is an interaction effect, which is expressed as follows: ， wherein is an unknown second derivative continuous function, is the single-index weight vector, and is the unknown coefficient function, representing the position information main effects of the two factors and the interaction effect of the two factors; morphological characteristics The morphological extraction module is independently extracted; the position information main effect is independently extracted by the position extraction module; and the interaction effect information of the two is independently extracted by the fusion module.

7. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 1, wherein, The metabolic molecules significantly different between the benign and malignant breast cancer patient groups specifically include: through data analysis software, the serum metabolomics spectrum of breast cancer patients is studied to find characteristic peaks with significant differences, and through the difference in metabolomics, significant different metabolic molecules are found.

8. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 7, wherein, The metabolomics model analyzes the metabolic fingerprint based on a metabolic database and pre-determined significant different metabolic molecules to obtain abnormal metabolic pathways of breast cancer malignancy, specifically including: based on the significant different metabolic molecules, a corresponding heat map and volcano plot are drawn for the metabolic fingerprint, up-regulated factors and down-regulated factors are determined, and then through metabolic database comparison, abnormal metabolic pathways, signaling pathways and related biochemical reactions of breast cancer malignancy are determined; the significant different molecular peak values are input into a fully connected neural network for feature fusion to obtain a metabolomics representation.

9. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 1, wherein, The ultrasound image group features and the metabolomics features are weighted according to the attention mechanism, and the weighted features are input into a fully connected network for classification and prediction of benign and malignant breast cancer, and the weights are automatically adjusted during the training process. The detection module further analyzes the attention weights of the screened ultrasound imageomics features and metabolomics features to determine the contribution of the ultrasound image group and the metabolomics group in the individual breast cancer prediction result.

10. The multi-modal based breast cancer benign or malignant detection system as claimed in claim 1, wherein, The metabolic fingerprint of the breast cancer patient is obtained by peripheral blood sampling, and 1 tube of fasting venous blood is collected using a BD 6ml coagulation tube; after blood sampling, gently invert and mix immediately, and pay attention to avoid hemolysis; the collected blood is placed at room temperature for about 30 minutes, and all blood sample separation is completed within 4 hours; centrifuge at 3000-3500 rpm for 10-15 minutes on a horizontal centrifuge; use a pipette to sequentially aspirate the upper serum, and divide it into pre-labeled nuclease-free cryotubes, 1000 µl per tube, and store them in a -80°C freezer to avoid repeated freezing and thawing; For metabolomics testing: 0.5 µl of serum sample is placed on a polished steel target plate MTP 384, and after air drying, 1 µl of three-dimensional nanomaterial mass spectrometry matrix is added for ionization; an autoflex max time-of-flight tandem mass spectrometer instrument is used for sample detection, a smartbeam-II laser is used in a reflection mode at 355 nm with a laser frequency of 1000 Hz to collect mass spectrometry spectra of 100-5000 Da; each sample is randomly measured 25 times at 20 different grating points.

Images