A breast mass detection system based on multi-view images

Abstract

The application discloses a breast lump detection system based on multi-view images, comprising: a data import module for loading multi-view images of a breast and pre-processing data; a feature extraction module for extracting image features of a main view and an auxiliary view; a region extraction module for obtaining a candidate region of a lump in the main view and ROI features, a Patch position and Patch features in the auxiliary view; an information fusion module for fusing the ROI and the Patch by weighted fusion to obtain enhanced ROI visual features; and a lump detection module for classifying and position regression according to the enhanced ROI visual features to obtain a lump detection result of the main view. The application simulates the thinking of a doctor analyzing multi-view images, constructs a lump detection system based on multi-view images, effectively fuses the information of multiple views, and improves the detection rate of breast lumps and reduces the false positive rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of breast cancer diagnosis and target detection, and particularly to a breast mass detection system based on multi-view images. BACKGROUND

[0002] Regular breast screening is the key to early detection of breast cancer and reduction of mortality. Among them, mammography is the main screening method for breast cancer, and mass is the main basis for diagnosing breast cancer. However, due to the different experiences of doctors, and the visual fatigue caused by long-time review of mammography images, the detection rate of mass in the clinic is not ideal, and mass detection has a very urgent need for computer-aided diagnosis technology. Through artificial intelligence to automatically detect mass, it can assist doctors in diagnosis, help relieve the pressure of doctors, and improve the detection rate of mass.

[0003] Thanks to the continuous improvement of computing performance, deep learning technology has developed rapidly and has been applied to a variety of practical scenarios. Using deep learning methods for mass detection on a single mammography image has made some progress, but the precision is too low. Clinically, mammography will image a single breast from multiple views, including CC view and MLO view, and doctors need to analyze the two views when diagnosing. The mass detection system based on a single mammography image ignores the associated information between the two views, resulting in less than ideal detection results and making it difficult to apply to practical scenarios. A few systems fuse the detection results of the two views, but still rely on the detection performance of a single view, and there is a problem of missing mass detection in a single view. How to simulate the doctor's diagnosis thinking and combine multiple view images for mass detection still has great challenges. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a breast mass detection system based on multi-view images, which can combine the information of CC view and MLO view, break through the limitations of traditional mass detection based on a single mammography image, improve the detection rate of mass and reduce the false positive rate, and more effectively assist doctors in diagnosis.

[0005] To achieve the above purpose, the technical scheme provided by the present application is as follows: a breast mass detection system based on multi-view images, comprising:

[0006] A data import module is used to load multi-view images of the breast and pre-process the data;

[0007] A feature extraction module is used to extract image features of the main view and the auxiliary view, and obtain four main view feature maps of different resolutions and four auxiliary view feature maps of different resolutions;

[0008] The region extraction module performs preliminary detection of the mass on four main view feature maps by using a region proposal network to obtain candidate regions of the mass, referred to as ROI positions, and inputs the four main view feature maps and the ROI positions into an ROI pooling layer to obtain ROI features of a unified size.

[0009] The information fusion module converts the ROI features and the Patch features into ROI visual features and Patch visual features by using a full connection layer, converts the ROI positions and the Patch positions into ROI position features and Patch position features by using position encoding, and performs information fusion of the ROI visual features, the ROI position features, the Patch visual features and the Patch position features by using weighted fusion to obtain enhanced ROI visual features.

[0010] The mass detection module is configured to perform classification and position regression based on the enhanced ROI visual features to obtain a mass detection result of the main view.

[0011] Further, the data import module comprises a data loading module and a data preprocessing module, wherein:

[0012] The data loading module reads the multi-view images of the breast from the local, including two images of the main view and the auxiliary view, the main view and the auxiliary view are images of different views of the same breast, including a breast X-ray image of a CC view, referred to as a CC view, and a breast X-ray image of an MLO view, referred to as an MLO view; wherein the main view refers to a view to be detected for the mass, and the auxiliary view refers to another view corresponding to the main view, if the main view is the CC view, then the auxiliary view is the MLO view, and if the main view is the MLO view, then the auxiliary view is the CC view.

[0013] The data preprocessing module is configured to scale the main view and the auxiliary view, and fill the two views to the same size.

[0014] Further, the feature extraction module is a deep convolutional neural network comprising 5 convolutional stages, the first convolutional stage is composed of a convolutional layer with a kernel size of 7x7 and a pooling layer with a size of 3x3, the second to fifth convolutional stages are composed of a plurality of convolutional blocks and a pooling layer with a size of 2x2, each convolutional block is composed of three convolutional layers with kernel sizes of 1x1, 3x3 and 1x1, the number of convolutional blocks in the second to fifth convolutional stages is 3, 4, 6 and 3 respectively, the primary view and the auxiliary view will be input into the first convolutional stage respectively, the output of the first convolutional stage will be input into the second convolutional stage, and so on, finally the feature map P5 output by the fifth convolutional stage is obtained, P5 will be up-sampled and fused with the output of the fourth convolutional stage to obtain the feature map P4, P4 will also be up-sampled and fused with the output of the third convolutional stage to obtain the feature map P3, P3 will also be up-sampled and fused with the output of the second convolutional stage to obtain the feature map P2, finally four primary view feature maps with different resolutions and four auxiliary view feature maps with different resolutions are obtained, and the four feature maps with different resolutions will be used to detect four different sizes of tumors.

[0015] Further, the region extraction module comprises a region proposal network, a ROI pooling layer and a region aggregation module, wherein:

[0016] The region proposal network is composed of 3 convolutional layers, the first convolutional layer has a kernel size of 3x3, and the other two convolutional layers have a kernel size of 1x1, the four primary view feature maps P2, P3, P4 and P5 obtained by the feature extraction module will be input into the first convolutional layer respectively, and the outputs thereof will be input into the other two convolutional layers, the output result of one of the convolutional layers is the ROI position, each ROI position is represented as (x, y, w, h), x, y, w and h represent the horizontal coordinate of the ROI center position on the image, the vertical coordinate of the ROI center position on the image, the width of the ROI and the height of the ROI respectively, the output result of the other convolutional layer is used to judge whether the ROI is background, the ROI judged as background will be filtered out, and the remaining ROIs will be corresponded to one of the four primary view feature maps according to their areas;

[0017] The ROI pooling layer is used to obtain ROI features with the same size, the ROI positions obtained by the region proposal network will be input into the ROI pooling layer together with the corresponding primary view feature maps to obtain ROI features R2, R3, R4 and R5 with the same size;

[0018] The region aggregation module is used for acquiring the same size of patch features and patch positions, dividing four auxiliary view feature maps into a plurality of rectangular blocks with the same size in the width and height dimensions respectively, to obtain patch features Pa2, Pa3, Pa4 and Pa5 and patch positions, each patch position being represented as (x', y', w', h'), x', y', w' and h' representing the horizontal coordinate of the patch center position on the image, the vertical coordinate of the patch center position on the image, the width of the patch and the height of the patch respectively.

[0019] Further, the information fusion module takes the ROI features, ROI positions, patch features and patch positions of the same level as inputs and takes the enhanced ROI visual features as outputs, and includes the following steps:

[0020] 1) The ROI features and the patch features respectively pass through a fully connected layer for dimension reduction to obtain ROI visual features and patch visual features with a length of 1024, and the ROI positions and the patch positions respectively pass through position coding for dimension increase to obtain ROI position features and patch position features with a length of 256, the position coding being represented as:

[0021]

[0022] In the formula, PE (pos,i) represents the value of the i-th dimension after the dimension increase of pos, pos represents any one of the ROI position (x, y, w, h) or the patch position (x', y', w', h'), i represents the index of the feature dimension, sin and cos represent the sine function and the cosine function respectively, dim represents the feature dimension after the dimension increase, and is 64, k is half of i, which is a parameter for judging whether i is odd or even, the ROI position (x, y, w, h) is sequentially subjected to dimension increase to obtain four features with a length of 64, and the four features are spliced to obtain the ROI position features with a length of 256, and the patch position (x', y', w', h') is sequentially subjected to dimension increase to obtain four features with a length of 64, and the four features are spliced to obtain the patch position features with a length of 256;

[0023] 2) Each ROI visual feature and ROI position feature are spliced to obtain an ROI total feature with a length of 1280, and each patch visual feature and patch position feature are spliced to obtain a patch total feature with a length of 1280, the n-th ROI total feature being defined as A n , the m-th patch total feature being defined as A m , and the correlation weight w n,m between the n-th ROI and the m-th patch being:

[0024]

[0025] In the formula, w n,m represents the correlation weight between the nth ROI and the mth patch, dot represents the inner product operation, d represents the dimension size of the total feature, is 1280, W q and W k are conversion matrices for transforming the feature space, both of which have a size of dxd, after calculating all the correlation weights, the weights need to be normalized, and are represented as:

[0026]

[0027] In the formula, exp represents the exponential function with the natural constant e as the base, and k' represents the k'th patch;

[0028] 3) the nth ROI visual feature vis n According to the correlation weight, all patch visual features are weighted and fused to obtain the enhanced nth ROI visual feature VIS n :

[0029]

[0030] In the formula, W v represents the conversion matrix for transforming the feature space, has a size of 1024x1024, vis m represents the mth patch visual feature, the patch visual features of the auxiliary view are weighted and fused, the information of the two views is combined, the enhanced ROI visual feature of the main view is obtained, and the accuracy of the subsequent mass detection result is improved.

[0031] Further, the mass detection module comprises two fully connected neural networks, which are respectively used for classification and position regression, both of which have only one layer, and the input is the enhanced ROI visual feature obtained by the information fusion module, the output dimension of the fully connected neural network used for classification is 2, which respectively represents the probability that the ROI is a mass and the probability that the ROI is background, and the output dimension of the fully connected neural network used for position regression is 8, which respectively represents the position of the ROI as a mass and the position of the ROI as background.

[0032] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0033] 1. By increasing the feature extraction of the auxiliary view, the system is given the ability to input multi-view images, solving the problem that only single breast X-ray images can be used for mass detection.

[0034] 2. Through the information fusion module, the information of the CC view and the MLO view is effectively fused, the detection rate of the mass is significantly improved, and the false positive rate is reduced. Unlike other mass detection systems, the auxiliary view of the present application does not have the problem of missing detection of the mass, which affects the mass detection of the main view.

[0035] 3. Through the information fusion module, the information fusion of the multi-view image is endowed with a certain degree of explainability, which is beneficial to the understanding of the forming principle of the result by the medical personnel, and is also helpful for the further verification work of the result. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a schematic diagram of the relationship between the modules of the system of the present application.

[0037] Figure 2 is a structural schematic diagram of the information fusion module. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below in combination with specific embodiments.

[0039] The present embodiment provides a breast mass detection system based on multi-view images, which is a breast mass detection system developed using Python language and can run on multiple platforms. The relationship between the modules of the system is shown in Figure 1 , which comprises:

[0040] A data import module is used to load the multi-view images of the breast and pre-process the data;

[0041] A feature extraction module is used to extract the image features of the main view and the auxiliary view, to obtain four main view feature maps of different resolutions and four auxiliary view feature maps of different resolutions;

[0042] A region extraction module uses a region proposal network to preliminarily detect the mass in the four main view feature maps to obtain the candidate region of the mass (referred to as the ROI position), and inputs the four main view feature maps and the ROI position into a ROI pooling layer to obtain the ROI features of the same size. The region aggregation module is used to divide the four auxiliary view feature maps respectively to obtain a plurality of Patch features and Patch positions of the same size;

[0043] An information fusion module is used to convert the ROI features and the Patch features into ROI visual features and Patch visual features by using a full connection layer, to convert the ROI position and the Patch position into ROI position features and Patch position features by using position coding, and to fuse the ROI visual features, the ROI position features, the Patch visual features and the Patch position features by using weighted fusion to obtain enhanced ROI visual features;

[0044] a mass detection module, configured to perform classification and position regression according to the enhanced ROI visual features, to obtain a mass detection result of the main view.

[0045] Specifically, the data import module comprises a data loading module and a data preprocessing module, wherein:

[0046] The data loading module reads the multi-view images of the breast from the local, including two images of the main view and the auxiliary view, the main view and the auxiliary view are images of different views of the same breast, including the CC-view mammogram image (referred to as the CC view) and the MLO-view mammogram image (referred to as the MLO view), the main view refers to the view to be subjected to mass detection, the auxiliary view refers to another view corresponding to the main view, if the main view is the CC view, the auxiliary view is the MLO view, if the main view is the MLO view, the auxiliary view is the CC view.

[0047] The data preprocessing module is configured to scale the main view and the auxiliary view, first, the width of the main view is scaled to 896, and the height is scaled proportionally; if the height after scaling is greater than 1792, the height is scaled to 1792, and the width is scaled proportionally, the auxiliary view is scaled to the same size as the main view, and then the two views are filled to 896x1792, and the pixel values of the filled part are 0.

[0048] Specifically, the feature extraction module is a deep convolutional neural network comprising five convolutional stages, the first convolutional stage is composed of a convolutional layer with a convolution kernel size of 7x7 and a pooling layer with a size of 3x3, the second to fifth convolutional stages are composed of a plurality of convolutional blocks and a pooling layer with a size of 2x2, each convolutional block is composed of three convolutional layers with convolution kernel sizes of 1x1, 3x3 and 1x1, respectively, the number of convolutional blocks in the second to fifth convolutional stages is 3, 4, 6 and 3, respectively, the main view and the auxiliary view will be input into the first convolutional stage, the output of the first convolutional stage will be input into the second convolutional stage, and so on, finally, the feature map output by the fifth convolutional stage is obtained, that is, P5, the resolution of P5 is 28x56, P5 will be up-sampled and fused with the output of the fourth convolutional stage to obtain a feature map P4, the resolution of P4 is 56x112, P4 will also be up-sampled and fused with the output of the third convolutional stage to obtain a feature map P3, the resolution of P3 is 112x224, P3 will also be up-sampled and fused with the output of the second convolutional stage to obtain a feature map P2, the resolution of P2 is 224x448, and the feature dimensions of the four feature maps are all 256, finally, four main view feature maps with different resolutions and four auxiliary view feature maps with different resolutions are obtained.

[0049] Specifically, the region extraction module comprises a region proposal network, an ROI pooling layer and a region aggregation module, wherein:

[0050] The region proposal network is composed of 3 convolutional layers, the first convolutional layer has a kernel size of 3x3, and the other two convolutional layers have a kernel size of 1x1, the four main view feature maps (P2, P3, P4, P5) obtained by the feature extraction module will be input into the first convolutional layer respectively, and the output of the first convolutional layer will be input into the other two convolutional layers, the output of one of the two convolutional layers is the ROI position, each ROI position is represented as (x, y, w, h), x, y, w, h represent the horizontal coordinate of the ROI center position on the image, the vertical coordinate of the ROI center position on the image, the width of the ROI and the height of the ROI respectively, the output of the other convolutional layer is used to judge whether the ROI is background, the ROI judged as background will be filtered out, and the remaining ROIs are corresponded to one of the four main view feature maps according to their areas;

[0051] The ROI pooling layer is used to obtain ROIs with the same size, the ROI positions obtained by the region proposal network are corresponded to one of the four feature maps according to their areas, the area of the ROI is the width of the ROI multiplied by the height of the ROI, the area range is 0-112 2 The ROI position corresponding to P2, the area range is 112 2 -224 2 The ROI position corresponding to P3, the area range is 224 2 -448 2 The ROI position corresponding to P4, the area is greater than 448 2 The ROI position corresponding to P5, the ROI position and the corresponding main view feature map are input into the ROI pooling layer to obtain ROI features R2, R3, R4, R5 with the same size, the size of the ROI feature is 7x7x256;

[0052] The region aggregation module is used to obtain Patch features and Patch positions with the same size, the four auxiliary view feature maps are divided into multiple rectangular blocks with the same size in the width and height dimensions, the width and height of the rectangular block are both 14, to obtain Patch features Pa2, Pa3, Pa4, Pa5 and Patch positions, the size of the Patch feature is 14x14x256, Pa2 has 512, Pa3 has 128, Pa4 has 32, and Pa5 has 8, the position of each Patch is represented as (x', y', w', h'), x', y', w', h' represent the horizontal coordinate of the Patch center position on the image, the vertical coordinate of the Patch center position on the image, the width of the Patch and the height of the Patch respectively.

[0053] Specifically, as Figure 2As shown, the information fusion module takes ROI features, ROI positions, patch features and patch positions of the same level as input, and enhanced ROI visual features as output, and includes the following steps:

[0054] 1) The ROI features and patch features are respectively reduced in dimension by a fully connected layer to obtain ROI visual features and patch visual features with a length of 1024, and the ROI positions and patch positions are respectively increased in dimension by position encoding to obtain ROI position features and patch position features with a length of 256, and the position encoding is represented as:

[0055]

[0056] In the formula, PE (pos,i) represents the value of the i-th dimension after the position pos is increased in dimension, the position pos represents any one of the ROI position (x, y, w, h) or the patch position (x', y', w', h'), i represents the index of the feature dimension, sin and cos represent the sine function and the cosine function respectively, dim represents the feature dimension after the increase in dimension, which is 64, k is half of i, which is a parameter for judging whether i is odd or even, the ROI position (x, y, w, h) is sequentially increased in dimension to obtain four features with a length of 64, and the four features are spliced to obtain the ROI position feature with a length of 256, and the patch position (x', y', w', h') is sequentially increased in dimension to obtain four features with a length of 64, and the four features are spliced to obtain the patch position feature with a length of 256;

[0057] 2) Each ROI visual feature and ROI position feature is spliced to obtain an ROI total feature with a length of 1280, and each patch visual feature and patch position feature is spliced to obtain a patch total feature with a length of 1280, and the n-th ROI total feature is defined as A n , and the m-th patch total feature is A m , and the correlation weight w n,m between the n-th ROI and the m-th patch is:

[0058]

[0059] In the formula, w n,m represents the correlation weight between the n-th ROI and the m-th patch, dot represents the inner product operation, d represents the dimension size of the total feature, which is 1280, W q and W k are conversion matrices for transforming the feature space, and the sizes are both dxd, and after calculating all the correlation weights, the weights need to be normalized, and represented as:

[0060]

[0061] In the formula, exp represents an exponential function with the natural constant e as the base, and k' represents the k'th patch;

[0062] 3) then'th ROI visual feature vis n According to the correlation weight, all patch visual features are weighted and fused to obtain then'th enhanced ROI visual feature VIS n :

[0063]

[0064] In the formula, W v represents a conversion matrix for transforming the feature space, with a size of 1024*1024, vis m represents the m'th patch visual feature, the patch visual features of the auxiliary view are weighted and fused, the information of the two views is combined, the ROI visual feature of the main view is enhanced, and the accuracy of the subsequent mass detection result is improved.

[0065] Specifically, the mass detection module includes two fully connected neural networks, which are respectively used for classification and position regression, and both of the two fully connected neural networks only have one layer, and the input of both is the enhanced ROI visual feature obtained by the information fusion module, the output dimension of the fully connected neural network used for classification is 2, which respectively represents the probability that the ROI is a mass and the probability that the ROI is background, the output dimension of the fully connected neural network used for position regression is 8, which respectively represents the position of the ROI when the ROI is a mass and the position of the ROI when the ROI is background, according to the classification result, the ROI with a background probability greater than a mass probability is filtered out, according to the position regression result, the position of the ROI classified as a mass is obtained, and the final mass detection result of the main view is obtained.

[0066] The above embodiment is a preferred embodiment of the present application, but the embodiment of the present application is not limited by the above embodiment, and any change, modification, substitution, combination, simplification made without departing from the spirit and principle of the present application should be an equivalent replacement mode, which is included in the protection scope of the present application.

Claims

1. A multi-view image based breast mass detection system, comprising: The application relates to a breast tumor detection method based on multi-view images. The data import module is used for loading multi-view images of a breast and pre-processing data; The feature extraction module is used for extracting image features of a main view and an auxiliary view, and obtaining four main view feature maps with different resolutions and four auxiliary view feature maps with different resolutions; The region extraction module performs preliminary detection on the four main view feature maps by using a region proposal network to obtain candidate regions of the tumor, referred to as ROI positions, and inputs the four main view feature maps and the ROI positions into an ROI pooling layer to obtain ROI features with a unified size; the region aggregation module is used for dividing the four auxiliary view feature maps to obtain a plurality of Patch features with the same size and Patch positions; The information fusion module is used for converting the ROI features and the Patch features into ROI visual features and Patch visual features by using a full connection layer, converting the ROI positions and the Patch positions into ROI position features and Patch position features by using position coding, and performing information fusion on the ROI visual features, the ROI position features, the Patch visual features and the Patch position features by using weighted fusion to obtain enhanced ROI visual features; The tumor detection module is used for performing classification and position regression according to the enhanced ROI visual features to obtain a tumor detection result of the main view.

2. The multi-view image based breast mass detection system of claim 1, wherein: The data import module comprises a data loading module and a data preprocessing module. The data loading module reads multi-view images of a breast from a local, and the multi-view images comprise a main view and an auxiliary view, the main view and the auxiliary view are images of different views of the same breast, and the main view and the auxiliary view comprise a breast X-ray image in a CC view, referred to as a CC view, and a breast X-ray image in an MLO view, referred to as an MLO view; wherein the main view refers to a view to be detected for a tumor, and the auxiliary view refers to another view corresponding to the main view; if the main view is the CC view, the auxiliary view is the MLO view; if the main view is the MLO view, the auxiliary view is the CC view; The data preprocessing module is used for scaling the main view and the auxiliary view, and filling the two views to the same size.

3. The multi-view image based breast mass detection system of claim 2, wherein: The feature extraction module is a deep convolutional neural network comprising five convolutional stages, the first convolutional stage is composed of a convolutional layer with a convolution kernel size of 7x7 and a pooling layer with a size of 3x3, the second to fifth convolutional stages are composed of a plurality of convolutional blocks and a pooling layer with a size of 2x2, each convolutional block is composed of three convolutional layers, the convolution kernel sizes of the three convolutional layers are 1x1, 3x3 and 1x1 respectively, the number of convolutional blocks in the second to fifth convolutional stages are 3, 4, 6 and 3 respectively, the primary view and the auxiliary view will be used as the input of the first convolutional stage respectively, the output of the first convolutional stage will be used as the input of the second convolutional stage, and so on, finally the feature map P5 output by the fifth convolutional stage is obtained, P5 will be up-sampled and fused with the output of the fourth convolutional stage to obtain the feature map P4, P4 will also be up-sampled and fused with the output of the third convolutional stage to obtain the feature map P3, P3 will also be up-sampled and fused with the output of the second convolutional stage to obtain the feature map P2, finally four primary view feature maps with different resolutions and four auxiliary view feature maps with different resolutions are obtained, and the four feature maps with different resolutions will be used to detect four different sizes of masses.

4. The multi-view image based breast mass detection system of claim 3, wherein: The region extraction module comprises a region proposal network, an ROI pooling layer and a region aggregation module, wherein: The region proposal network is composed of three convolutional layers, the convolution kernel size of the first convolutional layer is 3x3, and the convolution kernel sizes of the other two convolutional layers are 1x1, the four primary view feature maps P2, P3, P4 and P5 obtained by the feature extraction module will be used as the input of the first convolutional layer respectively, the output of the first convolutional layer will be input into the other two convolutional layers, the output result of one of the convolutional layers is the ROI position, each ROI position is represented as (x, y, w, h), x, y, w and h represent the horizontal coordinate of the ROI center position on the image, the vertical coordinate of the ROI center position on the image, the width of the ROI and the height of the ROI respectively, the output result of the other convolutional layer is used to judge whether the ROI is background, the ROI judged as background will be filtered out, and the remaining ROIs are corresponded to one of the four primary view feature maps according to their areas; The ROI pooling layer is used to obtain ROI features with the same size, the ROI positions obtained by the region proposal network will be input into the ROI pooling layer together with the corresponding primary view feature maps to obtain ROI features R2, R3, R4 and R5 with the same size; The region aggregation module is used to obtain Patch features and Patch positions with the same size, the four auxiliary view feature maps are divided into a plurality of rectangular blocks with the same size in the width and height dimensions to obtain Patch features Pa2, Pa3, Pa4 and Pa5 and Patch positions, each Patch position is represented as (x', y', w', h'), x', y', w' and h' represent the horizontal coordinate of the Patch center position on the image, the vertical coordinate of the Patch center position on the image, the width of the Patch and the height of the Patch respectively.

5. The multi-view image based breast mass detection system of claim 4, wherein: The information fusion module takes ROI features, ROI positions, patch features and patch positions of the same level as inputs, and takes enhanced ROI visual features as outputs, and comprises the following steps: 1) The ROI features and the patch features respectively pass through a full connection layer for dimension reduction to obtain ROI visual features and patch visual features with a length of 1024, and the ROI positions and the patch positions respectively pass through position coding for dimension increase to obtain ROI position features and patch position features with a length of 256, wherein the position coding is represented as: PE = sin (pi * i / dim) in which PE (pos,i) represents the value of the i-th dimension after dimensionality increasing of pos, pos represents any one of ROI position (x, y, w, h) or patch position (x', y', w', h'), i represents the index of feature dimension, sin and cos represent the sine function and the cosine function respectively, dim represents the feature dimension after dimensionality increasing, is 64, k is half of i, is a parameter for judging whether i is odd or even, the ROI position (x, y, w, h) is sequentially dimensionally increased to obtain four features with a length of 64, and the four features are spliced to obtain ROI position features with a length of 256, and the patch position (x', y', w', h') is sequentially dimensionally increased to obtain four features with a length of 64, and the four features are spliced to obtain patch position features with a length of 256; 2) each ROI visual feature and ROI position feature are spliced to obtain an ROI total feature with a length of 1280, each Patch visual feature and Patch position feature are spliced to obtain a Patch total feature with a length of 1280, the nth ROI total feature is defined as A n , the mth Patch total feature is defined as A m , and the correlation weight w n,m between the nth ROI and the mth Patch is: where w n,m denotes the correlation weight between the nth ROI and the mth patch, dot denotes the inner product operation, d denotes the dimension size of the total feature, which is 1280, W q and W k are the conversion matrices for transforming the feature space, both of which have a size of dxd, and after calculating all the correlation weights, the weights need to be normalized, which is denoted as: In the formula, exp represents an exponential function with the natural constant e as the base, and k' represents the k'th patch; 3) the nth ROI visual feature vis n According to the correlation weight, all Patch visual features are weighted and fused to obtain the nth enhanced ROI visual feature VIS n : In the formula, W v denotes the conversion matrix for transforming the feature space, with a size of 1024x1024, vis m denotes the mth Patch visual feature, by weighted fusion of the Patch visual features of the auxiliary view, the information of the two views is combined to obtain the enhanced ROI visual feature of the main view, and the accuracy of the subsequent mass detection result is improved.

6. The multi-view image based breast mass detection system of claim 5, wherein: The mass detection module comprises two full connection neural networks respectively for classification and position regression, both of which have only one layer, and the input of both is the enhanced ROI visual features obtained by the information fusion module, the output dimension of the full connection neural network for classification is 2, representing the probability that the ROI is a mass and the probability that the ROI is background respectively, and the output dimension of the full connection neural network for position regression is 8, representing the position of the ROI as a mass and the position of the ROI as background respectively.

Images