Auxiliary detection system and method for breast abnormality

Abstract

The present invention relates to an auxiliary detection system and method for breast abnormality, which collects and preprocesses breast color ultrasound images of patients, identifies and marks regions of interest, and extracts color ultrasound image features to perform preliminary auxiliary diagnosis of breast conditions; collects and preprocesses molybdenum target images, extracts molybdenum target image features after marking, fuses molybdenum target image features with color ultrasound image features to obtain a first fusion feature, performs auxiliary diagnosis based on the first fusion feature, and determines whether to obtain a CT image; performs preprocessing, marking, and feature extraction on CT images, fuses CT image features with the first fusion feature, performs auxiliary diagnosis on the condition, and determines whether to obtain an MRI image; performs preprocessing, marking, and feature extraction on breast MRI images, fuses MRI image features with the second fusion feature, and then performs final auxiliary diagnosis. Doctors are assisted in diagnosing the condition of breast patients, and the diagnostic efficiency and the accuracy of the diagnostic results are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to an auxiliary detection system and method for breast abnormality. Background Art

[0002] Breast diseases are diseases that originate from breast-related tissues such as breast glands, fat, lymph, blood vessels, and nipples. Among the new cases of female cancer, the number of new cases of breast cancer ranks first in the world and second in China. Early diagnosis and treatment of breast diseases, especially early breast cancer, are crucial to disease control. Currently, imaging diagnosis of breast diseases plays a vital role in early detection, lesion assessment, treatment decision-making, prognosis assessment, screening and auxiliary diagnosis of breast cancer.

[0003] Different medical imaging examinations may be required for different patients' breast diseases. Common breast imaging examination methods mainly include mammography, breast color ultrasound, CT and MRI. Breast ultrasound examination has good resolution for soft tissues and can detect small lesions of several millimeters. It is non-radioactive and is the preferred examination method for breast lesions in adolescents or pregnant and lactating women. Breast mammography can detect breast cancer early and has a high diagnostic accuracy rate when combined with clinical examination. Breast CT is generally used as a supplementary examination for mammography and ultrasound examination. It can detect lesions in dense breasts, abnormal changes in the chest wall, lesions in the tail of the breast, and enlarged axillary and internal mammary lymph nodes. MRI is a nuclear magnetic resonance imaging, which has a relatively high resolution for tiny lesions and can be used for breast cancer staging assessment, determining the range of ipsilateral breast tumors, and determining whether there are multifocal or multicentric tumors.

[0004] When a breast patient has a breast disease or a routine breast examination finds an abnormality, one or more breast imaging data may be obtained. The doctor will diagnose the breast condition based on one or more breast imaging data and the patient's symptoms. Since the doctor directly diagnoses through breast images during the diagnosis process and since there may be multiple breast imaging data, it is difficult to diagnose the breast condition and the accuracy of the diagnosis result is not high.

[0005] Therefore, how to develop a system for breast image processing to process breast images to assist doctors in diagnosing breast patients' conditions, reduce the difficulty of breast disease diagnosis, and improve diagnostic efficiency and the accuracy of diagnostic results is a technical problem that urgently needs to be solved. Summary of the invention

[0006] The purpose of the present invention is to provide a system and method for auxiliary detection of breast abnormalities, which are used to process breast images to assist doctors in diagnosing breast patients' conditions, reduce the difficulty of breast disease diagnosis, and improve diagnostic efficiency and the accuracy of diagnostic results.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, a method for auxiliary detection of breast abnormalities is provided, comprising the following steps:

[0009] S1: collecting breast color ultrasound images of patients, preprocessing the breast color ultrasound images, identifying regions of interest in the preprocessed breast color ultrasound images, and marking the regions of interest;

[0010] S2: Input the marked breast color ultrasound image into the feature extraction module to extract the color ultrasound image features, perform a preliminary auxiliary diagnosis of the patient's breast condition based on the extracted color ultrasound image features, and determine whether it is necessary to obtain X-ray mammography image data. If so, execute step S3; if not, output the preliminary auxiliary diagnosis result;

[0011] S3: collecting mammographic images of the patient, preprocessing the mammographic images, identifying and marking the regions of interest in the preprocessed mammographic images;

[0012] S4: Inputting the marked mammographic X-ray mammographic target image into a feature extraction module to extract mammographic target image features, fusing the extracted mammographic target image features with the color Doppler ultrasound image features to obtain a first fusion feature, further assisting in the diagnosis of the patient's breast condition based on the first fusion feature, and determining whether it is necessary to obtain a breast CT image. If so, executing step S5; if not, outputting the further assisting diagnosis result;

[0013] S5: acquiring breast CT images of the patient, preprocessing the breast CT images, identifying and marking regions of interest in the preprocessed breast CT images;

[0014] S6: Input the marked breast CT image into the feature extraction module to extract CT image features, further perform feature fusion processing on the extracted CT image features and the first fusion features to obtain second fusion features, perform further auxiliary diagnosis on the patient's breast condition based on the second fusion features, and determine whether it is necessary to obtain a breast MRI image. If so, execute step S7; if not, output the further auxiliary diagnosis result;

[0015] S7: Acquire a breast MRI image of the patient, preprocess the breast MRI image, identify and mark a region of interest in the preprocessed breast MRI image;

[0016] S8: Input the marked breast MRI image into the feature extraction module to extract MRI image features, further fuse the extracted MRI image features with the second fusion features to obtain a third fusion feature, perform a final auxiliary diagnosis on the patient's breast condition based on the third fusion feature, and output the final auxiliary diagnosis result.

[0017] Preferably, the specific process of preprocessing the collected breast color Doppler ultrasound images of the patient in step S1 is as follows:

[0018] S11: remove invalid images, duplicate images, and images with resolution lower than a preset threshold from the patient's breast color Doppler ultrasound images;

[0019] S12: Establishing a reference coordinate system, obtaining the device coordinate system of the color ultrasound device when acquiring the breast color ultrasound image, obtaining a transformation matrix of the device coordinate system relative to the reference coordinate system, and transforming the coordinates of all pixels in the breast color ultrasound image in the device coordinate system to the coordinates in the reference coordinate system based on the transformation matrix to obtain the breast color ultrasound image in the reference coordinate system;

[0020] S13: dividing the pixel into a plurality of interval pixels according to the grayscale range of the pixel, and setting a different grayscale transformation function for each interval pixel to perform different grayscale transformations;

[0021] S14: Segment the breast color ultrasound image under the reference coordinates into image blocks of a specified size, obtain a two-dimensional cosine wave of each image block, calculate the image transformation matrix under the reference coordinate system, and transform all image blocks based on the transformation matrix.

[0022] Preferably, the specific calculation formula for calculating the transformation matrix of the image is as follows:

[0023] A( i , j )= c ( i )cos[(j+0.5π) i / N ];

[0024] in, i is the horizontal frequency of the two-dimensional cosine wave, j is the frequency of the two-dimensional cosine wave in the vertical direction, N is the size of the image block, A( i , j ) is the transformation matrix of the image, c ( i ) is the compensation coefficient.

[0025] Preferably, in step S2, the specific process of inputting the marked breast color ultrasound image into the feature extraction module to extract the color ultrasound image features is as follows:

[0026] S21: converting the breast color ultrasound image into a grayscale image, obtaining a pixel value of each pixel in the grayscale image, and obtaining a dot matrix integral image by taking a designated corner pixel point of the grayscale image as a starting point and taking the sum of the pixel values ​​in a matrix area formed with any pixel point on the image as a pixel value of a pixel point at the same position in the constructed dot matrix integral image;

[0027] S22: constructing a Gaussian pyramid, taking the dot integral image as the bottom layer of the Gaussian pyramid, performing Gaussian blur on the dot integral image and then downsampling it as the upper layer of the bottom layer, and then iteratively executing to construct a scale space;

[0028] S23: taking a pixel point whose pixel value is smaller than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and smaller than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space, or which is larger than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and larger than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space as an extreme point;

[0029] S24: obtaining continuous extreme value points in the scale space by means of discrete space point interpolation, connecting the continuous extreme value points to obtain an extreme value point curve, and fitting the extreme value point curve by means of a preset scale space function to obtain stable feature points and corresponding scale values ​​of the color ultrasound image.

[0030] Preferably, in step S4, the process of inputting the marked breast X-ray mammography image into the feature extraction module for mammography image feature extraction is the same as the process of inputting the marked breast color ultrasound image into the feature extraction module for color ultrasound image feature extraction in step S2, and feature extraction is performed on the breast X-ray mammography image to obtain stable feature points and corresponding scale values ​​of the X-ray mammography image.

[0031] Preferably, the specific process of fusing the molybdenum target image feature and the color Doppler ultrasound image feature to obtain the first fusion feature in step S4 is as follows:

[0032] S41: constructing a color ultrasound image by using the stable feature points of the color ultrasound image and the corresponding scale values ​​to create a feature vector corresponding to each feature point of the color ultrasound image, and constructing a feature vector corresponding to each feature point of the X-ray molybdenum target image by using the stable feature points of the X-ray molybdenum target image and the corresponding scale values;

[0033] S42: calculating the Euclidean distance between the feature vector of each stable feature point in the color ultrasound image feature vector and the feature vectors corresponding to all stable feature points of the X-ray molybdenum target image, and selecting the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance as a matching point for matching;

[0034] S43: after retaining the matching points in the X-ray molybdenum target image, calculating the perspective transformation matrix of the X-ray molybdenum target image based on a preset RANSAC algorithm, and performing perspective transformation on the X-ray molybdenum target image by using the perspective transformation matrix to obtain a perspective view of the X-ray molybdenum target image;

[0035] S44: Based on the matching points between the color ultrasound image and the X-ray molybdenum target image, the color ultrasound image is superimposed on the X-ray molybdenum target image perspective view to achieve fusion of the color ultrasound image and the X-ray molybdenum target image.

[0036] Preferably, in step S42, after obtaining the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance by calculating the Euclidean distance, the Euclidean distance ratio of the feature point with the shortest Euclidean distance to the feature point with the second shortest Euclidean distance is calculated, the ratio is compared with a preset threshold, and the matching points whose ratio is greater than the threshold are filtered.

[0037] In a second aspect, a breast abnormality auxiliary detection system is provided, which is used to implement the breast abnormality auxiliary detection system, including an image acquisition module, a preprocessing module, an image recognition and marking module, a feature extraction module, a feature fusion module and an auxiliary diagnosis module; the image acquisition module is connected to the preprocessing module, the preprocessing module is connected to the image recognition and marking module, the image recognition and marking module is connected to the feature extraction module, the feature extraction module is connected to the feature fusion module, and the feature fusion module is connected to the auxiliary diagnosis module;

[0038] The image acquisition module is used to acquire the patient's breast color ultrasound image, breast X-ray mammography image, breast CT image, and breast MRI image;

[0039] The preprocessing module is used to preprocess breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images;

[0040] The image recognition and marking module is used to identify and mark the region of interest in the preprocessed breast color ultrasound image, breast X-ray mammography image, breast CT image, and breast MRI image;

[0041] The feature extraction module is used to extract the features of the labeled breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images;

[0042] The feature fusion module is used to perform feature fusion processing on the molybdenum target image feature and the color Doppler ultrasound image feature to obtain a first fusion feature, further perform feature fusion processing on the CT image feature and the first fusion feature to obtain a second fusion feature, and further perform feature fusion processing on the MRI image feature and the second fusion feature to obtain a third fusion feature;

[0043] The auxiliary diagnosis module is used to perform further auxiliary diagnosis on the patient's breast condition based on the first fusion feature, or to perform further auxiliary diagnosis on the patient's breast condition based on the second fusion feature, or to perform final auxiliary diagnosis on the patient's breast condition based on the third fusion feature, to obtain and output the auxiliary diagnosis results.

[0044] The beneficial effects of the present invention include:

[0045] The auxiliary detection system and method of breast abnormality provided by the present invention collects and preprocesses the breast color Doppler ultrasound images of patients, identifies and marks the regions of interest, and extracts the features of the color Doppler ultrasound images to perform preliminary auxiliary diagnosis of breast conditions; collects and preprocesses molybdenum target images, extracts molybdenum target image features after marking, fuses the molybdenum target image features with the color Doppler ultrasound image features to obtain a first fusion feature, performs auxiliary diagnosis based on the first fusion feature, and determines whether to obtain a CT image; performs preprocessing, marking, and feature extraction on the CT image, fuses the CT image features with the first fusion feature to obtain a second fusion feature, performs auxiliary diagnosis on the condition, and determines whether to obtain an MRI image; performs preprocessing, marking, and feature extraction on the breast MRI image, fuses the MRI image features with the second fusion feature, and then performs final auxiliary diagnosis. Doctors are assisted in diagnosing the condition of breast patients, improving the diagnostic efficiency and the accuracy of the diagnostic results.

[0046] First, by removing invalid images, duplicate images, and images with resolutions below a preset threshold from the patient's breast color ultrasound images, the influence of noise data on subsequent labeling and feature extraction is avoided. By establishing a reference coordinate system, obtaining the transformation matrix of the device coordinate system relative to the reference coordinate system, and transforming the breast image to obtain the breast image under the reference coordinates, all breast images are transformed into a unified coordinate system. This process greatly improves the accuracy and effectiveness of subsequent feature extraction, facilitates finding feature matching points for feature fusion, and improves feature fusion efficiency.

[0047] Second, by dividing the pixels into multiple interval pixels according to the grayscale range of the pixels, different grayscale transformation functions are set for each interval pixel to perform different grayscale transformations, thus achieving targeted grayscale transformation and improving the grayscale transformation effect. The breast color ultrasound images under the reference coordinates are segmented into image blocks of specified sizes, and the two-dimensional cosine wave of each image block is obtained. The image transformation matrix is ​​calculated under the reference coordinate system, and all image blocks are transformed based on the transformation matrix. Combined with image transformation, accurate image transformation is achieved, providing an accurate data basis for subsequent feature extraction and feature fusion.

[0048] Third, by obtaining the pixel value of each pixel in the grayscale image, and taking the designated corner pixel point of the grayscale image as the starting point, and the sum of the pixel values ​​in the matrix area formed with any pixel point on the image as the pixel value of the pixel point at the same position in the constructed dot integral image, a dot integral image is obtained, and the pixel sum of any rectangular area in the original image is calculated through the dot integral image, which greatly reduces the subsequent feature extraction calculation amount and improves processing efficiency.

[0049] Fourth, by constructing a Gaussian pyramid, the dot integral image is used as the bottom layer of the Gaussian pyramid, and the dot integral image is Gaussian blurred and then downsampled as the upper layer of the bottom layer. The scale space is iteratively constructed to facilitate the subsequent extraction of multi-scale features of the image and simplify the multi-resolution analysis in the image feature extraction process.

[0050] Fifth, the pixel value in the pixel point is smaller than or greater than the pixel value of the pixel point in the neighborhood of the same layer in the scale space and greater than the pixel value of the pixel point in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space. Continuous extreme points are obtained in the scale space by discrete space point interpolation. The continuous extreme points are connected to obtain an extreme point curve, and the extreme point curve is fitted by a preset scale space function to obtain stable feature points and corresponding scale values ​​of the color ultrasound image, thereby realizing the accurate extraction of image features in multiple dimensions.

[0051] Sixth, by constructing the color ultrasound image, a feature vector corresponding to each feature point of the color ultrasound image and the X-ray molybdenum target image is created, and the Euclidean distance between the feature vector of each stable feature point in the color ultrasound image feature vector and all the stable feature points of the X-ray molybdenum target image is calculated. The feature point with the shortest Euclidean distance is selected as the matching point, and the perspective transformation matrix of the X-ray molybdenum target image is calculated to obtain the perspective view of the X-ray molybdenum target image. The fusion of the color ultrasound image and the X-ray molybdenum target image is realized based on the matching points, and the fusion of different types of breast images is realized, so as to further obtain more accurate lesion feature data and provide more precise data support for auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1It is a schematic diagram of the flow chart of the auxiliary detection method of breast abnormality of the present invention.

[0053] Figure 2 It is a schematic diagram of the process of color ultrasound image feature extraction of the present invention.

[0054] Figure 3 It is a schematic diagram of the architecture of the auxiliary detection system for breast abnormalities of the present invention. DETAILED DESCRIPTION

[0055] The following is combined with Figure 1~Figure 3 The present invention is further described in detail:

[0056] Example 1

[0057] See attached Figure 1 As shown, a method for auxiliary detection of breast abnormalities comprises the following steps:

[0058] S1: Collect the breast color ultrasound image of the patient, pre-process the breast color ultrasound image, identify the region of interest in the pre-processed breast color ultrasound image, and mark the region of interest, where the region of interest is a lesion area or a suspected lesion area. Since the shooting area of ​​various breast images of patients is relatively large, but the area with abnormalities or symptoms is actually only a very small part of it, the present invention captures the lesion area and possible lesion area in advance before processing the image data, so that subsequent image processing is only performed on the lesion area and possible lesion area, which can greatly reduce the amount of image processing and improve processing efficiency.

[0059] S2: Input the marked breast color ultrasound image into the feature extraction module for color ultrasound image feature extraction, perform a preliminary auxiliary diagnosis of the patient's breast condition based on the extracted color ultrasound image features, and determine whether it is necessary to obtain X-ray molybdenum target image data. If so, execute step S3, if not, output the preliminary auxiliary diagnosis results. Since some patients only have routine examinations, the breast color ultrasound image is obtained at the beginning, and the preliminary auxiliary diagnosis of the breast is obtained by feature extraction, and it is determined according to the preliminary auxiliary diagnosis results whether further collection of breast X-ray molybdenum target images is required. If there are no abnormal features in the color ultrasound image, there is no need to execute the next step, and the preliminary auxiliary diagnosis results are directly input. If the features extracted from the color ultrasound image show that the patient has certain breast abnormalities, and because the information displayed by the color ultrasound image is limited, it is determined at this time that a breast X-ray molybdenum target image needs to be obtained to achieve further auxiliary diagnosis.

[0060] S3: Collect the patient's breast X-ray molybdenum target images, pre-process the breast X-ray molybdenum target images, identify and mark the regions of interest in the pre-processed breast X-ray molybdenum target images. The pre-processing and marking of the regions of interest here are the same as in step S1. Since the shooting area of ​​the various breast images of the patient is relatively large, but the area with abnormalities or symptoms is actually only a very small part of it, the present invention captures the lesion area and possible lesion area in advance before processing the image data, so that the image processing is only performed on the lesion area and possible lesion area in the future, which can greatly reduce the amount of image processing and improve processing efficiency.

[0061] S4: Input the marked breast X-ray mammography image into the feature extraction module to extract the mammography image features, fuse the extracted mammography image features with the color Doppler ultrasound image features to obtain a first fusion feature, perform further auxiliary diagnosis on the patient's breast condition based on the first fusion feature, and determine whether it is necessary to obtain a breast CT image. If so, execute step S5, if not, output the further auxiliary diagnosis results. Auxiliary diagnosis is not performed directly through the features of breast X-ray mammography images because the feature information reflected by a single breast image is relatively limited, and the display of the lesion may not be complete. Therefore, the present invention fuses the features of X-ray mammography images and color Doppler ultrasound images to obtain more accurate lesion feature data and improve the accuracy of auxiliary diagnosis.

[0062] S5: Collect the patient's breast CT image, pre-process the breast CT image, identify and mark the region of interest in the pre-processed breast CT image. When the color ultrasound image and X-ray mammography image cannot accurately obtain the patient's breast abnormality information, and further diagnosis is required, the patient's breast CT image data is taken. When the patient's breast abnormality information can be accurately obtained through color ultrasound images and X-ray mammography images, there is no need to obtain the patient's breast CT image. When an abnormality is found in the patient's breast, for example, the mammography image can accurately reflect the patient's abnormality such as breast calcification, but when there is a suspected breast tumor, it is necessary to further accurately judge and locate the suspected breast tumor area through CT images.

[0063] S6: Input the marked breast CT image into the feature extraction module to extract CT image features, further perform feature fusion processing on the extracted CT image features and the first fusion features to obtain second fusion features, perform further auxiliary diagnosis on the patient's breast condition based on the second fusion features, and determine whether it is necessary to obtain a breast MRI image. If so, execute step S7; if not, output the further auxiliary diagnosis result;

[0064] S7: Acquire the patient's breast MRI image, preprocess the breast MRI image, identify and mark the region of interest in the preprocessed breast MRI image. Since CT images focus on reflecting the calcification information of breast tumors, when it is necessary to obtain blood flow information of breast tumors, breast MRI is required.

[0065] S8: Input the marked breast MRI image into the feature extraction module to extract MRI image features, further fuse the extracted MRI image features with the second fusion features to obtain a third fusion feature, perform a final auxiliary diagnosis on the patient's breast condition based on the third fusion feature, and output the final auxiliary diagnosis result. Through the multi-level feature fusion processing process, it is possible to obtain complete information on the patient's breast lesions from the final feature fusion results, provide doctors with complete lesion feature information, and achieve accurate and comprehensive detection of breast abnormalities.

[0066] In this embodiment, in step S1, the region of interest in the preprocessed breast color ultrasound image is identified by the image recognition and marking module, and the region of interest is marked. The image recognition and marking module is a neural network model, including an input layer, a convolution layer + an activation layer + a pooling layer, a fully connected layer and an output layer. Before the image recognition and marking module identifies and marks the region of interest, the model is first trained using the labeled breast color ultrasound image.

[0067] After meeting the specified performance requirements, the preprocessed breast color ultrasound image is input into the model through the input layer. The convolution layer performs convolution operation on the breast color ultrasound image. Each convolution kernel in the convolution layer extracts specific features in the input data, including edge features and texture features shown in the image. After the convolution operation, a feature map is obtained, which represents the feature intensity at different positions in the output color ultrasound image. After obtaining the feature map, the feature map is nonlinearly transformed through the subsequent activation layer, so that the model can capture more complex feature representations in the color ultrasound image.

[0068] The feature map obtained after the convolution layer and activation function processing is input to the pooling layer, which downsamples the input feature map to reduce the dimension and computation of the data, and reduces the resolution of the feature map while retaining important feature information in the image. The fully connected layer is usually located at the end of the model. It receives the output of the pooling layer and maps it to the output space to combine and classify the features extracted by the previous layer.

[0069] Example 2

[0070] On the basis of Example 1, the specific process of preprocessing the collected breast color Doppler ultrasound images of the patient in step S1 is as follows:

[0071] S11: remove invalid images, duplicate images, and images with resolution lower than a preset threshold from the patient's breast color Doppler ultrasound images;

[0072] S12: Establishing a reference coordinate system, obtaining the device coordinate system of the color ultrasound device when acquiring the breast color ultrasound image, obtaining a transformation matrix of the device coordinate system relative to the reference coordinate system, and transforming the coordinates of all pixels in the breast color ultrasound image in the device coordinate system to the coordinates in the reference coordinate system based on the transformation matrix to obtain the breast color ultrasound image in the reference coordinate system;

[0073] S13: dividing the pixel into a plurality of interval pixels according to the grayscale range of the pixel, and setting a different grayscale transformation function for each interval pixel to perform different grayscale transformations;

[0074] S14: Segment the breast color ultrasound image under the reference coordinates into image blocks of a specified size, obtain a two-dimensional cosine wave of each image block, calculate the image transformation matrix under the reference coordinate system, and transform all image blocks based on the transformation matrix.

[0075] The specific calculation formula for calculating the transformation matrix of the image is as follows:

[0076] A( i , j )= c ( i )cos[(j+0.5π) i / N ];

[0077] in, i is the horizontal frequency of the two-dimensional cosine wave, j is the frequency of the two-dimensional cosine wave in the vertical direction, N is the size of the image block, A( i , j ) is the transformation matrix of the image, c ( i ) is the compensation coefficient.

[0078] In this embodiment, by removing invalid images, repeated images and images with resolutions lower than a preset threshold in the breast color ultrasound images of the patient, the influence of noise data on subsequent marking and feature extraction is avoided. By establishing a reference coordinate system, obtaining a transformation matrix of the device coordinate system relative to the reference coordinate system, transforming the breast image to obtain a breast image under the reference coordinates, and realizing the transformation of all breast images into a unified coordinate system, this process greatly improves the accuracy and effectiveness of subsequent feature extraction, facilitates finding feature matching points for feature fusion, and improves the efficiency of feature fusion. By dividing the pixels into multiple interval pixels according to the grayscale range of the pixels, setting different grayscale transformation functions for each interval pixel to perform different grayscale transformations, targeted grayscale transformation is achieved, and the grayscale transformation effect is improved. The breast color ultrasound image under the reference coordinates is segmented into image blocks of a specified size, and a two-dimensional cosine wave of each image block is obtained. The transformation matrix of the image is calculated in the reference coordinate system, and all image blocks are transformed based on the transformation matrix. The image is accurately transformed in combination with the image transformation, providing an accurate data basis for subsequent feature extraction and feature fusion.

[0079] Example 3

[0080] Based on Example 1 or Example 2, see Figure 2 The specific process of inputting the marked breast color ultrasound image into the feature extraction module for color ultrasound image feature extraction in step S2 is as follows:

[0081] S21: Set a linear transformation function, and use the linear transformation function to perform grayscale linear expansion on each pixel in the image to convert the breast color ultrasound image into a grayscale image, obtain the pixel value of each pixel in the grayscale image, and use the sum of the pixel values ​​in the matrix area formed by any pixel on the image and the specified corner pixel point of the grayscale image as the starting point to obtain a dot matrix integral map. The dot matrix integral map can greatly speed up the calculation of the pixel sum of any rectangular area in the image because it only needs to be calculated sequentially.

[0082] S22: constructing a Gaussian pyramid, taking the dot integral image as the bottom layer of the Gaussian pyramid, performing Gaussian blur on the dot integral image and then downsampling the image as the upper layer of the bottom layer, and then iteratively executing to construct a scale space.

[0083] S23: taking a pixel point whose pixel value is smaller than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and smaller than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space, or which is larger than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and larger than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space as an extreme point;

[0084] S24: obtaining continuous extreme value points in the scale space by means of discrete space point interpolation, connecting the continuous extreme value points to obtain an extreme value point curve, and fitting the extreme value point curve by means of a preset scale space function to obtain stable feature points and corresponding scale values ​​of the color ultrasound image.

[0085] In this embodiment, by obtaining the pixel value of each pixel in the grayscale image, and taking the specified corner pixel point of the grayscale image as the starting point, and taking the sum of the pixel values ​​in the matrix area formed with any pixel point on the image as the pixel value of the pixel point at the same position in the constructed dot integral map, a dot integral map is obtained, and the pixel sum of any rectangular area in the original image is calculated by the dot integral map, which greatly reduces the amount of subsequent feature extraction calculations and improves processing efficiency. By constructing a Gaussian pyramid, the dot integral map is used as the bottom layer of the Gaussian pyramid, and the dot integral map is Gaussian blurred and then downsampled as the upper layer of the bottom layer, and the scale space is iteratively constructed, which is convenient for subsequent extraction of multi-scale features of the image, and simplifies the multi-resolution analysis in the image feature extraction process.

[0086] In another implementation of this embodiment, a dot matrix integral image is also created first, and a box filter is used to approximate the Gaussian kernel for the dot matrix integral image. Since the amount of calculation when calculating the convolution is independent of the filter size, the algorithm operation speed can be greatly improved. By changing the size of the filter, a scale space is obtained. Then the Hessian matrix is ​​used to detect the extreme points of the image. First, the extreme points are judged according to the sign of the determinant calculated by the eigenvalue. The scale space is divided into several groups. A group represents a series of response images of the filter template that is gradually enlarged to filter the same input image. Each group is composed of several fixed layers. For a key point in a certain scale image, after the extreme value is obtained by the Hessian matrix, non-maximum suppression is performed in the cube neighborhood and then interpolation operations are performed in the scale space and image space to obtain the position of the stable feature point and the scale value at which it is located. Then, it is judged whether the point is a local extreme point based on whether the value is positive or negative. If the determinant is positive, then the eigenvalues ​​are all positive or all negative, so the point is an extreme point. And to exclude the peak point of the edge response function, the principal curvature is larger in the direction of the edge, and smaller in the direction perpendicular to the edge. The principal curvature can be obtained by calculating the Hessian matrix at the position scale of the point, and the derivative is estimated by the adjacent difference of the sampling points to achieve the removal of edge response.

[0087] Example 4

[0088] On the basis of Example 1 or Example 2 or Example 3, the process of inputting the marked breast X-ray molybdenum target image into the feature extraction module for molybdenum target image feature extraction in step S4 is the same as the process of inputting the marked breast color ultrasound image into the feature extraction module for color ultrasound image feature extraction in step S2, and feature extraction is performed on the breast X-ray molybdenum target image to obtain stable feature points and corresponding scale values ​​of the X-ray molybdenum target image.

[0089] The specific process of fusing the molybdenum target image feature and the color Doppler ultrasound image feature to obtain the first fusion feature in step S4 is as follows:

[0090] S41: constructing a color ultrasound image by using the stable feature points of the color ultrasound image and the corresponding scale values ​​to create a feature vector corresponding to each feature point of the color ultrasound image, and constructing a feature vector corresponding to each feature point of the X-ray molybdenum target image by using the stable feature points of the X-ray molybdenum target image and the corresponding scale values;

[0091] S42: Calculate the Euclidean distance between the feature vector of each stable feature point in the color ultrasound image feature vector and the feature vectors corresponding to all stable feature points of the X-ray molybdenum target image, and select the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance as the matching point for matching. Image matching is to find the one-to-one mapping relationship between the same position points of the X-ray molybdenum target image and the color ultrasound image, and then achieve the matching of the X-ray molybdenum target image and the color ultrasound image according to the point mapping relationship.

[0092] S43: after retaining the matching points in the X-ray molybdenum target image, calculating the perspective transformation matrix of the X-ray molybdenum target image based on a preset RANSAC algorithm, and performing perspective transformation on the X-ray molybdenum target image by using the perspective transformation matrix to obtain a perspective view of the X-ray molybdenum target image;

[0093] S44: Based on the matching points between the color ultrasound image and the X-ray molybdenum target image, the color ultrasound image is superimposed on the X-ray molybdenum target image perspective view to achieve fusion of the color ultrasound image and the X-ray molybdenum target image.

[0094] In step S42 of the above process, after obtaining the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance by calculating the Euclidean distance, the Euclidean distance ratio of the feature point with the shortest Euclidean distance to the feature point with the second shortest Euclidean distance is calculated, and the ratio is compared with a preset threshold, and the matching points whose ratio is greater than the threshold are filtered.

[0095] An auxiliary detection system for breast abnormalities, used to implement the auxiliary detection system for breast abnormalities, see Figure 3 , including an image acquisition module, a preprocessing module, an image recognition and marking module, a feature extraction module, a feature fusion module and an auxiliary diagnosis module; the image acquisition module is connected to the preprocessing module, the preprocessing module is connected to the image recognition and marking module, the image recognition and marking module is connected to the feature extraction module, the feature extraction module is connected to the feature fusion module, and the feature fusion module is connected to the auxiliary diagnosis module;

[0096] The image acquisition module is used to acquire the patient's breast color ultrasound image, breast X-ray mammography image, breast CT image, and breast MRI image;

[0097] The preprocessing module is used to preprocess breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images;

[0098] The image recognition and marking module is used to identify and mark the region of interest in the preprocessed breast color ultrasound image, breast X-ray mammography image, breast CT image, and breast MRI image;

[0099] The feature extraction module is used to extract the features of the labeled breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images;

[0100] The feature fusion module is used to perform feature fusion processing on the molybdenum target image feature and the color Doppler ultrasound image feature to obtain a first fusion feature, further perform feature fusion processing on the CT image feature and the first fusion feature to obtain a second fusion feature, and further perform feature fusion processing on the MRI image feature and the second fusion feature to obtain a third fusion feature;

[0101] The auxiliary diagnosis module is used to perform further auxiliary diagnosis on the patient's breast condition based on the first fusion feature, or to perform further auxiliary diagnosis on the patient's breast condition based on the second fusion feature, or to perform final auxiliary diagnosis on the patient's breast condition based on the third fusion feature, to obtain and output the auxiliary diagnosis results.

[0102] In summary, the auxiliary detection system and method of breast abnormality provided by the present invention collects the breast color Doppler ultrasound images of patients and performs preprocessing and feature extraction to perform preliminary auxiliary diagnosis of breast conditions; performs feature fusion of molybdenum target image features and color Doppler ultrasound image features to obtain a first fusion feature, and performs auxiliary diagnosis based on the first fusion feature; performs feature fusion of CT image features and the first fusion feature to obtain a second fusion feature, and performs auxiliary diagnosis of the condition; performs feature fusion of MRI image features and the second fusion feature, and then performs final auxiliary diagnosis. It can effectively assist doctors in diagnosing breast patients' conditions and improve diagnostic efficiency and the accuracy of diagnostic results.

[0103] By removing the noise image in the patient's breast color ultrasound image, the influence of noise data on subsequent marking and feature extraction is avoided. By establishing a reference coordinate system, obtaining the transformation matrix of the device coordinate system relative to the reference coordinate system, transforming the breast image to obtain the breast image under the reference coordinate, and realizing the transformation of all breast images into a unified coordinate system, this process greatly improves the accuracy and effectiveness of subsequent feature extraction, facilitates the finding of feature matching points for feature fusion, and improves the efficiency of feature fusion. The breast color ultrasound image under the reference coordinate is segmented into image blocks of specified size, and the two-dimensional cosine wave of each image block is obtained. The transformation matrix of the image is calculated in the reference coordinate system, and all image blocks are transformed based on the transformation matrix. Combined with image transformation, the precise transformation of the image is realized, providing an accurate data basis for subsequent feature extraction and feature fusion.

[0104] By creating a dot matrix integral map to calculate the pixel sum of any rectangular area in the original image, the amount of subsequent feature extraction calculations is greatly reduced. By constructing a Gaussian pyramid, it is convenient to extract the multi-scale features of the image in the subsequent process, and it simplifies the multi-resolution analysis in the image feature extraction process. The pixel value in the pixel point is less than or greater than the pixel value of the pixel point in the neighborhood of the same layer in the scale space, and greater than the pixel value of the pixel point in the layer where the pixel point is located in the scale space and the pixel value of the upper and lower adjacent layers. The continuous extreme points are connected to obtain the extreme point curve and fit it to obtain the stable feature points and corresponding scale values ​​of the color ultrasound image, realizing the accurate extraction of image features in multiple dimensions. The eigenvector corresponding to each feature point of the color ultrasound image and the X-ray molybdenum target image is created by constructing the color ultrasound image, and the Euclidean distance between the eigenvector of each stable feature point in the color ultrasound image feature vector and all the stable feature points of the X-ray molybdenum target image is calculated. The feature point with the shortest Euclidean distance is selected as the matching point, and the perspective transformation matrix of the X-ray molybdenum target image is calculated to obtain the perspective view of the X-ray molybdenum target image. The fusion of the color ultrasound image and the X-ray molybdenum target image is realized based on the matching points, and the fusion of different types of breast images is realized, so as to further obtain more accurate lesion feature data and provide more precise data support for auxiliary diagnosis.

Claims

1. A method for auxiliary detection of breast abnormalities, characterized in that: The following steps are involved: S1: Collect and preprocess the patient's breast color ultrasound images, identify and mark the region of interest in the preprocessed breast color ultrasound images; S2: Input the marked breast color ultrasound image into the feature extraction module to extract the color ultrasound image features, perform a preliminary auxiliary diagnosis of the patient's breast condition based on the color ultrasound image features, and determine whether it is necessary to obtain X-ray mammography image data. If so, execute step S3; if not, output the preliminary auxiliary diagnosis result; S3: collecting and preprocessing the patient's mammographic X-ray images, and identifying the region of interest markers in the preprocessed mammographic X-ray images; S4: Input the marked mammographic X-ray mammography image into the feature extraction module to extract the mammographic image features, fuse the mammographic image features with the color Doppler ultrasound image features to obtain a first fusion feature, and perform further auxiliary diagnosis on the patient's breast condition based on the first fusion feature to determine whether it is necessary to obtain a breast CT image. If so, execute step S5; if not, output the further auxiliary diagnosis result; S5: acquiring and preprocessing the breast CT images of the patient, and identifying the region of interest markers in the preprocessed breast CT images; S6: Input the marked breast CT image into the feature extraction module to extract CT image features, further fuse the CT image features with the first fusion features to obtain a second fusion feature, and perform further auxiliary diagnosis on the patient's breast condition based on the second fusion feature to determine whether it is necessary to obtain a breast MRI image. If so, execute step S7; if not, output the further auxiliary diagnosis result; S7: Acquire and preprocess the patient's breast MRI image, and identify a region of interest marker in the preprocessed breast MRI image; S8: Input the labeled breast MRI image into the feature extraction module to extract MRI image features, further fuse the MRI image features with the second fusion features to obtain a third fusion feature, perform a final auxiliary diagnosis on the patient's breast condition based on the third fusion feature, and output the final auxiliary diagnosis result.

2. The auxiliary detection method for breast abnormality according to claim 1, characterized in that: The specific process of preprocessing the collected breast color Doppler ultrasound images of the patient in step S1 is as follows: S11: remove invalid images, duplicate images, and images with resolution lower than a preset threshold from the patient's breast color Doppler ultrasound images; S12: Establishing a reference coordinate system, obtaining the device coordinate system of the color ultrasound device when acquiring the breast color ultrasound image, obtaining a transformation matrix of the device coordinate system relative to the reference coordinate system, and transforming the coordinates of all pixels in the breast color ultrasound image in the device coordinate system to the coordinates in the reference coordinate system based on the transformation matrix to obtain the breast color ultrasound image in the reference coordinate system; S13: dividing the pixel into a plurality of interval pixels according to the grayscale range of the pixel, and setting a different grayscale transformation function for each interval pixel to perform different grayscale transformations; S14: Segment the breast color ultrasound image under the reference coordinates into image blocks of a specified size, obtain a two-dimensional cosine wave of each image block, calculate the image transformation matrix under the reference coordinate system, and transform all image blocks based on the transformation matrix.

3. The auxiliary detection method for breast abnormality according to claim 2, characterized in that: The specific calculation formula for calculating the transformation matrix of the image is as follows: A( i , j )= c ( i )cos[(j+0.5π) i / N ]; in, i is the horizontal frequency of the two-dimensional cosine wave, j is the frequency of the two-dimensional cosine wave in the vertical direction, N is the size of the image block, A( i , j ) is the transformation matrix of the image, c ( i ) is the compensation coefficient.

4. The auxiliary detection method for breast abnormality according to claim 1, characterized in that: The specific process of inputting the marked breast color ultrasound image into the feature extraction module to extract the color ultrasound image features in step S2 is as follows: S21: converting the breast color ultrasound image into a grayscale image, obtaining a pixel value of each pixel in the grayscale image, and obtaining a dot matrix integral image by taking a designated corner pixel point of the grayscale image as a starting point and taking the sum of the pixel values ​​in a matrix area formed with any pixel point on the image as a pixel value of a pixel point at the same position in the constructed dot matrix integral image; S22: constructing a Gaussian pyramid, taking the dot integral image as the bottom layer of the Gaussian pyramid, performing Gaussian blur on the dot integral image and then downsampling it as the upper layer of the bottom layer, and then iteratively executing to construct a scale space; S23: taking a pixel point whose pixel value is smaller than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and smaller than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space, or which is larger than the pixel value of a pixel point in the neighborhood of the same layer in the scale space and larger than the pixel value of a pixel point in a corresponding 3×3 position in the layer where the pixel point is located and the upper and lower adjacent layers in the scale space as an extreme point; S24: obtaining continuous extreme value points in the scale space by means of discrete space point interpolation, connecting the continuous extreme value points to obtain an extreme value point curve, and fitting the extreme value point curve by means of a preset scale space function to obtain stable feature points and corresponding scale values ​​of the color ultrasound image.

5. The auxiliary detection method for breast abnormality according to claim 4, characterized in that: The process of inputting the marked breast X-ray molybdenum target image into the feature extraction module for molybdenum target image feature extraction in step S4 is the same as the process of inputting the marked breast color ultrasound image into the feature extraction module for color ultrasound image feature extraction in step S2. Feature extraction is performed on the breast X-ray molybdenum target image to obtain stable feature points and corresponding scale values ​​of the X-ray molybdenum target image.

6. The auxiliary detection method for breast abnormality according to claim 5, characterized in that: The specific process of fusing the molybdenum target image feature and the color Doppler ultrasound image feature to obtain the first fusion feature in step S4 is as follows: S41: constructing a color ultrasound image by using the stable feature points of the color ultrasound image and the corresponding scale values ​​to create a feature vector corresponding to each feature point of the color ultrasound image, and constructing a feature vector corresponding to each feature point of the X-ray molybdenum target image by using the stable feature points of the X-ray molybdenum target image and the corresponding scale values; S42: calculating the Euclidean distance between the feature vector of each stable feature point in the color ultrasound image feature vector and the feature vectors corresponding to all stable feature points of the X-ray molybdenum target image, and selecting the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance as a matching point for matching; S43: after retaining the matching points in the X-ray molybdenum target image, calculating the perspective transformation matrix of the X-ray molybdenum target image based on a preset RANSAC algorithm, and performing perspective transformation on the X-ray molybdenum target image by using the perspective transformation matrix to obtain a perspective view of the X-ray molybdenum target image; S44: Based on the matching points between the color ultrasound image and the X-ray molybdenum target image, the color ultrasound image is superimposed on the X-ray molybdenum target image perspective view to achieve fusion of the color ultrasound image and the X-ray molybdenum target image.

7. The auxiliary detection method for breast abnormality according to claim 6, characterized in that: In step S42, after obtaining the stable feature point of the X-ray molybdenum target image with the shortest Euclidean distance by calculating the Euclidean distance, the Euclidean distance ratio of the feature point with the shortest Euclidean distance to the feature point with the second shortest Euclidean distance is calculated, the ratio is compared with a preset threshold, and the matching points whose ratio is greater than the threshold are filtered.

8. An auxiliary detection system for breast abnormality, used to implement an auxiliary detection method for breast abnormality according to any one of claims 1 to 7, characterized in that: It includes an image acquisition module, a preprocessing module, an image recognition and marking module, a feature extraction module, a feature fusion module and an auxiliary diagnosis module; the image acquisition module is connected to the preprocessing module, the preprocessing module is connected to the image recognition and marking module, the image recognition and marking module is connected to the feature extraction module, the feature extraction module is connected to the feature fusion module, and the feature fusion module is connected to the auxiliary diagnosis module; The image acquisition module is used to acquire breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images of patients; The preprocessing module is used to preprocess breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images; The image recognition and marking module is used to identify and mark the region of interest in the preprocessed breast color ultrasound image, breast X-ray mammography image, breast CT image, and breast MRI image; The feature extraction module is used to extract the features of the labeled breast color ultrasound images, breast X-ray mammography images, breast CT images, and breast MRI images; The feature fusion module is used to perform feature fusion processing on the molybdenum target image feature and the color Doppler ultrasound image feature to obtain a first fusion feature, further perform feature fusion processing on the CT image feature and the first fusion feature to obtain a second fusion feature, and further perform feature fusion processing on the MRI image feature and the second fusion feature to obtain a third fusion feature; The auxiliary diagnosis module is used to perform preliminary auxiliary diagnosis of the patient's breast condition based on the color ultrasound image features, perform further auxiliary diagnosis of the patient's breast condition based on the first fusion feature, or perform further auxiliary diagnosis of the patient's breast condition based on the second fusion feature, or perform final auxiliary diagnosis of the patient's breast condition based on the third fusion feature, obtain auxiliary diagnosis results and output them.

Images