A breast image processing method, model, device, equipment and storage medium

Abstract

The present application relates to the technical field of image processing, in particular to a breast image processing method, model, device, equipment and storage medium. The present application collects the gray image and the elastic image of the breast, extracts the feature map of the gray image and the feature map of the elastic image, since the gray image and the elastic image of the present application contain the same target object, the hardness feature map of the target object can be screened from the elastic image feature map, and the gray feature map of the target object can be screened from the gray image feature map, then the elastic image feature map and the target gray feature map are fused to obtain a first fusion feature map; the gray image feature map and the target hardness feature map are fused to obtain a second fusion feature map, the first fusion feature map and the second fusion feature map contain a plurality of target-related features, so combining the first fusion feature map and the second fusion feature map can realize more accurate classification of the target object.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, model, apparatus, device, and storage medium for processing breast images. Background Technology

[0002] A grayscale image of the breast (obtained via ultrasound) is acquired and then subjected to machine learning analysis to classify the image. The classification results can help doctors analyze whether the breast has lesions and the extent of those lesions. However, grayscale images only show the edge information of the breast; classifying the image based solely on this single piece of information will lead to inaccurate classification results.

[0003] In summary, existing breast image classification methods result in inaccurate classification results.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a breast image processing method, model, apparatus, device, and storage medium, which solves the problem of inaccurate classification results caused by existing breast image classification methods.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a breast image processing method, comprising:

[0008] Features are extracted from the grayscale image of the breast to obtain a grayscale image feature map, and features are extracted from the elastic image of the breast to obtain an elastic image feature map, which is used to characterize the hardness of the breast.

[0009] Determine the target grayscale feature map occupied by the same target object in the grayscale image feature map and the target hardness feature map occupied in the elasticity image feature map, wherein the same target object is a target object that appears in both the mammary gland grayscale image and the mammary gland elasticity image;

[0010] The elastic image feature map and the target grayscale feature map are fused to obtain a first fused feature map;

[0011] The grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map;

[0012] The classification result of the target object is obtained based on the first fusion feature map and the second fusion feature map.

[0013] In one implementation, determining the target grayscale feature map occupied by the same target object in the grayscale image feature map and the target hardness feature map occupied in the elasticity image feature map, wherein the same target object is a target object that appears in both the breast grayscale image and the breast elasticity image, includes:

[0014] By applying a class activation map to the grayscale image feature map and the elastic image feature map, a first target region of the same target object on the grayscale image feature map and a second target region on the elastic image feature map are obtained.

[0015] Based on the first target region and the grayscale image feature map, a target grayscale feature map is obtained;

[0016] Based on the second target region and the elastic image feature map, the target hardness feature map is obtained.

[0017] In one implementation, fusing the elastic image feature map and the target grayscale feature map to obtain a first fused feature map includes:

[0018] Extract the horizontal and vertical edge features of the target from the target grayscale feature map;

[0019] The horizontal edge features and vertical edge features of the target are combined to obtain the edge features of the target on the grayscale image feature map, which are denoted as the target grayscale edge features;

[0020] Cross-attention calculation is performed on the target grayscale edge features and the elastic image feature map to obtain the first fused feature map.

[0021] In one implementation, the step of performing cross-attention calculation on the target grayscale edge features and the elastic image feature map to obtain a first fused feature map includes:

[0022] The target grayscale edge feature is multiplied by a set first weight matrix to obtain the query vector of the cross attention;

[0023] The elastic image feature map is multiplied by a set second weight matrix to obtain the key vector of the cross attention;

[0024] The elastic image feature map is multiplied by a set third weight matrix to obtain the value vector of the cross attention;

[0025] Multiply the query vector by the transpose of the key vector to obtain the product vector;

[0026] Determine the vector dimension of the key vector, and divide the resulting vector of the product by the root mean square of the vector dimension to obtain the intermediate vector;

[0027] The normalized intermediate vector is multiplied by the value vector to obtain the first fused feature map.

[0028] In one implementation, fusing the grayscale image feature map and the target hardness feature map to obtain a second fused feature map includes:

[0029] Perform a convolution operation on the elastic image feature map to obtain the convolved elastic image feature map;

[0030] By connecting the target hardness feature map and the convolved elastic image feature map, a connected feature map is obtained;

[0031] Perform a convolution operation on the connection feature map to obtain the hardness distribution feature of the target object, which is denoted as the target hardness distribution feature;

[0032] A second fused feature map is obtained by applying a cross-attention mechanism to the target hardness distribution features and the grayscale image feature map.

[0033] In one implementation, the extraction method of the target hardness feature map includes:

[0034] Determine each target pixel point corresponding to the target object on the breast elastography image;

[0035] Calculate the mean, variance, and bias of each target pixel.

[0036] The mean, variance, and deviation are connected using a join function to obtain a target hardness feature map.

[0037] Secondly, embodiments of the present invention also provide a model for processing breast images, used to implement the breast image processing method described above. The model is a trained model, and the optimization method of the loss function applied during the training process of the model includes:

[0038] During training, the accuracy, sensitivity, and classification specificity of the model are statistically analyzed.

[0039] The reward value for reinforcement learning is obtained based on the accuracy, sensitivity, and classification specificity.

[0040] Based on the current adjustment factor of the loss function, the Q value of reinforcement learning is obtained;

[0041] Based on the Q-value and the reward value, the adjustment factor of the loss function is optimized using reinforcement learning.

[0042] Thirdly, embodiments of the present invention also provide a breast image processing apparatus, wherein the apparatus comprises the following components:

[0043] The feature extraction module is used to extract features from the grayscale image of the breast to obtain a grayscale image feature map, and to extract features from the elastic image of the breast to obtain an elastic image feature map, wherein the elastic image feature map is used to characterize the hardness of the breast.

[0044] The feature map calculation module is used to determine the target grayscale feature map occupied by the same target object on the grayscale image feature map and the target hardness feature map occupied on the elastic image feature map, wherein the same target object is a target object that appears on both the mammary gland grayscale image and the mammary gland elastic image;

[0045] The first fusion module is used to fuse the elastic image feature map and the target grayscale feature map to obtain a first fused feature map;

[0046] The second fusion module is used to fuse the grayscale image feature map and the target hardness feature map to obtain a second fused feature map;

[0047] The classification module is used to obtain the classification result of the target object based on the first fusion feature map and the second fusion feature map.

[0048] Fourthly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a breast image processing program stored in the memory and executable on the processor, wherein when the processor executes the breast image processing program, it implements the steps of the breast image processing method described above.

[0049] Fifthly, embodiments of the present invention also provide a computer-readable storage medium storing a breast image processing program, wherein when the breast image processing program is executed by a processor, it implements the steps of the breast image processing method described above.

[0050] Beneficial effects: This invention acquires grayscale and elasticity images of breast tissue, extracts feature maps from the grayscale and elasticity images, and since the grayscale and elasticity images contain the same target object, the hardness feature map of the target object can be selected from the elasticity image feature map, and the grayscale feature map of the target object (target grayscale feature map) can be selected from the grayscale image feature map. Then, the elasticity image feature map and the target grayscale feature map are fused to obtain a first fused feature map; the grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map. Since the first fused feature map includes features of the entire elasticity image and the target grayscale features, and the second fused feature map includes features of the entire grayscale image and the target hardness features, both contain multiple target-related features. Therefore, combining the first and second fused feature maps enables more accurate classification of the target object. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of the present invention;

[0052] Figure 2 This is a schematic diagram of tumor classification in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram illustrating the acquisition of the first fused feature map in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram illustrating the acquisition of the second fused feature map in an embodiment of the present invention;

[0055] Figure 5 This is a structural diagram of the breast image processing device provided by the present invention;

[0056] Figure 6 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0058] Research has found that acquiring grayscale images of the breast (obtained via ultrasound) and then performing machine learning analysis to classify these images can help doctors analyze whether the breast has lesions and the extent of those lesions. However, grayscale images only show the edge information of the breast; classifying grayscale images based solely on this single piece of information can lead to inaccurate classification results.

[0059] To address the aforementioned technical problems, this invention provides a breast image processing method, model, apparatus, device, and storage medium, resolving the issue of inaccurate classification results caused by existing breast image classification methods. Specifically, the invention first extracts features from the breast grayscale image to obtain a grayscale image feature map, and then extracts features from the breast elasticity image to obtain an elasticity image feature map, which characterizes the firmness of the breast. Next, it determines the target grayscale feature map occupied by the same object in the grayscale image feature map and the target firmness feature map occupied in the elasticity image feature map; the same object is defined as the object appearing in both the breast grayscale image and the breast elasticity image. Then, it fuses the elasticity image feature map and the target grayscale feature map to obtain a first fused feature map; finally, it fuses the grayscale image feature map and the target firmness feature map to obtain a second fused feature map; and finally, based on the first and second fused feature maps, it obtains the classification result of the target object. This invention improves the accuracy of classification results.

[0060] For example, if grayscale and elastography images of the same breast are acquired sequentially, a tumor (target object) on the breast might be located in the lower left corner of the grayscale image and the upper right corner of the elastography image. Features are extracted from the entire grayscale image to obtain a grayscale image feature map; features are also extracted from the entire elastography image to obtain an elastography image feature map. The grayscale and elastography image feature maps are then aligned. Alignment means locating the same tumor's position on the grayscale image feature map (denoted as position one) and its position on the elastography image feature map (denoted as position two). The feature map containing position one is segmented from the grayscale image feature map and denoted as the target grayscale feature map; the feature map containing position two is segmented from the elastography image feature map and denoted as the target hardness feature map. The elastography image feature map and the target grayscale feature map are then fused to obtain a first fused feature map; the grayscale image feature map and the target hardness feature map are then fused to obtain a second fused feature map. Analyzing the first and second fusion feature maps yields the classification result for the tumor, which is used to characterize the probability that the tumor is benign.

[0061] Exemplary methods

[0062] The breast image processing method of this embodiment can be applied to a terminal device, which can be a terminal product with image acquisition function, such as a disease diagnostic instrument. In this embodiment, for example... Figure 1 As shown, the breast image processing method specifically includes the following steps:

[0063] S100, extract features from the grayscale image of the breast to obtain a grayscale image feature map, and extract features from the elastic image of the breast to obtain an elastic image feature map, wherein the elastic image feature map is used to characterize the hardness of the breast.

[0064] S200, determine the target grayscale feature map occupied by the same target object on the grayscale image feature map and the target hardness feature map occupied on the elastic image feature map, wherein the same target object is a target object that appears on both the mammary gland grayscale image and the mammary gland elastic image.

[0065] S300, the elastic image feature map and the target grayscale feature map are fused to obtain the first fused feature map.

[0066] S400, the grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map.

[0067] S500, the classification result of the target object is obtained based on the first fusion feature map and the second fusion feature map.

[0068] The method of obtaining the classification result of the target object based on the first fused feature map and the second fused feature map in step S500 is the prior art.

[0069] The breast grayscale image and breast elastography image involved in steps S100 to S500 are both unregistered images. Unregistered means that the same target contained in the image is located at different positions in the two images. For example, the target is located in the middle of the breast grayscale image, but the target is located in a non-middle position in the breast grayscale image.

[0070] In one embodiment, such as Figure 2 As shown, step S100 involves using two parallel convolutional neural networks (CNNs) to extract feature maps F from the grayscale image of the breast. g (Feature map of grayscale image) and feature map of breast elasticity image F e (Elastic image feature map)

[0071] In one embodiment, before performing step S200, such as Figure 2 As shown, the feature map F is aligned using the class activation map CAM. g and feature map F eAlignment, in this context, refers to locating the target object (tumor) within the feature map F. g The position of the target object on the feature map F and the location of the target object e The position above. The principle of CAM is explained as follows:

[0072] For feature map F g and feature map F e Two fusion networks are applied separately. First, a 10-cycle warm-up phase is performed using a feature matrix consisting entirely of 1s. After the 10-cycle warm-up phase (after which the activation map gradually converges to the tumor target region), the two fusion networks are then applied to the feature map F. g and feature map F e The output features are concatenated to obtain the feature map F. k The class activation map (CAM) is obtained using the following formula:

[0073]

[0074]

[0075] CAM g Class activation map corresponding to the grayscale image of the breast, CAM e Class activation map corresponding to breast elasticity image.

[0076] In the formula, w k These are the weights corresponding to the output of the final convolutional layer, which represent the importance of the feature map for a specific category.

[0077] (x1, y1) = argmax(CAM) g )

[0078] (x2, y2) = argmax(CAM) e )

[0079] Constructing an affine network based on class activation graphs (CAM):

[0080]

[0081] t x =x2-x1

[0082] t y =y2-y1

[0083] (x1,y1) and (x2,y2) are the same marker in feature map F. g and feature map F e The coordinates on the feature map F, where (x, y) are the coordinates of the target object. g If the coordinates on the feature map F are given, then (x′, y′) represents the target object's position on the feature map F. gThe coordinates after the up-transformation; when (x,y) is the target object in the feature map F e If the coordinates on the feature map F are given, then (x′, y′) represents the target object's position on the feature map F. e The coordinates after the up-transformation. Based on (x′, y′), the target object can be located in feature map F. e Or feature map F g It's in the upper position.

[0084] In one embodiment, such as Figure 3 As shown, step S300 includes the following specific steps S301 to S308:

[0085] S301, Extract the target grayscale feature map F g目标 The target's horizontal edge features and vertical edge features.

[0086] like Figure 3 As shown, the feature map F is processed through an affine network. g Perform affine transformation ( Figure 2 In this context, ATB stands for affine transformation, used to capture the target object in the feature map F. g The position of F makes F g The location of the target on F e Align the target object's location on the map. In other words, alignment allows us to determine the feature map F. g The grayscale feature map F of the target object belongs to the target object. g目标 .

[0087] Obtain the target grayscale feature map F g目标 Then, it is processed using a Gaussian first-order partial derivative kernel, respectively from F g目标 Extract the horizontal edge features F of the target g目标 *Gx and the vertical edge features of the target F g目标 *G y .

[0088] in,

[0089]

[0090] The filter size determines the number of pixels considered in each convolution, while σ is the standard deviation of the Gaussian function, which controls the filter scale. Size can be 3, 5, or 7, and σ can be 0.05, 0.5, 1, 5, or 10.

[0091] S302, merge the target's horizontal edge features and target's vertical edge features to obtain the target's edge features on the grayscale image feature map, denoted as target grayscale edge feature F. edge (x,y).

[0092] Edge feature extraction in Figure 2 It is performed in the EEM module.

[0093] Step S301 yields the horizontal edge feature F g目标 *G x and vertical edge features F g目标 *G y Then, merging these two edge features yields the edge features F of the target object on the grayscale image feature map. edge (x,y):

[0094]

[0095] R is the field of real numbers.

[0096] S303, the target grayscale edge feature F edge (x,y) multiplied by the set first weight matrix W Q1 To obtain the query vector Q of the cross-attention. edge .

[0097] Q edge =F edge W Q1

[0098] S304, the elastic image feature map F e Multiply by the set second weight matrix W K1 To obtain the key vector K of the cross attention. e .

[0099] K e =F e W K1

[0100] S305, multiply the elastic image feature map by the set third weight matrix W. V1 To obtain the value vector V of the cross attention. e .

[0101] V e =F e W V1

[0102] W Q1 W K1 and W V1 ∈R N×C×dk It is a weight matrix.

[0103] S306, the query vector Q edge Multiplied by the transpose of the key vector K e TThe product result vector Q is obtained. edge K e T .

[0104] S307, Determine the key vector K e vector dimension d k The product result vector Q edge K e T Divided by the root mean square of the vector dimension Obtain the intermediate vector

[0105] S308, normalize the intermediate vector Multiply by the value vector V e The first fused feature map F is obtained. fusion1 :

[0106]

[0107] In one embodiment, such as Figure 4 As shown, step S400 includes the following specific steps S401 to S404:

[0108] S401, for the elastic image feature map F e Perform a convolution operation to obtain the convolved elastic image feature map Conv(F). e ).

[0109] S402, Connect the target hardness feature map C moments The elastic image feature map Conv(F) after convolution e ), to obtain the connection feature map cat[Conv(F e ),C moments ].

[0110] S403, for the connection feature map cat[Conv(F e ),C moments Perform a convolution operation to obtain the hardness distribution characteristics of the target object, denoted as the target hardness distribution characteristic F. color :

[0111] F color =Conv(cat[Conv(F e ),C moments ])

[0112] Target hardness distribution characteristics F color Extraction in Figure 2 It is performed in the CEM module.

[0113] S404, apply a cross-attention mechanism to the target hardness distribution features and the grayscale image feature map to calculate (the cross-attention mechanism is in...) Figure 2 (Executed in MAF) to obtain the second fused feature map F fusion2 .

[0114] The detailed process of step S404 is as follows:

[0115] Q color =F color W Q2

[0116] K g =F g W K2

[0117] V g =F g W V2

[0118]

[0119] In the formula, W Q2 W K2 and W V2 ∈R N×C×dk The weight matrix.

[0120] In one embodiment, from the elastic image feature map F e Extracting the target hardness feature map C moments The specific process is as follows: Determine each target pixel point x corresponding to the target object on the breast elasticity image. ij Calculate the mean C of each target pixel. mean Variance C var and deviation C skew The mean, variance, and deviation are connected using a join function to obtain the target hardness feature map C. moments .

[0121]

[0122]

[0123]

[0124] C moments =cat[C mean C var C skew ]

[0125] In one embodiment, a trained model is provided, the trained model ( Figure 2The portion enclosed in the large rectangle represents the model used to implement the aforementioned breast image processing method. During the training of this model, the loss function L used for training needs to be continuously optimized. focal (p,y):

[0126]

[0127] p represents the probability of a tumor being benign, output by the model for the grayscale and elastography images of the breast tissue in the training samples. y = 1 indicates that the tumor is actually benign, and y = 0 indicates that the tumor is actually malignant. α and γ are two adjustment factors that need to be optimized in the loss function.

[0128] The number of positive and negative samples (the grayscale and elastic images corresponding to malignant tumors are negative samples, and the grayscale and elastic images corresponding to benign tumors are positive samples) can be adjusted according to the size of p. Adjusting the number of positive and negative samples has the following effects:

[0129] Imbalanced distributions of benign and malignant data (samples) can make model training more challenging. When data is severely imbalanced, models often bias towards the majority class because this yields higher accuracy on the training data. Various strategies can be employed to address data imbalance, such as resampling (oversampling or undersampling), using class weights, generating synthetic samples (e.g., the SMOTE method), and calculating weighted loss functions.

[0130] For binary classification problems (beneficial and malignant classifications), a weight can be assigned to each class, with the majority class having a lower weight and the minority class having a higher weight. When calculating the loss, the loss for each sample is multiplied by the weight of its corresponding class. This makes the model pay more attention to the minority class when optimizing the loss function.

[0131] The optimization process for α and γ is as follows: During training, the accuracy PRE, sensitivity SEN, and classification specificity SPEC of the model are statistically analyzed; based on the accuracy PRE, sensitivity SEN, and classification specificity SPEC, the reward value R for reinforcement learning is obtained; based on the current adjustment factor α′ of the loss function, the Q value for reinforcement learning is obtained; based on the Q value and the reward value, the adjustment factor of the loss function is optimized using reinforcement learning.

[0132] R = 0.5 × F1 + 0.5 × Youden

[0133]

[0134] Youden = SEN + SPEC - 1

[0135]

[0136]

[0137] w is the weight of DQN, β is the learning rate, R is the reward, λ is the discount factor, and s′ represents the new state generated after taking action a in state s.

[0138] After obtaining L(w), L(w) is input into a known neural network (existing technology). Based on the output value of the neural network, the combination value of α and γ is selected from the preset set of α [-0.1, -0.05, 0, 0.05, 0.1] and the preset set of γ [-0.1, -0.05, 0, 0.05, 0.1].

[0139] In one embodiment, the accuracy of the proposed breast image processing method for classifying benign and malignant tumors is evaluated using the following test data.

[0140] Test data: 357 patients, including 312 with benign tumors and 45 with malignant tumors. This dataset was divided into 284 training samples and 73 test samples. For 5-fold cross-validation, the 284 training samples were divided into five different groups, with a training-to-validation ratio of 4:1 for each group. All images were initially resized to 384x384 pixels. The training images were further randomly cropped to 256x256 pixels, while the test images were cropped to the same size. Finally, the training data was randomly rotated to obtain the rotated training data.

[0141] The breast image processing method of this invention was run on five TITAN RTX GPUs equipped with 24GB of memory, with the input image patch size set to [256, 256], the batch size set to 16, and the initial learning rate set to 5 × 10⁻⁶. -4 In addition, the learning rate decay weight is set to 0.98, the decay period is set to 5, and the maximum period is set to 5000.

[0142] Accuracy (ACC), Precision (PRE), F1 score (F1), Youden score, and area under the curve (AUC) are used as classification evaluation metrics. These metrics are used to evaluate the accuracy of the model of this invention in classifying benign and malignant diseases.

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] Youden = SEN + SPEC - 1

[0149] In the formula, TP (true positive), TN (true negative), FP (false positive), and FN (false negative) represent the number of true positive, true negative, false positive, and false negative samples, respectively. TP indicates the number of pixels predicted as positive (i.e., benign tumors) and actually labeled as positive in the tumor classification task; TN (True Negative) indicates the number of pixels predicted as negative (i.e., malignant tumors) and actually labeled as negative in the tumor classification task; FP (False Positive) indicates the number of pixels predicted as positive and actually labeled as negative in the tumor classification task; and FN (False Negative) indicates the number of pixels predicted as negative and actually labeled as positive in the tumor classification task.

[0150] In summary, this invention acquires grayscale and elastic images of the breast, extracts feature maps from the grayscale and elastic images, and since the grayscale and elastic images contain the same target object, the hardness feature map of the target object can be selected from the elastic image feature map, and the grayscale feature map of the target object (target grayscale feature map) can be selected from the grayscale image feature map. Then, the elastic image feature map and the target grayscale feature map are fused to obtain a first fused feature map; the grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map. Because the first fused feature map includes features of the entire elastic image and the target's grayscale features, and the second fused feature map includes features of the entire grayscale image and the target's hardness features, both contain multiple target-related features. Therefore, combining the first and second fused feature maps enables more accurate classification of the target object.

[0151] Exemplary device

[0152] This embodiment also provides a breast image processing device, such as... Figure 5 As shown, the device comprises the following components:

[0153] The feature extraction module 01 is used to extract features from the grayscale image of the breast to obtain a grayscale image feature map, and to extract features from the elastic image of the breast to obtain an elastic image feature map, wherein the elastic image feature map is used to characterize the hardness feature of the breast.

[0154] The feature map calculation module 02 is used to determine the target grayscale feature map occupied by the same target object on the grayscale image feature map and the target hardness feature map occupied on the elastic image feature map. The same target object is a target object that appears on both the mammary gland grayscale image and the mammary gland elastic image.

[0155] The first fusion module 03 is used to fuse the elastic image feature map and the target grayscale feature map to obtain a first fused feature map;

[0156] The second fusion module 04 is used to fuse the grayscale image feature map and the target hardness feature map to obtain a second fused feature map;

[0157] The classification module 05 is used to obtain the classification result of the target object based on the first fusion feature map and the second fusion feature map.

[0158] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 6 As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a breast image processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0159] Those skilled in the art will understand that Figure 6 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a breast image processing program stored in the memory and executable on the processor. When the processor executes the breast image processing program, it implements the following operation instructions:

[0161] Features are extracted from the grayscale image of the breast to obtain a grayscale image feature map, and features are extracted from the elastic image of the breast to obtain an elastic image feature map, which is used to characterize the hardness of the breast.

[0162] Determine the target grayscale feature map occupied by the same target object in the grayscale image feature map and the target hardness feature map occupied in the elasticity image feature map, wherein the same target object is a target object that appears in both the mammary gland grayscale image and the mammary gland elasticity image;

[0163] The elastic image feature map and the target grayscale feature map are fused to obtain a first fused feature map;

[0164] The grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map;

[0165] The classification result of the target object is obtained based on the first fusion feature map and the second fusion feature map.

[0166] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing breast images, characterized in that, include: Features are extracted from the grayscale image of the breast to obtain a grayscale image feature map, and features are extracted from the elastic image of the breast to obtain an elastic image feature map, which is used to characterize the hardness of the breast. Determine the target grayscale feature map occupied by the same target object in the grayscale image feature map and the target hardness feature map occupied in the elasticity image feature map, wherein the same target object is a target object that appears in both the mammary gland grayscale image and the mammary gland elasticity image; The elastic image feature map and the target grayscale feature map are fused to obtain a first fused feature map; The grayscale image feature map and the target hardness feature map are fused to obtain a second fused feature map; Based on the first fusion feature map and the second fusion feature map, the classification result of the target object is obtained; The process of fusing the elastic image feature map and the target grayscale feature map to obtain a first fused feature map includes: Extract the horizontal and vertical edge features of the target from the target grayscale feature map; The horizontal edge features and vertical edge features of the target are combined to obtain the edge features of the target on the grayscale image feature map, which are denoted as the target grayscale edge features; The target grayscale edge features are multiplied by a set first weight matrix to obtain the query vector for cross-attention; The elastic image feature map is multiplied by a set second weight matrix to obtain the cross-attention key vector; The elastic image feature map is multiplied by a set third weight matrix to obtain the cross-attention value vector; Multiply the query vector by the transpose of the key vector to obtain the product vector; Determine the vector dimension of the key vector, and divide the resulting vector of the product by the root mean square of the vector dimension to obtain the intermediate vector; The normalized intermediate vector is multiplied by the value vector to obtain the first fused feature map.

2. The breast image processing method as described in claim 1, characterized in that, The determination of the target grayscale feature map occupied by the same target object in the grayscale image feature map and the target hardness feature map occupied in the elasticity image feature map, wherein the same target object is a target object that appears in both the breast grayscale image and the breast elasticity image, includes: By applying a class activation map to the grayscale image feature map and the elastic image feature map, a first target region of the same target object on the grayscale image feature map and a second target region on the elastic image feature map are obtained. Based on the first target region and the grayscale image feature map, a target grayscale feature map is obtained; Based on the second target region and the elastic image feature map, the target hardness feature map is obtained.

3. The breast image processing method as described in claim 1, characterized in that, The process of fusing the grayscale image feature map and the target hardness feature map to obtain a second fused feature map includes: Perform a convolution operation on the elastic image feature map to obtain the convolved elastic image feature map; By connecting the target hardness feature map and the convolved elastic image feature map, a connected feature map is obtained; Perform a convolution operation on the connection feature map to obtain the hardness distribution feature of the target object, which is denoted as the target hardness distribution feature; A second fused feature map is obtained by applying a cross-attention mechanism to the target hardness distribution features and the grayscale image feature map.

4. The breast image processing method as described in claim 1, characterized in that, The extraction method of the target hardness feature map includes: Determine each target pixel point corresponding to the target object on the breast elastography image; Calculate the mean, variance, and bias of each target pixel. The mean, variance, and deviation are connected using a join function to obtain a target hardness feature map.

5. A model for processing breast images, used to implement the breast image processing method according to any one of claims 1-4, characterized in that, The model is a trained model, and the optimization methods of the loss function applied in the training process of the model include: During training, the accuracy, sensitivity, and classification specificity of the model are statistically analyzed. The reward value for reinforcement learning is obtained based on the accuracy, sensitivity, and classification specificity. Based on the current adjustment factor of the loss function, the Q value of reinforcement learning is obtained; Based on the Q-value and the reward value, the adjustment factor of the loss function is optimized using reinforcement learning.

6. A breast image processing device, characterized in that, The device comprises the following components: The feature extraction module is used to extract features from the grayscale image of the breast to obtain a grayscale image feature map, and to extract features from the elastic image of the breast to obtain an elastic image feature map, wherein the elastic image feature map is used to characterize the hardness of the breast. The feature map calculation module is used to determine the target grayscale feature map occupied by the same target object on the grayscale image feature map and the target hardness feature map occupied on the elastic image feature map, wherein the same target object is a target object that appears on both the mammary gland grayscale image and the mammary gland elastic image; The first fusion module is used to fuse the elastic image feature map and the target grayscale feature map to obtain a first fused feature map; The second fusion module is used to fuse the grayscale image feature map and the target hardness feature map to obtain a second fused feature map; The classification module is used to obtain the classification result of the target object based on the first fusion feature map and the second fusion feature map; The process of fusing the elastic image feature map and the target grayscale feature map to obtain a first fused feature map includes: Extract the horizontal and vertical edge features of the target from the target grayscale feature map; The horizontal edge features and vertical edge features of the target are combined to obtain the edge features of the target on the grayscale image feature map, which are denoted as the target grayscale edge features; The target grayscale edge features are multiplied by a set first weight matrix to obtain the query vector for cross-attention; The elastic image feature map is multiplied by a set second weight matrix to obtain the cross-attention key vector; The elastic image feature map is multiplied by a set third weight matrix to obtain the cross-attention value vector; Multiply the query vector by the transpose of the key vector to obtain the product vector; Determine the vector dimension of the key vector, and divide the resulting vector of the product by the root mean square of the vector dimension to obtain the intermediate vector; The normalized intermediate vector is multiplied by the value vector to obtain the first fused feature map.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a breast image processing program stored in the memory and executable on the processor. When the processor executes the breast image processing program, it implements the steps of the breast image processing method as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a breast image processing program, which, when executed by a processor, implements the steps of the breast image processing method as described in any one of claims 1-4.

Images