Multi-mode ultrasonic mammary gland and multi-mode fusion method

Abstract

The invention relates to the technical field of medical image processing, and discloses a multi-mode ultrasonic mammary gland and multi-mode fusion method. The method comprises the following steps: S1, acquiring medical images of at least two modalities of a mammary gland area, wherein the medical images comprise a first modal image and a second modal image; s2, performing standardization preprocessing on the first modal image and the second modal image; and S3, constructing a physical modeling model of the breast tissue based on the second modal image, and generating a tissue deformation displacement field to obtain tissue response information. By adopting a multi-modal ultrasonic mammary gland image registration and fusion technology and combining feature fusion and deformation field optimization based on an image neural network, the image registration precision is remarkably improved. Compared with a traditional image registration method in the prior art, the multi-modal image registration method not only optimizes the alignment effect among the multi-modal images, but also effectively reduces registration errors caused by modal differences, and particularly provides more accurate alignment on detail structures of breast tissues.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multi-mode ultrasound breast imaging and multi-mode fusion method. Background Art

[0002] Conventional ultrasound breast image registration methods typically rely on simple image alignment algorithms, such as feature point matching or region-based registration. These methods often overlook the underlying differences between images of different modalities, limiting registration accuracy. Especially in cases of complex breast tissue structure and significant modality differences, these methods are prone to significant errors, making it difficult to ensure the accuracy of registration results. Existing methods also suffer from an inability to effectively handle the nonlinear deformation of multimodal images, particularly with regard to detail preservation.

[0003] Traditional image registration methods lack modeling of the physical properties of breast tissue and typically rely solely on image grayscale values ​​or texture features for registration. However, the structural and elastic properties of breast tissue vary significantly between different modal images. Existing technologies fail to fully account for this variability, resulting in registration results that are susceptible to the influence of imaging equipment and acquisition angles, failing to accurately reflect the true physical characteristics of breast tissue. This lack of physically sound modeling makes image registration unsuitable for practical applications in some scenarios.

[0004] In existing multimodal image fusion techniques, registration and feature fusion are typically performed separately. After image registration, feature fusion relies solely on the alignment of image pixels. However, this approach ignores structural consistency at the tissue level, resulting in the registered image failing to fully preserve the individual strengths of the multimodal images. Existing techniques are prone to structural distortion during the fusion process, particularly in the representation of breast tissue boundaries and texture details. Consequently, existing techniques suffer from significant deficiencies in structural consistency and the accuracy of multimodal information fusion. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a multimodal ultrasound breast registration and multimodal fusion method, which solves the problems of low registration accuracy, ineffective processing of modal differences, and image fusion structure distortion existing in the existing multimodal ultrasound breast image registration methods.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-mode ultrasound breast imaging and multi-mode fusion method, comprising the following steps; S1. Acquire at least two modal medical images of a breast region, including a first modality image and a second modality image; S2. performing standardization preprocessing on the first modality image and the second modality image; S3. constructing a physical model of breast tissue based on the second modality image, generating a tissue deformation displacement field to obtain tissue response information; S4. Constructing a graph structure to represent the image region, and extracting regional features of the first modality image and the second modality image respectively; S5. Based on graph structure and tissue response information, graph neural network is used to perform structural fusion of multimodal image features; S6. Based on the fused features, guide the registration network to predict the deformation field of the second modality image and perform image registration on it; S7. Optimize the registration and fusion results based on the joint loss function and output the fused image.

[0007] Preferably, the first modality image and the second modality image in step S1 include B-ultrasound images and elastic ultrasound images collected using ultrasound equipment, wherein the B-ultrasound image is used to display the structural characteristics of breast tissue, and the elastic ultrasound image is used to obtain elasticity information of breast tissue.

[0008] Preferably, in step S1, the first modality image and the second modality image adopt the same scanning position, and the two modality images are aligned by an image registration algorithm so that the two images have a consistent spatial distribution in the spatial coordinate system.

[0009] Preferably, the step S2 specifically includes the following steps: performing size normalization processing on the first modality image and the second modality image so that the sizes of the first modality image and the second modality image are consistent; The grayscale values ​​of the first modality image and the second modality image are normalized to ensure that the images are within the same grayscale range; the first modality image and the second modality image are denoised to remove the noise introduced by the imaging device to ensure the clarity and accuracy of the image.

[0010] Preferably, the physical model in step S3 is an elastic mechanics model established based on the finite element method: determining an initial position of each node of the breast tissue according to the local deformation region of the second modality image; Set boundary conditions and define the elastic modulus and Poisson's ratio of breast tissue to calculate the mechanical response; Based on the boundary conditions, the deformation displacement field of each node is calculated using a finite element solver.

[0011] Preferably, the displacement field is further used to calculate the local structural response field, and the response field is calculated by the following formula: Among them, S(x,y) represents the response field at a specific position (x,y), u(x,y) represents the displacement field at the point (x,y), represents the gradient of the displacement field, ||·|| F is the representation of the Frobenius norm.

[0012] Preferably, constructing the graph structure in step S4 specifically includes the following steps: Dividing the first modality image and the second modality image into a plurality of small regions, each of which is a node in a graph structure; Extract node features based on pixel intensity, texture features, and shape information of each small area; The adjacency relationship between nodes is established in the graph structure, and the adjacency relationship is calculated based on the similarity between adjacent regions, where the similarity measurement is based on the convolutional features and local feature vectors of the region.

[0013] Preferably, the graph neural network in step S5 adopts a graph neural structure based on an attention mechanism, and its edge weights are adaptively calculated according to the feature similarity between regions and the corresponding structural response field. The specific calculation method is as follows; Among them, α ij is the attention weight between node i and node j, indicating the degree of attention of node i to node j, exp(·) exponential function, is the dot product of the feature vectors of node i and node j, ∑ is the summation symbol, k is node k, which is a neighbor node of the current node i, N(i) is the set of neighbor nodes of node i, that is, the nodes connected to i, h i ,h k is the feature vector of node i and node k, and T is the transposed symbol.

[0014] Preferably, the registration network in step S6 is a convolutional neural network, which gradually refines the deformation field of the predicted image by performing multiple convolution operations on the fused feature image. The deformation field is calculated by the following formula: in, represents the displacement vector at the node (x,y), Represents the displacement component of the node (x, y) in the horizontal direction, Represents the displacement component of the node (x, y) in the vertical direction.

[0015] Preferably, the joint loss function in step S7 specifically includes the following steps: Image similarity loss, which calculates the difference between the fused image and the target image; Loss of smoothness of deformation field, smoothing the deformation field through regularization method to reduce local irregularities; Multimodal regional feature consistency loss ensures consistency in regional features between the first and second modality images; structural response guidance loss aligned with physical modeling results ensures that the calculated results are consistent with the response information generated by physical modeling of breast tissue; The joint loss function is expressed as; in, represents the total loss function, represents the similarity loss, represents the smoothing loss, represents structural loss, represents the response loss, and λ1, λ2, λ3, and λ4 represent weighting coefficients.

[0016] The present invention provides a multi-mode ultrasound breast imaging and multi-mode fusion method. It has the following beneficial effects: 1. This invention significantly improves image registration accuracy by employing multimodal ultrasound breast image registration and fusion technology, combined with feature fusion and deformation field optimization based on graph neural networks. Compared with traditional image registration methods in the prior art, this invention not only optimizes the alignment between multimodal images but also effectively reduces registration errors caused by modality differences, providing more precise alignment of detailed breast tissue structures.

[0017] 2. This invention effectively enhances the physical rationality of the registration process by introducing a breast tissue response field based on physical modeling and using it as input features for the graph neural network. Unlike existing registration methods that lack physical constraints, this invention addresses the inability of traditional methods to accurately reflect the elastic properties of breast tissue. It improves the consistency of breast tissue responses across different modalities, thereby enhancing the stability and quality of the fused images.

[0018] 3. This invention utilizes a multi-factor joint loss function, comprehensively considering image similarity, deformation field smoothness, structural consistency, and response feature alignment. Compared to existing loss function designs, this invention incorporates multimodal region consistency loss and response guidance loss into the optimization process, avoiding the amplification of local errors and ensuring more precise regional alignment between images of different modalities, thereby enhancing the quality and reliability of the final fused image.

[0019] 4. By incorporating the attention mechanism within graph neural networks, this paper effectively optimizes the modeling of relationships between nodes within the graph structure, enabling the network to more accurately capture the deep connections between images of different modalities. Compared to existing image registration methods based on traditional convolutional neural networks, this technical solution not only achieves refined feature fusion at a detailed level, but also improves the model's adaptability to changes in image structure, further mitigating the impact of feature differences between modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Please see the attached Figure 1 , an embodiment of the present invention provides a multi-mode ultrasound breast imaging and multi-mode fusion method, comprising the following steps: S1, acquiring at least two modal medical images of a breast region, including a first modal image and a second modal image; Specifically, the first step in this embodiment is to acquire at least two modal medical images of the breast region. Typically, these images include a first modality image and a second modality image. Acquiring medical images of different modalities is a prerequisite for multimodal fusion of breast images. These images, derived from different imaging technologies, can provide different types of tissue characteristic information for breast tissue.

[0023] Typically, the first modality image is a conventional B-ultrasound image, and the second modality image is an elastic ultrasound image. B-ultrasound images are used to display the structural characteristics of breast tissue, such as lumps, glandular tissue, and other important structures. Elastic ultrasound images, on the other hand, are used to obtain information about the elasticity of breast tissue, reflecting the firmness or rigidity of the tissue, and can help identify different types of tissue lesions, such as tumors.

[0024] B-ultrasound images: Ultrasound equipment is used to capture B-ultrasound images of the breast area. B-ultrasound images clearly show the structural characteristics of breast tissue and are particularly useful for initial screening for breast cancer. Generally, B-ultrasound images are obtained non-invasively using an ultrasound probe.

[0025] Elastic ultrasound images: Elastic ultrasound equipment is used to acquire elastic ultrasound images of breast tissue. Elastic ultrasound technology assesses tissue elasticity or rigidity by measuring tissue deformation and is often used to determine tissue hardness or tumor boundaries. Acquiring elastic ultrasound images relies on applying a certain external force to the breast tissue and detecting the tissue's response to produce the image.

[0026] First, the two modal image acquisition devices are set to the same scanning position. This setup ensures spatial consistency between the two images, making subsequent image registration more accurate. To further improve the accuracy of spatial alignment between the images, an image registration algorithm is used to align the two modal images. This method ensures that the two images have a consistent distribution in the spatial coordinate system, avoiding image registration errors caused by slight differences in scanning angle or position.

[0027] Specifically, during image registration, either feature-based or region-based registration algorithms are used. These algorithms match feature points or regions within the images to align the two modalities. This ensures that critical tissue regions within the images are aligned in the same coordinate system, providing a precise foundation for subsequent fusion and analysis.

[0028] In this embodiment, step S1 accurately acquires and registers breast images of different modalities, thereby laying a solid foundation for subsequent image processing, feature fusion, and analysis.

[0029] S2. performing standardization preprocessing on the first modality image and the second modality image; Specifically, in step S1 of this embodiment, we have acquired two modal images of the breast region and ensured their spatial consistency through image registration. The next step is to perform standardization preprocessing on these two modal images, which is a key step in image fusion and subsequent processing. The purpose of standardization preprocessing is to ensure that the data of the two images can be compared and processed at the same scale, thereby improving the quality of image fusion and the accuracy of subsequent analysis.

[0030] Generally, image normalization preprocessing includes grayscale normalization, size normalization, and noise removal. These processes effectively reduce background noise, grayscale variations, and intermodal influences that may occur in the image, improving image stability and consistency. Normalized images not only enhance the robustness of image processing algorithms but also provide more accurate input for subsequent tasks such as image fusion and feature extraction.

[0031] First, grayscale normalization is performed on both the first and second modality images. Grayscale normalization compresses the pixel values ​​in an image to a uniform range, typically between [0, 1] or [0, 255]. Normalized images effectively eliminate grayscale differences caused by factors such as imaging equipment and imaging methods, allowing comparison of data from different modalities at the same scale.

[0032] Specifically, let the pixel value of the original image be I(x,y), and the normalized image be I norm (x,y), its normalized formula is; in: I min and I max are the minimum and maximum pixel values ​​in the image respectively; I norm (x,y) is the normalized pixel value.

[0033] Through this formula, all pixel values ​​are mapped to the range of [0,1], thus eliminating the grayscale deviation caused by different imaging devices.

[0034] In some embodiments, the second modality image may involve different imaging techniques or equipment. For example, if the second modality image is an ultrasound elasticity image, its grayscale distribution may differ from that of a conventional B-ultrasound image. Therefore, during image normalization, the grayscale range of the ultrasound elasticity image may require separate normalization based on the characteristics of the equipment used.

[0035] Size Normalization: To ensure consistent image size, both the primary and secondary modality images are typically size-normalized. This process involves resizing both images to a uniform, standard size through interpolation, typically using bilinear or cubic interpolation. This size normalization ensures spatial consistency between the images, ensuring that subsequent operations such as image fusion, feature extraction, and analysis are unaffected by size differences.

[0036] Noise Removal: Image noise can affect subsequent processing accuracy, particularly in medical images, where background noise can obscure true structural features. Therefore, denoising algorithms, such as median filtering, Gaussian filtering, or wavelet transform denoising, are often employed during the standardization process. These methods effectively remove high-frequency noise from images while retaining useful low-frequency information, thereby improving image quality.

[0037] In one possible implementation, the noise removal algorithm can be based on Gaussian filtering: The original image is I(x,y), and the image after Gaussian filtering is I gf(x,y), the calculation formula is: I gf (x,y)=∑ i,j I(x+i,y+j)·G(i,j,σ); in: I gf (x,y) represents the pixel value at position (x,y) of the image after Gaussian filtering, which is the result of smoothing; ∑ i,j I performs the sum operation on all pixel offsets (i, j) within the convolution kernel window; I(x+i,y+j) represents the grayscale value of the neighborhood pixel at position (x+i,y+j) in the original image; G(i,j,σ) represents the value of the two-dimensional Gaussian kernel function at position (i,j), which is the weight of the corresponding position; σ represents the standard deviation of the Gaussian kernel and controls the smoothness of the filter.

[0038] Gaussian filtering can effectively smooth images and remove high-frequency noise without significantly affecting the structural information of the image.

[0039] The standardization preprocessing step in this embodiment makes the first modality image and the second modality image tend to be consistent in spatial scale and data distribution through operations such as normalization, resizing, and noise removal.

[0040] S3. constructing a physical model of breast tissue based on the second modality image, generating a tissue deformation displacement field to obtain tissue response information; Specifically, in this embodiment, after completing the standardized preprocessing of the first and second modality images, a physically plausible tissue modeling mechanism is necessary to achieve structurally guided image registration and fusion. This describes the elastic response characteristics of the breast region under load. This step not only provides the response information foundation for subsequent graph construction and multimodal fusion, but also provides physical support for enhancing the registration network's understanding of actual tissue deformation.

[0041] In this embodiment, a two-dimensional elastic mechanics model of breast tissue is established based on the tissue elasticity distribution reflected in the second modal image. The model is constructed and solved based on the finite element method and can more realistically reflect the deformation behavior of the tissue under stress.

[0042] Specifically, the topology of the finite element mesh within the breast tissue is first determined based on the tissue boundaries and local deformation regions displayed in the second modality image I(x,y). Common methods include using a regular rectangular grid or a Delaunay triangulation method based on region boundaries for discretization, forming multiple finite element units with unit properties, each corresponding to a set of pixels within the image region.

[0043] The initial node position (x0, y0) of each element can be determined by the center point in the image coordinate system, or the geometric centroid of the region can be used as the node reference point. In one possible implementation, to ensure that the boundary conditions conform to the actual imaging conditions, the bottom or peripheral area of ​​the image can be set as a fixed boundary, and a unit normal load can be applied to the upper boundary to simulate the mechanical pressure probe process.

[0044] Furthermore, the material properties of the tissue to which each cell belongs are defined, including the elastic modulus E and Poisson's ratio ν. These parameters can be assigned values ​​based on the pixel intensity in the second modality image, for example, assigning higher E values ​​to brighter areas (representing hard tissue) and lower E values ​​to darker areas (representing soft tissue).

[0045] The elastic modulus E is used to reflect the rigidity of breast tissue; Poisson's ratio ν represents the ratio of the transverse to longitudinal deformation of a material.

[0046] Then, according to the elastic mechanics equilibrium equation, the stiffness matrix K and load vector f of the system are constructed, and the following linear equations are solved.

[0047] Ku=f; in: K system stiffness matrix, generated based on element type and node connection method; u global displacement vector, which contains the deformation displacement of each degree of freedom in the x and y directions; f is the load vector, which defines the externally applied unit force. Its magnitude and direction are set according to the simulation conditions.

[0048] In some embodiments, to suppress high-frequency noise or non-physical solutions, a Gaussian kernel is used to perform smooth regularization processing on the displacement results to improve the continuity and numerical stability of the displacement field.

[0049] Based on this displacement field, in order to further explore the local response behavior of the tissue structure, the present invention introduces the definition of the response field and uses the gradient norm of the displacement field as the response quantity calculation indicator. In a preferred embodiment, the following response field formula is used for calculation: in: S(x,y) represents the response field at a specific location (x,y); u(x,y) represents the displacement field at the point (x,y). The displacement field is usually used to describe the degree to which a point on an object is offset from its original position. Represents the gradient of the displacement field, which shows the rate of change of displacement in space and describes the change of displacement in the x and y directions; ||·|| F is the representation of the Frobenius norm.

[0050] The above response field is not only used to characterize the local strain degree of each region under mechanical deformation, but also provides a basis for weighting the importance of nodes in the subsequent graph structure, and can be used as a reference indicator for edge weight adjustment in graph neural networks.

[0051] Step S3 establishes an elastic mechanics model based on the finite element method and calculates the deformation displacement field and response field, which not only improves the physical rationality of image registration but also provides a solid theoretical basis for structure-guided image feature fusion.

[0052] S4. Constructing a graph structure to represent the image region, and extracting regional features of the first modality image and the second modality image respectively; Specifically, in this embodiment, after completing the construction of the tissue-level response field, the displacement information and response value in the image pixel space alone are still insufficient to effectively support the fine registration and deep fusion of multimodal images. Since the structural boundaries and texture features of breast tissue in different modal images are significantly different, there is an urgent need for a unified expression method that fuses image spatial information and mechanical response information. To this end, in step S4, the present invention further constructs a graph structure expression mechanism driven by a fusion response field to enhance the consistency of the structural hierarchy between multimodal images and provide a structural basis for subsequent graph matching and feature fusion.

[0053] Traditional graph registration methods typically use image edges, corners, or intensity extremes as graph nodes, making it difficult to reflect tissue-level structural behavior. Alternatively, this embodiment incorporates displacement and response field results into the design of graph nodes and edge weights, enabling the graph structure to reflect the true tissue response characteristics and local structural change trends.

[0054] In this embodiment, the breast tissue region in the second modality image I(x, y) is firstly used as a basis, and a graph structure is constructed according to the calculation results of the displacement field and the response field. The graph structure is denoted as in: A collection of nodes representing a graph: ε represents the set of edges between nodes in the graph.

[0055] For the edge ε in the graph structure, the spatial position relationship between nodes and the consistency of response gradients are considered during the construction process. In one implementation, a k-nearest neighbor connection strategy is adopted to connect each node to its k neighboring nodes in Euclidean space. The edge weight w ij It is defined comprehensively based on the response similarity and displacement difference between nodes i and j, for example; in: w ij represents the edge weight between nodes i and j; u i represents the displacement vector of node i; S i represents the response value of node i; σ u Parameters representing the effects of the control displacement difference; σ s Parameters that represent the effects of poor control response; ||u i -u j || 2 represents the square of the displacement difference; |S i -S j | 2 Represents the square of the response difference.

[0056] The above-mentioned weight design enables the graph structure to introduce mechanical response features while maintaining spatial connectivity, which is conducive to focusing on deformation information in the subsequent fusion stage.

[0057] As an extension method, after constructing the graph structure, the Laplace matrix of the graph can be further calculated. Used for frequency domain analysis of graph signals or as the input structure of graph convolutional neural networks. The graph Laplacian matrix can be obtained by the standard definition: in: Represents the weighted adjacency matrix of the graph, whose elements are edge weights w ij ; Represents the degree matrix of the graph, which is a diagonal matrix.

[0058] This structure-enhanced graph construction method not only retains the spatial distribution information of the image structure, but also integrates the response information obtained based on physical modeling in the previous step, providing theoretical support for the stability modeling of key structural areas in cross-modal registration.

[0059] In step S4, a graph structure model driven by fusion response information is constructed, so that the multimodal image fusion method proposed in the present invention has a structural priori modeling capability.

[0060] S5. Based on the graph structure and tissue response information, a graph neural network is used to structurally fuse multimodal image features. Specifically, after completing the response field-driven graph structure construction in step S4, in order to further improve the registration accuracy and structural consistency expression capabilities between multimodal images, this embodiment of the present invention introduces a graph neural network structure based on an attention mechanism in step S5 to achieve adaptive edge weight calculation and structural information fusion between graph nodes. This module models regional response features, graph node embedding information, and their similarities through an end-to-end learning approach, thereby completing the dynamic update of attention weights between nodes and providing efficient structural support for subsequent graph registration and alignment.

[0061] In this embodiment, the graph neural network model is constructed in step S4. is the basic input, where the feature vector h i It combines image spatial information with structural properties guided by the response field.

[0062] To learn node i and its neighbor nodes The correlation strength between them is calculated by introducing the following attention weight calculation formula.

[0063] in: α ij is the attention weight between node i and node j, indicating the degree of attention of node i to node j; exp(·) exponential function, which amplifies the difference in similarity between nodes; is the dot product of the feature vectors of node i and node j, which measures the similarity between the two nodes; ∑ is the summation symbol, which means summing all elements in a set; k is node k, which is a neighbor node of the current node i; N(i) is the set of neighbor nodes of node i, that is, the nodes connected to i; h i ,h k is the feature vector of node i and node k, representing the embedded information of the node; T is the transpose symbol, which means converting the eigenvector into a row vector or a column vector.

[0064] This embodiment further adopts a weighted aggregation strategy to update the feature representation of node i: in: is the updated feature of node i in the l+1th layer graph neural network; W is a learnable linear transformation matrix used to map node input features; σ(·) activation function; α ij Calculation results from the aforementioned attention mechanism; Input features of neighbor node j in layer l.

[0065] Generally speaking, by stacking several layers of such graph attention layers, the propagation and deep interaction of full-graph structural information can be achieved, so that corresponding regions in different modal graph structures have consistent representation in the embedding space.

[0066] Step S5 realizes the adaptive learning and updating of edge weights by introducing the attention mechanism graph neural network structure. It not only models the response characteristics of the region, the similarity between nodes and the structural adjacency relationship, but also provides theoretical support and engineering basis for subsequent image registration optimization and structural consistency maintenance.

[0067] S6. Based on the fused features, guide the registration network to predict the deformation field of the second modality image and perform image registration on it; Specifically, in this embodiment, after completing the optimization of the attention weights between nodes through the graph neural network in the aforementioned step S5, the next step is to further expand the accuracy and stability of the image registration. This step focuses on the deformation field optimization based on the convolutional neural network (CNN), and uses multiple convolution operations to perform refined calculations and iterative optimization of the image deformation field, thereby enhancing the registration effect and ensuring global structural consistency. The main goal of step S6 is to further adjust and refine the deformation field between images based on the aforementioned graph structure and response field information, combined with global optimization technology, to ensure accurate registration between cross-modal images.

[0068] In this embodiment, the key operation in step S6 is to first correct the initial image registration results through deformation field optimization based on a convolutional neural network. Specifically, the deformation field of the registered image is iteratively updated multiple times using the convolutional neural network to improve the accuracy and stability of the transformation. This process, based on the graph structure and attention weights generated in step S5, utilizes deep learning techniques to model the image deformation and obtain a refined deformation field.

[0069] During the optimization process, the desired deformation field It is represented as a two-dimensional vector, containing displacement information in the x-axis and y-axis directions. The specific expression is as follows: in: Represents the displacement vector at the node (x, y), which contains the displacement of the point in the x and y directions; Represents the displacement component of the node (x, y) in the horizontal direction (x-axis direction); Represents the displacement component of the node (x,y) in the vertical direction (y-axis direction).

[0070] During the optimization process, the convolutional neural network uses these displacements for backpropagation, adjusting the various parameters of the deformation field. Typically, the network will undergo multiple convolution operations, continuously adjusting the weights of the convolution kernels to refine the image deformation and reduce deformation errors during the registration process.

[0071] Alternatively, the convolutional neural network in step S6 can use multiple convolutional layers to gradually learn the structural features of the image, and apply nonlinear activation functions such as ReLU (Rectified Linear Unit) or LeakyReLU to each layer to enhance the network's learning ability and prevent overfitting. Through this multi-layer structure, the network can gradually learn more complex deformation patterns in the image, thereby improving the accuracy of registration.

[0072] Specifically, in one possible implementation, the network output is not just the deformation field itself, but may also include a confidence measure of the deformation field, which can help the subsequent registration process assess the reliability of the deformation. Furthermore, to further improve the registration effect, the graph node similarity and response field features obtained in step S5 can be combined as additional input features to the convolutional neural network, thereby strengthening the prediction ability of the deformation field.

[0073] In the further optimization process, this embodiment also considers regularization constraints. To prevent overfitting and ensure smooth and consistent deformation fields, some embodiments add regularization terms to the optimization objective function, such as gradient smoothing constraints. Specifically, the following regularization terms can be introduced: in: represents the regularization term, which controls the smoothness of the deformation; ∑ i ||·|| means summing all nodes i; Represents node x i The gradient of the deformation field at , which represents the rate of change of deformation; Represents the square of the gradient, which measures the strength of the deformation.

[0074] The resulting deformation field, derived from the optimization process in step S6, serves as the core result of image registration, accurately aligning key structural regions across multiple modal images. This optimization step ensures global and local structural consistency, effectively improving image registration accuracy. This provides more stable and accurate registration results, particularly when processing medical images with significant modality differences.

[0075] S7, optimizing the registration and fusion results based on the joint loss function, and outputting the fused image; Specifically, in this embodiment, after completing the deformation field obtained through convolutional neural network optimization in step S6, step S7 further undertakes the task of quantitatively optimizing the fusion effect of the registered images. This step not only concludes the entire multimodal image registration process but also serves as a centralized evaluation and reverse correction of the results of the previous operations, including graph structure modeling, response map generation, and deformation field prediction. Its primary goal is to optimize the image registration network as a whole through a joint loss function, thereby improving the consistency and stability of the fused image across multiple dimensions.

[0076] In this embodiment, a multi-factor collaborative joint loss function is introduced in step S7. This loss function comprehensively considers the similarity between images, the smoothness of the deformation field, the regional consistency between multimodal structures, and the model's ability to guide the response graph, ensuring that the fused image has good multimodal semantic consistency and structural fidelity.

[0077] In general, to achieve the above goals, the joint loss function can be set as follows: in: Represents the total loss function, which is a comprehensive consideration of multiple aspects of loss in model training; Represents similarity loss, which is used to measure the similarity between the predicted results and the real data; Represents smoothing loss, which is used to smooth the output of the model and reduce noise or discontinuity; Represents structural loss, which measures the structural performance of the model; Represents the response loss, which is used to measure the model's ability to respond to external inputs; λ1,λ2,λ3,λ4 represent weighting coefficients, which are used to adjust the relative importance of each loss term.

[0078] In some embodiments, the response feature map may be derived from the graph structure node weight distribution generated in step S5, or a sparse representation of the local region response map in step S4. By aligning these response maps with the network prediction results, the model's physical structure adaptability can be further enhanced.

[0079] As an option, to address the problem of localized low-texture regions in breast ultrasound images, during actual training, the settings of λ3 and λ4 can be prioritized to enhance the ability to capture structural information. Furthermore, to prevent the model from overfitting to background regions, a masking mechanism can be introduced into the loss function to calculate response consistency only for key structural regions, thereby improving training efficiency and enhancing model generalization.

[0080] Step S7 achieves multi-dimensional supervision of image similarity, structural rationality, and response consistency in the multimodal registration task through the construction and optimization of the above-mentioned joint loss function, effectively improving the stability and generalization of the registration model.

[0081] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-mode ultrasound breast imaging and multi-mode fusion method, characterized in that: The following steps are included: S1. Acquire at least two modal medical images of a breast region, including a first modality image and a second modality image; S2. performing standardization preprocessing on the first modality image and the second modality image; S3. constructing a physical model of breast tissue based on the second modality image, generating a tissue deformation displacement field to obtain tissue response information; S4. Constructing a graph structure to represent the image region, and extracting regional features of the first modality image and the second modality image respectively; S5. Based on graph structure and tissue response information, graph neural network is used to perform structural fusion of multimodal image features; S6. Based on the fused features, guide the registration network to predict the deformation field of the second modality image and perform image registration on it; S7. Optimize the registration and fusion results based on the joint loss function and output the fused image.

2. A multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The first modality image and the second modality image in step S1 include B-ultrasound images and elastic ultrasound images collected using ultrasound equipment, wherein the B-ultrasound image is used to display the structural characteristics of breast tissue, and the elastic ultrasound image is used to obtain elasticity information of breast tissue.

3. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: In step S1 , the first modality image and the second modality image adopt the same scanning position, and the two modality images are aligned by an image registration algorithm so that the two images have a consistent spatial distribution in a spatial coordinate system.

4. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The step S2 specifically includes the following steps: performing size normalization processing on the first modality image and the second modality image so that the sizes of the first modality image and the second modality image are consistent; The grayscale values ​​of the first modality image and the second modality image are normalized to ensure that the images are within the same grayscale range; the first modality image and the second modality image are denoised to remove the noise introduced by the imaging device to ensure the clarity and accuracy of the image.

5. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The physical model in step S3 is an elastic mechanics model established based on the finite element method; determining an initial position of each node of the breast tissue according to the local deformation region of the second modality image; Set boundary conditions and define the elastic modulus and Poisson's ratio of breast tissue to calculate the mechanical response; Based on the boundary conditions, the deformation displacement field of each node is calculated using a finite element solver.

6. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The displacement field is further used to calculate the local structural response field, which is calculated by the following formula: Among them, S(x,y) represents the response field at a specific position (x,y), u(x,y) represents the displacement field at the point (x,y), represents the gradient of the displacement field, ||·|| F is the representation of the Frobenius norm.

7. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: Constructing the graph structure in step S4 specifically includes the following steps: Dividing the first modality image and the second modality image into a plurality of small regions, each of which is a node in a graph structure; Extract node features based on pixel intensity, texture features, and shape information of each small area; The adjacency relationship between nodes is established in the graph structure, and the adjacency relationship is calculated based on the similarity between adjacent regions, where the similarity measurement is based on the convolutional features and local feature vectors of the region.

8. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: In step S5, the graph neural network adopts a graph neural structure based on the attention mechanism, and its edge weights are adaptively calculated based on the feature similarity between regions and the corresponding structural response field. The specific calculation method is as follows; Among them, α ij is the attention weight between node i and node j, indicating the degree of attention of node i to node j, exp(·) exponential function, is the dot product of the feature vectors of node i and node j, ∑ is the summation symbol, k is node k, which is a neighbor node of the current node i, N(i) is the set of neighbor nodes of node i, that is, the nodes connected to i, h i ,h k is the feature vector of node i and node k, and T is the transposed symbol.

9. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The registration network in step S6 is a convolutional neural network, which gradually refines the deformation field of the predicted image by performing multiple convolution operations on the fused feature image. The deformation field is calculated by the following formula: in, represents the displacement vector at the node (x,y), Represents the displacement component of the node (x, y) in the horizontal direction, Represents the displacement component of the node (x, y) in the vertical direction.

10. The multi-mode ultrasound breast imaging and multi-mode fusion method according to claim 1, characterized in that: The joint loss function in step S7 specifically includes the following steps: Image similarity loss, which calculates the difference between the fused image and the target image; Loss of smoothness of deformation field, smoothing the deformation field through regularization method to reduce local irregularities; Multimodal regional feature consistency loss ensures the consistency of regional features between the first modality image and the second modality image; A structural response-guided loss aligned with physical modeling results ensures that the computational results are consistent with the response information generated by physical modeling of breast tissue; The joint loss function is expressed as; in, represents the total loss function, represents the similarity loss, represents the smoothing loss, represents structural loss, represents the response loss, and λ1, λ2, λ3, and λ4 represent weighting coefficients.

Images