Mammary gland ultrasound image end-to-end segmentation method based on image weight reduction

Abstract

The invention discloses a breast ultrasound image end-to-end segmentation method based on image weight reduction, which comprises the following steps: firstly, processing a breast ultrasound image by using UNet to obtain a high-dimensional feature image of the image; performing tumor area rough prediction according to the feature map, and performing map data construction to obtain sparse map data of the breast ultrasound image; and through a data loading method of splitting the sparse graph into sub-graphs and sparse features, the data is transferred from the CPU to the GPU, so that the amount of data occupied and transmitted by the memory of the CPU is reduced, the GPU computing resources can be fully utilized, and the training process is accelerated. According to the method, the graph neural network can be utilized to assist in improving the segmentation precision, end-to-end training acceleration is provided, the model precision and the training efficiency are considered, and the problem that the high dimension and the large data volume of the medical image enable a graph structure to consume a large amount of resources during calculation, and consequently the calculation cost is high is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical ultrasonic image processing, and in particular relates to an end-to-end segmentation method of breast ultrasonic images based on graph lightweighting. Background Art

[0002] Breast ultrasound image segmentation is to more clearly show the changes in anatomical or pathological structures, and plays an important role in computer-aided diagnosis and intelligent medicine. Breast ultrasound image segmentation can help doctors identify lesions more accurately and improve the accuracy and efficiency of breast cancer screening, thus occupying a key position in medical image analysis.

[0003] Traditional breast ultrasound segmentation methods mainly include threshold segmentation, edge detection, level set method, etc. With the development of deep learning, graph data has gradually been applied to breast ultrasound image segmentation. Because it can represent the non-Euclidean structure in the data, it is widely used in various backbone networks to assist in improving segmentation accuracy.

[0004] However, medical images usually have complex and irregular structures, including both local details and global morphological changes. How to effectively construct the graph data corresponding to the original image without losing the original information is still a major challenge in medical image segmentation. At the same time, due to the large amount of data and high dimension of graph data, its training process often consumes a lot of computing resources, resulting in a very high computational cost of the network. This high consumption of computing resources has become one of the bottlenecks in the current application of graph data in medical image segmentation.

[0005] In real life, breast medical images, as a kind of complex medical data, can usually be modeled and analyzed through graph data structures to capture the structural relationships and characteristics between tissues. The vertices in the graph can be used to represent points with medical significance in the image, such as masses, calcifications or other lesion areas, while the edges represent the anatomical or functional connections between these areas.

[0006] In recent years, with the continuous expansion of data scale, the magnitude of graph data has also grown exponentially. In the field of image processing, the original graph data often contains tens of thousands of vertices and millions of edges. The processing of large-scale graph data faces challenges such as high computational complexity and low efficiency. Therefore, how to effectively compress and sparse these graph data to ensure analysis accuracy while reducing computational overhead has become a hot topic of research. Hariharan proposed to sample edges by using the edge association as the upper bound of edge connectivity, and proved the feasibility of the sparse algorithm within the error range; Cohen proposed a random algorithm for generating graphs, using the distance between vertices in the original graph as an edge measurement method; Tanwistha and Huzefa proposed an adaptive global sparse algorithm for set classification, which compares the similarity of the attribute values ​​of two vertices and samples edges with higher similarity according to the proportional requirements; if the remaining edges can increase the number of connected parts in the graph, they will be added to the sparse graph.

[0007] Although these methods have improved the efficiency of graph sparse construction to a certain extent and can perform edge sampling within the error range, they still have some shortcomings when dealing with the complex structure of breast medical images. First, existing sparse algorithms are usually based on randomness or simple similarity metrics, which are difficult to fully capture the complex anatomical relationships and subtle connections between lesions in breast images. Second, over-simplification of edge structures may cause important diagnostic information to be ignored, affecting the accurate analysis of the disease.

[0008] Graph Neural Networks (GNNs) are a type of deep learning model based on graph structured data. GNNs can capture the high-order relationships between nodes in the graph structure, thereby learning richer node embeddings. However, when processing graph data with a large number of nodes and dense connections between nodes, the amount of computation will increase dramatically, which will have a huge negative impact on the training efficiency and memory consumption of the model. To solve this problem, researchers have proposed a variety of training strategies, including subgraph-wise sampling, which reduces computational complexity by sampling only part of the subgraph during training. Although these methods have alleviated the neighbor explosion problem to a certain extent, they often sacrifice the accuracy of gradient estimation, which in turn affects the convergence and final performance of the model. To improve training efficiency, Defferrar et al. proposed a local convolution operator based on a fast filtering recursive formula, usually denoted as the Chebyshev convolution operator, which greatly reduces the computational complexity of training parameters, especially in sparse graph processing; Levie et al. proposed the CayleyNet algorithm, proving that the time complexity is linearly related to the number of edges in a sparse connected graph; Kipf et al. proposed a semi-supervised GCN node classification model, which linearly scales the number of edges in the graph and learns the hidden layer representation, deepens the depth of the network, reduces the neighborhood width, and further improves the accuracy and learning ability. However, this method is still limited to undirected and unweighted graphs, and the entire graph must be loaded for training, which requires high machine storage space, so large-scale graphs may take a long time to load and calculate.

[0009] These methods work well for graphs with smaller feature sizes, but they often consume a lot of computing resources when faced with larger-scale graph data, resulting in a very high computational cost for the network. The subgraph feature loading for each iteration limits the overall training speed and GPU resource utilization. Moreover, with the rapid development of GPU technology and the optimization of computing methods in recent years, the mismatch between computing power and data loading is becoming increasingly serious. Summary of the invention

[0010] In order to solve the above technical defects in the prior art, the purpose of the present invention is to propose a lightweight graph neural network breast ultrasound image segmentation method. UNet is used as the backbone network for feature map extraction, and the amount of data transmitted by CPU memory is significantly reduced by sparse construction and graph sampling learning of graph data, which greatly shortens the data loading time. The method is integrated into the GNN training process to accelerate GNN training and increase the scale of processable graphs, so that it can make full use of GPU computing resources and improve training accuracy and efficiency.

[0011] To achieve the above object, the technical solution adopted by the present invention is: an end-to-end segmentation method of breast ultrasound images based on graph lightweight, characterized in that the steps include data set construction, segmentation backbone network construction, and graph neural network acceleration;

[0012] The dataset is constructed by using the BUSI dataset, and by flipping, rotating, changing brightness and contrast, the subsequent network can more effectively extract cancer features. The segmentation backbone network is constructed by using a UNet-based backbone network to obtain high-dimensional feature maps of breast ultrasound images, thereby making a rough prediction of the tumor area and reducing interference from clutter noise and fuzzy cancer boundaries. The UNet method adds a padding strategy to the convolution operation to ensure that the size of the feature map will not be reduced after the convolution operation, making the feature map after sampling at each layer easy to process. The graph neural network is accelerated by sparsifying the input features, significantly reducing the amount of data loaded, thereby accelerating GNN training and increasing the scale of processable graphs.

[0013] The UNet-based backbone network consists of two symmetrical parts: an encoder and a decoder. The encoder part gradually downsamples to extract feature maps, and the decoder restores the spatial resolution of the feature maps by upsampling layer by layer and fuses the features of the encoder. At the same time, skip connections are introduced between the corresponding layers of the encoder and decoder to transfer low-level features to the decoder stage, retaining detail information to enhance the segmentation boundary effect.

[0014] A sparse graph inference module is added after the fourth stage of the backbone network decoder in the upsampling stage to process the feature maps, convert them into graph data and perform sparsification, thus achieving more comprehensive information extraction.

[0015] The sparse graph reasoning module uses GNN for graph reasoning. The process of applying the attention mechanism in GNN can be described as follows:

[0016]

[0017] in Represents node v i The neighbor set of Represents node v i is the hidden state of the system, σ(·) is the activation function (e.g. ReLU), and W k is the weight matrix of the kth layer, α ij is node v i and v j The attention coefficient between .

[0018] During the sparse graph construction process, the feature graph generated by the segmentation backbone is represented as X∈R h*w*c, where h×w is the image size and c is the number of channels; the node feature matrix H can be obtained by reshaping X to N×c size, where N=h×w; after mapping each pixel to a graph node, the constructed graph G=(A,H) retains the inherent information of each pixel and can provide accurate pixel-level information for global reasoning; the adjacency matrix A encodes the connection pattern between nodes, indicating that each node v i The information of a node is aggregated from those nodes in its neighborhood.

[0019] The sparse graph reasoning module has a coarse prediction branch, which performs auxiliary sparse operations. The specific process is divided into two steps. One is to use the random edge deletion method to randomly delete the edges of the fully connected graph constructed by the feature graph, and then use the feature information of the graph after random edge deletion to combine various sparse methods to sparse the graph.

[0020] The graph neural network acceleration is described in detail. In each training iteration, the sampling mechanism of the sparse graph is used to generate subgraphs. First, sampling is performed based on the previous sparse graph to extract the subgraphs and their corresponding sparse feature data. These data are then collected and transmitted to the GPU, where data recovery is performed and the sparse graph data and features are converted into the original format for forward and backward calculations in the GNN.

[0021] The beneficial effects of the above technical solution: The present invention is based on a graph-lightweight end-to-end segmentation method for breast ultrasound images, and uses UNet to process breast ultrasound images to obtain a high-dimensional feature map of the image; then, a rough prediction of the tumor area is made based on the feature map, and graph data is constructed by combining methods based on node importance, methods based on global edge weights, methods based on local neighborhoods, and methods based on graph segmentation to obtain sparse graph data of breast ultrasound images, and by splitting the sparse graph into subgraphs and loading data of sparse features, the data is transferred from the CPU to the GPU, reducing the amount of data transmitted by the CPU memory, so that it can make full use of GPU computing resources and accelerate the training process. This method can use graph neural networks to assist in improving segmentation accuracy and provide end-to-end training acceleration. It is a practical method that takes into account both model accuracy and training efficiency.

[0022] The BUSI dataset contains 780 breast ultrasound images, which are labeled with different types of breast tumors, including benign and malignant tumors. The images are collected by multiple medical institutions and provide high-quality samples for breast ultrasound image research. The UNet model is used as the segmentation backbone network, which demonstrates convincing and robust performance in a variety of medical image segmentation tasks and has excellent performance in the field of computer vision. The present invention adds a padding strategy in the convolution operation to ensure that the size of the feature map will not be reduced after the convolution operation. The segmentation backbone network construction uses a backbone network based on UNet to obtain high-dimensional feature maps of breast ultrasound images. It consists of two parts: a symmetrical encoder and a decoder. After the fourth stage of the backbone network decoder in the upsampling stage, a coarse prediction branch and a sparse graph reasoning module are added. The sparse graph reasoning module consists of two parts: sparse graph construction and long-range information reasoning of GNN. Its purpose is to improve the judgment accuracy of uncertain segmentation pixels, improve the model training speed, and prevent the propagation of noise in long-range information reasoning, thereby further improving the segmentation performance. This module is designed to capture the long-distance dependency between pixels, so it needs to operate at a level with sufficiently rich semantic information. In UNet, the lower the layer features are, the more global information they contain, and the larger the spatial range of the features, which improves the model's understanding of global contextual information; but applying these modules at too low a resolution may lead to the loss of detail information, resulting in poor recovery of local fine structures, which is not conducive to the effect of segmentation tasks. Therefore, the present invention places the sparse graph reasoning module after the fourth stage of the backbone network decoder in the upsampling stage to improve segmentation performance. Reducing the size of feature data is the main method to solve the bottleneck of existing data loading, but the usual lossless compression method is not suitable for graph feature data. This is because its compression ratio (the ratio of the original data size to the compressed size) is not fixed and is low, and the decompression process is relatively complicated, making it difficult to implement efficiently on the GPU. In the construction of the sparse graph, methods based on node importance, methods based on global edge weights, methods based on local neighborhoods, and methods based on graph segmentation are used to perform sparse strategies to construct a sparsely connected graph G. s, and evaluate them. Apply the attention mechanism in GNN. In this attention-based GNN, each node has different weights for aggregating information from its neighbors, allowing the network to focus on the most important relationships in the graph. This improves the ability to capture complex patterns in the data, making GNNs more powerful in tasks such as node classification, link prediction, and graph classification. They can be used directly for downstream tasks or further processed by pooling or reading functions to generate global graph representations. This attention mechanism not only improves the expressive power of GNNs, but also provides better interpretability by revealing which nodes and edges contribute most to the final prediction. Graph neural networks accelerate medical image segmentation, accelerate GNN training and increase the size of graphs that can be processed by sparse input features and significantly reduce the amount of data loaded. Before training begins, the features of the graph dataset are sparsed. In each training iteration, the previous sparse graph is sampled to obtain a subgraph, then the subgraph and the corresponding sparse features are collected and transmitted, and the graph data and features are restored to their original format on the GPU, and finally forward and backward calculations are performed in the GNN. The graph neural network acceleration method is used to sample and train the sparse graph data, reducing the amount of data involved in feature collection and data transmission. The sparseness is one-time, while the decompression is repeated in each iteration. The decompression step is necessary for seamless integration with the GNN training framework. It keeps the shape and format of the final loaded data unchanged, so that it can be compatible with most common graph neural network models without any modification to the model structure. Therefore, unlike the optimization method that modifies the model structure, the sparse data loading method can be easily applied to the existing GNN training process. The method saves data loading overhead, improves training speed and GPU resource utilization by splitting feature data, and saves CPU memory resources, so that the training speed of trainable graphs is improved under the same scale. Therefore, reducing the size of feature data is the main method to fundamentally accelerate data loading and improve GPU resource utilization in GNN training.

[0023] Experimental methods and results analysis

[0024] 1 Experimental Environment

[0025] The experimental environment is Intel(R)Core(TM)i5 CPU 2.4GHz, 16GB memory, Ubuntu 20.04 operating system, and the running software is pycharm2022.

[0026] 2 Dataset Selection

[0027] The BUSI dataset is used for experiments and verification. Examples of ultrasound images are as follows: Figure 4 .

[0028] 3Baseline selection

[0029] Several models widely used in medical image segmentation were selected as baselines for comparison, including UNet, GraphSAGE, DANet (Dual Attention Network) and CCNet (Cross Attention Network). These models were experimented on the datasets and comprehensively compared with the method of the present invention in multiple dimensions such as accuracy (using mIOU), memory usage and time consumption.

[0030] 4 Experimental results analysis

[0031] 4.1 Performance comparison of different segmentation methods

[0032] Example of BUSI breast ultrasound image segmentation results

[0033] The result of the method of the present invention is obtained by the sparse method of atlas segmentation, such as Figure 5 The results of BUSI breast ultrasound image segmentation are shown in Figure 1. The performance evaluation data of each model is shown in Table 1, where the bold value is the optimal value. It can be seen that compared with other models, this method has the least memory usage and its accuracy is slightly higher than that of the optimal DANet, but the training time is shortened by nearly half, second only to UNet.

[0034] Table 1 Performance evaluation of each model

[0035]

[0036] In summary, the algorithm of the present invention shortens the model training time while taking into account the image segmentation accuracy and memory usage, and has higher practical application value.

[0037] 4.2 Performance comparison of different sparsification methods

[0038] In this experiment, five different sparse processing methods are compared, including the random edge deletion sparse method, the node importance-based method, the global edge weight-based method, the local neighborhood-based method, and the graph segmentation-based method. The specific effects on the segmentation results are shown in Table 2.

[0039] Table 2 Performance evaluation of various sparsification methods

[0040]

[0041] Experimental results show that different sparsification methods have a significant impact on the segmentation performance of GNN. Specifically, the random edge deletion sparsification method can better preserve local features while maintaining the stability of the network structure, so that the model's mIOU reaches 62.62%, and the training time is the shortest, only 6.82 hours. However, although random sparsification can effectively reduce the computational cost, the randomness of its feature propagation leads to unstable model sensitivity, and the SPE drops to 75.31%, indicating that this method has low accuracy in boundary segmentation.

[0042] In contrast, the DSC and mIoU of the atlas segmentation sparse method are the best. This result shows that the atlas segmentation-based sparse method can better adapt to the local complexity in medical images in terms of information extraction and feature propagation, especially in the breast ultrasound image segmentation task, and has obvious advantages.

[0043] In general, different sparsification methods have a significant impact on the segmentation performance of GNN, among which the sparsification method based on atlas segmentation performs best and can significantly improve the accuracy and generalization ability of segmentation results. In future work, we plan to further optimize the sparsification strategy and combine it with multi-scale feature extraction methods to further improve the robustness of the model in processing complex medical image tasks.

[0044] 4.3 Performance comparison of sparse graph reasoning module locations

[0045] In this experiment, the segmentation network was structurally adjusted to verify the network's detail capture capability and segmentation accuracy by moving the lightweight graph reasoning module to different upsampling paths. In the UNet infrastructure, the encoding path is used to extract features layer by layer, while the decoding path is responsible for upsampling layer by layer and restoring the spatial resolution of the image. If the lightweight graph reasoning module can interact with higher-resolution feature maps, it may be more conducive to capturing detailed information in lesion images with complex boundaries.

[0046] The lightweight graph inference module is moved after other upsampling layers to allow it to work when processing higher resolution feature maps, thereby improving the accuracy of boundary segmentation.

[0047] Table 3 Performance evaluation of the location of the sparse graph reasoning module

[0048]

[0049] Experimental results show that different sparse graph reasoning module positions have a significant impact on the segmentation performance of GNN. Specifically, the first sampling position can better complete the basic segmentation task while maintaining the reasoning speed, making the model's mIOU reach 60.23%, and the reasoning time is the shortest, only 8.23 ​​hours. However, although the sparsification of this position effectively reduces the computational cost, its feature extraction ability is limited, and the sensitivity (SEN) is low, only 73.57%, indicating that this position has certain deficiencies in capturing more complex boundary features.

[0050] In contrast, the DSC and mIOU of the fourth sampling position are both the best. This result shows that the fourth sampling position can better adapt to the local complexity of breast ultrasound images while retaining global structural information, which significantly improves the segmentation accuracy, especially mIOU reaches 68.49% and DSC is 81.24%. However, its inference time increased significantly, reaching 18.47 hours, indicating that this position is suitable for tasks that require high accuracy but are not time-sensitive.

[0051] In general, different sampling positions have a great impact on the segmentation performance of the sparse graph reasoning module, among which the fourth sampling position performs best and can significantly improve the accuracy and stability of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0053] Figure 1 The overall structure of the segmentation model;

[0054] Figure 2 It is a lightweight graph reasoning module;

[0055] Figure 3 Accelerate the method flow for graph neural networks;

[0056] Figure 4 This is an example of a BUSI breast ultrasound image;

[0057] Figure 5 This is an example of BUSI breast ultrasound image segmentation results. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0059] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as in the embodiments of this application.

[0061] like Figure 1 The overall structure of the segmentation model shown in the figure adopts a multi-layer encoder-decoder structure, and the whole process from input feature extraction to final prediction optimization is completed through jump connections, bridge layers and subsequent processing modules. First, the input data is sequentially passed through encoders 1 to 4 for multi-layer feature extraction, and semantic features from fine-grained to coarse-grained are extracted layer by layer. At the same time, the low-level features are directly passed to the corresponding decoder through jump connections to retain key detail information. After the feature extraction is completed, the high-level features of the encoder are globally modeled and compressed through the bridge layer to adapt them to the input requirements of the decoder. Subsequently, the decoder upsamples the features layer by layer, adds a padding strategy to the convolution operation, and gradually restores a higher-resolution feature map in combination with the low-level features in the jump connection. In the output part of the decoder, a preliminary prediction result is generated through a coarse prediction branch to assist the sparse operation of the graph data. The sparse graph reasoning module uses the coarse prediction results to construct a sparse graph structure, and further models the global dependency relationship between nodes and edges through a graph neural network, thereby generating a more refined and globally consistent final prediction output.

[0062] A graph-lightweight end-to-end segmentation method for breast ultrasound images includes data set construction, segmentation backbone network construction, and graph neural network acceleration. The data set construction uses the BUSI data set, which can more effectively extract cancer features by flipping, rotating, changing brightness, and contrast. The segmentation backbone network construction: uses a UNet-based backbone network to obtain high-dimensional feature maps of breast ultrasound images, thereby making a rough prediction of the tumor area and reducing interference from clutter noise and fuzzy cancer boundaries; the UNet method adds a padding strategy to the convolution operation to ensure that the size of the feature map will not be reduced after the convolution operation, so that the feature maps after sampling at each layer are easy to process.

[0063] like Figure 2 As shown in the figure, the UNet-based backbone network structure combines a multi-layer encoder-decoder architecture and a sparse graph reasoning module to achieve refined prediction optimization through feature extraction, sparsification operations, graph neural network (GNN) reasoning and feature fusion. The input features are first converted into a sparse graph structure through a sparsification operator to reduce the computational complexity and retain the structural information of the graph. Then, the GNN module is used to further optimize the sparse graph features by using the graph neural network training acceleration method, and the global relationship and long-distance dependency between nodes are captured through the message passing mechanism to generate graph features that have been optimized through reasoning. These optimized features are fused with the high-resolution features from the encoder (Encoder3), and the detailed information is maintained and the global consistency is enhanced through weighted summation or other fusion methods. Finally, they are input to the decoder (Decoder3) to generate high-resolution output features.

[0064] The UNet-based backbone network consists of two symmetrical parts: an encoder and a decoder. The encoder part gradually downsamples to extract feature maps, and the decoder restores the spatial resolution of the feature map by upsampling layer by layer, and fuses the features of the encoder. At the same time, a jump connection is introduced between the corresponding layers of the encoder and the decoder to pass low-level features to the decoder stage, retaining detail information to enhance the segmentation boundary effect. A sparse graph reasoning module is added after the fourth stage of the backbone network decoder in the upsampling stage to process the feature map, convert the feature map into graph data and perform sparseness, thereby achieving more comprehensive information extraction. The sparse graph reasoning module uses GNN for graph reasoning. The process of applying the attention mechanism in GNN can be described as follows:

[0065]

[0066] in Represents node v i The neighbor set of Represents node v i is the hidden state of the system, σ(·) is the activation function (e.g. ReLU), and W k is the weight matrix of the kth layer, α ij is node v i and v jThe attention coefficient between . In this attention-based GNN, each node has different weights for aggregating information from its neighbors, allowing the network to focus on the most important relationships in the graph. This improves the ability to capture complex patterns in the data, making GNNs stronger in tasks such as node classification, link prediction, and graph classification. They can be used directly for downstream tasks or further processed by pooling or read functions to generate a global graph representation. This attention mechanism not only improves the expressiveness of GNNs, but also provides better interpretability by revealing which nodes and edges contribute most to the final prediction.

[0067] During the sparse graph construction process, the feature graph generated by the segmentation backbone is represented as X∈R h*w*c , where h×w is the image size and c is the number of channels; the node feature matrix H can be obtained by reshaping X to N×c size, where N=h×w; after mapping each pixel to a graph node, the constructed graph G=(A,H) retains the inherent information of each pixel and can provide accurate pixel-level information for global reasoning; the adjacency matrix A encodes the connection pattern between nodes, indicating that each node v i The information of a node is aggregated from those nodes in its neighborhood.

[0068] The coarse prediction branch performs auxiliary sparse operation, which is divided into two steps in the specific process. The first step is to use the random edge deletion method to randomly delete the fully connected graph constructed by the feature map, and then use the feature information of the randomly deleted graph to combine various sparse methods to perform graph sparseness. The coarse prediction branch is placed after the fourth stage of the backbone network decoder, and the size of the feature map is 1 / 8 of the input image. A layer sequence is applied, including 3×3 convolution+BN+ReLU+1×1 convolution, to generate a coarse prediction map Fc, and the auxiliary map is sparse.

[0069] Taking the breast ultrasound medical image with an input of 3*600*600 as an example, after passing through the upstream encoder, the input feature map dimensions of encoder 4 are 256*75*75 of encoder 4 and 512*37*37 of the bridge layer, and the output is a feature map of 256*75*75. First, the feature map is converted into a fully connected graph with features as nodes, and the feature map passes through the coarse prediction branch to obtain a coarse prediction feature map with a dimension of 1*75*75. The prediction information of the coarse prediction feature map is combined with the node importance-based method, the global edge weight-based method, the local neighborhood-based method, and the graph segmentation-based method to perform a sparse strategy to convert the fully connected graph into a sparse graph. The sparse graph features are input into the GNN module, and the relationship between nodes is modeled using the graph neural network to capture long-distance dependencies and global consistency. In the training process, the graph neural network is accelerated, and information is propagated on the graph through the message passing mechanism to enhance the context feature expression of each node. The output of the GNN module is a feature optimized by sparse graph reasoning, with a dimension of 256*75*75. It is fused with the output of the sparse reasoning module, the output of encoder 4 (256*75*75), and the high-resolution feature 56*150*150 from Encoder3, and the feature information is input into the downstream network for segmentation tasks.

[0070] like Figure 3 The graph neural network acceleration shown in the figure significantly reduces the amount of data loaded by sparsifying the input features, thereby accelerating GNN training and increasing the scale of graphs that can be processed. In each training iteration, the sampling mechanism of the sparse graph is used to generate subgraphs. First, sampling is performed based on the previous sparse graph to extract the subgraphs and their corresponding sparse feature data. These data are then collected and transmitted to the GPU, where data recovery is performed to convert the sparse graph data and features into the original format for forward and reverse calculations in the GNN.

[0071] The graph neural network acceleration method is used to sample and train the sparse graph data, reducing the amount of data involved in feature collection and data transmission. The sparseness is one-time, while the decompression is repeated in each iteration. The decompression step is necessary for seamless integration with the GNN training framework, which keeps the shape and format of the final loaded data unchanged, so that it can be compatible with most common graph neural network models without any modification to the model structure. Therefore, unlike the optimization method that modifies the model structure, the sparse data loading method can be easily applied to the existing GNN training process. By splitting the feature data, the data loading overhead is saved, the training speed and GPU resource utilization are improved, and the CPU memory resources are saved, so that the training speed of the trainable graph is improved under the same scale.

[0072] Reducing the size of feature data is the main way to solve the bottleneck of existing data loading. Considering that the dimensions containing more important information in the feature vector often have larger absolute values, secondary compression is used to reduce the data size. When using sparse compression to compress graph features, the compression ratio does not change as the scale of the graph increases. Therefore, using sparse compression on graphs of any size can bring significant acceleration effects.

[0073] First, the sparse graph data that has undergone the sparsification operation is used to generate multiple subgraphs using a subgraph sampling method based on node feature values, including the subgraph features and subgraph structure, which are saved in node feature matrices and adjacency matrices respectively to ensure that the subgraphs are representative in terms of structural integrity and feature coverage. Subsequently, the sampled subgraph structure and sparse features are transferred from the CPU to the GPU, and their computing performance is utilized to restore them to their original format through efficient decompression on the GPU side. Next, the graph features and structure are input into the GNN model for forward and backward propagation, and graph convolution operations are used to complete information aggregation and optimize model parameters. This method significantly improves training efficiency through sparse feature processing and subgraph sampling, and reduces memory requirements while ensuring model accuracy, with high scalability and practical value. The final output is the prediction result of GNN.

[0074] Through the above steps, this graph neural network acceleration method reduces memory and computing requirements while effectively improving training efficiency by using sparse features and subgraph sampling technology, and ensures the accuracy and scalability of the model through feature decompression and subgraph calculation on the GPU side.

Claims

1. A graph-based end-to-end segmentation method for breast ultrasound images, characterized by The steps include dataset construction, segmentation backbone network construction and graph neural network acceleration; The dataset is constructed using the BUSI dataset, and by flipping, rotating, changing brightness and contrast, the subsequent network can more effectively extract cancer features; The segmentation backbone network is constructed by using a UNet-based backbone network to obtain high-dimensional feature maps of breast ultrasound images, thereby making a rough prediction of the tumor area while reducing interference from clutter noise and blurred cancer boundaries. The UNet method incorporates a padding strategy into the convolution operation to ensure that the size of the feature map does not decrease after the convolution operation, making the feature maps after sampling at each layer easier to process. The graph neural network acceleration significantly reduces the amount of data loaded by sparsifying the input features, thereby accelerating GNN training and increasing the scale of graphs that can be processed.

2. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 1, characterized in that: The UNet-based backbone network consists of two symmetrical parts: an encoder and a decoder. The encoder part gradually downsamples to extract feature maps, and the decoder restores the spatial resolution of the feature maps by upsampling layer by layer and fuses the features of the encoder. At the same time, skip connections are introduced between the corresponding layers of the encoder and decoder to transfer low-level features to the decoder stage, retaining detailed information to enhance the segmentation boundary effect.

3. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 1, characterized in that: A sparse graph inference module is added after the fourth stage of the backbone network decoder in the upsampling stage to process the feature map, convert the feature map into graph data and perform sparsification, thereby achieving more comprehensive information extraction.

4. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 3, characterized in that: The sparse graph reasoning module uses GNN for graph reasoning. The process of applying the attention mechanism in GNN can be described as follows: in Represents node v i The neighbor set of Represents node v i The hidden state of , σ(·) is the activation function (such as ReLU), W k is the weight matrix of the kth layer, α ij is node v i and v j The attention coefficient between .

5. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 4, characterized in that: During the sparse graph construction process, the feature graph generated by the segmentation backbone is represented as X∈R h*w*c , where h×w is the image size and c is the number of channels; the node feature matrix H can be obtained by reshaping X to N×c size, where N=h×w; after mapping each pixel to a graph node, the constructed graph G=(A,H) retains the inherent information of each pixel and can provide accurate pixel-level information for global reasoning; the adjacency matrix A encodes the connection pattern between nodes, indicating that each node v i The information of a node is aggregated from those nodes in its neighborhood.

6. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 4, characterized in that: The sparse graph inference module has a coarse prediction branch, which performs auxiliary sparse operations. In the specific process, it is divided into two steps. The first step is to use the random edge deletion method to randomly delete the edges of the fully connected graph constructed by the feature graph, and then use the feature information of the graph after random edge deletion to combine various sparse methods to perform graph sparseness.

7. The end-to-end segmentation method for breast ultrasound images based on graph lightweighting according to claim 1, characterized in that: The graph neural network acceleration is that in each training iteration, the sampling mechanism of the sparse graph is used to generate subgraphs. First, sampling is performed based on the previous sparse graph to extract the subgraphs and their corresponding sparse feature data. These data are then collected and transmitted to the GPU. On the GPU, data recovery is performed, and the sparse graph data and features are converted into the original format for forward and backward calculations in the GNN.

Images