An end-to-end multi-modal ultrasound image lesion automatic positioning method

Abstract

The application relates to an end-to-end multi-modal ultrasound image lesion automatic positioning method, which comprises the following steps: acquiring an ultrasound image sequence, and pre-processing the ultrasound image sequence; performing feature extraction on the pre-processed ultrasound image sequence to obtain optical flow motion features, global semantic features, time sequence enhancement features and colored information auxiliary features; inputting the optical flow motion features, the global semantic features, the time sequence enhancement features and the colored information auxiliary features into a cross-modal self-attention fusion encoder to obtain fusion features; generating a candidate box set based on the fusion features through a detection head, and constructing a graph structure based on the candidate box set; inputting the graph structure into a Transformer-based graph neural network to obtain a lesion position and a confidence degree. The application can improve the robustness, real-time performance and cross-device adaptability of an AI-assisted ultrasound diagnosis system.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an end-to-end multimodal ultrasound image lesion automatic localization method. Background Technology

[0002] Ultrasound imaging is a widely used non-invasive diagnostic tool in clinical practice, covering the screening and diagnosis of diseases in multiple organs such as the thyroid, breast, liver, kidneys, and blood vessels. With the continuous development of AI technology, computer vision-assisted detection technology can greatly improve the standardization and efficiency of clinical testing. In ultrasound image analysis, dynamic localization and real-time tracking of lesions are fundamental to accurate diagnosis. Doctors need to quickly identify and locate the target area from the dynamic video stream during continuous scanning, while simultaneously making a comprehensive judgment based on probe movement, tissue deformation characteristics, and Doppler ultrasound data.

[0003] However, existing technologies have the following technical problems in practical applications:

[0004] First, traditional target detection methods are ill-suited to the dynamic characteristics of ultrasound imaging. Unlike static images such as CT and MR, ultrasound imaging is a real-time dynamic process, with the location of lesions constantly changing as the probe moves. Detection methods that rely solely on single-frame images cannot effectively utilize temporal information, resulting in severe jitter and poor temporal consistency in the detection results.

[0005] Second, the feature extraction methods of existing AI-assisted systems are disconnected from doctors' observation behaviors. Most solutions separate tasks such as motion analysis, target detection, and feature classification, and use manual rules to integrate the outputs of each module. This fails to achieve end-to-end behavior learning, resulting in high system complexity, significant information transmission loss, and difficulty in adapting to the scanning habits of different operators.

[0006] Third, the detection stability is insufficient in complex noise environments. Ultrasound images generally suffer from motion artifacts, speckle noise, and local blurring. Traditional post-processing methods such as NMS only consider spatial overlap and cannot model the temporal dependencies between candidate boxes, which can easily lead to the false filtering of key lesions or the retention of redundant false detections.

[0007] Fourth, the ability to generalize across devices is limited. The imaging parameters, gain adjustment, and color Doppler display methods of ultrasound equipment from different brands vary significantly. Existing methods are often optimized for specific models and lack effective utilization of multimodal auxiliary information (such as blood flow signals and measurement scales), resulting in high model transfer costs and limited clinical applicability. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide an end-to-end multimodal ultrasound imaging lesion automatic localization method, which can improve the robustness, real-time performance and cross-device adaptability of AI-assisted ultrasound diagnostic system.

[0009] The technical solution adopted by this invention to solve its technical problem is: to provide an end-to-end multimodal ultrasound imaging automatic lesion localization method, comprising the following steps:

[0010] Acquire ultrasound image sequences in real time and preprocess the ultrasound image sequences;

[0011] Feature extraction is performed on the preprocessed ultrasound image sequence to obtain optical flow motion features, global semantic features, temporal enhancement features, and color information auxiliary features;

[0012] The optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features are input into a cross-modal self-attention fusion encoder to obtain fused features.

[0013] Based on the fusion features, a set of candidate boxes is generated using the detection head, and a graph structure is constructed based on the set of candidate boxes;

[0014] The graph structure is input into a Transformer-based graph neural network for dynamic relationship learning to obtain the lesion location and confidence level.

[0015] The preprocessing of the ultrasound image sequence specifically involves: setting a region of interest for each frame in the ultrasound image sequence and performing Gaussian filtering on the region of interest.

[0016] The optical flow motion characteristics are obtained in the following way:

[0017] The optical flow field of each region of interest in the preprocessed ultrasound image sequence was calculated by combining Shi-Tomasi corner detection with LK optical flow method to obtain optical flow motion characteristics.

[0018] The global semantic features are obtained in the following way:

[0019] Each frame of ultrasound image in the preprocessed ultrasound image sequence is divided into several non-overlapping windows, and all feature pixels in each window are linearly projected to obtain a serialized embedding vector.

[0020] Self-attention is calculated for the serialized embedding vectors within each window. Feature interaction between different windows is achieved through a shift window mechanism or spatial reduction attention. A learnable sparse mask is introduced to suppress background noise regions and obtain features at different levels.

[0021] A feature pyramid structure is constructed to fuse features at different levels, resulting in a multi-scale fused global feature.

[0022] A query-based feature matching strategy is adopted, and attention interaction is performed between the learnable target query vector and the global features fused by the multi-scale to obtain the final global semantic features.

[0023] The temporal enhancement features are obtained through the following methods:

[0024] The dynamic motion features and fine-grained spatial features of the preprocessed ultrasound image sequence are captured by a dual-channel CNN feature extraction network. Then, the dynamic motion features and fine-grained spatial features are complementaryly fused through cross-information transmission and time attention mechanisms to obtain temporal enhancement features.

[0025] The colored information auxiliary features are obtained through the following method:

[0026] Gaussian filtering is applied to each frame of the preprocessed ultrasound image sequence to remove high-frequency noise.

[0027] Convert the color space of each frame of ultrasound image after removing high-frequency noise to HSV color space or Lab color space.

[0028] The Sobel operator was used to detect abrupt changes in pixel values ​​in ultrasound images after color space conversion, and preset threshold ranges were set for different target colored features based on HSV or Lab separation channels to obtain several blood flow Doppler color patches.

[0029] Isolated blood flow Doppler patches are removed, and the geometric features of the remaining blood flow Doppler patches are calculated.

[0030] Time series analysis is performed on the blood flow Doppler color patches extracted from each frame of ultrasound image to calculate the temporal change trend, and the geometric features are fused to obtain colored information auxiliary features.

[0031] The cross-modal self-attention fusion encoder includes:

[0032] The feature alignment module is used to perform linear transformations on the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features, so that they are mapped to the same feature space.

[0033] An adaptive weighted fusion module is used to assign weights to the optical flow motion features, global semantic features, temporal enhancement features and colored information auxiliary features through a gating attention mechanism and then fuse them to obtain preliminary fused features.

[0034] The cross-modal self-attention module is used to establish the interrelationships between the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features using a self-attention mechanism to obtain cross-modal features.

[0035] The layer-by-layer fusion module is used to fuse fused features and cross-modal features to obtain fused features.

[0036] When constructing a graph structure based on the candidate box set, the candidate boxes in each frame of the ultrasound image sequence are used as graph nodes, and the edge connections of the graph nodes are constructed through spatial proximity, temporal consistency and feature similarity; the features of the graph nodes include category information, location information and temporal features.

[0037] The Transformer-based graph neural network includes:

[0038] The GNN propagation module is used to model cross-frame relationships using Transformer and introduces a dynamic attention mechanism to adaptively adjust the relationships between candidate boxes.

[0039] The target node guidance module is used to take the preset key node bounding boxes as target nodes, calculate the feature similarity weight between the feature information of the target node and the feature information of the graph node, and adjust the confidence of the candidate box according to the feature similarity weight to obtain the corrected confidence.

[0040] The temporal alignment module is used to align the features of the candidate boxes in the current frame with the features of the candidate boxes in the previous frame after adaptive adjustment by the GNN propagation module and calculate the feature similarity. Based on the feature similarity and optical flow prediction position deviation, the temporal stability score is calculated, and the temporal stability score and the corrected confidence score are weighted and fused to obtain the final confidence score.

[0041] The technical solution adopted by the present invention to solve its technical problem is: to provide a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-mentioned end-to-end multimodal ultrasound imaging lesion automatic localization method are implemented.

[0042] Beneficial effects

[0043] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0044] This invention integrates multi-dimensional features such as optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features to align with the multi-dimensional observation behaviors of doctors during clinical scanning, avoiding information loss from manual integration and improving the overall system integrity and clinical interpretability. Furthermore, this invention uses graph neural networks to model the spatiotemporal dependence of candidate boxes, replacing traditional hard thresholding post-processing, and combines temporal alignment to optimize confidence, effectively suppressing detection jitter and single-frame false detections caused by noise and artifacts, thereby improving the robustness, real-time performance, and cross-device adaptability of the AI-assisted ultrasound diagnostic system.

[0045] This invention also designs a lightweight Transformer and a dual-channel system with fast and slow frames, significantly reducing computational overhead while balancing long-distance feature modeling and real-time performance, adapting to dynamic scanning scenarios of ultrasound equipment. This invention extracts common visual elements from the equipment, such as color information, and combines this with intelligent classification of the scanned area, reducing dependence on parameters of specific ultrasound equipment, significantly improving the model's cross-equipment generalization ability, and expanding its clinical applicability. Attached Figure Description

[0046] Figure 1 This is a flowchart of the end-to-end multimodal ultrasound imaging automatic lesion localization method according to the first embodiment of the present invention;

[0047] Figure 2 This is a block diagram of the end-to-end multimodal ultrasound imaging automatic lesion localization method according to the first embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0049] The first embodiment of the present invention relates to an end-to-end multimodal ultrasound imaging automatic lesion localization method. This method integrates optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features through an end-to-end network, and combines graph structure modeling of the spatiotemporal dependencies between candidate boxes to achieve intelligent understanding and real-time accurate localization of the dynamic ultrasound scanning process. It can be widely applied to ultrasound image analysis of superficial and deep tissues such as thyroid, breast, abdominal organs, and blood vessels, significantly improving the robustness, real-time performance, and cross-device adaptability of AI-assisted ultrasound diagnostic systems.

[0050] like Figure 1 and Figure 2 As shown, the end-to-end multimodal ultrasound imaging lesion automatic localization method of this embodiment includes the following steps:

[0051] Step 1: Acquire ultrasound image sequences in real time and preprocess the ultrasound image sequences.

[0052] In this step, the preprocessing of the ultrasound image sequence specifically involves: setting a region of interest (ROI) for each frame in the ultrasound image sequence, and performing Gaussian filtering on the ROI to reduce noise interference, thus providing basic data for subsequent multimodal feature extraction.

[0053] Step 2: Extract features from the preprocessed ultrasound image sequence to obtain optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features.

[0054] In this step, the optical flow field of the region of interest in each frame of the preprocessed ultrasound image sequence is calculated using Shi-Tomasi corner detection combined with the Lucas-Kanade (LK) optical flow method to obtain the optical flow motion characteristics. The basic principle of the LK optical flow method is as follows:

[0055] Let the grayscale value of the ultrasound image be... Assuming the feature point is in The motion of time is caused by If we express this, then we have:

[0056] ;

[0057] Take a first-order Taylor expansion:

[0058] ;

[0059] in, , These are the spatial gradients of the image. This represents the time gradient. Extending it to a local window yields a system of linear equations:

[0060] ;

[0061] in, , By solving Optical flow characteristics can be obtained.

[0062] In this step, a lightweight Transformer structure is introduced to extract features from each frame of the ultrasound image sequence to obtain global semantic features, as detailed below:

[0063] First, the ultrasound image is divided into several non-overlapping windows of fixed size. Then, all feature pixels within each window are linearly projected, converting them into one-dimensional sequential embedding vectors. This results in a sequence of multiple embedding vectors. This step fuses the spatial features within a window into a single embedding vector, preserving local spatial information while reducing feature computation redundancy and lowering the dimensionality of subsequent calculations.

[0064] Next, self-attention calculation is performed on the serialized embedding vectors within each window to capture local details; then, feature interaction between different windows is achieved through a shift window mechanism or spatial reduction attention, realizing cross-window feature fusion, maintaining global dependency modeling capability, while introducing learnable sparse masks to suppress background noise regions, obtaining features at different levels, and strengthening the attention weight of lesion-related features.

[0065] Then, a feature pyramid structure is constructed to fuse features at different levels. This means that shallow features retain high-resolution spatial detail information (such as the edge of the lesion and the pixel features of small-scale lesions), while deep features extract high-order semantic information (such as the category attributes of the lesion and the semantic association with surrounding tissues). Deep low-resolution features are restored to shallow resolution through upsampling, and shallow and deep features of the same resolution are spliced ​​and fused through lateral connection to finally obtain the global features after multi-scale fusion.

[0066] Finally, a query-based feature matching strategy is adopted. The learnable target query vector interacts with the multi-scale fused global features through attention interaction to obtain the final global semantic features. This step abandons the dense prediction and decoding methods of native Transformer and DETR models, and adopts a query-based feature matching strategy. Specifically, a set of learnable target query vectors is preset, and these vectors directly interact with the global features output by the encoder through attention interaction. The final global semantic features are output only through updates to the query vectors. This approach maintains the advantages of Transformer's long-range modeling while significantly reducing computational overhead. It utilizes a global self-attention mechanism to model long-range pixel relationships, enhancing feature extraction capabilities under complex backgrounds and noise interference, and outputting global semantic features.

[0067] In this step, a dual-channel CNN feature extraction network is used to capture the dynamic motion features and fine-grained spatial features of the preprocessed ultrasound image sequence. Then, the dynamic motion features and fine-grained spatial features are complementaryly fused through cross-information transmission and temporal attention mechanisms to obtain temporal enhancement features.

[0068] In this embodiment, the dual-channel CNN feature extraction network includes a fast frame channel and a slow frame channel, which are designed in parallel and each undertakes a different feature extraction task.

[0069] The fast frame channel employs a high sampling rate to sample ultrasound image sequences frame by frame, preserving continuous inter-frame information as much as possible. It avoids or performs only minimal downsampling to ensure the integrity of dynamic information. It can utilize a lightweight CNN network with fewer layers, fewer convolutional kernels, and lower computational cost, avoiding the impact of complex calculations on real-time performance. This fast frame channel focuses on the dynamic temporal information of ultrasound images, including probe motion trajectory, real-time tissue deformation, and the trend of lesion position changes between frames. This information is crucial for achieving dynamic lesion tracking and is a core element missing from traditional single-frame detection methods.

[0070] The slow frame channel uses a low sampling rate to sample ultrasound image sequences, reducing the number of input frames and lowering the computational overhead of subsequent deep networks. It also allows for more refined feature extraction from individual frames. This can be achieved using a deeper CNN network, employing multiple convolutional and pooling operations to progressively extract high-dimensional spatial features, preserving more fine-grained texture information. This slow frame channel focuses on the spatial texture details of individual ultrasound images, including the edge contours of lesions, internal echo characteristics, boundary differences between lesions and surrounding normal tissues, and anatomical features of organs. This information is fundamental for accurate lesion localization and feature discrimination.

[0071] The features extracted from the fast frame channel and the slow frame channel are independent of each other. Directly concatenating them would lead to feature redundancy and insufficient information interaction. Therefore, this implementation adopts a cross-information transfer mechanism and a temporal attention mechanism to achieve deep fusion and intelligent weighting of the two types of features. The cross-information transfer mechanism can combine the motion features of spatial texture to effectively eliminate motion artifacts without texture background and retain only the true dynamic information of the lesion area. The temporal attention mechanism can adaptively filter effective temporal information, reduce the interference of invalid frames and blurry frames on features, and strengthen the key frame information with the clearest lesion features, thereby improving the robustness of temporal features.

[0072] Color Doppler, measurement scales, elastography, and other non-grayscale information are crucial components of ultrasound scanning. This information not only helps doctors accurately determine lesion characteristics but also constructs more complete logical relationships in image analysis. Therefore, this step also extracts colored information auxiliary features using the following methods:

[0073] First, Gaussian filtering is applied to each frame of the preprocessed ultrasound image sequence to remove high-frequency noise. This method eliminates noise interference and provides a basis for subsequent color patch extraction.

[0074] Next, the color space of each frame of ultrasound image after removing high-frequency noise is converted to the HSV color space or Lab color space. This method decouples the color of the target color features (red / blue Doppler, black and white scale) from the background, greatly improving the distinction between the target colored features (red / blue Doppler, black and white scale) and the background.

[0075] Then, the Sobel operator is used to detect abrupt changes in pixel values ​​in the ultrasound image after color space conversion, enhancing the color boundaries of colored information (such as the boundaries between Doppler red / blue areas and surrounding grayscale tissues, and the line boundaries of the measurement scale), making the blurred feature boundaries clear; then, based on the HSV or Lab separation channel, preset threshold ranges are set for different target colored features to achieve the extraction of different color blocks, resulting in several blood flow Doppler color blocks (obvious red and blue areas in ultrasound images) and measurement scale color blocks (black or white areas in ultrasound images).

[0076] Subsequently, isolated blood flow Doppler patches are removed, and the geometric features (shape, area, positional distribution, etc.) of the remaining blood flow Doppler patches are calculated. These geometric features can be used to determine the size, orientation, and dynamic characteristics of the blood flow Doppler patches. Simultaneously, the measured scale patches can be obtained, their relative positions in the ultrasound image are analyzed, and the scale intervals are identified to assist in the spatial scale correction of the ultrasound image.

[0077] Finally, time-series analysis (such as changes in the area and position of the blood flow Doppler color patch) is performed on the blood flow Doppler color patch extracted from each frame of ultrasound image, the temporal trend is calculated, and the geometric features are fused to obtain the colored information auxiliary features.

[0078] Step 3: Input the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features into the cross-modal self-attention fusion encoder to obtain the fused features.

[0079] To address the spatial, temporal, and semantic differences among multimodal features, the cross-modal self-attention fusion encoder in this embodiment achieves adaptive fusion of different modal features through self-attention and gating mechanisms. This cross-modal self-attention fusion encoder includes:

[0080] The feature alignment module is used to perform a linear transformation on the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features, so that they are mapped to the same feature space, which can be represented as:

[0081] ;

[0082] in, and These are learnable parameters that ensure all features are fused along the same dimension. These are the features after linear transformation. Indicates the characteristics of optical flow motion. Represents global semantic features. Indicates temporal enhancement features, This represents auxiliary features for colored information.

[0083] The adaptive weighted fusion module is used to assign weights to the optical flow motion features, global semantic features, temporal enhancement features and colored information auxiliary features through a gating attention mechanism and then fuse them to obtain preliminary fused features.

[0084] The weights of each feature in this embodiment can be calculated using the following formula:

[0085] ;

[0086] Preliminary fusion characteristics can be represented as:

[0087] ;

[0088] in, For learnable attention weights, The weights for each feature ensure that the model automatically focuses on key features. This represents the initial fusion characteristics.

[0089] The cross-modal self-attention module is used to establish the interrelationships between the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features using a self-attention mechanism, thereby obtaining cross-modal features.

[0090] After completing the initial weighted fusion, this implementation further employs a self-attention mechanism to establish the interrelationships between modalities, thereby enhancing information interaction between different modalities. Let the initial fusion features be... After transformation through Query, Key, and Value, that is: ;

[0091] The cross-modal feature can then be expressed as:

[0092] ;

[0093] ;

[0094] in, It is the feature dimension. This indicates the calculation of self-attention weights, which enables the model to focus on the most relevant modal features.

[0095] The layer-by-layer fusion module is used to fuse the initial fusion features and cross-modal features to obtain the fused features. To further improve the fusion quality, this implementation stacks multiple fusion layers inside the cross-modal self-attention fusion encoder. The output of each layer is used as the input of the next layer, which can be represented as:

[0096] ;

[0097] in, For the first The output of the layer, when the first When the layer is the last layer, its output is the fused feature. This layer-by-layer fusion method can continuously optimize the feature representation, making the final fused feature more stable and robust.

[0098] Step 4: Based on the fused features, a candidate bounding box set is generated using the detection head, and a graph structure is constructed based on the candidate bounding box set. In this step, when constructing the graph structure based on the candidate bounding box set, the candidate bounding boxes in each frame of the ultrasound image sequence are used as graph nodes, and edge connections between the graph nodes are constructed using spatial proximity, temporal consistency, and feature similarity. The features of the graph nodes include category information, location information (normalized bounding box coordinates), and temporal features (inter-frame variation trends). Spatial proximity can be calculated using IoU, temporal consistency can be measured using KL divergence, and feature similarity can be calculated using CNN visual feature distance.

[0099] Step 5: Input the graph structure into a Transformer-based graph neural network for dynamic relationship learning to obtain the lesion location and confidence level, and determine the scanned area and state category of each frame using an MLP classification head. The Transformer-based graph neural network in this step includes:

[0100] The GNN propagation module is used to model cross-frame relationships using Transformer and introduces a dynamic attention mechanism to adaptively adjust the relationships between candidate boxes. It can be represented as:

[0101] ;

[0102] in, These are the node features after being processed by the GNN propagation module. The adjacency matrix describes the relationships between candidate boxes and is represented as follows: , The vector generated from the feature maps of the candidate boxes. This is the learnable output projection matrix.

[0103] The target node guidance module is used to select preset key node bounding boxes as target nodes, calculate the feature similarity weights between the feature information of the target nodes and the feature information of the graph nodes, and adjust the confidence of the candidate boxes according to the feature similarity weights to obtain the corrected confidence. Among them, feature similarity weight The calculation method is as follows:

[0104] ;

[0105] in, For the first The features of the candidate box, i.e., the th candidate box in the graph structure. Features of each graph node Features of the target node.

[0106] The temporal alignment module is used to align the features of the candidate bounding boxes in the current frame. Features of the candidate boxes after adaptive adjustment by the GNN propagation module in the previous frame Perform alignment and calculate feature similarity Based on feature similarity Optical flow prediction position deviation Calculate the time series stability score And time series stability score and the corrected confidence level We perform weighted fusion to obtain the final confidence level.

[0107] Among them, feature similarity The calculation method is as follows: Temporal stability score The calculation method is as follows: ,in, This is the balance coefficient.

[0108] The Transformer-based graph neural network operation in this step effectively suppresses false detections in a single frame, enhances target locking stability between consecutive frames, and obtains the lesion location and confidence level. Finally, the MLP classification head can output the scanning location and status category of the current frame (left thyroid, right thyroid, thyroid isthmus, breast longitudinal section, carotid artery transverse section, etc.), thereby assisting doctors in standardizing the scanning procedure.

[0109] It is easy to see that this invention, by fusing multi-dimensional features such as optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features, aligns with the multi-dimensional observation behaviors of doctors during clinical scanning, avoiding information loss from manual integration and improving the overall system integrity and clinical interpretability. Furthermore, this invention uses graph neural networks to model the spatiotemporal dependence of candidate boxes, replacing traditional hard thresholding post-processing, and combines temporal alignment to optimize confidence, effectively suppressing detection jitter and single-frame false detections caused by noise and artifacts, improving the robustness, real-time performance, and cross-device adaptability of the AI-assisted ultrasound diagnostic system. This invention also designs a lightweight Transformer and fast / slow frame dual-channel, significantly reducing computational overhead, balancing long-distance feature modeling and real-time performance, and adapting to dynamic scanning scenarios of ultrasound equipment. This invention extracts common visual elements of the equipment, such as colored information, and combines them with intelligent classification of the scanned area, reducing dependence on parameters of specific ultrasound equipment, significantly improving the model's cross-device generalization ability, and expanding its clinical applicability.

[0110] The second embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the end-to-end multimodal ultrasound imaging lesion automatic localization method of the first embodiment.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0112] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An end-to-end multimodal ultrasound imaging method for automatic lesion localization, characterized in that, Includes the following steps: Acquire ultrasound image sequences in real time and preprocess the ultrasound image sequences; Feature extraction is performed on the preprocessed ultrasound image sequence to obtain optical flow motion features, global semantic features, temporal enhancement features, and color information auxiliary features; The optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features are input into a cross-modal self-attention fusion encoder to obtain fused features. Based on the fusion features, a set of candidate boxes is generated using the detection head, and a graph structure is constructed based on the set of candidate boxes; The graph structure is input into a Transformer-based graph neural network for dynamic relationship learning to obtain the lesion location and confidence level.

2. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, The preprocessing of the ultrasound image sequence specifically involves: setting a region of interest for each frame in the ultrasound image sequence and performing Gaussian filtering on the region of interest.

3. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 2, characterized in that, The optical flow motion characteristics are obtained in the following way: The optical flow field of each region of interest in the preprocessed ultrasound image sequence was calculated using Shi-Tomasi corner detection combined with LK optical flow method to obtain optical flow motion characteristics.

4. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, The global semantic features are obtained in the following way: Each frame of ultrasound image in the preprocessed ultrasound image sequence is divided into several non-overlapping windows, and all feature pixels in each window are linearly projected to obtain a serialized embedding vector. Self-attention is calculated for the serialized embedding vectors within each window. Feature interaction between different windows is achieved through a shift window mechanism or spatial reduction attention. A learnable sparse mask is introduced to suppress background noise regions and obtain features at different levels. A feature pyramid structure is constructed to fuse features at different levels, resulting in a multi-scale fused global feature. A query-based feature matching strategy is adopted, and attention interaction is performed between the learnable target query vector and the global features fused by the multi-scale to obtain the final global semantic features.

5. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, The temporal enhancement features are obtained through the following methods: The dynamic motion features and fine-grained spatial features of the preprocessed ultrasound image sequence are captured by a dual-channel CNN feature extraction network. Then, the dynamic motion features and fine-grained spatial features are complementaryly fused through cross-information transmission and time attention mechanisms to obtain temporal enhancement features.

6. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, The colored information auxiliary features are obtained through the following method: Gaussian filtering is applied to each frame of the preprocessed ultrasound image sequence to remove high-frequency noise. Convert the color space of each frame of ultrasound image after removing high-frequency noise to HSV color space or Lab color space. The Sobel operator was used to detect abrupt changes in pixel values ​​in ultrasound images after color space conversion, and preset threshold ranges were set for different target colored features based on HSV or Lab separation channels to obtain several blood flow Doppler color patches. Isolated blood flow Doppler patches are removed, and the geometric features of the remaining blood flow Doppler patches are calculated. Time series analysis is performed on the blood flow Doppler color patches extracted from each frame of ultrasound image to calculate the temporal change trend, and the geometric features are fused to obtain colored information auxiliary features.

7. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, The cross-modal self-attention fusion encoder includes: The feature alignment module is used to perform linear transformations on the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features, so that they are mapped to the same feature space. An adaptive weighted fusion module is used to assign weights to the optical flow motion features, global semantic features, temporal enhancement features and colored information auxiliary features through a gating attention mechanism and then fuse them to obtain preliminary fused features. The cross-modal self-attention module is used to establish the interrelationships between the optical flow motion features, global semantic features, temporal enhancement features, and colored information auxiliary features using a self-attention mechanism to obtain cross-modal features. The layer-by-layer fusion module is used to fuse fused features and cross-modal features to obtain fused features.

8. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 1, characterized in that, When constructing a graph structure based on the candidate box set, the candidate boxes in each frame of the ultrasound image sequence are used as graph nodes, and the edge connections of the graph nodes are constructed through spatial proximity, temporal consistency and feature similarity; the features of the graph nodes include category information, location information and temporal features.

9. The end-to-end multimodal ultrasound imaging lesion automatic localization method according to claim 8, characterized in that, The Transformer-based graph neural network includes: The GNN propagation module is used to model cross-frame relationships using Transformer and introduces a dynamic attention mechanism to adaptively adjust the relationships between candidate boxes. The target node guidance module is used to take the preset key node bounding boxes as target nodes, calculate the feature similarity weight between the feature information of the target node and the feature information of the graph node, and adjust the confidence of the candidate box according to the feature similarity weight to obtain the corrected confidence. The temporal alignment module is used to align the features of the candidate boxes in the current frame with the features of the candidate boxes in the previous frame after adaptive adjustment by the GNN propagation module and calculate the feature similarity. Based on the feature similarity and optical flow prediction position deviation, the temporal stability score is calculated, and the temporal stability score and the corrected confidence score are weighted and fused to obtain the final confidence score.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the end-to-end multimodal ultrasound imaging lesion auto-localization method as described in any one of claims 1-9.

Images