A multi-modal deep learning driven multi-dimensional ultrasound image intelligent processing system

Abstract

The application discloses a kind of multi-modal deep learning driven multi-dimensional ultrasonic image intelligent processing systems, it is related to image processing technical field, for solving the problem of multi-modal ultrasonic fusion analysis difference;The application is by collecting B mode, color doppler, elasticity and radio frequency image, and constructs uniform structured tensor, realizes the time synchronization and spatial registration of multi-modal image, introduces cross attention mechanism to align and fuse modal characteristics, enhances feature expression ability, extracts dynamic characteristics and spatial topological relationship of tissue evolution with time in combination with time series modeling and graph structure modeling method, form joint space-time feature representation, in output end, system supports regional segmentation, structure classification and function parameter extraction etc. Multi-task path, satisfy clinical diversification analysis demand, overall architecture is end-to-end deep learning process, with high automation and cross-modal adaptability, improve ultrasonic image analysis efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images. Background Technology

[0002] With the development of medical imaging technology, ultrasound examination has been widely used in clinical diagnosis, disease screening and functional assessment due to its safety, non-invasiveness and real-time performance. In recent years, ultrasound equipment has gradually evolved from traditional two-dimensional imaging to three-dimensional and four-dimensional imaging, while integrating multiple imaging modalities such as elastography, color Doppler and radio frequency raw signals to form multidimensional ultrasound image data with rich spatiotemporal structure and physiological information.

[0003] The shortcomings of existing technologies: Most existing ultrasound image processing systems focus on the processing and analysis of single-modal data and lack a deep collaborative modeling mechanism for information from different modalities. Specifically, in the fusion of multimodal ultrasound images, current mainstream methods rely on traditional image processing techniques such as template matching, interpolation reconstruction, or image registration, which make it difficult to achieve semantic consistency and structural alignment between modalities. This often results in problems such as blurred boundaries and tissue mismatch in the fused images, which seriously affects the subsequent analysis results. Summary of the Invention

[0004] To overcome the aforementioned deficiencies in the prior art, the following solution is proposed to address the problem of poor multimodal ultrasound fusion analysis in the aforementioned background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images includes an image data acquisition module, a fusion feature module, an image spatial topology module, and a result output module, with each module connected by a signal.

[0007] The image data acquisition module acquires B-mode images, color Doppler images, elastic images, and radio frequency signal images, performs time synchronization and spatial alignment operations, and converts them into structured tensors of a unified dimension through a modal embedding encoder.

[0008] The feature fusion module takes a structured tensor as input to the cross-attention fusion network, performs inter-modal feature alignment and joint modeling, and outputs a multimodal fusion feature sequence.

[0009] The image spatial topology module takes the multimodal fusion feature sequence as input to the temporal modeling network, extracts the dynamic features of the image as it changes over time, and combines graph structure modeling methods to generate spatial topological constraint representations, thus constructing a joint spatiotemporal feature sequence.

[0010] The results output module takes the joint spatiotemporal feature sequence as input to the multi-task decoding path and outputs the target region segmentation map, organizational structure category label, and functional analysis parameters according to the task type.

[0011] In a preferred embodiment, the image data acquisition module includes the following:

[0012] Acquire multimodal ultrasound image data, including B-mode images, color Doppler images, elastography images, and radiofrequency signal images;

[0013] The optimal set of frame pairs is constructed based on the inter-modal structural similarity, and time-coordinated frame selection is performed to achieve time synchronization of images.

[0014] During spatial registration, a pseudo-image mask is generated to guide alignment and spatial registration is performed.

[0015] The synchronously registered image is input into a modality-aware embedding network, which integrates the modality indication vector and the spatial coordinate position encoding to output a structured tensor of uniform dimension.

[0016] In a preferred embodiment, the feature fusion module includes the following:

[0017] The structured tensor sequence is split modally along the channel dimension to obtain multiple single-modal feature tensors;

[0018] Perform a shared convolution mapping on each modality feature tensor to generate a query vector, a key vector, and a value vector, respectively.

[0019] Attention weights are calculated based on the similarity between queries and keys across modalities, and weighted combinations are performed on the value vectors to generate cross-modal fusion features.

[0020] In a preferred embodiment, the calculation process for attention weights includes:

[0021] For each modality's feature representation, generate a query vector, a key vector, and a value vector respectively;

[0022] Calculate the feature dimension dot product between the query vector and the key vector for any two modalities, and divide the dot product result by the square root of the vector dimension as a scaling factor;

[0023] The scaled dot product of all modal pairs is input into a row-normalized exponential weighting function to generate the relative attention weights between modal pairs.

[0024] Relative attention is applied to the value vector of the corresponding modality, and a weighted summation operation is performed to obtain the fused feature output.

[0025] In a preferred embodiment, the image spatial topology module includes the following:

[0026] Multimodal fusion feature sequences are collected, and the fusion features of each frame are input into a temporal modeling network in chronological order to extract the dynamic evolution features of the image in the time dimension;

[0027] The fused features of each frame are expanded into a vector sequence in the spatial dimension and input into the temporal modeling network frame by frame to perform state updates and generate the temporal feature representation of the corresponding frame.

[0028] Within the image space dimension, feature points of the target region are selected as nodes of the graph structure. Edge connections are constructed based on the spatial proximity or feature similarity between nodes to generate a graph structure representation.

[0029] The graph structure is input into a graph neural network for modeling, and through several rounds of information transmission between nodes, a graph feature representation containing spatial topological constraints is obtained;

[0030] The temporal feature representation and graph feature representation of each frame are fused along the position dimension to construct a joint spatiotemporal feature sequence.

[0031] In a preferred embodiment, within the image spatial dimension, feature points of the target region are selected as graph structure nodes. Edge connections are constructed based on the spatial proximity or feature similarity between nodes to generate a graph structure representation. Specifically, this includes:

[0032] Within the spatial dimension of each frame image, feature points in the target region with response intensity higher than a set response threshold are selected as graph structure nodes.

[0033] Calculate the Euclidean distance between each pair of nodes and the cosine similarity between their corresponding feature vectors. The Euclidean distance reflects the geometric proximity of their spatial locations, while the cosine similarity reflects the consistency of their semantic features.

[0034] When the Euclidean distance between two nodes is lower than a preset spatial proximity threshold, or the feature similarity is higher than a preset similarity threshold, an edge connection is established between the two nodes.

[0035] Generate a graph structure representation containing node attributes and edge weights based on all valid edge connections.

[0036] In a preferred embodiment, the graph structure is input into a graph neural network for modeling. Through several rounds of inter-node information transfer, a graph feature representation containing spatial topological constraints is obtained. Specifically, this includes:

[0037] The graph neural network iteratively updates by using the feature vector of each graph node as the initial node representation and the edge weights between nodes as the modulation factor for information transmission.

[0038] In each round of information transmission, the feature representations of the target node and its directly adjacent nodes are aggregated, weighted and fused according to the edge connection weights, and the current feature state of the target node is updated.

[0039] Repeatedly perform information transmission and state update operations, and aggregate the features of directly adjacent nodes;

[0040] After all rounds of updates are completed, the output is a graph feature representation containing spatial topological relationships, which is used for the spatiotemporal joint representation of image processing tasks.

[0041] In a preferred embodiment, the result output module includes the following:

[0042] When the task type is region segmentation, the joint spatiotemporal features of each frame are input into the upsampling decoding structure, the resolution is restored according to a fixed level, and the target region segmentation map with the same size as the original image is output for pixel-level structural boundary recognition.

[0043] When the task type is structural classification, the temporal dimension aggregation operation is performed on the joint spatiotemporal feature sequence to extract the global feature vector of the sequence, and input into the fully connected classification structure to output the category label of the organizational structure;

[0044] When the task type is functional parameter extraction, the joint spatiotemporal feature sequence is input into the regression neural network structure, and the output is a numerical functional parameter that reflects the evolution of the tissue region over time, including compression ratio, elasticity change value or echo amplitude fluctuation rate.

[0045] Based on the task type settings, select the corresponding decoding path and output the corresponding analysis results.

[0046] The technical effects and advantages of the multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images of this invention are as follows:

[0047] This invention addresses the challenge of registering images across different modalities in terms of time and space by acquiring B-mode, color Doppler, elastography, and radio frequency images and constructing a unified structured tensor representation. The system introduces a cross-attention mechanism to achieve feature alignment and information enhancement between modalities, improving the expressiveness and discriminative power of fused features. Combining temporal modeling networks and graph structure modeling techniques, it jointly models the dynamic evolution and spatial topological relationships of tissue states in image sequences, accurately depicting the structure and changes of target regions. At the output end, the system supports multiple task paths, including region segmentation, structural classification, and functional parameter extraction, providing targeted support for different types of clinical analysis. The overall solution employs an end-to-end deep learning processing flow, possessing excellent automation capabilities and cross-modal adaptability, making it widely applicable to intelligent ultrasound image processing needs in complex clinical scenarios, improving the accuracy and efficiency of image-assisted analysis. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of a multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In order to achieve the above objectives, Figure 1 A schematic diagram of the structure of a multimodal deep learning-driven multidimensional ultrasound image intelligent processing system of the present invention is given, which specifically includes an image data acquisition module, a fusion feature module, an image spatial topology module, and a result output module, and the modules are connected by signals.

[0051] The image data acquisition module acquires B-mode images, color Doppler images, elastic images, and radio frequency signal images, performs time synchronization and spatial alignment operations, and converts them into structured tensors of a unified dimension through a modal embedding encoder.

[0052] The feature fusion module takes a structured tensor as input to the cross-attention fusion network, performs inter-modal feature alignment and joint modeling, and outputs a multimodal fusion feature sequence.

[0053] The image spatial topology module takes the multimodal fusion feature sequence as input to the temporal modeling network, extracts the dynamic features of the image as it changes over time, and combines graph structure modeling methods to generate spatial topological constraint representations, thus constructing a joint spatiotemporal feature sequence.

[0054] The results output module takes the joint spatiotemporal feature sequence as input to the multi-task decoding path and outputs the target region segmentation map, organizational structure category label, and functional analysis parameters according to the task type.

[0055] The specific steps of the image data acquisition module are as follows:

[0056] First, multimodal ultrasound image data of the area under examination are acquired. The acquired images include grayscale B-mode images, color Doppler images, elastography images, and radiofrequency signal images, which correspond to multiple dimensions such as anatomical structure, blood flow status, tissue elasticity, and acoustic reflection characteristics, respectively. All images are continuously acquired through a unified ultrasound platform to ensure synchronous time stamps and spatial coverage.

[0057] To achieve temporal synchronization of images, image frame sequences of each modality are extracted sequentially. Frame pair matching of different modalities is performed based on the inter-frame structural similarity index. The calculation of structural similarity includes, but is not limited to, the structural similarity index (SSIM) or gradient direction consistency measure. By constructing the optimal set of frame pairs, the frame group that is structurally closest is selected, and temporal collaborative frame selection is performed accordingly to ensure that the image sequences in subsequent processing have a consistent temporal reference benchmark.

[0058] After time synchronization is completed, the images are spatially aligned. The grayscale B-mode image is used as a reference image to extract its grayscale distribution and edge features, generating a pseudo-image mask for structure guidance. Using this mask as a registration template, other modal images are spatially registered according to edge structure and grayscale similarity. Rigid or elastic registration algorithms are used to correct scale, rotation and local deformation, so that different modal images maintain structural consistency in spatial dimension.

[0059] All synchronized and registered images are encoded according to their modality labels. The modality identification information is jointly encoded with the two-dimensional spatial coordinates corresponding to each pixel in the image. A positional encoding vector is introduced to enhance spatial perception. For each image, feature extraction, encoding transformation and dimension mapping are performed in sequence. Finally, a structured tensor with a unified format is output. This tensor serves as the input basis for subsequent multimodal fusion and spatiotemporal modeling.

[0060] Using a grayscale B-mode image as a reference image, its grayscale distribution and edge features are extracted to generate a pseudo-image mask for structure guidance. The specific process is as follows:

[0061] First, the original B-mode image is normalized by mapping the gray value of each pixel in the image to the [0, 1] interval according to the maximum gray value range. This eliminates the influence of the quantization deviation of the imaging device on subsequent edge detection. This normalization process ensures that the image brightness values ​​are distributed on a uniform scale, which is beneficial to the stability and accuracy of edge detection.

[0062] Gaussian filtering is used to smooth the normalized image to remove high-frequency noise. The filter window size is typically set to 3×3 or 5×5, and the standard deviation is determined based on the image noise level, generally between 0.8 and 1.2. This processing aims to preserve structural contours while suppressing random texture noise, ensuring that the edge extraction stage is not affected by spurious signals.

[0063] A multi-directional gradient response analysis method is employed to detect intensity changes in image grayscale values ​​along the horizontal, vertical, and diagonal directions. The gradient magnitude of each pixel in each direction is assessed, and if the gradient magnitude in a certain direction exceeds a preset threshold, the pixel is marked as a candidate edge point. This threshold can be adaptively set based on the standard deviation of the overall image grayscale changes to balance the sensitivity and stability of edge detection. Connectivity analysis is performed on the candidate edge point set to identify continuous edge segments, and linear interpolation is used to complete discontinuous regions, ensuring that the target region edges present a closed loop or a complete structure. This completion process is achieved by tracking the edge direction and the grayscale consistency with adjacent pixels, avoiding registration errors caused by edge breaks.

[0064] The resulting edge structure is then dilated to appropriately expand the coverage area of ​​the edge region. The dilation radius is set according to the image resolution, typically between 1 and 3 pixels. The resulting binary mask image is the pseudo-image mask, where edge regions are marked as 1 and non-edge regions as 0. This mask serves as a structural guidance signal in subsequent modal image registration, guiding other image modalities to spatially align with the reference image.

[0065] After completing the temporal synchronization and spatial registration of multimodal ultrasound images, further encoding processing is performed on the image sequences of each modality to construct a structured tensor in a unified format, as follows:

[0066] Each image is already associated with a specific modality category (such as B-mode image, color Doppler image, elasticity image, or radio frequency signal image) at the time of acquisition. To preserve the feature differences between modalities, a unique modality identifier code is assigned to each modality before image encoding. For example, four-dimensional vectors with values ​​[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], and [0, 0, 0, 1] can be assigned to the four modalities respectively as modality recognition labels. Before performing feature processing on the image, this modality label is jointly embedded with the image data. That is, a set of corresponding modality identifier information is attached to each pixel in the image, thereby guiding the subsequent feature extraction process to retain modality perception capability.

[0067] Spatial location coding is introduced, which converts the two-dimensional spatial coordinates (row number i, column number j) of each pixel into a set of numerical vectors as supplementary location information for that pixel. Specific processing includes numerical projection of the horizontal and vertical coordinates, such as linear normalization to the [0, 1] interval, or the use of a location embedding form based on sine and cosine functions (e.g., converting i to sin(i) or cos(i) forms), to enhance the neural network's ability to analyze location information. This location coding, together with the original image grayscale values ​​and modality identifiers, forms a pixel-level multi-channel input vector, thereby guiding the model to simultaneously focus on image content, modality type, and spatial location during the feature extraction stage.

[0068] An image incorporating modal identifiers and spatial locations is input into a feature extraction structure, where multi-scale convolution operations are performed to extract image features such as edges, textures, and deformations. The feature extraction structure can employ a network architecture consisting of stacked convolutional layers and activation functions, processing the image layer by layer and outputting intermediate feature maps across multiple channels. These feature maps contain both local texture details and global structural features, possessing strong expressive power.

[0069] Since the original dimensions of images from different modalities may differ in resolution, number of channels, or grayscale depth, a specific projection layer is needed to map the features of all images to a unified tensor representation format. This projection layer can use a 1×1 convolutional or fully connected structure to compress or expand all channels to a specified number (e.g., uniformly map to 64 channels). The mapped tensor structure is: image width × image height × unified number of channels.

[0070] Modal decomposition of structured tensors: The structured tensor sequence obtained in the preceding sequence is divided according to modal labels in the channel dimension. Assuming that the structured tensor set contains four modes from B-mode images, color Doppler images, elastic images and radio frequency signal images, the unified tensor sequence is split into four separate tensor sets. Each set retains only the data of the corresponding modal channel, forming four single-modal feature tensors. Each single-modal tensor retains its spatial dimension information (such as width and height) unchanged, and is decomposed only in the channel dimension.

[0071] Shared convolution mapping generates feature vectors: The four single-modal feature tensors mentioned above are mapped through a shared convolutional network with the same structure to obtain a set of three vector forms corresponding to each modality: query vector, key vector, and value vector. The convolutional structure usually includes a set of 3×3 or 1×1 convolutional kernels, and combines non-linear activation functions (such as ReLU or GELU) for feature transformation processing. The output is a set of feature vectors with the same size as the input tensor but with a higher degree of semantic abstraction. The use of shared structure helps to establish a comparable semantic space between different modalities and enhances the stability and robustness of cross-modal interaction.

[0072] Calculate attention weights and generate fused features:

[0073] For each unimodal feature tensor, its corresponding query vector, key vector, and value vector are extracted through three independent sets of convolutional projection operations. Each projection maintains the spatial dimensions of the input feature tensor, transforming only the channel dimension, thereby generating three equal-dimensional representation tensors.

[0074] Interactive connections are established between modal pairs. Taking any two modalities A and B as an example, the query vector of modal A and the key vector of modal B are subjected to an element-wise dot product operation. To avoid the numerical amplification effect of different dimensional features in the dot product result, the dot product result is divided by the square root of the channel dimension of the vector, which is used as a scaling factor to obtain a stable attention score. This scaled dot product operation is used to represent the attention intensity of modal A to modal B at a specific spatial location.

[0075] The scaled dot product results between all modal pairs are input into a set of row-normalized exponential weighting functions. Specifically, for each spatial location, the score of the current modal pair is exponentialized and then normalized with the sum of the scores of all modal pairs at that location to form the relative attention weight at that location. The normalization operation ensures that at each spatial point, the attention distributions between all modal pairs can be summed to 1, thus satisfying the weight summation constraint.

[0076] For each modality's value vector, a weighted sum is performed based on the attention weights established between it and other modalities. That is, for a certain fusion target modality, the value vectors of its related modalities are weighted and superimposed according to the corresponding attention weights, and the cross-modal fusion feature representation of the modality in the current fusion step is output.

[0077] All fused features will retain their original spatial distribution structure and incorporate information from other modalities in the channel dimension, forming the final fused feature tensor, which serves as the input basis for the next stage of temporal modeling or downstream decoding tasks.

[0078] The specific contents of the image spatial topology module include:

[0079] The fused feature tensors obtained from the preceding modules are arranged in temporal order to form a frame-level feature sequence. Each frame in this sequence contains fused multimodal information while maintaining the spatial layout structure of the original image. Before being input into the temporal modeling network, each frame is first unfolded into a two-dimensional feature vector sequence in the spatial dimension. Each vector corresponds to a spatial location point in the image. The feature vectors of each frame are input into the temporal modeling network based on a recurrent neural structure, such as a gated recurrent unit (GRU) or a long short-term memory network (LSTM). The network models the dynamic evolution pattern of the image over time through recursive updates of its internal state and outputs the temporal feature representation corresponding to each frame.

[0080] In each frame of the image, several key points with reasonable spatial distribution and stable features within the target area are selected as nodes of the graph structure. These nodes can be obtained through methods such as feature saliency screening, gradient peak detection, or uniform sampling of the region.

[0081] Specifically, within the spatial dimension of each image frame, all pixels are traversed based on the response map, and pixels with response intensities higher than a set response threshold are selected as candidate nodes. This ensures that the selected feature points are concentrated in the target tissue region of the image, thereby enhancing the targeting and discriminative power of graph modeling. Each selected point will serve as a node in the graph structure, and its spatial coordinates and joint spatiotemporal feature vector will be recorded as basic attributes.

[0082] For any two nodes, calculate their Euclidean distance in the image plane and the cosine similarity between their feature vectors. If the Euclidean distance between the two nodes is less than a set spatial proximity threshold, or their feature similarity is higher than a set similarity threshold, then establish an edge connection between the two nodes. In this way, a graph structure representation containing nodes and connecting edges is generated, which fully reflects the local connectivity and feature coupling relationship in the spatial structure.

[0083] Two weight adjustment factors are set: a spatial distance adjustment factor (which can be set as a monotonically decreasing function to suppress the connection strength of distant connections) and a semantic similarity adjustment factor (which can be set as a monotonically increasing function to enhance the connection strength of similar features). The above two adjustment factors are applied to the Euclidean distance and cosine similarity results respectively, and the final edge weights are formed by weighted fusion.

[0084] For example, the edge weight can be defined as a normalized value, expressed as follows: the negative Euclidean distance between nodes is fed into an exponential decay function to generate an edge contribution score based on spatial distance; at the same time, the cosine similarity is fed into a sigmoid function to generate an edge contribution score based on feature similarity; the two scores are combined by weighting coefficients to form the final weight of the edge.

[0085] The constructed graph structure is input into the graph neural network for modeling. The graph neural network propagates and aggregates feature information between nodes according to the connection relationship through multiple information propagation layers. In each round of propagation, the node receives the feature information of the neighboring nodes and merges it with its own information to form a context-enhanced structural representation.

[0086] Specifically, the feature vector of each graph structure node is used as the initial node representation, and the edge weights between nodes are used as the modulation factor for information transmission to perform iterative update operations of the graph neural network.

[0087] In each round of information transmission, the feature representations of the target node and its directly adjacent nodes are aggregated, weighted and fused according to the edge connection weights, and the current feature state of the target node is updated.

[0088] Repeatedly perform information transmission and state update operations, and aggregate the features of directly adjacent nodes;

[0089] After multiple rounds of propagation and updates, each node obtains a graph feature representation embedded with its neighborhood information, thus forming a feature set containing spatial topological constraints globally.

[0090] For the same frame image, its temporal feature representation in the time dimension and its graph structure feature representation in the spatial dimension are obtained. The two are matched and fused according to their spatial positions. That is, at each node or spatial point, the corresponding temporal features and graph features are combined by vector concatenation, attention fusion or transformation mapping to generate a joint spatiotemporal feature representation. The joint features of all frames are arranged in sequence to form a joint spatiotemporal feature sequence.

[0091] For example, taking a target region in a frame of joint spatiotemporal feature image as an example, five key feature points are selected to form nodes in the graph structure, named A, B, C, D and E respectively. Assume that the spatial connection relationship between these nodes is as follows: A is connected to B and C; B is connected to A and D; C is connected to A and E; D is connected to B; E is connected to C. At this time, the constructed graph structure contains nodes and their connection relationship, reflecting the spatial topological relationship of the region in the image.

[0092] In the graph neural network modeling stage:

[0093] Layer 1 propagation: Node A receives initial feature vectors from B and C, and fuses this information with its own feature vector, for example, by weighted summation or mapping function to complete feature update; Node B also receives features from A and D and performs fusion; Node C receives features from A and E, and so on.

[0094] After this round of propagation, the feature representation of each node not only includes its own initial information, but also the feature context information of its directly adjacent nodes;

[0095] Second-level propagation: Node A will now have updated features about B and C, while B also incorporates information from D. Therefore, A's features begin to indirectly contain information from D. Because node C connects A and E, it will have the influence of its more distant neighbors in the second round of propagation. As the number of propagation layers increases, nodes can perceive a broader graph structure context, enabling spatial dependency modeling.

[0096] After, for example, 2 to 3 layers of propagation and feature updates, the feature vector of each node in the graph (e.g., node A) has been fused with the information of its first-order, second-order, or even more distant neighbor nodes, forming a context-enhanced graph feature representation that includes spatial topological constraints.

[0097] The specific content of the results output module includes:

[0098] When the task type is region segmentation, the feature map of each frame in the joint spatiotemporal feature sequence is read frame by frame and fed into the resolution restoration structure. This structure consists of multiple upsampling units. Each upsampling operation improves the spatial resolution through interpolation or transposed convolution. At the same time, cross-layer connections are used to fuse low-level shallow features with high-level deep features, so that the restored target region segmentation map retains spatial boundary information and has high-level semantic expression capabilities. The size of the finally restored region segmentation map is consistent with the original input image. The region segmentation map is normalized in the channel dimension or classified by softmax, and each pixel position is mapped to the corresponding tissue category label. The output is a two-dimensional region segmentation map, where each pixel value represents the tissue structure category at that position, which is used for boundary-level recognition of the region of interest.

[0099] When the task type is structural classification, a temporal aggregation operation is performed on the entire joint spatiotemporal feature sequence. Specifically, average pooling or max pooling is performed on the feature vectors of all frames along the time axis to obtain a fixed-dimensional global feature vector. This vector retains the core features of temporal information while possessing overall semantic abstraction capabilities. This global feature vector is input into a set of fully connected neural network structures. The network consists of several linear mapping layers and activation functions. Through layer-by-layer calculation and nonlinear transformation, a vector representing the predicted probability of the category is output. Finally, the category label corresponding to the highest probability is taken as the structural type determination result of the region, such as "connective tissue", "solid tissue" or "lesion tissue", which is used to assist in tissue attribute identification and medical classification.

[0100] When the task type is functional parameter extraction, the complete joint spatiotemporal feature sequence is fed into a regressive neural network structure in chronological order. This regressive neural network structure is usually a sequence model with temporal recursion or attention mechanisms, which can capture the response changes of tissue state over continuous time segments. The input to the regressive neural network structure is the spatiotemporal feature vector of each frame of the image arranged in chronological order in the joint spatiotemporal feature sequence. These feature vectors contain information in two dimensions:

[0101] Temporal dimension information: represents the dynamic changes of the image between different frames, used to characterize the response of tissues over time in the ultrasound sequence;

[0102] Spatial dimensional information: includes spatial topological features obtained from graph neural networks, used to characterize the structural state, boundary connectivity, and internal distribution patterns of the target region;

[0103] The feature vectors of each frame of the image maintain the same dimension. The length of the input sequence is consistent with the number of image frames involved in the analysis. The entire sequence is composed of the input tensor in chronological order.

[0104] During processing, the model identifies the elastic response, echo signal fluctuations, and other dynamic changes of tissue at different time points. In the model output stage, specific output dimensions are set for the required functional indicators and mapped to quantitative values. The output results are a set of continuous numerical parameters that evolve over time, including but not limited to: tissue compression ratio (representing the rate of change of tissue thickness under external force), elasticity change value (representing the amount of change in structural elasticity between different time points), and echo amplitude fluctuation rate (representing the degree of dynamic change in ultrasound signal intensity). This set of parameters is output in the form of a table or time series graph for subsequent clinical interpretation or auxiliary analysis.

[0105] The system automatically switches decoding paths based on user settings or task scheduling instructions, activating only the processing flow corresponding to the currently required analysis task, thus avoiding the waste of irrelevant computing resources.

[0106] For example, in liver mass boundary detection (task type: region segmentation), after receiving an ultrasound image sequence of the liver region, the system sets the task type to region segmentation. In the processing flow, the system inputs the joint spatiotemporal features of each frame into an upsampling decoding structure, progressively restoring the image resolution and enhancing boundary information by combining shallow features, ultimately outputting a pixel-level segmentation map of the liver mass. Doctors can then visually see the precise boundaries of the mass, aiding in subsequent interventional procedures or surgical planning.

[0107] The above formulas are all dimensionless calculations. The formulas are derived from software simulations using a large amount of collected data, and are the closest to the real situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0108] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0109] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images, characterized in that: It includes an image data acquisition module, a fusion feature module, an image spatial topology module, and a result output module, and the modules are connected by signals. The image data acquisition module acquires B-mode images, color Doppler images, elastic images, and radio frequency signal images, performs time synchronization and spatial alignment operations, and converts them into structured tensors of a unified dimension through a modal embedding encoder. The feature fusion module takes a structured tensor as input to the cross-attention fusion network, performs inter-modal feature alignment and joint modeling, and outputs a multimodal fusion feature sequence. The image spatial topology module takes the multimodal fusion feature sequence as input to the temporal modeling network, extracts the dynamic features of the image as it changes over time, and combines graph structure modeling methods to generate spatial topological constraint representations, thus constructing a joint spatiotemporal feature sequence. The results output module takes the joint spatiotemporal feature sequence as input to the multi-task decoding path and outputs the target region segmentation map, organizational structure category label and functional analysis parameters according to the task type. The image data acquisition module includes the following: Acquire multimodal ultrasound image data, including B-mode images, color Doppler images, elastography images, and radiofrequency signal images; The optimal set of frame pairs is constructed based on the inter-modal structural similarity, and time-coordinated frame selection is performed to achieve time synchronization of images. During spatial registration, a pseudo-image mask is generated to guide alignment and spatial registration is performed. The synchronously registered image is input into a modality-aware embedding network, which integrates the modality indication vector and the spatial coordinate position encoding to output a structured tensor of uniform dimension.

2. The multimodal deep learning-driven multidimensional ultrasound image intelligent processing system according to claim 1, characterized in that: The feature fusion module includes the following: The structured tensor sequence is split modally along the channel dimension to obtain multiple single-modal feature tensors; Perform a shared convolution mapping on each modality feature tensor to generate a query vector, a key vector, and a value vector, respectively. Attention weights are calculated based on the similarity between queries and keys across modalities, and weighted combinations are performed on the value vectors to generate cross-modal fusion features.

3. The multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to claim 2, characterized in that: The calculation process for attention weights includes: For each modality's feature representation, generate a query vector, a key vector, and a value vector respectively; Calculate the feature dimension dot product between the query vector and the key vector for any two modalities, and divide the dot product result by the square root of the vector dimension as a scaling factor; The scaled dot product of all modal pairs is input into a row-normalized exponential weighting function to generate the relative attention weights between modal pairs. Relative attention is applied to the value vector of the corresponding modality, and a weighted summation operation is performed to obtain the fused feature output.

4. The multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to claim 3, characterized in that: The image spatial topology module includes the following: Multimodal fusion feature sequences are collected, and the fusion features of each frame are input into a temporal modeling network in chronological order to extract the dynamic evolution features of the image in the time dimension; The fused features of each frame are expanded into a vector sequence in the spatial dimension and input into the temporal modeling network frame by frame to perform state updates and generate the temporal feature representation of the corresponding frame. Within the image space dimension, feature points of the target region are selected as nodes of the graph structure. Edge connections are constructed based on the spatial proximity or feature similarity between nodes to generate a graph structure representation. The graph structure is input into a graph neural network for modeling, and graph feature representation containing spatial topological constraints is obtained through information transmission between nodes. The temporal feature representation and graph feature representation of each frame are fused along the position dimension to construct a joint spatiotemporal feature sequence.

5. The multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to claim 4, characterized in that: Within the image spatial dimension, feature points of the target region are selected as nodes in the graph structure. Edge connections are constructed based on spatial proximity or feature similarity between nodes to generate a graph structure representation. Specific details include: Within the spatial dimension of each frame image, feature points in the target region with response intensity higher than a set response threshold are selected as graph structure nodes. Calculate the Euclidean distance between each pair of nodes and the cosine similarity between their corresponding feature vectors. The Euclidean distance reflects the geometric proximity of their spatial locations, while the cosine similarity reflects the consistency of their semantic features. When the Euclidean distance between two nodes is lower than a preset spatial proximity threshold, or the feature similarity is higher than a preset similarity threshold, an edge connection is established between the two nodes. Generate a graph structure representation containing node attributes and edge weights based on all valid edge connections.

6. The multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to claim 5, characterized in that: The graph structure is input into a graph neural network for modeling. Through information transfer between nodes, a graph feature representation containing spatial topological constraints is obtained. Specifically, this includes: The graph neural network iteratively updates by using the feature vector of each graph node as the initial node representation and the edge weights between nodes as the modulation factor for information transmission. In each round of information transmission, the feature representations of the target node and its directly adjacent nodes are aggregated, weighted and fused according to the edge connection weights, and the current feature state of the target node is updated. Repeatedly perform information transmission and state update operations, and aggregate the features of directly adjacent nodes; After all rounds of updates are completed, the output is a graph feature representation containing spatial topological relationships, which is used for the spatiotemporal joint representation of image processing tasks.

7. The multimodal deep learning-driven intelligent processing system for multidimensional ultrasound images according to claim 6, characterized in that: The results output module includes the following: When the task type is region segmentation, the joint spatiotemporal features of each frame are input into the upsampling decoding structure, the resolution is restored according to a fixed level, and the target region segmentation map with the same size as the original image is output for pixel-level structural boundary recognition. When the task type is structural classification, the temporal dimension aggregation operation is performed on the joint spatiotemporal feature sequence to extract the global feature vector of the sequence, and input into the fully connected classification structure to output the category label of the organizational structure; When the task type is functional parameter extraction, the joint spatiotemporal feature sequence is input into the regression neural network structure, and the output is a numerical functional parameter that reflects the evolution of the tissue region over time, including compression ratio, elasticity change value or echo amplitude fluctuation rate. Based on the task type settings, select the corresponding decoding path and output the corresponding analysis results.

Images