Ultrasonic image processing device and method suitable for ultrasound-guided interventional surgery robot

Abstract

The embodiments of the present application relate to the field of image analysis, and in particular to an ultrasonic image processing device and method suitable for an ultrasound-guided interventional surgical robot. The ultrasonic image processing device includes: an acquisition module for continuously acquiring cardiac ultrasonic images of a patient at multiple time points; a preprocessing module for segmenting the cardiac ultrasonic images and obtaining segmentation results; a basic feature extraction module for extracting static features of the segmentation results; an associated feature extraction module for extracting dynamic features of the segmentation results; a feature fusion module for combining static features with dynamic features to determine the volume change trend of pericardial effusion; and a lightweight deployment optimization module for combining the modules so that the modules can process ultrasonic images in real time. By extracting the static features and dynamic features of the ultrasonic image separately, information of two different states can be captured more effectively, so as to avoid information interference between different modules and improve the pertinence of feature extraction; and avoid processing information of all dimensions at the same time, thus saving computing resources.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image analysis, and in particular to an ultrasound image processing device and method suitable for an ultrasound-guided interventional surgery robot. Background Art

[0002] The statements herein merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Normally, the pericardial cavity contains a small amount of fluid, which primarily serves as a lubricant and friction reliever. If the amount of fluid in the pericardial cavity increases beyond the normal range, it is called a pericardial effusion, and the amount of pericardial effusion can be visualized using ultrasound imaging. Summary of the Invention

[0004] A brief overview of the present application is provided below to provide a basic understanding of certain aspects of the present application. It should be understood that this overview is not an exhaustive overview of the present application. It is not intended to identify key or important portions of the present application, nor is it intended to limit the scope of the present application. Its purpose is simply to present certain concepts in a simplified form as a prelude to the more detailed description that will be discussed later.

[0005] One aspect of an embodiment of the present application provides an ultrasonic image processing device suitable for an ultrasound-guided interventional surgical robot, wherein the ultrasonic image is used to reflect the state of pericardial effusion, and the device includes: an acquisition module for continuously acquiring a patient's cardiac ultrasonic image at multiple time points; a preprocessing module for segmenting the cardiac ultrasonic image to obtain a segmentation result; a basic feature extraction module for extracting static features of the segmentation result; an associated feature extraction module for extracting dynamic features of the segmentation result; a feature fusion module for combining static features with dynamic features, and determining the volume change trend of pericardial effusion based on the combination; and a deployment optimization module for making the combination lightweight so that it can process ultrasonic images in real time.

[0006] Another aspect of an embodiment of the present application provides an ultrasound image processing method suitable for an ultrasound-guided interventional surgical robot, wherein the ultrasound image is used to reflect the state of pericardial effusion, and the method includes the following steps: S10: continuously acquiring cardiac ultrasound images of the patient at multiple time points; S20: segmenting the cardiac ultrasound image to obtain the segmentation result of the cardiac ultrasound image; S30: extracting static features of the segmentation result; S40: extracting dynamic features of the segmentation result; S50: combining the static features with the dynamic features, and determining the volume change trend of the pericardial effusion based on the combination; S60: making the combination lightweight so that it can process cardiac ultrasound images in real time.

[0007] In the ultrasonic image processing device suitable for an ultrasound-guided interventional surgical robot provided in the embodiments of the present application, cardiac ultrasound images acquired in multiple time periods contain both spatial information and time information. By extracting the static features and dynamic features of the ultrasound images separately, such separate processing can more effectively capture information in these two different states, so as to avoid information interference between different modules and improve the targeted feature extraction; at the same time, it avoids processing information of all dimensions at the same time, saving computing resources.

[0008] The embodiments of the present application provide an ultrasound image processing method suitable for an ultrasound-guided interventional surgical robot. By extracting static features and dynamic features separately, feature extraction is targeted, interference between multi-dimensional information is avoided, and computing efficiency is improved, so as to realize real-time processing of image sequences on resource-limited edge devices, thereby realizing dynamic monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To further illustrate the above and other advantages and features of the present application, the following detailed description of specific embodiments of the present application is provided in conjunction with the accompanying drawings. The accompanying drawings, together with the detailed description below, are incorporated into and form a part of this specification. Elements with the same function and structure are denoted by the same reference numerals. It should be understood that these drawings depict only typical examples of the present application and should not be construed as limiting the scope of the present application.

[0010] Figure 1 is a structural block diagram of an ultrasonic image processing device provided in an embodiment of the present application;

[0011] Figure 2 It is a flowchart of the ultrasound image processing method provided in an embodiment of the present application.

[0012] Description of reference numerals:

[0013] 10. Acquisition module; 20. Preprocessing module; 30. Basic feature extraction module; 40. Correlation feature extraction module; 50. Feature fusion module; 60. Deployment optimization module; 100. Ultrasonic image processing device. DETAILED DESCRIPTION

[0014] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary depending on the implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is a routine task for those skilled in the art who benefit from the content of this application.

[0015] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure and / or processing steps that are closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0016] The disclosure below provides a plurality of different embodiments or examples for implementing the present application. In order to simplify the disclosure of the present application, the components and methods of specific examples are described below. Of course, they are merely examples and are not intended to limit the present application. In the description of the embodiments of the present application, the meaning of "plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

[0017] Pericardial effusion is an important indicator of heart disease. Traditional monitoring relies on manual measurement of the effusion area in ultrasound images, which has problems such as low efficiency, strong subjectivity, and insufficient dynamic tracking capabilities.

[0018] The embodiment of the present application provides an ultrasound image processing device suitable for ultrasound-guided interventional surgery robot, wherein the ultrasound image is used to reflect the state of pericardial effusion. Figure 1 FIG. 1 shows a structural block diagram of an ultrasonic image processing device provided in an embodiment of the present application, as shown in FIG. Figure 1 As shown, it includes: an acquisition module 10, used to continuously acquire the patient's cardiac ultrasound images at multiple time points; a preprocessing module 20, used to segment the cardiac ultrasound images and obtain segmentation results; a basic feature extraction module 30, used to extract static features of the segmentation results; an associated feature extraction module 40, used to extract dynamic features of the segmentation results; a feature fusion module 50, used to combine the static features with the dynamic features, and determine the volume change trend of the pericardial effusion based on the combination; a deployment optimization module 60, used to make the combination lightweight so that it can process ultrasound images in real time.

[0019] In the ultrasonic image processing device 100 suitable for an ultrasound-guided interventional surgical robot provided in an embodiment of the present application, cardiac ultrasound images obtained in multiple time periods contain both spatial information and time information. By extracting the static features and dynamic features of the ultrasound images separately, such separate processing can more effectively capture information in these two different states, so as to avoid information interference between different modules and improve the targeted feature extraction; at the same time, it avoids processing information of all dimensions at the same time, saving computing resources.

[0020] In some embodiments, the preprocessing module 20 is configured to receive a cardiac ultrasound image and to output a segmentation result of the ultrasound image, the segmentation result including a binary mask of the area where the pericardial effusion is located in the cardiac ultrasound image and a sequence of predicted values ​​of the pericardial effusion volume in the cardiac ultrasound image.

[0021] In some embodiments, the cardiac ultrasound image and the segmentation result satisfy the following relationship:

[0022] .

[0023] Where T represents the number of time points; t represents the time period number; Represents the features of the cardiac ultrasound images corresponding to T time periods, which include texture, edge, and morphological feature information; Model represents the relationship between them; It represents the pixel-level segmentation result of the pericardial effusion area at time t, and the output result is a binary mask; Represents the sequence of predicted pericardial effusion volume values ​​from time period t+1 to time period t+10, in mL, which is used to dynamically monitor the growth trend of effusion.

[0024] In some embodiments, the output result of the pre-processing module 20 is the segmented pericardial effusion ultrasound image and the predicted sequence value of the pericardial effusion volume, so as to facilitate the subsequent pericardial effusion region analysis and pericardial effusion volume prediction.

[0025] In some embodiments, the basic feature extraction module 30 includes: a first basic feature extraction unit, used to extract color and edge features of the cardiac ultrasound image; a second basic feature extraction unit, used to extract texture features of the cardiac ultrasound image; a third basic feature extraction unit, used to extract morphological features of the cardiac ultrasound image, and the first basic feature extraction unit, the second basic feature extraction unit and the third basic feature extraction unit are set in parallel.

[0026] In some embodiments, multiple parallel extraction units are set up to extract static features of cardiac ultrasound images, so that the features extracted by each unit are complementary, which facilitates feature splicing and outputs a consistent and comprehensive feature map.

[0027] In some embodiments, the first basic feature extraction unit includes a pooling layer and two dilated convolution layers to enhance the grayscale distribution contrast and edge features of the pericardial effusion area. The edge feature extraction satisfies the following relationship:

[0028] .

[0029] in, represents the raw pixel matrix of the input cardiac ultrasound image; represents the maximum pooling operation to reduce the resolution of cardiac ultrasound images and thus enhance edge saliency; represents a dilation rate of 2, which expands the receptive field to capture long-distance edge features; and Represents the convolution kernel weight matrix, which is used to extract edge features in different directions; Represents the output edge feature map, which characterizes the grayscale difference boundary between pericardial effusion and surrounding tissue.

[0030] In some embodiments, setting a pooling layer can retain the most obvious color and edge features while reducing the size of the feature map, reducing the amount of calculation and speeding up subsequent processing; setting two layers of hole convolution layers can avoid information loss caused by pooling and capture a wider range of context information.

[0031] In some embodiments, the second basic feature extraction unit includes three convolutional layers and one channel attention layer to quantify the internal intensity heterogeneity of the pericardial effusion. The extraction of texture features satisfies the following relationship:

[0032] .

[0033] .

[0034] in, The feature map representing the output of the third convolution layer represents the local texture details of the pericardial effusion area; Represents global average pooling, compressing the feature map into a channel description vector; Represents the weight of the fully connected layer and generates the channel attention weight; Represents the Sigmoid activation function, which maps the weights to the [0,1] interval; Represents the channel attention weight, quantifying the importance of different texture feature channels; Represents the weighted texture feature map, which characterizes the response of the heterogeneous area.

[0035] In some embodiments, three convolutional layers enable layer-by-layer understanding of images, from simple to complex, to gradually extract multi-level features; a single channel attention layer enables intelligent feature screening, highlighting key features. This combination balances the depth of the convolutional layers with the location of attention, enabling understanding of complex patterns while avoiding computational overhead and making attention judgments more accurate.

[0036] In some embodiments, the third basic feature extraction unit includes four convolutional layers and a dynamic receptive field adjustment module. The convolution kernel size is gradually reduced to capture the global geometric morphology of the pericardial effusion area. The morphological feature extraction satisfies the following relationship:

[0037] .

[0038] .

[0039] in, The feature map representing the output of the fourth convolution layer represents the geometric shape information of the effusion area; Represents the dynamic convolution kernel weight, which is used to calculate the feature point offset; Represents the spatial offset of the feature point, and adjusts the receptive field to adapt to the changes in the effusion morphology; and Indicates the initial feature point position and adjacent point coordinates; Represents the learnable weight coefficient, which controls the contribution of different positions to the morphological features; represents the raw pixel matrix of the input cardiac ultrasound image; Indicates the number of feature points, which is used to describe the total number of feature points considered or calculated during the feature extraction process; n represents the number of feature points, which is used to distinguish different feature points in the set; The output morphological feature map represents the global contour and size variation of the pericardial effusion area.

[0040] In some embodiments, four convolutional layers enable progressive analysis from global to local perspectives, while a dynamic receptive field adjustment module adapts to objects of varying sizes. This combination of hierarchical feature extraction and adaptive perception not only optimizes computational efficiency, making the feature extraction process suitable for on-device deployment, but also enhances the model's robustness to complex scenarios and improves the accuracy and stability of real-time inference.

[0041] In some embodiments, the extraction of associated features satisfies the following relationship:

[0042] .

[0043] in, Represents the associated feature map extracted by the 3D convolution operation, which contains dynamic information on the time series; Represents a three-dimensional convolution operation that can process information in both spatial and temporal dimensions simultaneously; Represents the feature map sequence from tk to t time, which is the input of 3D convolution, where t represents the time point and k represents the size of the time window; Represents the weight of the 3D convolution kernel. The weight performs dot product operation with the input feature map during the convolution process to extract related features.

[0044] In some embodiments, the correlation feature extraction module 40 may include an edge-texture correlation feature extraction unit, a spatial correlation feature extraction unit, and a temporal correlation feature extraction unit.

[0045] The spatial correlation feature extraction unit can include two convolution layers and one spatial attention layer, which are used to model the edge continuity of the effusion area, where the convolution kernel size can be set to 1×1; the edge-texture correlation feature extraction unit can include 0 pooling layers and 3 cross-channel convolution layers. By completely discarding the pooling layer, the spatial information loss caused by downsampling is completely avoided, the original resolution of the feature map is maintained, and cross-channel feature interaction is realized at the same time. The output of the spatial correlation feature extraction unit is integrated to achieve modeling of the edge and texture of the effusion area; the temporal correlation feature extraction unit can be set to use 3, 4, and 5 temporal convolution layers for 3D convolution respectively, which can capture the high-frequency changes between consecutive frames and expand the receptive field, realize multi-granularity temporal modeling, and thus extract the dynamic features of the effusion thickness and area changes between consecutive frames.

[0046] In some embodiments, the feature fusion module 50 includes: a spatial modeling unit for generating a spatial feature map by fusing static features through adaptive weights; and a temporal modeling unit for combining the spatial feature map with dynamic features to obtain the relationship of pericardial effusion between ultrasound images.

[0047] In some embodiments, by processing static features and dynamic features separately, spatial modeling can refine local details, single-frame image processing does not require cross-frame calculations, reducing memory usage and achieving lightweight spatial modeling; and enabling temporal modeling to capture global dynamics, only serializing related features, avoiding the high complexity and large amount of computation required for processing all time and space dimensions.

[0048] In some embodiments, the relationship between the ultrasound images of the pericardial effusion satisfies the following expression:

[0049] .

[0050] in, Represents the query matrix, which represents the feature information that needs to be paid attention to at present; Represents the key matrix, which is used to match the query matrix and determine the weight of the focus; Represents the value matrix, which contains the actual information content and is weighted summed according to the attention weight; Indicates the dimension of the key, which is used to scale the dot product result to prevent the gradient from disappearing or exploding; represents a normalization function that converts the input value into a probability distribution to ensure that the sum of all attention weights is 1; Represents the attention mechanism, which is used to calculate the correlation between different inputs and dynamically adjust the weight of information.

[0051] In some embodiments, the correlation feature map extracted by the 3D convolution operation can be input into the spatiotemporal Transformer module, and the long-term dependency of the evolution of pericardial effusion between adjacent cardiac ultrasound images can be modeled through the multi-head self-attention mechanism, thereby breaking through the processing limitations of local feature relationships and modeling the global interactions between all positions and lines in space, providing more powerful modeling capabilities for multi-line convolution feature fusion, so as to clearly understand and reflect complex spatiotemporal relationships.

[0052] In some embodiments, the spatiotemporal Transformer module comprises a dual-branch architecture: a spatial encoding branch and a temporal encoding branch. The spatial encoding branch employs progressive downsampling convolutional layers to extract effusion edges and echo intensity features from single-frame ultrasound images, while the temporal encoding branch employs a multi-head self-attention mechanism to model the correlation between adjacent ultrasound image frames, thereby capturing the dynamic changes in effusion volume.

[0053] In some embodiments, the feature fusion module 50 further includes a dynamic prediction unit for predicting the volume change trend of pericardial effusion in future ultrasound image sequences; wherein the predicted values ​​of pericardial effusion in the next 10 frames satisfy the following relationship:

[0054] .

[0055] .

[0056] Represents a recurrent neural network, which is used to capture long-term dependencies in sequential data; Represents the hidden state at time t, the internal state of LSTM, which contains all the information up to the current moment; Represents the cell state at time t, the core memory unit of LSTM, used to store long-term dependency information; represents the hidden state at time t-1; represents the cell state at time t-1; Represents fusion features, combining information of spatial and temporal features as input to LSTM; represents the output weight matrix, which is used to transform the hidden state into the final predicted value; Represents the predicted value of the effusion volume for the next 10 frames, which is calculated based on the hidden state and output weights at the current moment.

[0057] In some embodiments, the dynamic prediction unit can predict the pericardial effusion volume trend for the next 10 frames based on the LSTM network and the fused features of the current frame, namely the effusion segmentation mask of the current frame. This helps predict future effusion volume trends and estimate the probability of abnormal fluctuations, providing strong support for dynamic monitoring and anomaly detection of pericardial effusion.

[0058] In some embodiments, the deployment optimization module 60 may include a lightweight modeling unit, a dynamic quantization unit, and an operator fusion optimization unit.

[0059] The lightweight modeling unit is used to train a lightweight student model based on the relationship between pericardial effusion between ultrasound images and the output of the relationship between the predicted values ​​of pericardial effusion in the next 10 frames, so that the lightweight student model can realize real-time processing of image sequences on the edge computing device through the dynamic quantization unit and the operator fusion optimization unit.

[0060] The dynamic quantization unit dynamically adjusts quantization parameters based on the actual distribution of input data, reducing model size and computation, thereby achieving lightweight student models. The operator fusion optimization unit combines multiple computational operations into a single composite operation, reducing memory accesses and computational overhead. Through dynamic quantization and operator fusion optimization, the student model can achieve efficient, real-time pericardial effusion monitoring on wearable ultrasound devices.

[0061] In some embodiments, the acquisition module 10, the preprocessing module 20, the basic feature extraction module 30, the associated feature extraction module 40 and the feature fusion module 50 can be integrated into the ultrasound monitoring device by deploying the optimization module 60, so that the wearable ECG monitoring device can realize real-time processing of the cardiac ultrasound image sequence, thereby facilitating timely feedback on the patient's heart function.

[0062] Another aspect of the embodiment of the present application provides an ultrasound image processing method suitable for an ultrasound-guided interventional surgery robot, wherein the ultrasound image is used to reflect the state of pericardial effusion. Figure 2 A schematic diagram showing a flow chart of an ultrasound image processing method provided by an embodiment of the present application is shown in FIG. Figure 2As shown, it includes the following steps: S10: continuously acquiring cardiac ultrasound images of the patient at multiple time points; S20: segmenting the cardiac ultrasound images to obtain segmentation results of the cardiac ultrasound images; S30: extracting static features of the segmentation results; S40: extracting dynamic features of the segmentation results; S50: combining the static features with the dynamic features, and determining the volume change trend of the pericardial effusion based on the combination; S60: making the combination lightweight so that it can process cardiac ultrasound images in real time.

[0063] The embodiments of the present application provide an ultrasound image processing method suitable for an ultrasound-guided interventional surgical robot. By extracting static features and dynamic features separately, feature extraction is targeted, interference between multi-dimensional information is avoided, and computing efficiency is improved, so as to realize real-time processing of image sequences on resource-limited edge devices, thereby realizing dynamic monitoring.

[0064] In some embodiments, in step S20, the segmentation result includes a binary mask of the region where the pericardial effusion is located in the cardiac ultrasound image and a sequence of predicted values ​​of the pericardial effusion volume in the cardiac ultrasound image.

[0065] In some embodiments, the cardiac ultrasound image and the segmentation result satisfy the following relationship:

[0066] .

[0067] Where T represents the number of time points; t represents the time period number; Represents the features of the cardiac ultrasound images corresponding to T time periods, which include texture, edge, and morphological feature information; Model represents the relationship between them; It represents the pixel-level segmentation result of the pericardial effusion area at time t, and the output result is a binary mask; Represents the sequence of predicted pericardial effusion volume values ​​from time period t+1 to time period t+10, in mL, which is used to dynamically monitor the growth trend of effusion.

[0068] In some embodiments, a multi-line convolutional neural network calculation can be performed on the pixel-level segmentation results. In step S30, the following steps are also included: S31: extracting the color and edge features of the cardiac ultrasound image; S32: extracting the texture features of the cardiac ultrasound image; S33: extracting the morphological features of the cardiac ultrasound image.

[0069] In some embodiments, in step S31, the grayscale distribution contrast and edge features of the pericardial effusion region are enhanced by a pooling layer and two dilated convolution layers. The edge features are extracted to satisfy the following relationship:

[0070] .

[0071] in, represents the raw pixel matrix of the input cardiac ultrasound image; represents the maximum pooling operation to reduce the resolution of cardiac ultrasound images and thus enhance edge saliency; represents a dilation rate of 2, which expands the receptive field to capture long-distance edge features; and Represents the convolution kernel weight matrix, which is used to extract edge features in different directions; Represents the output edge feature map, which characterizes the grayscale difference boundary between pericardial effusion and surrounding tissue.

[0072] In some embodiments, in step S32, three convolutional layers and one channel attention layer are used to quantify the internal intensity heterogeneity of the pericardial effusion, and the texture features are extracted to satisfy the following relationship:

[0073] .

[0074] .

[0075] in, The feature map representing the output of the third convolution layer represents the local texture details of the pericardial effusion area; Represents global average pooling, compressing the feature map into a channel description vector; Represents the weight of the fully connected layer and generates the channel attention weight; Represents the Sigmoid activation function, which maps the weights to the [0,1] interval; Represents the channel attention weight, quantifying the importance of different texture feature channels; Represents the weighted texture feature map, which characterizes the response of the heterogeneous area.

[0076] In some embodiments, in step S33, the global geometric morphology of the pericardial effusion region is captured by four convolutional layers and a dynamic receptive field adjustment module, wherein the convolution kernel size decreases step by step, and the morphological feature extraction satisfies the following relationship:

[0077] .

[0078] .

[0079] in, The feature map representing the output of the fourth convolution layer represents the geometric shape information of the effusion area; Represents the dynamic convolution kernel weight, which is used to calculate the feature point offset; Represents the spatial offset of the feature point, and adjusts the receptive field to adapt to the changes in the effusion morphology; and Indicates the initial feature point position and adjacent point coordinates; Represents the learnable weight coefficient, which controls the contribution of different positions to the morphological features; The output morphological feature map represents the global contour and size variation of the pericardial effusion area.

[0080] In some embodiments, in step S40, the extraction of dynamic features satisfies the following relationship:

[0081] .

[0082] in, Represents the associated feature map extracted by the 3D convolution operation, which contains dynamic information on the time series; Represents a three-dimensional convolution operation that can process information in both spatial and temporal dimensions simultaneously; Represents the feature map sequence from tk to t time, which is the input of 3D convolution, where t represents the time point and k represents the size of the time window; Represents the weight of the 3D convolution kernel. The weight performs dot product operation with the input feature map during the convolution process to extract dynamic features.

[0083] In some embodiments, step S50 further includes the following steps: S51: generating a spatial feature map by adaptively weighting static features; S52: combining the spatial feature map with dynamic features to obtain the relationship of pericardial effusion between ultrasound images.

[0084] In some embodiments, the relationship between the ultrasound images of the pericardial effusion satisfies the following expression:

[0085] .

[0086] in, Represents the query matrix, which represents the feature information that needs to be paid attention to at present; Represents the key matrix, which is used to match the query matrix and determine the weight of the focus; Represents the value matrix, which contains the actual information content and is weighted summed according to the attention weight; Indicates the dimension of the key, which is used to scale the dot product result to prevent the gradient from disappearing or exploding; represents a normalization function that converts the input value into a probability distribution to ensure that the sum of all attention weights is 1; Represents the attention mechanism, which is used to calculate the correlation between different inputs and dynamically adjust the weight of information.

[0087] In some embodiments, step S50 further includes step S53: predicting the volume change trend of pericardial effusion in future ultrasound image sequences; wherein the predicted values ​​of pericardial effusion in the next 10 frames satisfy the following relationship:

[0088] .

[0089] .

[0090] in, Represents a recurrent neural network, which is used to capture long-term dependencies in sequential data; Represents the hidden state at time t, the internal state of LSTM, which contains all the information up to the current moment; Represents the cell state at time t, the core memory unit of LSTM, used to store long-term dependency information; represents the hidden state at time t-1; represents the cell state at time t-1; Represents fusion features, combining information of spatial and temporal features as input to LSTM; represents the output weight matrix, which is used to transform the hidden state into the final predicted value; Represents the predicted value of the pericardial effusion volume in the next 10 frames, which is calculated based on the hidden state and output weight at the current moment.

[0091] Regarding the embodiments of the present application, it should also be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other to obtain new embodiments.

[0092] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An ultrasonic image processing device suitable for an ultrasound-guided interventional surgery robot, wherein the ultrasonic image is used to reflect the state of pericardial effusion, characterized in that: It includes: an acquisition module, configured to continuously acquire cardiac ultrasound images of the patient at multiple time points; A preprocessing module, configured to segment the cardiac ultrasound image and obtain a segmentation result; A basic feature extraction module, used to extract static features of the segmentation result; An associated feature extraction module, used to extract dynamic features of the segmentation result; a feature fusion module, configured to combine the static feature with the dynamic feature, and determine a volume change trend of the pericardial effusion based on the combination; deploying an optimization module for making the combination lightweight so as to enable the combination to process the ultrasound image in real time; The feature fusion module includes: A spatial modeling unit, configured to generate a spatial feature map by fusing the static features through adaptive weighting; a time series modeling unit, configured to combine the spatial feature map with the dynamic feature to obtain a relationship of the pericardial effusion between the ultrasound images; A dynamic prediction unit is used to predict the volume change trend of the pericardial effusion in the future ultrasound image sequence; wherein the predicted value of the pericardial effusion in the next 10 frames satisfies the following relationship: ； ； Represents a recurrent neural network, which is used to capture long-term dependencies in sequential data; Represents the hidden state at time t. The internal state of the LSTM contains all the information up to the current moment; Represents the cell state at time t, the core memory unit of the LSTM, used to store long-term dependency information; represents the hidden state at time t-1; represents the cell state at time t-1; Represents the fusion feature, which combines the information of spatial and temporal features as the input of the LSTM; represents the output weight matrix, which is used to transform the hidden state into the final predicted value; Represents the predicted value of the effusion volume for the next 10 frames, which is calculated based on the hidden state and output weights at the current moment.

2. The device according to claim 1, characterized in that The preprocessing module is configured to receive a cardiac ultrasound image and output a segmentation result of the ultrasound image. The segmentation result includes a binary mask of the region where the pericardial effusion is located in the cardiac ultrasound image and a sequence of predicted values ​​of the pericardial effusion volume in the cardiac ultrasound image.

3. The device according to claim 2, characterized in that The cardiac ultrasound image and the segmentation result satisfy the following relationship: ； Where T represents the number of time points; t represents the time period number; Indicates the features of the cardiac ultrasound image corresponding to T time periods, wherein the features include texture, edge, and morphological feature information; Model representation the relationship between them; represents the pixel-level segmentation result of the pericardial effusion area at time period t, and the output result is the binary mask; It represents the sequence of predicted values ​​of the pericardial effusion volume from time period t+1 to time period t+10, in mL, and is used to dynamically monitor the growth trend of the effusion.

4. The device according to claim 3, characterized in that The basic feature extraction module includes: a first basic feature extraction unit, configured to extract color and edge features of the cardiac ultrasound image; a second basic feature extraction unit, configured to extract texture features of the cardiac ultrasound image; A third basic feature extraction unit is configured to extract morphological features of the cardiac ultrasound image. The first basic feature extraction unit, the second basic feature extraction unit, and the third basic feature extraction unit are configured in parallel.

5. The device according to claim 4, characterized in that The first basic feature extraction unit includes a pooling layer and two atrous convolution layers to enhance the grayscale distribution contrast and edge features of the pericardial effusion area. The extraction of the edge features satisfies the following relationship: ； in, represents the original pixel matrix of the input cardiac ultrasound image; represents a maximum pooling operation to reduce the resolution of the cardiac ultrasound image, thereby enhancing edge saliency; represents a dilation rate of 2, which expands the receptive field to capture long-distance edge features; and Represents the convolution kernel weight matrix, which is used to extract edge features in different directions; The output edge feature map represents the grayscale difference boundary between the pericardial effusion and the surrounding tissue.

6. The device according to claim 4, characterized in that The second basic feature extraction unit includes three convolutional layers and one channel attention layer to quantify the internal intensity heterogeneity of the pericardial effusion. The extraction of the texture feature satisfies the following relationship: ； ； in, The feature map representing the output of the third convolution layer represents the local texture details of the pericardial effusion area; represents global average pooling, compressing the feature map into a channel description vector; Represents the weight of the fully connected layer and generates the channel attention weight; Represents the Sigmoid activation function, which maps the weights to the [0,1] interval; Represents the channel attention weight, quantifying the importance of different texture feature channels; Represents the weighted texture feature map, which characterizes the response of the heterogeneous area.

7. The device according to claim 4, characterized in that The third basic feature extraction unit includes four convolution layers and a dynamic receptive field adjustment module. The convolution kernel size decreases step by step to capture the global geometric morphology of the pericardial effusion area. The extraction of the morphological features satisfies the following relationship: ； ； in, The feature map representing the output of the fourth convolution layer represents the geometric shape information of the effusion area; Represents the dynamic convolution kernel weight, which is used to calculate the feature point offset; represents the spatial offset of the feature point, and adjusts the receptive field to adapt to the changes in the effusion morphology; and Indicates the initial feature point position and adjacent point coordinates; Represents the learnable weight coefficient, which controls the contribution of different positions to the morphological features; represents the original pixel matrix of the input cardiac ultrasound image; Indicates the number of feature points, which is used to describe the total number of feature points considered or calculated during the feature extraction process; n represents the number of the feature point, which is used to distinguish different feature points in the set; The output morphological feature map represents the global contour and size change of the pericardial effusion area.

8. The device according to claim 1, characterized in that The extraction of the associated features satisfies the following relationship: ； in, Represents the associated feature map extracted by the 3D convolution operation, which contains dynamic information on the time series; Represents a three-dimensional convolution operation that can process information in both spatial and temporal dimensions simultaneously; Represents the feature map sequence from tk to t time, which is the input of 3D convolution, where t represents the time point and k represents the size of the time window; Represents the weight of the 3D convolution kernel, which performs a dot product operation with the input feature map during the convolution process to extract the associated features.

9. The device according to claim 1, characterized in that The relationship between the ultrasound images of the pericardial effusion satisfies the following expression: ； in, Represents the query matrix, which represents the feature information that needs to be paid attention to at present; represents a key matrix, which is used to match the query matrix and determine the weight of the focus point; Represents a value matrix containing the actual information content, which is weighted summed according to the attention weight; Indicates the dimension of the key, which is used to scale the dot product result to prevent the gradient from disappearing or exploding; represents a normalization function that converts the input value into a probability distribution to ensure that the sum of all attention weights is 1; Represents the attention mechanism, which is used to calculate the correlation between different inputs and dynamically adjust the weight of information.

Images