Ultrasonic image respiration signal identification method, device, equipment and medium

Abstract

The invention discloses an ultrasonic image respiration signal identification method, device and equipment and a medium, and relates to the technical field of medical image analysis, the method comprises the following steps: representing respiration movement correlation and spatial position correlation between two-dimensional ultrasonic images as a heterogeneous graph, and carrying out image feature fusion embedding on the heterogeneous graph by introducing manifold constraint, so as to obtain a two-dimensional image respiration signal; and obtaining image breathing motion manifold embedding features as two-dimensional ultrasonic image breathing signals. The relationship between the respiratory movement and the spatial position of the ultrasonic image is fused through the respiratory movement manifold embedding feature, so that the stable and effective respiratory signal can be obtained by randomly changing the pose of the ultrasonic probe in the acquisition process, the respiratory signal identification precision is improved, and the acquisition flexibility of the ultrasonic probe is ensured.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a method, apparatus, device, and medium for identifying respiratory signals in ultrasound images based on heterogeneous information alignment and embedding unsupervised methods. Background Technology

[0002] The incidence and mortality rates of malignant liver tumors are trending upwards, while image-guided interventional surgery is a major treatment approach for malignant tumors of abdominal organs. However, thoracic and abdominal organs are subject to displacement and deformation due to respiratory motion. These displacements and deformations can generate image artifacts during image acquisition or introduce uncertainty in lesion location during interventional surgery, making it difficult to improve surgical precision and potentially causing damage to surrounding healthy tissues. To avoid the influence of respiratory motion on surgery, it is necessary to monitor the respiratory status of the organs during image acquisition or surgery.

[0003] As a numerical representation of the respiratory state of organs, the extraction of respiratory signals has become a hot research topic. Clinically, external devices can be used to acquire real-time respiratory signals, such as using a spirometer to measure tidal changes or using a belt with a pressure sensor or optical / electromagnetic tracking equipment to measure changes in the chest and abdomen. However, the introduction of external devices firstly increases costs, and secondly, the inherent asynchrony between the external device and the acquisition device can affect the accuracy of the respiratory signals.

[0004] Image-based respiratory signal extraction techniques can effectively avoid the aforementioned problems. Ultrasound images are widely used in guiding clinical interventional surgeries for abdominal tumors due to their high real-time performance, safety, and flexibility. Therefore, it is particularly important to obtain respiratory signals by monitoring abdominal organ respiratory movements through clinically acquired ultrasound images.

[0005] However, existing methods largely assume that the imaging angle and position remain constant during the acquisition process. Ultrasound probes, on the other hand, are highly flexible, allowing doctors to freely change their probe pose to better track respiratory motion. Changes in ultrasound probe pose introduce complex and random grayscale variations into the image sequence. When these variations are superimposed on respiratory motion, existing methods fail to distinguish between respiratory and spatial motion, leading to respiratory signal extraction failures. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method, apparatus, device and medium for identifying respiratory signals in ultrasound images to overcome or at least partially solve the above problems.

[0007] This invention provides the following solution:

[0008] A method for identifying respiratory signals in ultrasound images, comprising:

[0009] Two-dimensional ultrasound image sequences are obtained by extracting two-dimensional ultrasound images from the acquired four-dimensional ultrasound images;

[0010] A respiratory spatial heterogeneity map is constructed using the two-dimensional ultrasound image sequence. The respiratory spatial heterogeneity map is used to associate two-dimensional ultrasound images with unchanged respiratory state but changed spatial position using spatial correlation edge sets, and to associate two-dimensional ultrasound images with unchanged spatial position but changed respiratory state using respiratory correlation edge sets.

[0011] Neighbor node sampling is performed on the breathing space heterogeneous graph to obtain a heterogeneous subgraph containing a subset of neighbor nodes with two edge types;

[0012] A node feature encoder is used to extract node features from the heterogeneous subgraph to obtain new node features of the neighboring heterogeneous subgraph nodes.

[0013] The new node features and the heterogeneous subgraph are embedded into an attention network for heterogeneous graph learning, and the connection relationships of the two edge types are combined to obtain the embedded features;

[0014] Low-dimensional respiratory motion features are obtained by dimensionality reduction of the embedded features using a fully connected network.

[0015] The low-dimensional respiratory motion features and the two-dimensional ultrasound images acquired manually in real time are input into the reconstruction model. The reconstruction model includes a conditional variational encoder module, which is used to map the two-dimensional ultrasound images acquired manually in real time with the low-dimensional respiratory motion features to obtain respiratory signals, using the images as conditional variables and the respiratory motion manifold embedding features as data variables.

[0016] Preferably, the connection relationship between node i and node j in the respiratory space heterogeneous graph is represented by the following formula:

[0017]

[0018] In the formula: Indicates a connection relationship. Let i represent the set of first-order neighbor nodes of node i.

[0019] Preferably: the step of sampling neighbor nodes of the respiratory space heterogeneous graph to obtain a heterogeneous subgraph containing a subset of neighbor nodes of two edge types includes:

[0020] Define the set of neighbor nodes according to the edge type; for each set of neighbor nodes, collect k nodes from each set through random sampling. res and k sp The neighbor node sequence consists of 1 neighbor node. and and its edge subset E sub_res and E sub_spMerge into S sub S formed in sub_new ;

[0021] To the original The process of repeatedly sampling all nodes is used to generate the heterogeneous subgraph containing a subset of neighbor nodes of both edge types.

[0022] Preferably, the node feature encoder comprises a fully convolutional network.

[0023] Preferably, the method for obtaining new node features includes:

[0024] After performing neighbor sampling on the heterogeneous respiratory space graph, all nodes in the heterogeneous subgraph are input into the model for node feature extraction.

[0025] A one-dimensional feature vector set is obtained by processing the feature sequence through a fully connected network. New node features that serve as neighboring subgraph nodes.

[0026] Preferably, a contrastive loss function is used to impose manifold constraints on node embedding features during the training of the attention network and node feature encoder for heterogeneous graph learning; the embedding loss function based on contrastive learning is shown in the following equation:

[0027]

[0028] In the formula: Represents a node and The distance between respiratory signals The label indicates nodes with similar respiratory states within the graph.

[0029] Preferably, the loss function of the conditional variational encoder module is as follows:

[0030]

[0031] In the formula: y i This indicates that the conditional variational encoder module generates a breathing signal. This represents the corresponding values ​​for y obtained by the attention network and the fully connected network learned from the heterogeneous graph. i respiratory signals; This represents the root mean square error between the two; KL(N(0,1),N(g(y) i ),h(y i ))) represents the Kullback-Leibler divergence, which measures the difference between N(0,1) and N(g(y). i ),h(y iThe distribution similarity of )) where N(0,1) is a Gaussian distribution, g and h represent the z generator in the variational encoder module, g(y i ) and h(y i ) represents y i The corresponding mean and equation; Weight KL This represents the weight that controls the KL divergence, and is a constant.

[0032] An ultrasound image respiratory signal recognition device is used to perform the above-described ultrasound image respiratory signal recognition method, the device comprising:

[0033] The two-dimensional ultrasound image sequence acquisition unit is used to extract two-dimensional ultrasound images from the acquired four-dimensional ultrasound images to obtain a two-dimensional ultrasound image sequence.

[0034] A respiratory spatial heterogeneity map construction unit is used to construct a respiratory spatial heterogeneity map using the two-dimensional ultrasound image sequence. The respiratory spatial heterogeneity map is used to associate two-dimensional ultrasound images with unchanged respiratory state but changed spatial position using a spatial association edge set, and to associate two-dimensional ultrasound images with unchanged spatial position but changed respiratory state using a respiratory association edge set.

[0035] The heterogeneous subgraph acquisition unit is used to sample neighbor nodes of the breathing space heterogeneous graph to obtain a heterogeneous subgraph containing a subset of neighbor nodes of two edge types.

[0036] The new node feature extraction unit is used to extract new node features of neighboring heterogeneous subgraph nodes by using the node feature encoder to extract node features of the heterogeneous subgraph.

[0037] The embedding feature acquisition unit is used to embed the new node features and the heterogeneous subgraph into an attention network for heterogeneous graph learning and combine the connection relationships of the two edge types to obtain the embedding features;

[0038] A low-dimensional respiratory motion feature output unit is used to reduce the dimensionality of the embedded features through a fully connected network to obtain low-dimensional respiratory motion features;

[0039] The mapping unit is used to input the low-dimensional respiratory motion features and the two-dimensional ultrasound images acquired manually in real time into the reconstruction model. The reconstruction model includes a conditional variational encoder module, which is used to map the two-dimensional ultrasound images acquired manually in real time with the low-dimensional respiratory motion features to obtain respiratory signals, using the images as conditional variables and the respiratory motion manifold embedding features as data variables.

[0040] An ultrasound image respiratory signal recognition device, the device comprising a processor and a memory:

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is used to execute the above-described ultrasound image respiratory signal recognition method according to the instructions in the program code.

[0043] A computer-readable storage medium for storing program code for performing the above-described ultrasound image respiratory signal recognition method.

[0044] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0045] This application provides a method, apparatus, device, and medium for identifying respiratory signals in ultrasound images. The method represents the respiratory motion correlation and spatial position correlation between two-dimensional ultrasound images as a heterogeneous graph. By introducing manifold constraints, image features are fused and embedded into the heterogeneous graph to obtain the respiratory motion manifold embedding features as the respiratory signal in the two-dimensional ultrasound image. By fusing the relationship between respiratory motion and spatial position in the ultrasound image through the respiratory motion manifold embedding features, a stable and effective respiratory signal can be obtained even if the ultrasound probe pose is arbitrarily changed during acquisition, improving the accuracy of respiratory signal identification and ensuring the flexibility of ultrasound probe acquisition.

[0046] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0048] Figure 1 This is a flowchart of the ultrasound image respiratory signal recognition method provided in the embodiments of the present invention;

[0049] Figure 2 This is a flowchart of the ultrasound image respiratory signal recognition method provided in the embodiments of the present invention;

[0050] Figure 3 This is a schematic diagram of the ultrasound image respiratory signal recognition device provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the ultrasound image respiratory signal recognition device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0053] See Figure 1 This invention provides a method for recognizing respiratory signals in ultrasound images, such as... Figure 1 As shown, the method may include:

[0054] S101: Extract two-dimensional ultrasound images from the acquired four-dimensional ultrasound images to obtain a two-dimensional ultrasound image sequence;

[0055] S102: A respiratory spatial heterogeneity map is constructed using the two-dimensional ultrasound image sequence. This respiratory spatial heterogeneity map is used to associate two-dimensional ultrasound images with unchanged respiratory state but changed spatial position using spatial association edge sets, and also to associate two-dimensional ultrasound images with unchanged spatial position but changed respiratory state using respiratory association edge sets. Specifically, in this application embodiment, the connection relationship between node i and node j in the respiratory spatial heterogeneity map can be represented by the following formula:

[0056]

[0057] In the formula: Indicates a connection relationship. Let i represent the set of first-order neighbor nodes of node i.

[0058] S103: Sampling of neighbor nodes is performed on the heterogeneous graph of the breathing space to obtain a heterogeneous subgraph containing subsets of neighbor nodes of two edge types; in specific implementation, embodiments of this application can provide a set of neighbor nodes defined according to edge type; for each set of neighbor nodes, k are sampled from each set by random sampling. res and k sp The neighbor node sequence consists of 1 neighbor node. and and its edge subset E sub_res and E sub_sp Merge into S sub S formed in sub_new ;

[0059] To the original The above sampling process is repeated for all nodes to generate the heterogeneous subgraph containing a subset of neighbor nodes of both edge types.

[0060] S104: Use a node feature encoder to extract node features from the heterogeneous subgraph to obtain new node features of neighboring heterogeneous subgraph nodes; in specific implementation, the node feature encoder in this application embodiment may include a fully convolutional network.

[0061] The method for obtaining the new node features includes:

[0062] After performing neighbor sampling on the heterogeneous respiratory space graph, all nodes in the heterogeneous subgraph are input into the model for node feature extraction.

[0063] A one-dimensional feature vector set is obtained by processing the feature sequence through a fully connected network. New node features that serve as neighboring subgraph nodes.

[0064] S105: Embed the new node features and the heterogeneous subgraph into an attention network for heterogeneous graph learning and combine the connection relationships of the two edge types to obtain the embedded features;

[0065] S106: Dimensionality reduction of the embedded features is performed using a fully connected network to obtain low-dimensional respiratory motion features;

[0066] S107: Input the low-dimensional respiratory motion features and the two-dimensional ultrasound images acquired manually in real time into the reconstruction model. The reconstruction model includes a conditional variational encoder module. The conditional variational encoder module is used to map the two-dimensional ultrasound images acquired manually in real time with the low-dimensional respiratory motion features to obtain respiratory signals, using the images as conditional variables and the respiratory motion manifold embedding features as data variables.

[0067] During specific training, a contrastive loss function is used to impose manifold constraints on the node embedding features during the training of the heterogeneous graph learning attention network and node feature encoder; the embedding loss function based on contrastive learning is shown in the following equation:

[0068]

[0069] In the formula: Represents a node and The distance between respiratory signals The label indicates nodes with similar respiratory states within the graph.

[0070] The loss function of the conditional variational encoder module is shown in the following equation:

[0071]

[0072] In the formula: y i This indicates that the conditional variational encoder module generates a breathing signal. This represents the corresponding values ​​for y obtained by the attention network and the fully connected network learned from the heterogeneous graph. i respiratory signals; This represents the root mean square error between the two; KL(N(0,1),N(g(y) i ),h(y i ))) represents the Kullback-Leibler divergence, which measures the difference between N(0,1) and N(g(y). i ),h(y i The distribution similarity of )) where N(0,1) is a Gaussian distribution, g and h represent the z generator in the variational encoder module, g(y i ) and h(y i ) represents y i The corresponding mean and equation; Weight KL This represents the weight that controls the KL divergence, and is a constant.

[0073] The ultrasound image respiratory signal recognition method provided in this application is specifically designed for the unique characteristics of manual acquisition of abdominal ultrasound images. Considering that both organ respiratory motion and probe pose changes exist simultaneously in manually acquired two-dimensional abdominal ultrasound images, this method will improve the accuracy of respiratory motion recognition reduced by probe pose changes. It is adaptable to various manual ultrasound acquisition methods, thus expanding the applicability of this method.

[0074] The ultrasound image respiratory signal recognition method provided in this application constructs a heterogeneous map of the ultrasound image to separate organ respiratory motion from the spatial motion of the ultrasound probe. It employs hierarchical attention aggregation to independently learn the correlation between respiration and space, mapping the image to a unified respiratory feature space, thereby avoiding respiratory signal recognition errors caused by probe pose interference. To achieve real-time respiratory signal recognition relying solely on the ultrasound image, this method transforms the respiratory signal recognition task into a learning problem that maps the image to a unified respiratory motion feature space through conditional generative learning; the respiratory signal is the image feature within this unified space.

[0075] The method for identifying respiratory signals from ultrasound images provided in the embodiments of this application will be described in detail below.

[0076] This application provides a real-time respiratory signal recognition method based on respiratory motion heterogeneous graph embedding and mapping. This method represents the respiratory motion correlation and spatial location correlation between two-dimensional images as a spatiotemporal heterogeneous graph, and effectively embeds the respiratory motion heterogeneous graph by introducing manifold constraints to obtain the respiratory motion manifold embedding features of the image as the respiratory signal of the two-dimensional ultrasound image. Since only two-dimensional ultrasound images are available during model application, the respiratory motion correlation and spatial location correlation between images are difficult to obtain. Therefore, this invention incorporates a conditional variational encoder module, aiming to construct a generative mapping model with ultrasound images as conditional variables and respiratory motion manifold embedding features as data variables. During model application, only a two-dimensional ultrasound image needs to be input to generate the corresponding respiratory signal.

[0077] See Figure 2 The steps of an example implementation provided in this application are described in detail. The ultrasound image respiratory signal recognition method based on heterogeneous information embedding and alignment provided in this application includes the following steps:

[0078] Step 1: The four-dimensional ultrasound image is divided into numerous two-dimensional ultrasound images according to the ultrasound acquisition plane. This application simulates a probe scanning method to extract two-dimensional images from each three-dimensional ultrasound image in the sequence. This application considers multiple sets of four-dimensional ultrasound image sequences acquired from different positions and angles in the region of interest. Each set of sequences contains three-dimensional volume data at T time points, distributed across different states of the respiratory cycle.

[0079] Each three-dimensional ultrasound is denoted as V. t ,t∈T. For each V t The image is uniformly divided into N two-dimensional ultrasound images. Each image is denoted as . This is represented as a two-dimensional ultrasound image acquired at position m, rotation angle n, and acquisition time t. From all V... t The images extracted from the model are uniformly distributed in a low-dimensional respiratory motion feature space, and will constitute the input dataset S of the model provided in this application.

[0080] Step 2: In dataset S, from the same V t All the images that were split Regardless of whether the grayscale features are similar, they will belong to the same breathing state. However, in different V... t The split images are obtained only when (m,n) are the same. The similarity of grayscale features between them is used to determine the similarity of breathing states.

[0081] Therefore, to accurately represent the two types of correlations mentioned above, this method constructs the dataset S into a respiratory spatial heterogeneous graph data G containing the two types of connection edges. SRIn this heterogeneous diagram, the respiratory state remains unchanged while the spatial position changes. Spatial correlation edge set E sp Correlation. And for those whose spatial location remains unchanged but whose breathing state changes... Using the breathing-related edge set E res Related. In G SR The data contains only one type of node: the image. in

[0082] Let the set of first-order neighbors of node i be . Connection relationship between node i and node j for:

[0083]

[0084] Since the respiratory motion relation subgraph is an undirected graph, therefore for The connection matrix.

[0085] Step 3: Due to G SR To reduce computational complexity, given the large number of nodes, this method samples neighbor nodes of the respiratory spatial heterogeneous graph before inputting it into the embedding module. The sampled node objects consist of sequences of nodes with the same position, as this can form a subgraph that contains the respiratory cycle.

[0086] Assume the sequence of nodes with the same position is For nodes First, based on edge type E res and E sp Let's define its set of neighboring nodes. Then, for each set of neighboring nodes, we randomly sample k nodes from each set. res and k sp The neighbor node sequence consists of 1 neighbor node. and and its edge subset E sub_res and E sub_sp Merge into S sub S formed in sub_new . To the original By repeating the above process for all nodes, a heterogeneous subgraph containing a subset of neighbor nodes of both edge types can be generated.

[0087] Step 4: The node features obtained by the node feature encoder can better meet the requirements of subsequent modules for ultrasound image features, capture the intrinsic structure and important information of ultrasound, adapt to new ultrasound data, and improve the model's generalization ability.

[0088] This method employs a fully convolutional network as the node feature encoder. For G... SR After neighbor sampling, all nodes in the heterogeneous subgraph are input into the model for node feature extraction. That is, the node sequence... Inputting the data into a convolutional network yields feature sequences containing respiratory motion structure information. This feature sequence will be processed by a fully connected network to obtain a one-dimensional feature vector set. New node features that serve as neighboring subgraph nodes.

[0089] Step 5: Utilize an attention network for heterogeneous graph learning to process the breathing spatial heterogeneous graph G. SR Learning is performed. By effectively learning the representations between nodes through a hierarchical attention mechanism, it can handle heterogeneous graphs with various types of nodes and edges well, capturing complex relationships and interactions between nodes. Based on the given connection relationship E... sub_res and E sub_sp New node characteristics After being embedded, the embedding features are obtained. Those h that exhibit respiratory movement characteristics and have the same respiratory state will approach each other.

[0090] Finally, H is accessed through a fully connected network. sub_new Dimensionality reduction to obtain low-dimensional respiratory motion characteristics In order to make G SR Embedding preserves the respiratory manifold features, meaning that the distance between node embedding results reflects the similarity of the respiratory states of nodes within the graph.

[0091] This application uses a contrastive loss function to impose manifold constraints on node embedding features during the training of the attention network and node feature encoder for heterogeneous graph learning. The embedding loss function based on contrastive learning is as follows:

[0092]

[0093] in, Represents a node and The distance between respiratory signals. The labels representing nodes with similar breathing states within the graph contain only 0 and 1. If a node... and If they are connected, then otherwise, Margin > 0 indicates the boundary value of the loss function.

[0094] Step Six: The goal of this method is to perform real-time respiratory signal estimation on two-dimensional ultrasound image sequences acquired manually. Since only two-dimensional images can be obtained in a real-time acquisition environment, graph connectivity cannot be established, and HAN-based spatiotemporal graph encoders cannot meet the requirements for real-time respiratory signal estimation.

[0095] Therefore, this method introduces a conditional variational encoder (CVAE) to learn the output features of the spatiotemporal graph embedding encoder. The aim is to obtain the embedding features of the HAN (Hypertrathecal Angiography) in a generative manner using only two-dimensional ultrasound images. This allows the model to use only the generation module during operation, avoiding the need for real-time acquisition of spatiotemporal graph connections.

[0096] The general workflow of a Conditional Variational Encoder (CVAE) is as follows: It takes node embeddings as input and image-encoded features as conditions, then encodes them into a probability distribution or latent variable in the latent space using a Z generator. The decoder then reconstructs the input node embeddings using the latent variables and conditions. The loss function of the Conditional Variational Encoder (CVAE) is as follows:

[0097]

[0098] In the formula: y i This indicates that the conditional variational encoder module generates a breathing signal. This represents the corresponding values ​​for y obtained by the attention network and the fully connected network learned from the heterogeneous graph. i respiratory signals; This represents the root mean square error between the two; KL(N(0,1),N(g(y) i ),h(y i ))) represents the Kullback-Leibler divergence, which measures the difference between N(0,1) and N(g(y). i ),h(y i The distribution similarity of )) where N(0,1) is a Gaussian distribution, g and h represent the z generator in the variational encoder module, g(y i ) and h(y i ) represents y i The corresponding mean and equation; Weight KL This represents the weight that controls the KL divergence, and is a constant.

[0099] Step 7: Overall model training consists of two steps.

[0100] The first step involves training the node encoder and the heterogeneous graph encoder using neighbor sampling subgraphs of the breathing spatial graph, and then optimizing the first two networks using the loss function (2). Training is stopped when the local breathing signals gradually align.

[0101] The second step is to freeze the weights of the node encoder and the heterogeneous graph encoder, train the variational encoder network, and use the loss function (3) as the loss term until the model converges.

[0102] Step 8: In the process of model application, only a single frame of two-dimensional ultrasound image needs to be encoded with its grayscale features by a node feature encoder and a conditional encoder. After sampling the Gaussian distribution using the encoded features, the corresponding respiratory signal can be obtained through the decoder.

[0103] This method represents the respiratory motion correlation and spatial location correlation between two-dimensional ultrasound images as a heterogeneous graph. By introducing manifold constraints and fusing image features into the heterogeneous graph, the respiratory motion manifold embedding features are obtained as the respiratory signal of the two-dimensional ultrasound image. A conditional variational encoder module is used to construct a reconstruction model with the image as the conditional variable and the respiratory motion manifold embedding features as the data variable. The entire process includes constructing and embedding the image respiratory motion manifold space using the heterogeneous graph, and then learning and reconstructing this space using the variational encoder. In this process, the variational encoder acts as an explicit mapping function from image to respiratory signal, thus better adapting to new data mappings and possessing strong processing capabilities.

[0104] In summary, the ultrasound image respiratory signal recognition method provided in this application fuses the relationship between respiratory motion and spatial position in ultrasound images by embedding respiratory motion manifold features. This allows for the acquisition of stable and effective respiratory signals even when the ultrasound probe pose is arbitrarily changed during the acquisition process, thereby improving the accuracy of respiratory signal recognition and ensuring the flexibility of ultrasound probe acquisition.

[0105] See Figure 3 This application embodiment can also provide an ultrasound image respiratory signal recognition device, such as... Figure 3 As shown, the apparatus for performing the above-described ultrasound image respiratory signal recognition method may include:

[0106] The two-dimensional ultrasound image sequence acquisition unit 301 is used to extract two-dimensional ultrasound images from the acquired four-dimensional ultrasound images to obtain a two-dimensional ultrasound image sequence.

[0107] The respiratory spatial heterogeneity map construction unit 302 is used to construct a respiratory spatial heterogeneity map using the two-dimensional ultrasound image sequence. The respiratory spatial heterogeneity map is used to associate two-dimensional ultrasound images with unchanged respiratory state but changed spatial position using a spatial association edge set, and to associate two-dimensional ultrasound images with unchanged spatial position but changed respiratory state using a respiratory association edge set.

[0108] The heterogeneous subgraph acquisition unit 303 is used to sample neighbor nodes of the breathing space heterogeneous graph to obtain a heterogeneous subgraph containing a subset of neighbor nodes of two edge types.

[0109] The new node feature extraction unit 304 is used to extract new node features of neighboring heterogeneous subgraph nodes by using a node feature encoder to extract node features of the heterogeneous subgraph.

[0110] The embedding feature acquisition unit 305 is used to embed the new node features and the heterogeneous subgraph into an attention network for heterogeneous graph learning and combine the connection relationships of the two edge types to obtain the embedding features;

[0111] The low-dimensional respiratory motion feature output unit 306 is used to reduce the dimensionality of the embedded features through a fully connected network to obtain low-dimensional respiratory motion features.

[0112] The mapping unit 307 is used to input the low-dimensional respiratory motion features and the two-dimensional ultrasound images acquired manually in real time into the reconstruction model. The reconstruction model includes a conditional variational encoder module, which is used to map the two-dimensional ultrasound images acquired manually in real time with the low-dimensional respiratory motion features to obtain respiratory signals, using the images as conditional variables and the respiratory motion manifold embedding features as data variables.

[0113] This application embodiment can also provide an ultrasound image respiratory signal recognition device, the device including a processor and a memory:

[0114] The memory is used to store program code and transmit the program code to the processor;

[0115] The processor is used to execute the steps of the ultrasound image respiratory signal recognition method described above according to the instructions in the program code.

[0116] like Figure 4 As shown in the figure, an ultrasound image respiratory signal recognition device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0117] In this embodiment, the processor 10 may be a central processing unit (CPU and graphics processing unit, GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0118] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the ultrasound image respiratory signal recognition method.

[0119] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0120] Two-dimensional ultrasound image sequences are obtained by extracting two-dimensional ultrasound images from the acquired four-dimensional ultrasound images;

[0121] A respiratory spatial heterogeneity map is constructed using the two-dimensional ultrasound image sequence. The respiratory spatial heterogeneity map is used to associate two-dimensional ultrasound images with unchanged respiratory state but changed spatial position using spatial correlation edge sets, and to associate two-dimensional ultrasound images with unchanged spatial position but changed respiratory state using respiratory correlation edge sets.

[0122] Neighbor node sampling is performed on the breathing space heterogeneous graph to obtain a heterogeneous subgraph containing a subset of neighbor nodes with two edge types;

[0123] A node feature encoder is used to extract node features from the heterogeneous subgraph to obtain new node features of the neighboring heterogeneous subgraph nodes.

[0124] The new node features and the heterogeneous subgraph are embedded into an attention network for heterogeneous graph learning, and the connection relationships of the two edge types are combined to obtain the embedded features;

[0125] Low-dimensional respiratory motion features are obtained by dimensionality reduction of the embedded features using a fully connected network.

[0126] The low-dimensional respiratory motion features and the two-dimensional ultrasound images acquired manually in real time are input into the reconstruction model. The reconstruction model includes a conditional variational encoder module, which is used to map the two-dimensional ultrasound images acquired manually in real time with the low-dimensional respiratory motion features to obtain respiratory signals, using the images as conditional variables and the respiratory motion manifold embedding features as data variables.

[0127] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0128] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0129] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0130] Of course, it should be noted that, Figure 4 The structure shown does not constitute a limitation on the ultrasound image respiratory signal recognition device in the embodiments of this application. In practical applications, the ultrasound image respiratory signal recognition device may include devices that are more advanced than those described above. Figure 4 More or fewer components as shown, or combinations of certain components.

[0131] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the ultrasound image respiratory signal recognition method described above.

[0132] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An ultrasound image respiratory signal identification method, characterized by, The method comprises the following steps: extracting two-dimensional ultrasound images from the collected four-dimensional ultrasound images to obtain a two-dimensional ultrasound image sequence; constructing a respiratory spatial heterogeneous graph using the two-dimensional ultrasound image sequence, the respiratory spatial heterogeneous graph being used to associate two-dimensional ultrasound images with unchanged respiratory states and changed spatial positions using a spatial correlation edge set, and to associate two-dimensional ultrasound images with unchanged spatial positions and changed respiratory states using a respiratory correlation edge set; performing neighbor node sampling on the respiratory spatial heterogeneous graph to obtain a heterogeneous subgraph containing a neighbor node subset of two edge types; extracting node features of the heterogeneous subgraph using a node feature encoder to obtain new node features of neighbor heterogeneous subgraph nodes; embedding the new node features and the heterogeneous subgraph into an attention network for heterogeneous graph learning and combining the connection relationship of the two edge types to obtain embedding features; dimensionally reducing the embedding features through a fully connected network to obtain low-dimensional respiratory motion features; inputting the low-dimensional respiratory motion features and real-time collected two-dimensional ultrasound images into a reconstruction model, the reconstruction model comprising a conditional variational encoder module, the conditional variational encoder module being used to map the real-time collected two-dimensional ultrasound images and the low-dimensional respiratory motion features to obtain a respiratory signal by taking the two-dimensional ultrasound images as conditional variables and taking the respiratory motion manifold embedding features as data variables.

2. The ultrasound image respiratory signal identification method of claim 1, wherein, The connection relationship between node i and node j in the respiratory spatial heterogeneous graph is represented by the following formula: In the formula: represents a connection relationship, represents a first-order neighbor node set of node i.

3. The ultrasound image respiratory signal identification method of claim 1, wherein, The neighbor node sampling on the respiratory spatial heterogeneous graph to obtain a heterogeneous subgraph containing a neighbor node subset of two edge types comprises: According to the edge type, define the neighbor node set; for each neighbor node set, through the way of random sampling, collect k res and k sp neighbor nodes from each set to form a neighbor node sequence and and its edge subset E sub_res and E sub_sp merge into S sub to form S sub_new ; To the original The process of repeatedly sampling all nodes is used to generate the heterogeneous subgraph containing a subset of neighbor nodes of both edge types.

4. The ultrasound image respiratory signal identification method of claim 1, wherein, The node feature encoder comprises a full convolutional network.

5. The ultrasound image respiratory signal identification method of claim 4, wherein, The new node feature acquisition method comprises: After neighbor sampling on the respiratory spatial heterogeneous graph, all nodes in the heterogeneous subgraph are input into a model for node feature extraction; A set of one-dimensional feature vectors is obtained after processing the feature sequence by the fully connected network New node features as neighbor subgraph nodes.

6. The ultrasound image respiratory signal identification method of claim 1, wherein, In the process of training the attention network for heterogeneous graph learning and the node feature encoder, a contrastive loss function is used to constrain the node embedding features in the manifold; the embedding loss function based on contrastive learning is shown in the following formula: In the formula: represents a node and the distance between the respiratory signals, represents the label of the node with similar respiratory state in the graph.

7. The ultrasound image respiratory signal identification method of claim 1, wherein, The loss function of the conditional variational encoder module is shown in the following formula: In the formula: y i This indicates that the conditional variational encoder module generates a breathing signal. This represents the corresponding values ​​for y obtained by the attention network and the fully connected network learned from the heterogeneous graph. i respiratory signals; This represents the root mean square error between the two; KL(N(0,1),N(g(y) i ),h(y i ))) represents the Kullback-Leibler divergence, which measures the difference between N(0,1) and N(g(y). i ),h(y i The distribution similarity of )) where N(0,1) is a Gaussian distribution, g and h represent the z generator in the variational encoder module, g(y i ) and h(y i ) represents y i The corresponding mean and equation; Weight KL This represents the weight that controls the KL divergence, and is a constant.

8. An ultrasound image respiratory signal identification apparatus, characterized by, The device for performing the ultrasound image respiratory signal identification method of any one of claims 1-7 comprises: a two-dimensional ultrasound image sequence acquisition unit configured to extract two-dimensional ultrasound images from the collected four-dimensional ultrasound images to obtain a two-dimensional ultrasound image sequence; a respiratory spatial heterogeneous graph construction unit configured to construct a respiratory spatial heterogeneous graph using the two-dimensional ultrasound image sequence, the respiratory spatial heterogeneous graph being used to associate two-dimensional ultrasound images with unchanged respiratory states and changed spatial positions using a spatial correlation edge set, and to associate two-dimensional ultrasound images with unchanged spatial positions and changed respiratory states using a respiratory correlation edge set; a heterogeneous subgraph acquisition unit configured to perform neighbor node sampling on the respiratory spatial heterogeneous graph to obtain a heterogeneous subgraph containing a neighbor node subset of two edge types; a new node feature extraction unit configured to extract node features of the heterogeneous subgraph using a node feature encoder to obtain new node features of neighbor heterogeneous subgraph nodes; The embedding feature acquisition unit is configured to embed the new node features and the heterogeneous subgraph into an attention network for heterogeneous graph learning, and obtain embedding features by combining the connection relationship of the two edge types. The low-dimensional respiratory motion feature output unit is configured to obtain low-dimensional respiratory motion features by dimension reduction of the embedding features through a fully connected network. The mapping unit is configured to input the low-dimensional respiratory motion features and the real-time two-dimensional ultrasound image collected manually into a reconstruction model, the reconstruction model comprising a conditional variational encoder module, the conditional variational encoder module being configured to map the real-time two-dimensional ultrasound image collected manually and the low-dimensional respiratory motion features to obtain a respiratory signal by taking an image as a conditional variable and taking a respiratory motion manifold embedding feature as a data variable.

9. An ultrasound image respiratory signal identification device, characterized by The device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1-7 according to instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is configured to execute the method according to any one of claims 1-7. The computer readable storage medium is configured to store program code, and the program code is configured to execute the method according to any one of claims 1-7.

Images