A spatiotemporal consistency-based cross-modality fetal heart visual navigation method, device, equipment, medium and product

Abstract

The application discloses a cross-modal fetal heart visual navigation method and device based on space-time consistency, equipment, medium and product, relates to the field of fetal heart ultrasound, and the method comprises the following steps: acquiring a current ultrasound scanning section in a fetal heart ultrasound scanning process; based on the current ultrasound scanning section, a trained relative pose predictor is used to predict a pose transformation matrix and a relative pose transformation matrix of the current ultrasound scanning section; and based on the pose transformation matrix and the relative pose transformation matrix, the probe is rotated to a target ultrasound scanning section, so that fetal heart visual navigation is realized. The application can guide clinicians to navigate and acquire sections based on a real fetal heart scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fetal heart ultrasound, and in particular to a cross-modal fetal heart visual navigation method and device based on spatiotemporal consistency, equipment, medium and product. BACKGROUND

[0002] Medical imaging is the collection of internal tissues and organs of the human body in time or space, and plays an important role in capturing structures and lesions, but is usually limited by time or spatial complexity. For example, when obtaining a diagnostic section, a large amount of time is usually required, and the experience of an expert is relied on, which seriously restricts the efficiency and accuracy of medical diagnosis. Taking the fetal heart, a prenatal organ, as an example, it has both time and spatial complexity. The operator needs to master rich cardiac anatomy knowledge and have strong spatial imagination, and needs to convert the correct spatial stereoscopic relationship at any time according to the maternal environment, fetal position and gestational age, so as to quickly and accurately obtain the diagnostic section, but this is quite difficult for most prenatal screening physicians. Therefore, it is urgent to study fetal heart visual navigation to reduce the operator's subjective intervention and experience dependence, so as to solve the problem of section acquisition in early diagnosis of congenital heart disease. SUMMARY

[0003] Based on this, the purpose of the present application is to provide a cross-modal fetal heart visual navigation method, device and equipment based on spatiotemporal consistency.

[0004] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a cross-modal fetal heart visual navigation method based on spatiotemporal consistency, comprising:

[0006] obtaining a current ultrasound scanning section in a fetal heart ultrasound scanning process;

[0007] based on the current ultrasound scanning section, using a trained relative pose predictor to predict a pose transformation matrix and a relative pose transformation matrix of the current ultrasound scanning section;

[0008] based on the pose transformation matrix and the relative pose transformation matrix, rotating the probe to a target ultrasound scanning section to realize fetal heart visual navigation.

[0009] In a second aspect, the present application provides a cross-modal fetal heart visual navigation device based on spatiotemporal consistency, comprising:

[0010] an acquisition module configured to obtain a current ultrasound scanning section in a fetal heart ultrasound scanning process;

[0011] a prediction module configured to predict, based on the current ultrasound scan section, a pose transformation matrix and a relative pose transformation matrix of the current ultrasound scan section by using the trained relative pose predictor;

[0012] a rotation module configured to rotate the probe to the target ultrasound scan section based on the pose transformation matrix and the relative pose transformation matrix, so as to realize visual navigation of the fetal heart.

[0013] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the above-mentioned spatio-temporal consistency-based cross-modal visual navigation method for fetal heart.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the above-mentioned spatio-temporal consistency-based cross-modal visual navigation method for fetal heart.

[0015] In a fifth aspect, the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the above-mentioned spatio-temporal consistency-based cross-modal visual navigation method for fetal heart.

[0016] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0017] The present application provides a spatio-temporal consistency-based cross-modal visual navigation method for fetal heart, device, equipment, medium and product, and the trained relative pose predictor can predict the pose transformation matrix and the relative pose transformation matrix of the current ultrasound scan section, and obtain the pose transformation relationship between the current section and the target section, so as to accurately rotate the probe to the target section and realize visual navigation of the fetal heart. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0019] Figure 1 The flowchart of the spatio-temporal consistency-based cross-modal visual navigation method for fetal heart provided by an embodiment of the present application is shown in the figure.

[0020] Figure 2 The training process of the relative pose predictor is shown in the figure.

[0021] Figure 3A structural schematic diagram of a computer device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0023] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0024] Existing researches on the acquisition of fetal heart sections can be divided into two categories. One is the automatic recognition of fetal heart sections, but the prerequisite for section recognition is the data collection of the section, and the problem of section acquisition has not been fundamentally solved. The other is the acquisition of sections based on virtual ultrasound, such as the method proposed by Yang et al. which determines the pose relationship between fetal heart sections to establish virtual ultrasound navigation, but virtual ultrasound cannot be directly applied to real fetal heart. The advantage of the present application is to guide clinicians to navigate and acquire sections based on the real scenario of fetal heart.

[0025] In an exemplary embodiment, as shown in Figure 1 A spatiotemporal consistency-based cross-modal fetal heart visual navigation method is provided, which is executed by a computer device, specifically by a terminal or a server, or by both a terminal and a server. In the embodiments of the present application, the method is applied to a server, and includes the following steps S1 to S3. Wherein:

[0026] S1: acquiring a current ultrasound scanning section in the process of fetal heart ultrasound scanning.

[0027] S2: based on the current ultrasound scanning section, predicting the pose transformation matrix and the relative pose transformation matrix of the current ultrasound scanning section by using a trained relative pose predictor.

[0028] S3: rotating the probe to a target ultrasound scanning section based on the pose transformation matrix and the relative pose transformation matrix, to realize fetal heart visual navigation.

[0029] In a specific embodiment, the relative pose predictor comprises a first linear layer, a first Relu activation function, a second linear layer, a second Relu activation function and a third linear layer connected in sequence. The first linear layer and the second linear layer comprise 256 hidden units, and the output layer of the third linear layer contains 9 hidden units.

[0030] In a specific embodiment, as shown in FIG. 6, the training process of the relative pose predictor comprises: Figure 2

[0031] (1) obtaining a training sample; the training sample comprises a sample image and an actual pose transformation matrix of the sample image; the sample image comprises a sample ultrasound image of a fetal heart ultrasound scan and a sample CT image of a resampled large blood vessel cast CT.

[0032] By collecting L fetal heart ultrasound scans and resampled large blood vessel cast CTs, a series of ultrasound videos and CT images are obtained. Each ultrasound video and CT image comprises N images and their corresponding pose data, represented by a set For the images extracted from the ultrasound video and the resampled large blood vessel cast CT, they are represented as x vi and x ci , respectively, both of which have pose transformation matrices p vi and p ci in the absolute coordinate system. The pose transformation matrix of the target section in the absolute coordinate system is positioned as p t , where R vt is the pose transformation matrix of x i from p i to p t in the local coordinate system. Given any input frame x i , the target of visual navigation is to predict the relative pose transformation matrix from p i to the target section pose p t , such as the four-chamber heart section or the three-vessel section of the fetal heart. The true situation of the pose motion is calculated as:

[0033] R it = p i -1 p t

[0034] wherein, R it is the relative pose transformation matrix of x i to the target section x t .

[0035] (2) extracting feature vectors of adjacent two frames of sample images by a feature extractor.

[0036] Randomly extracting image pairs x i and x​j The image pair is subjected to feature extraction by two feature extractors E sharing parameters. The extracted feature vectors f i ,f j Subtraction is performed, whether it is subtraction of features of ultrasound scan images or subtraction of features of CT images, and the fetal heart sections have relative invariance in rotation.

[0037] (3) The feature vectors of the adjacent two sample images are input into the relative pose predictor to obtain a predicted relative pose transformation matrix of the adjacent two sample images and a predicted pose transformation matrix of the sample CT image.

[0038] The feature vectors of the image pair are input into the relative pose predictor P. P predicts the relative pose transformation matrix

[0039] (4) A loss function is constructed based on the predicted relative pose transformation matrix of the adjacent two sample images and the predicted pose transformation matrix of the sample CT image. Specifically, a first loss function is constructed based on the predicted relative pose transformation matrix of the adjacent two sample images and the actual relative pose transformation matrix of the adjacent two sample images; a second loss function is constructed based on the predicted pose transformation matrix of the sample CT image and the actual pose transformation matrix of the sample CT image; and a third loss function is constructed based on the similarity between the adjacent two sample images.

[0040] The first loss function L1 is the mean square error (MSE) between the predicted relative pose transformation matrix and the true value.

[0041]

[0042] Here is the predicted relative pose transformation matrix of the sample image x i to the sample image x j , R ij is the actual relative pose transformation matrix of the sample image x i to the sample image x j , MSE represents the mean square error, P represents the relative pose predictor, E(x i ), E(x j ) respectively represent the feature vectors of the sample image x i and the sample image x j extracted by the feature extractor.

[0043] Because the large blood vessel cast CT can obtain the 3D pose of the real fetal heart, the pose transformation matrix of a certain section is also predicted by regression.

[0044]

[0045] wherein, L2 is a second loss function, are the predicted pose transformation matrixes of the sample CT images x ci and x cj are the actual pose transformation matrixes of the sample CT images x ci , p cj , p ci and x cj .

[0046] In order to ensure that the distance between the predicted poses and the relative distance between the features remain isometric, a contrastive learning loss function based on feature distance is introduced, and cosine similarity or Euclidean distance is used to calculate the distance between two features:

[0047]

[0048] wherein, L3 is a third loss function, D f is the similarity between the adjacent two frame sample images, a is a hyperparameter between the feature distance and the actual distance, and τ is a parameter for adjusting the scale of the similarity score. Finally, the loss function L total is the fusion of the ultrasound part and the CT part:

[0049] L total = L1+L2+L 3。

[0050] (5) Adjusting the parameters of the relative pose predictor based on the loss function, and completing the training of the relative pose predictor.

[0051] The present application takes the four-chamber heart section of the fetal heart as the starting section, uses the relative pose predictor to obtain the relative rotation matrix from the four-chamber heart section to other sections as the rough pose to any section, and then trains the regression model of the specific section through the pairing data of the ultrasound scanning section and its corresponding pose data, and performs regression of the accurate section of the real fetal heart.

[0052] Based on the same inventive concept, the embodiments of the present application also provide a device for implementing the above-mentioned cross-modality fetal heart visual navigation method based on spatio-temporal consistency. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more spatio-temporal consistency-based cross-modality fetal heart visual navigation device embodiments provided below can be referred to the limitations of the spatio-temporal consistency-based cross-modality fetal heart visual navigation method in the foregoing, which will not be repeated here.

[0053] In one exemplary embodiment, a spatio-temporal consistency-based cross-modality fetal heart visual navigation device is provided, comprising:

[0054] an acquisition module configured to acquire a current ultrasound scanning section in a fetal heart ultrasound scanning process.

[0055] a prediction module configured to predict, based on the current ultrasound scanning section, a pose transformation matrix and a relative pose transformation matrix of the current ultrasound scanning section by using a trained relative pose predictor.

[0056] a rotation module configured to rotate the probe to a target ultrasound scanning section based on the pose transformation matrix and the relative pose transformation matrix, so as to realize fetal heart visual navigation.

[0057] The device further includes a training module, which specifically includes:

[0058] a training sample acquisition unit configured to acquire training samples, wherein the training samples include sample images and pose transformation matrices of the sample images, and the sample images include sample ultrasound images of fetal heart ultrasound scanning and sample CT images of resampled large blood vessel cast CT.

[0059] a feature vector extraction unit configured to extract feature vectors of adjacent two frames of sample images by using a feature extractor.

[0060] a prediction unit configured to input the feature vectors of the adjacent two frames of sample images into a relative pose predictor, so as to obtain a predicted relative pose transformation matrix of the adjacent two frames of sample images and a predicted pose transformation matrix of the sample CT images.

[0061] a loss function construction unit configured to construct a loss function based on the predicted relative pose transformation matrix of the adjacent two frames of sample images and the predicted pose transformation matrix of the sample CT images.

[0062] a parameter adjustment unit configured to adjust parameters of the relative pose predictor based on the loss function, so as to complete training of the relative pose predictor.

[0063] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in each of the above method embodiments when executing the computer program. The computer device can be a server or a terminal, and its internal structure diagram can be as follows: Figure 3As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be processed. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to implement a cross-modal fetal heart visual navigation method based on space-time consistency.

[0064] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0065] In one exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0066] In one exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0067] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0069] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0070] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0071] The principles and implementation modes of the present application are described by applying specific examples herein. The above description of the embodiments is only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A cross-modal fetal cardiac visual navigation method based on spatiotemporal consistency, characterized in that, include: Obtain the current ultrasound scan section during a fetal cardiac ultrasound scan; Based on the current ultrasound scanning section, the pose transformation matrix and relative pose transformation matrix of the current ultrasound scanning section are predicted using a trained relative pose predictor. The probe is rotated to the target ultrasound scanning plane based on the pose transformation matrix and the relative pose transformation matrix to achieve visual navigation of the fetal heart. The loss function of the relative pose predictor during the training process includes a first loss function, a second loss function, and a third loss function; The loss function is constructed based on the predicted relative pose transformation matrix of two adjacent sample images and the predicted pose transformation matrix of the sample CT image, specifically including: The first loss function is constructed based on the predicted relative pose transformation matrix of two adjacent sample images and the actual relative pose transformation matrix of two adjacent sample images. A second loss function is constructed based on the predicted pose transformation matrix of the sample CT image and the actual pose transformation matrix of the sample CT image; A third loss function is constructed based on the similarity between two adjacent sample images; The formula for calculating the third loss function L3 is as follows: Among them, D f The similarity between two adjacent sample images is represented by α, a hyperparameter, τ, a scale parameter, N, and R. ij For sample image x i To sample image x j The actual relative pose transformation matrix.

2. The cross-modal fetal cardiac visual navigation method based on spatiotemporal consistency according to claim 1, characterized in that, The relative pose predictor includes a first linear layer, a first ReLU activation function, a second linear layer, a second ReLU activation function, and a third linear layer connected in sequence.

3. The cross-modal fetal cardiac visual navigation method based on spatiotemporal consistency according to claim 1, characterized in that, The training process of the relative pose predictor includes: Acquire training samples; the training samples include sample images and the actual pose transformation matrix of the sample images; the sample images include sample ultrasound images from fetal cardiac ultrasound scans and sample CT images from resampled large vessel cast CT scans; The feature vectors of two adjacent sample images are extracted using a feature extractor. The feature vectors of two adjacent sample images are input into the relative pose predictor to obtain the predicted relative pose transformation matrix of the two adjacent sample images and the predicted pose transformation matrix of the sample CT image. A loss function is constructed based on the predicted relative pose transformation matrix of two adjacent sample images and the predicted pose transformation matrix of the sample CT image. The parameters of the relative pose predictor are adjusted based on the loss function to complete the training of the relative pose predictor.

4. The cross-modal fetal cardiac visual navigation method based on spatiotemporal consistency according to claim 3, characterized in that, The formula for calculating the loss function is as follows: L total L1+L2+L3 Among them, L total Let L1, L2, and L3 be the first, second, and third loss functions, respectively. For the sample image x i To sample image x j The predicted relative pose transformation matrix, R ij For sample image x i To sample image x j The actual relative pose transformation matrix, MSE represents the mean square error, P represents the relative pose predictor, and E(x) represents the mean square error. i ), E(x) j ) represent the sample images x extracted by the feature extractor. i and sample image x j eigenvectors, The sample CT images x ci and x cj The predicted pose transformation matrix, p ci p cj The sample CT images x ci and x cj The actual pose transformation matrix.

5. A cross-modal fetal cardiac visual navigation device based on spatiotemporal consistency, characterized in that, include: The acquisition module is used to acquire the current ultrasound scan section during fetal cardiac ultrasound scanning. The prediction module is used to predict the pose transformation matrix and relative pose transformation matrix of the current ultrasound scanning section based on the current ultrasound scanning section using a trained relative pose predictor. The rotation module is used to rotate the probe to the target ultrasound scanning plane based on the pose transformation matrix and the relative pose transformation matrix to achieve visual navigation of the fetal heart. The loss function of the relative pose predictor during the training process includes a first loss function, a second loss function, and a third loss function; The loss function is constructed based on the predicted relative pose transformation matrix of two adjacent sample images and the predicted pose transformation matrix of the sample CT image, specifically including: The first loss function is constructed based on the predicted relative pose transformation matrix of two adjacent sample images and the actual relative pose transformation matrix of two adjacent sample images. A second loss function is constructed based on the predicted pose transformation matrix of the sample CT image and the actual pose transformation matrix of the sample CT image; A third loss function is constructed based on the similarity between two adjacent sample images; The formula for calculating the third loss function L3 is as follows: Among them, D f The similarity between two adjacent sample images is represented by α, a hyperparameter, τ, a scale parameter, N, and R. ij For sample image x i To sample image x j The actual relative pose transformation matrix.

6. The cross-modal fetal cardiac visual navigation device based on spatiotemporal consistency according to claim 5, characterized in that, Also includes: Training module; The training module specifically includes: A training sample acquisition unit is used to acquire training samples; the training samples include sample images and the actual pose transformation matrix of the sample images; the sample images include sample ultrasound images from fetal cardiac ultrasound scans and sample CT images from resampled large vessel cast CT scans; The feature vector extraction unit is used to extract the feature vectors of two adjacent sample images through the feature extractor. The prediction unit is used to input the feature vectors of two adjacent sample images into the relative pose predictor to obtain the predicted relative pose transformation matrix of the two adjacent sample images and the predicted pose transformation matrix of the sample CT image. The loss function construction unit is used to construct a loss function based on the predicted relative pose transformation matrix of two adjacent sample images and the predicted pose transformation matrix of the sample CT image. The parameter adjustment unit is used to adjust the parameters of the relative pose predictor based on the loss function, thereby completing the training of the relative pose predictor.

7. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the spatiotemporally consistent cross-modal fetal cardiac visual navigation method according to any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the spatiotemporal consistency-based cross-modal fetal cardiac visual navigation method as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the spatiotemporal consistency-based cross-modal fetal cardiac visual navigation method as described in any one of claims 1-4.

Images