Ultrasonic section positioning method, equipment and product for echocardiography

Abstract

The invention discloses an ultrasonic section positioning method, device and product for echocardiography, and relates to the field of echocardiography navigating.The method comprises the steps that an image pair composed of any two frames of images in the same echocardiography video with probe posture information is used for training an ultrasonic probe position rotation model; the ultrasonic probe position rotation model comprises a feature extractor and a rotation predictor; the feature extractor is used for extracting attitude feature vectors of the image pairs respectively; the rotation predictor is used for predicting relative rotation of the position of the ultrasonic probe according to the attitude feature vector of the image pair; training an ultrasonic section positioning model by using the echocardiogram video data set containing the target section; an ultrasonic section positioning model is constructed based on a feature extractor in the trained ultrasonic probe position rotation model; and performing ultrasonic section positioning according to the trained ultrasonic section positioning model. According to the invention, universal and automatic navigation positioning of the ultrasonic section can be realized.

Description

Technical Field

[0001] The present application relates to the field of echocardiography navigation, and in particular to an ultrasonic section positioning method, device and product for echocardiography. Background Art

[0002] Ultrasound section positioning is designed to help ultrasound physicians quickly find the required ultrasound section in actual clinical practice, and in robotic ultrasound, it is used to provide control of the target section for the robot control program, thereby realizing the automatic acquisition of the ultrasound section. Echocardiography is the most commonly used examination method for diagnosing cardiovascular diseases due to its advantages such as real-time imaging, non-invasiveness, low cost, and no side effects. However, there are many difficulties in the acquisition of echocardiograms. The biggest difficulty is that ultrasound physicians need to manually find the correct section position by moving the ultrasound probe. In this process, ultrasound physicians need to have a high degree of understanding of the three-dimensional structure of the heart. Therefore, it is difficult to train ultrasound physicians, which leads to the inability to promote echocardiography examinations to primary medical units.

[0003] In view of the above problems, the existing technical solutions are divided into two categories. One is a method based on image-to-action regression, which realizes the regression from ultrasound images to the next hand movement of ultrasound physicians by training dual backbone networks. This method is easily affected by the personal subjective preferences of ultrasound physicians in the training data during training; when using it, users need to imitate the actions predicted by the model one by one, which has defects in the experience of human-computer interaction. The other is a method based on reinforcement learning. Through the reinforcement learning method, the dual backbone network model is trained to predict the hand movement to be taken next from the current ultrasound imaging. This method is dependent on the environment and is limited by the reinforcement learning method. It usually works better in the environment used for training, but performs poorly on patients who did not appear in the training. At the same time, due to safety issues, it is usually impossible to perform online reinforcement learning training on real patients, which limits the performance of the algorithm. In addition, the current methods are methods and models designed for specific target sections. For any new target section, relevant data needs to be re-collected and the navigation algorithm needs to be re-trained. The navigation models between different sections are not universal.

[0004] Therefore, in order to realize universal and automatic navigation positioning of ultrasound sections, it is urgent to provide a new ultrasound section positioning method. Summary of the invention

[0005] The purpose of the present application is to provide an ultrasonic section positioning method, device and product for echocardiography, which can realize universal and automatic navigation positioning of the ultrasonic section.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an ultrasonic section positioning method for echocardiography, the ultrasonic section positioning method for echocardiography comprising:

[0008] An ultrasound probe position rotation model is trained using an image pair consisting of any two frames of images in the same ultrasound cardiography video with probe posture information; the ultrasound probe position rotation model comprises: a feature extractor and a rotation predictor; the feature extractor is used to extract posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasound probe position according to the posture feature vectors of the image pair;

[0009] Using an echocardiogram video data set containing a target section to train an ultrasound section localization model; the ultrasound section localization model is constructed based on a feature extractor in a trained ultrasound probe position rotation model;

[0010] Ultrasonic section positioning is performed according to the trained ultrasonic section positioning model.

[0011] Optionally, the method of training the ultrasound probe position rotation model using an image pair consisting of any two frames of images in the same ultrasound cardiography video with probe posture information may also include:

[0012] The same echocardiogram video output by the ultrasound machine is captured by a video capture device;

[0013] The sensor is used to obtain the probe posture corresponding to the echocardiogram video, and the same echocardiogram video with the probe posture information is obtained.

[0014] Optionally, the structure of the feature extractor is a dual-backbone network.

[0015] Optionally, the rotation predictor is a multi-layer cascaded fully connected network with an activation function.

[0016] Optionally, the loss functions of the ultrasound probe position rotation model and the ultrasound section positioning model are both mean square error functions MSE.

[0017] In a second aspect, the present application provides an ultrasonic section positioning device for echocardiography, wherein the ultrasonic section positioning device for echocardiography comprises:

[0018] The ultrasonic probe position rotation model training module is used to train the ultrasonic probe position rotation model using an image pair consisting of any two frames of images in the same ultrasonic cardiography video with probe posture information; the ultrasonic probe position rotation model includes: a feature extractor and a rotation predictor; the feature extractor is used to extract the posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasonic probe position according to the posture feature vectors of the image pair;

[0019] An ultrasound section positioning model training module is used to train an ultrasound section positioning model using an ultrasound cardiogram video data set containing a target section; the ultrasound section positioning model is constructed based on a feature extractor in a trained ultrasound probe position rotation model;

[0020] The ultrasonic section plane positioning module is used to perform ultrasonic section plane positioning according to the trained ultrasonic section plane positioning model.

[0021] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the ultrasonic section positioning method for echocardiography.

[0022] In a fourth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the ultrasonic section positioning method for echocardiography.

[0023] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0024] The present application provides an ultrasonic section positioning method, device and product for echocardiography, which trains an ultrasonic probe position rotation model by using an image pair consisting of any two frames of images in the same echocardiography video with probe posture information, that is, extracting features unrelated to the target section from the image pair; then constructing an ultrasonic section positioning model using a feature extractor in the trained ultrasonic probe position rotation model, and training it; the present application uses echocardiography data including multiple ultrasonic sections in the process of training the feature extractor unrelated to the target section, so the present application can achieve good results on different target sections. Furthermore, the present application uses a small amount of echocardiography video data sets containing target sections for training, which can reduce the training time while adding a small amount of parameters, and facilitate the deployment of the ultrasonic section positioning model in actual use scenarios, and can achieve universal and automatic navigation positioning of ultrasonic sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a diagram of an application environment of an ultrasound section positioning method for echocardiography in one embodiment of the present application;

[0027] Figure 2 A schematic diagram of a flow chart of an ultrasonic section positioning method for echocardiography provided in one embodiment of the present application;

[0028] Figure 3 A schematic diagram of the overall process of an ultrasonic section positioning method for echocardiography provided in one embodiment of the present application;

[0029] Figure 4 This is a schematic diagram of the automatic section positioning effect. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] The ultrasound section positioning method for echocardiography provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the ultrasound video data to be processed to the server 104. After the server 104 receives the ultrasound video data to be processed, the server 104 uses the trained ultrasound section positioning model to locate the ultrasound section for the ultrasound video data to be processed. The server 104 can feedback the obtained ultrasound section positioning result to the terminal 102. In addition, in some embodiments, the ultrasound section positioning method for ultrasound cardiography can also be implemented separately by the server 104 or the terminal 102, such as the terminal 102 can directly perform ultrasound section positioning on the ultrasound video data to be processed, or the server 104 can obtain the ultrasound video data to be processed from the data storage system and perform ultrasound section positioning on the ultrasound video data to be processed.

[0033] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, and tablet computers. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0034] In an exemplary embodiment, Figure 2 and Figure 3 As shown, a method for locating an ultrasound section of an echocardiogram is provided. The method is executed by a computer device, and specifically can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following S201 to S203. Among them:

[0035] S201, using an image pair consisting of any two frames of images in the same echocardiography video with probe posture information to train an ultrasound probe position rotation model; the ultrasound probe position rotation model includes: a feature extractor and a rotation predictor; the feature extractor is used to extract the posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasound probe position according to the posture feature vectors of the image pair; the structure of the feature extractor is a dual-backbone network; the rotation predictor is a multi-layer cascaded fully connected network with an activation function; each layer of the fully connected network includes: 128 to 512 neurons, cascaded 1 to 3 layers.

[0036] Before S201, it also included:

[0037] The same echocardiogram video output by the ultrasound machine is captured by a video capture device;

[0038] The probe posture information of the echocardiography video is positioned to obtain the same echocardiography video with the probe posture information.

[0039] S202, training an ultrasound section localization model using an ultrasound video data set containing a target section; the ultrasound section localization model is constructed based on a feature extractor in a trained ultrasound probe position rotation model;

[0040] S203, performing ultrasound section positioning according to the trained ultrasound section positioning model.

[0041] like Figure 3 As shown, the overall process of this application is:

[0042] (1) A target slice-independent feature extractor is trained in a self-supervised manner; wherein the training set includes: an echocardiogram video {x1, x2, ..., x N};

[0043] Using a large number of random echocardiogram videos from the same v Two randomly selected frames of images x vi and x vjThe image pairs formed by x are used as training samples. The feature extractor adopts the structure of dual backbone network to effectively extract the features unrelated to the target section from these image pairs. Two dual backbone networks E with the same structure and shared parameters are used to extract the features from the image x and x respectively. vi , x vj Extract the feature vector f vi and f vj .

[0044] f vi =E(x vi );

[0045] f vj =E(x vj );

[0046] The two extracted feature vectors f vi and f vj Merge and form a complete feature representation of an image pair, and input the merged feature vector into the rotation predictor P to predict the image x vi and x vj Corresponding ultrasound probe position p vi and p vj The relative rotation between The rotation predictor P predicts the relative rotation as In the process of training the dual backbone network E and the rotation extractor P, the true value R of the relative rotation is vij and predicted values Construct the loss function L1, and optimize L1 to train the dual backbone networks E and P. The loss function L1 is the minimum mean square error function MSE.

[0047] L1=MSE(P(E(x vi ), E(x vj )), R vij );

[0048] After training, the obtained dual backbone network E is a slice-independent feature extractor, which can extract parameters related to the probe position from the ultrasound image. Based on these parameters, universal target slice navigation is achieved.

[0049] (2) Construction of a universal ultrasound slice positioning model for the target slice

[0050] When constructing an ultrasound section localization model for a target section, the training data used is a set of ultrasound cardiogram videos {x1, x2, ...x M}. Each video x i All have a standard section position p it, which can determine the relative rotation between the probe position and the standard section position corresponding to any frame of image

[0051] When constructing an ultrasound slice localization model for a target slice, the previously trained slice-independent feature extractor and the parameters of the dual backbone network E are kept unchanged, so that ultrasound slice localization models for multiple different target slices can use the same dual backbone network E. The ultrasound slice localization model for a target slice consists of two parts, the first part of which is a pre-trained slice-independent feature extractor, namely the dual backbone network E. The second part is a rotation predictor G related to the target slice. The image to be predicted x vi First, input it into the previously trained dual backbone network E to obtain the image feature f vi . Then the image feature f vi is input into G, and the final output is the relative rotation between the standard section position and the current probe position The loss function L2 is constructed using the real relative rotation and the predicted relative rotation. The training of G is achieved by optimizing the loss function L2. The loss function L2 is also the mean square error function MSE.

[0052]

[0053] like Figure 4 As shown, automatic section positioning can be quickly obtained based on the trained ultrasound section positioning model, where the blue prism is the ultrasound probe position corresponding to the standard section, the white prism is the ultrasound probe position corresponding to the left image, and the green prism is the standard section position predicted by the automatic navigation model.

[0054] Based on the same inventive concept, the embodiment of the present application also provides an ultrasonic section positioning device for echocardiography for implementing the ultrasonic section positioning method for echocardiography involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more embodiments of the ultrasonic section positioning device for echocardiography provided below can refer to the limitations of the ultrasonic section positioning method for echocardiography above, and will not be repeated here.

[0055] In an exemplary embodiment, an ultrasound slice positioning device for echocardiography is provided, comprising:

[0056] The ultrasonic probe position rotation model training module is used to train the ultrasonic probe position rotation model using an image pair consisting of any two frames of images in the same ultrasonic cardiography video with probe posture information; the ultrasonic probe position rotation model includes: a feature extractor and a rotation predictor; the feature extractor is used to extract the posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasonic probe position according to the posture feature vectors of the image pair;

[0057] An ultrasound section positioning model training module is used to train an ultrasound section positioning model using an ultrasound cardiogram video data set containing a target section; the ultrasound section positioning model is constructed based on a feature extractor in a trained ultrasound probe position rotation model;

[0058] The ultrasonic section plane positioning module is used to perform ultrasonic section plane positioning according to the trained ultrasonic section plane positioning model.

[0059] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store echocardiographic video data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for positioning an ultrasound section for echocardiography is implemented.

[0060] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0061] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0063] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0064] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0065] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for positioning an ultrasonic section in an echocardiogram, characterized in that: The ultrasonic section positioning method for echocardiography comprises: An ultrasound probe position rotation model is trained using an image pair consisting of any two frames of images in the same ultrasound cardiography video with probe posture information; the ultrasound probe position rotation model comprises: a feature extractor and a rotation predictor; the feature extractor is used to extract posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasound probe position according to the posture feature vectors of the image pair; Using an echocardiogram video data set containing a target section to train an ultrasound section localization model; the ultrasound section localization model is constructed based on a feature extractor in a trained ultrasound probe position rotation model; Ultrasound section positioning is performed according to the trained ultrasound section positioning model.

2. The ultrasonic section positioning method for echocardiography according to claim 1, characterized in that: The method of training the ultrasound probe position rotation model by using an image pair consisting of any two frames of images in the same ultrasound cardiography video with probe posture information also includes: The same echocardiogram video output by the ultrasound machine is captured by a video capture device; The sensor is used to obtain the probe posture corresponding to the echocardiogram video, and the same echocardiogram video with the probe posture information is obtained.

3. The ultrasonic section positioning method for echocardiography according to claim 1, characterized in that: The structure of the feature extractor is a dual-backbone network.

4. The ultrasonic section positioning method for echocardiography according to claim 1, characterized in that: The rotation predictor is a multi-layer cascaded fully connected network with an activation function.

5. The ultrasonic section positioning method for echocardiography according to claim 1, characterized in that: The loss functions of the ultrasound probe position rotation model and the ultrasound section positioning model are both mean square error function MSE.

6. An ultrasonic section positioning device for echocardiography, characterized in that: The ultrasonic section positioning device for echocardiography comprises: The ultrasonic probe position rotation model training module is used to train the ultrasonic probe position rotation model using an image pair consisting of any two frames of images in the same ultrasonic cardiography video with probe posture information; the ultrasonic probe position rotation model includes: a feature extractor and a rotation predictor; the feature extractor is used to extract the posture feature vectors of the image pair respectively; the rotation predictor is used to predict the relative rotation of the ultrasonic probe position according to the posture feature vectors of the image pair; An ultrasound section positioning model training module is used to train an ultrasound section positioning model using an ultrasound cardiogram video data set containing a target section; the ultrasound section positioning model is constructed based on a feature extractor in a trained ultrasound probe position rotation model; The ultrasonic section plane positioning module is used to perform ultrasonic section plane positioning according to the trained ultrasonic section plane positioning model.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the ultrasonic section positioning method for echocardiography described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for ultrasonic section positioning for echocardiography described in any one of claims 1 to 5 is implemented.

Images