Method, apparatus and device for generating echocardiogram based on left ventricular volume curve

Abstract

The present invention provides a method, apparatus, and device for generating an echocardiogram based on a left ventricular volume curve, which can be applied to the technical field of medical data generation. The method for generating an echocardiogram based on a left ventricular volume curve includes: obtaining an initial echocardiogram image and a left ventricular volume-time curve; based on the left ventricular volume-time curve, using an optical flow encoding synthesis model to perform feature distortion mapping processing on the initial echocardiogram image to obtain multiple frames of target echocardiogram images, wherein the optical flow encoding synthesis model is trained from an initial model using multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video according to sample labels corresponding to the multiple frames of sample echocardiogram images, the multiple frames of target echocardiogram images indicate the temporal variation characteristics of multiple cardiac functions within a predetermined time period, and the multiple frames of target echocardiogram images are used to generate a target echocardiogram video.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data generation, and more particularly to a method, apparatus, and device for generating an echocardiogram based on a left ventricular volume curve. Background Art

[0002] An echocardiogram video is one of the current important biomedical images. Relevant professionals can use the echocardiogram video as intermediate information to perform targeted diagnoses on the target. Therefore, relevant professionals need a large amount of clinical echocardiogram video learning data as training data for simulation training to improve relevant professional knowledge and capabilities.

[0003] In the process of implementing the above inventive concept, it is found that the echocardiogram training videos generated by the existing technology cannot contain comprehensive and accurate cardiac activity characteristics, resulting in the inability to train based on the generated echocardiogram training videos and a large error in the simulation training process, affecting clinical judgment in practice and having poor applicability. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method, apparatus, and device for generating an echocardiogram based on a left ventricular volume curve.

[0005] According to a first aspect of the present invention, there is provided a method for generating an echocardiogram based on a left ventricular volume curve, including: obtaining an initial echocardiogram image and a left ventricular volume-time curve, where the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined period; based on the left ventricular volume-time curve, using an optical flow encoding synthesis model to perform feature distortion mapping processing on the initial echocardiogram image to obtain multiple frames of target echocardiogram images, where the optical flow encoding synthesis model is obtained by training an initial model using multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video according to sample labels corresponding to the multiple frames of sample echocardiogram images, the multiple frames of target echocardiogram images indicate the temporal change characteristics of multiple cardiac functions within a predetermined period, and the multiple frames of target echocardiogram images are used to generate a target echocardiogram video.

[0006] According to an embodiment of the present invention, the optical flow encoding synthesis model includes an image encoder, an optical flow generation sub-module, and an image decoder. Based on the left ventricular volume-time curve, the initial echocardiogram image is subjected to feature distortion mapping processing using the optical flow encoding synthesis model to obtain multiple frames of target echocardiogram images, including: obtaining a Gaussian noise matrix for generating the target echocardiogram images; inputting the initial echocardiogram image into the image encoder of the optical flow encoding synthesis model for feature extraction processing to obtain initial cardiac spatio-temporal features corresponding to the heart; inputting the Gaussian noise matrix, the left ventricular volume-time curve, and the initial cardiac spatio-temporal features into the optical flow generation sub-module of the optical flow encoding synthesis model. Based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-model performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatio-temporal features; inputting the multiple distorted cardiac spatio-temporal features into the image decoder of the optical flow encoding synthesis model for feature decoding mapping processing to obtain multiple frames of target echocardiogram images.

[0007] According to an embodiment of the present invention, based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-model performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatio-temporal features, including: based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-module performs denoising processing on the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve, where each target optical flow in the target optical flow sequence represents the displacement change feature between the pixel points in the initial echocardiogram image and the pixel points in each frame of the target echocardiogram image, and each target occlusion in the target occlusion sequence represents the region change feature of the initial echocardiogram image occluded by each frame of the target echocardiogram image; using the target optical flow sequence and the target occlusion sequence to perform distortion processing on the initial cardiac spatio-temporal features to obtain multiple distorted cardiac spatio-temporal features.

[0008] According to an embodiment of the present invention, obtaining the left ventricular volume-time curve includes: obtaining an echocardiogram video, where the echocardiogram video includes multiple frames of echocardiogram single-modal images; using a left ventricular depth segmentation model to perform depth estimation processing on the echocardiogram video to obtain relative depth estimation values corresponding to each pixel point located in the left ventricular region in each frame of the echocardiogram single-modal image, where the relative depth estimation value represents the depth of each pixel point in the left ventricular region; generating a left ventricular volume-time curve according to the multiple relative depth estimation values in each frame of the echocardiogram single-modal image.

[0009] According to an embodiment of the present invention, the optical flow encoding synthesis model is trained in the following manner, including: obtaining an initial model and a training sample data set, where the initial model includes an initial optical flow auto-encoding sub-module and an optical flow generation sub-module to be trained, the initial optical flow auto-encoding sub-module includes a pre-trained image encoder, a pre-trained optical flow prediction sub-module, and a pre-trained image decoder, and the training sample data set includes sample echocardiogram videos, sample left ventricular volume-time curves, and sample Gaussian noises; obtaining multiple frames of sample echocardiogram images from the sample echocardiogram videos based on a predetermined frame sequence; respectively inputting the multiple frames of sample echocardiogram images into the pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatio-temporal features; respectively inputting the multiple sample cardiac spatio-temporal features into the pre-trained optical flow prediction sub-module for optical flow prediction processing to obtain sample labels, where the sample labels include a sample optical flow sequence and a sample occlusion sequence; inputting the sample Gaussian noise, the sample optical flow sequence, and the sample occlusion sequence into the optical flow generation sub-module to be trained for feature distortion processing to obtain sample optical flow occlusion sequence noise features; obtaining an optical flow occlusion sequence noise loss value based on an optical flow occlusion sequence noise loss function according to the sample left ventricular volume-time curve, the sample optical flow occlusion sequence noise features, and the sample Gaussian noise; adjusting the module parameters of the optical flow generation sub-module to be trained according to the optical flow occlusion sequence noise loss value to obtain a trained optical flow generation sub-module; and obtaining a trained optical flow encoding synthesis model according to the pre-trained image encoder, the trained optical flow generation sub-module, and the pre-trained image decoder.

[0010] According to an embodiment of the present invention, inputting the sample Gaussian noise, the sample optical flow sequence, and the sample occlusion sequence into the optical flow generation sub-module to be trained for feature distortion processing to obtain sample optical flow occlusion sequence noise features includes: performing a merging process on the sample optical flow sequence and the sample occlusion sequence to generate a sample optical flow occlusion sequence; adding the sample Gaussian noise to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence; and inputting the optical flow occlusion noise sequence into the optical flow generation sub-module to be trained for feature generation processing to obtain sample optical flow occlusion sequence noise features.

[0011] According to an embodiment of the present invention, based on an optical flow occlusion sequence noise loss function, an optical flow occlusion sequence noise loss value is obtained according to a sample left ventricular volume-time curve, a sample optical flow occlusion sequence noise feature, and a sample Gaussian noise, including: obtaining a plurality of left ventricular volume values from the sample left ventricular volume-time curve; normalizing the plurality of left ventricular volume values and the first left ventricular volume value among the plurality of left ventricular volume values to obtain a plurality of left ventricular volume ratios; generating a left ventricular volume ratio curve within a predetermined time period according to the plurality of left ventricular volume ratios; inputting the left ventricular volume ratio curve into an enhanced embedding sequential model for time feature extraction to obtain left ventricular volume-time features within a predetermined time period; based on the optical flow occlusion sequence noise loss function, obtaining the optical flow occlusion sequence noise loss value according to a sample initial cardiac spatio-temporal feature, the left ventricular volume-time features, the sample optical flow occlusion sequence noise feature, and the sample Gaussian noise, where the sample initial cardiac spatio-temporal feature represents the first sample cardiac spatio-temporal feature among a plurality of sample cardiac spatio-temporal features.

[0012] According to an embodiment of the present invention, the initial optical flow auto-encoder sub-module is trained by the following method, including: obtaining a to-be-trained initial optical flow auto-encoder sub-module and a to-be-trained image discriminator, where the to-be-trained initial optical flow auto-encoder model includes a to-be-trained image encoder, a to-be-trained optical flow prediction sub-module, and a to-be-trained image decoder; obtaining a sample reference echocardiogram image and a sample target echocardiogram image from a sample echocardiogram video; inputting the sample reference echocardiogram image into the to-be-trained image encoder for feature extraction processing to obtain a sample reference cardiac spatio-temporal feature; inputting the sample reference echocardiogram image and the sample target echocardiogram image into the to-be-trained optical flow prediction sub-module for optical flow prediction processing to obtain a sample reference optical flow and a sample reference occlusion, where the sample reference optical flow represents the displacement change feature between pixel points in the sample reference echocardiogram image and pixel points in the sample target echocardiogram image, and the sample reference occlusion represents the region change feature of the sample reference echocardiogram image occluded by the sample target echocardiogram image; using the sample reference optical flow sequence and the sample reference occlusion sequence to distort the sample reference cardiac spatio-temporal feature to obtain a sample distorted cardiac spatio-temporal feature; inputting the sample distorted cardiac spatio-temporal feature into the to-be-trained image decoder for feature decoding mapping processing to obtain a reconstructed echocardiogram image; using the to-be-trained image discriminator to perform image adversarial processing on the reconstructed echocardiogram image to obtain an adversarial echocardiogram image; based on the optical flow auto-encoder loss function, obtaining an optical flow auto-encoder loss value according to the reconstructed echocardiogram image, the sample target echocardiogram image, the adversarial echocardiogram image, and a balance parameter; adjusting the module parameters of the to-be-trained initial optical flow auto-encoder sub-module according to the optical flow auto-encoder loss value to obtain a trained initial optical flow auto-encoder sub-module.

[0013] The second aspect of the present invention provides a device for generating an echocardiogram video based on a left ventricular curve, comprising: an acquisition module for acquiring an initial echocardiogram image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined time period; a feature distortion mapping module for performing feature distortion mapping processing on the initial echocardiogram image based on the left ventricular volume-time curve by using an optical flow encoding synthesis model to obtain multiple frames of target echocardiogram images, wherein the optical flow encoding synthesis model is obtained by training an initial model by using multiple frames of echocardiogram images, a sample left ventricular volume-time curve and sample Gaussian noise in a sample echocardiogram video according to the sample labels corresponding to the multiple frames of echocardiogram images, the multiple frames of target echocardiogram images indicate the temporal change characteristics of multiple cardiac functions within a predetermined time period, and the multiple frames of target echocardiogram images are used to generate a target echocardiogram video.

[0014] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above method.

[0015] The fourth aspect of the present invention further provides a computer-readable storage medium having executable instructions stored thereon, and when the instructions are executed by a processor, the processor is caused to execute the above method.

[0016] The fifth aspect of the present invention further provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0017] The method, apparatus, and device for generating an echocardiogram based on a left ventricular volume curve according to the present invention obtain an initial echocardiogram image and a left ventricular volume-time curve, and based on the left ventricular volume-time curve, use an optical flow encoding synthesis model to perform feature distortion mapping processing on the initial echocardiogram image to obtain multiple frames of target echocardiogram images, so as to perform synthesis processing on the multiple frames of target echocardiogram images to obtain a target echocardiogram video. It realizes obtaining a high-fine-grained target echocardiogram video containing important cardiac semantic information such as heart rate and filling velocity within a predetermined time period based on the left ventricular volume-time curve containing various cardiac function characteristics, which is convenient for helping relevant professionals to conduct a large number of simulation trainings through a large number of accurate and more comprehensive target echocardiogram videos containing cardiac activity characteristics. Further, during the process of using the optical flow encoding synthesis model to generate multiple frames of target echocardiogram images, the generated target echocardiogram images can also be adjusted, or cardiac-related activity characteristics can be added to obtain a target echocardiogram video with rich and real cardiac activity characteristics, improving the complexity of the training environment and training process for relevant professionals, thereby helping to improve the accuracy of the clinical cardiac function assessment of the actually collected echocardiogram images and echocardiogram videos of patient objects by relevant professionals.

[0018] According to an embodiment of the present invention, further, since multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video are used, and based on the sample labels corresponding to the multiple frames of sample echocardiogram images, a large number of model trainings are performed on an initial model to obtain an optical flow encoding synthesis model, so as to use the trained optical flow encoding synthesis model to process the initial echocardiogram image and the left ventricular volume-time curve, improving the efficiency of generating the target echocardiogram images and the target echocardiogram video and the training efficiency, reducing the training learning cost, and improving the practicability and applicability of the optical flow encoding synthesis model in a clinical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Through the following description of the embodiments of the present invention with reference to the drawings, the above content and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0020] Figure 1 An application scenario diagram of the method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown;

[0021] Figure 2 A flowchart of the method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown;

[0022] Figure 3 A schematic diagram of using an optical flow encoding synthesis model to generate target echocardiogram images according to an embodiment of the present invention is shown;

[0023] Figure 4 Shows a schematic diagram of the working process of the present invention according to an embodiment of the present invention;

[0024] Figure 5 Shows a schematic diagram of the effect comparison between the target echocardiogram video and the real video generated by using the method of the present invention according to an embodiment of the present invention;

[0025] Figure 6a Shows a schematic diagram of the waveform correlation of the left ventricular volume-time curve between the target echocardiogram video and the real video generated by using the method of the present invention for 1277 echocardiogram videos at different times according to an embodiment of the present invention;

[0026] Figure 6b Shows a schematic diagram of the mean square error of the left ventricular volume-time curve between the target echocardiogram video and the real video generated by using the method of the present invention for 1277 echocardiogram videos at different times according to an embodiment of the present invention;

[0027] Figure 6c Shows a schematic diagram of the waveform correlation of the left ventricular volume-time curve between the target echocardiogram video and the real video generated by using the method of the present invention for 1277 echocardiogram videos at different frame rates according to an embodiment of the present invention;

[0028] Figure 6d Shows a schematic diagram of the mean square error of the left ventricular volume-time curve between the target echocardiogram video and the real video generated by using the method of the present invention for 1277 echocardiogram videos at different frame rates according to an embodiment of the present invention;

[0029] Figure 7 Shows a structural block diagram of a device for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention;

[0030] Figure 8 Shows a block diagram of an electronic device for a method of generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention. Detailed implementation manners

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, many specific details are set forth to provide a thorough understanding of the embodiments of the present invention. However, obviously, one or more embodiments can be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present invention.

[0032] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0034] In cases where expressions similar to "at least one of A, B, and C, etc." are used, generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include, but not be limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0035] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties. Moreover, the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure, and application, all comply with relevant laws, regulations, and standards, adopt necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or reject.

[0036] Echocardiographic video is widely used in medical examinations as a non-invasive, safe, and effective method to help provide targeted diagnoses as intermediate results. By observing the propagation of ultrasonic waves in the human body, echocardiographic video can display the motion state of the heart, thereby providing video image information that can assist in the diagnosis of diseases. Since the interpretation of echocardiographic video often highly depends on the professional knowledge of relevant professionals, relevant professionals need to use a large number of echocardiographic videos as training data for simulation training from time to time to improve their professional capabilities.

[0037] However, the existing echocardiogram videos for simulation training are usually generated only based on the echocardiogram images generated by the left ventricular ejection fraction, resulting in the generated echocardiogram videos being unable to contain comprehensive cardiac activity characteristics. For example, important characteristics such as heart rate, heart rate variability, and ventricular filling velocity during diastole are not included. During the R & D process, it was found that the echocardiogram training videos generated using existing technologies cannot contain comprehensive and accurate cardiac activity characteristics, resulting in the inability to train based on the generated echocardiogram training videos and a large error during the simulation training process, affecting clinical judgment in practice and having poor applicability.

[0038] In view of this, embodiments of the present invention provide a method for generating an echocardiogram based on a left ventricular volume curve, including: obtaining an initial echocardiogram image and a left ventricular volume-time curve, where the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined period; based on the left ventricular volume-time curve, using an optical flow encoding synthesis model to perform feature distortion mapping processing on the initial echocardiogram image to obtain multiple frames of target echocardiogram images. The optical flow encoding synthesis model is trained from an initial model using multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video according to the sample labels corresponding to the multiple frames of sample echocardiogram images. The multiple frames of target echocardiogram images indicate the sequential change characteristics of multiple cardiac functions of the heart within a predetermined period, and the multiple frames of target echocardiogram images are used to generate a target echocardiogram video.

[0039] Figure 1 Fig. shows an application scenario diagram of the method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention.

[0040] As Figure 1 shown, the application scenario according to this embodiment may include a first terminal device 101, a user 102, and a second terminal device 103. The first terminal device 101 is used to send a request signal to the second terminal device 103, and the second terminal device 103 generates training data according to the received request signal.

[0041] The user 102 can use the first terminal device 101 to interact with the second terminal device 103 to receive or send signals, etc.

[0042] The second terminal device 103 may be a processor that receives the request signal, for example, receiving and processing the request signal sent by the first terminal device 101 (only as an example). The second terminal device 103 can analyze and process data such as the received request signal, and feedback the processing result (such as obtaining or generating relevant training data according to the user request, etc.) to the first terminal device 101.

[0043] It should be noted that the method for generating an echocardiogram based on the left ventricular volume curve provided by the embodiments of the present invention can generally be executed by the second terminal device 103. Correspondingly, the apparatus for generating an echocardiogram based on the left ventricular volume curve provided by the embodiments of the present invention can generally be disposed in the second terminal device 103. The method for generating an echocardiogram based on the left ventricular volume curve provided by the embodiments of the present invention can also be executed by a receiver or a cluster of receivers different from the second terminal device 103 and capable of communicating with the first terminal device 101 and / or the second terminal device 103. Correspondingly, the apparatus for generating an echocardiogram based on the left ventricular volume curve provided by the embodiments of the present invention can also be disposed in a receiver or a cluster of receivers different from the second terminal device 103 and capable of communicating with the first terminal device 101 and / or the second terminal device 103.

[0044] It should be understood that Figure 1 the numbers of the first terminal device, the user, and the second terminal device in

[0045] are merely illustrative. According to the implementation requirements, there can be any number of first terminal devices, users, and second terminal devices. Figure 1 Based on the scenario described below Figures 2 to 6d the method for generating an echocardiogram based on the left ventricular volume curve according to the embodiments of the invention will be described in detail through

[0046] Figure 2 FIG. shows a flowchart of a method for generating an echocardiogram based on the left ventricular volume curve according to an embodiment of the present invention.

[0047] As Figure 2 shown, the method for generating an echocardiogram based on the left ventricular volume curve of this embodiment includes operations S210~S220.

[0048] In operation S210, an initial echocardiogram image and a left ventricular volume-time curve are obtained.

[0049] According to an embodiment of the present invention, the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined time period.

[0050] According to an embodiment of the present invention, in the case where simulation training is required, the initial echocardiogram image and the left ventricular volume-time curve can be obtained from a database, and the initial echocardiogram image is used as a conditional image, that is, an initial template for generating a target echocardiogram video, and then based on the change characteristics of the left ventricle within the predetermined time period in the left ventricular volume-time curve, for example, cardiac activity characteristics such as the ventricular filling speed, heart rate, and heart rate variability during diastole within the predetermined time period of the heart, so as to generate an accurate target echocardiogram video with high fine-grainedness that includes relatively comprehensive cardiac activity characteristics.

[0051] In operation S220, based on the left ventricular volume-time curve, the initial echocardiogram image is processed by feature warping mapping using an optical flow encoding synthesis model to obtain multiple frames of target echocardiogram images.

[0052] According to an embodiment of the present invention, the optical flow encoding synthesis model is obtained by training an initial model using multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video, according to sample labels corresponding to the multiple frames of sample echocardiogram images.

[0053] According to an embodiment of the present invention, the multiple frames of target echocardiogram images can be used to generate a target echocardiogram video, that is, the multiple frames of target echocardiogram images, i.e., the target video frame sequence, can be synthesized to obtain a target echocardiogram video containing multiple cardiac function semantic information.

[0054] According to an embodiment of the present invention, the multiple frames of target echocardiogram images indicate the temporal variation characteristics of multiple cardiac functions of the heart within a predetermined period.

[0055] According to an embodiment of the present invention, by processing the initial echocardiogram image with an optical flow encoding synthesis model for feature warping mapping, a target video frame sequence, i.e., multiple frames of target echocardiogram images, is obtained. The multiple frames of target echocardiogram images include multiple consecutive cardiac function-related change characteristics within a predetermined period. For example, continuous cardiac activity change characteristics such as ventricular filling velocity, heart rate, and heart rate variability during diastole within a predetermined period of the heart.

[0056] According to an embodiment of the present invention, based on the obtained target echocardiogram video, it can help relevant personnel conduct simulation training of professional knowledge, thereby helping to judge the related functions of the heart.

[0057] For example, in a case where simulation training is required, an initial echocardiogram image and a left ventricular volume-time curve that can be used to generate a target echocardiogram video are obtained. The initial echocardiogram image and the left ventricular volume-time curve are input into the optical flow encoding synthesis model. The optical flow encoding synthesis model processes the initial echocardiogram image by feature warping mapping based on the left ventricular volume-time curve to obtain a target video frame sequence, i.e., multiple frames of target echocardiogram images, so as to synthesize the multiple frames of target echocardiogram images to obtain a target echocardiogram video for training, enabling scientific personnel or professionals in related fields to conduct simulation training based on the target echocardiogram video.

[0058] According to an embodiment of the present invention, by obtaining an initial echocardiogram image and a left ventricular volume-time curve, based on the left ventricular volume-time curve, using an optical flow encoding synthesis model to perform feature distortion mapping processing on the initial echocardiogram image, multiple frames of target echocardiogram images are obtained, so as to facilitate the synthesis processing of the multiple frames of target echocardiogram images to obtain a target echocardiogram video. Based on the left ventricular volume-time curve containing various cardiac function features, a high-fine-grained target echocardiogram video containing important cardiac semantic information such as heart rate and filling speed within a predetermined period is obtained, which is convenient for helping relevant professionals to conduct a large number of simulation trainings through a large number of accurate target echocardiogram videos containing more comprehensive cardiac activity features. Further, during the process of generating multiple frames of target echocardiogram images using the optical flow encoding synthesis model, the generated target echocardiogram images can also be adjusted, or cardiac-related activity features can be added to obtain a target echocardiogram video with rich and real cardiac activity features, improving the complexity of the training environment and training process for relevant professionals, thereby helping to improve the accuracy of the clinical cardiac function assessment of the actually collected echocardiogram images and echocardiogram videos of patient objects by relevant professionals.

[0059] According to an embodiment of the present invention, further, since multiple frames of sample echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video are used, and based on the sample labels corresponding to the multiple frames of sample echocardiogram images, a large number of model trainings are performed on an initial model to obtain an optical flow encoding synthesis model, so as to facilitate the use of the trained optical flow encoding synthesis model to process the initial echocardiogram image and the left ventricular volume-time curve, improving the efficiency of generating the target echocardiogram images and the target echocardiogram video as well as the training efficiency, reducing the training learning cost, and improving the practicality and applicability of the optical flow encoding synthesis model in a clinical environment.

[0060] It should be noted that the target echocardiogram images and the target echocardiogram video obtained by the present invention are only used as an intermediate result, and a diagnostic result or a health condition cannot be directly obtained from the target echocardiogram images and the target echocardiogram video obtained by the method according to the present invention.

[0061] According to an embodiment of the present invention, the method for obtaining a left ventricular volume-time curve includes the following operations.

[0062] According to an embodiment of the present invention, an initial echocardiogram video is obtained, and the initial echocardiogram video includes multiple frames of echocardiogram unimodal images.

[0063] According to an embodiment of the present invention, by constructing a left ventricular depth segmentation label based on the Simpson's method, an automated left ventricular depth segmentation model is trained and optimized, so as to use the left ventricular depth segmentation model to segment a historical initial echocardiogram video, and obtain multiple left ventricular volume-time curves.

[0064] According to an embodiment of the present invention, the left ventricular depth segmentation model is used to perform depth estimation processing on the initial echocardiogram video, and relative depth estimation values corresponding to each pixel point located in the left ventricular region in each frame of echocardiogram single-modal image are obtained.

[0065] According to an embodiment of the present invention, the relative depth estimation value represents the depth of each pixel point in the left ventricular region.

[0066] According to an embodiment of the present invention, the left ventricular depth segmentation model is used to segment each frame of echocardiogram image in the obtained initial echocardiogram video, and the depth value corresponding to each pixel point in each frame of initial echocardiogram image is obtained. Whether the pixel point is located in the left ventricular region can be judged according to the depth value corresponding to each pixel point, so as to obtain the relative depth estimation value corresponding to each pixel point located in the left ventricular region.

[0067] According to an embodiment of the present invention, a left ventricular volume-time curve is generated according to the multiple relative depth estimation values in each frame of echocardiogram single-modal image.

[0068] According to an embodiment of the present invention, the relative depth estimation values of each pixel point in each frame of initial echocardiogram image are summed, and the candidate volume of the left ventricle in each frame of initial echocardiogram image is calculated. A left ventricular volume-time curve within a predetermined time period is generated according to the multiple candidate volumes of multiple frames of initial echocardiogram images. The volume curve graph includes multiple minimum points, multiple maximum points and other numerical points. Among them, the minimum points and maximum points can correspond to the states of target contraction or target relaxation, for example, the motion states of cardiac contraction or cardiac relaxation. According to the entire left ventricular volume-time curve, important cardiac activity characteristics such as the ventricular filling speed during diastole can also be observed.

[0069] According to an embodiment of the present invention, the left ventricular depth segmentation model is trained by using an initial echocardiogram sample video, and based on a sample diastolic image with segmentation labels, a sample systolic image with segmentation labels, and a sample echocardiogram image without segmentation labels, to obtain an intermediate segmentation model with relative depth estimation values for each pixel point. Then, using the sample left ventricular volume-time curve as a label, the volume-time loss value is calculated for the training left ventricular volume-time curve generated from the relative depth estimation values of the pixel points located in the left ventricle obtained according to the intermediate segmentation model, so as to adjust the model parameters of the intermediate segmentation model again to obtain the left ventricular depth segmentation model.

[0070] According to an embodiment of the present invention, by acquiring an initial echocardiogram video, performing depth estimation processing on the initial echocardiogram video using the left ventricular depth segmentation model, obtaining relative depth estimation values corresponding to each pixel point located in the left ventricular region in each frame of the echocardiogram unimodal image, and generating a left ventricular volume-time curve based on the multiple relative depth estimation values in each frame of the echocardiogram unimodal image, it realizes the extraction and generation of the left ventricular volume-time curve for multiple historical initial echocardiogram videos using the pre-trained left ventricular depth segmentation model. While obtaining an accurate left ventricular volume-time curve, this curve can be used as training data to train the optical flow encoding synthesis model, thereby improving the ability and accuracy of the optical flow encoding synthesis model to generate a target echocardiogram video based on the left ventricular volume-time curve.

[0071] According to an embodiment of the present invention, the optical flow encoding synthesis model includes an image encoder, an optical flow generation sub-module, and an image decoder.

[0072] According to an embodiment of the present invention, the method for performing feature distortion mapping processing on an initial echocardiogram image using the optical flow encoding synthesis model based on the left ventricular volume-time curve to obtain multiple frames of target echocardiogram images includes the following operations.

[0073] According to an embodiment of the present invention, a Gaussian noise matrix for generating a target echocardiogram image is acquired.

[0074] According to an embodiment of the present invention, a set of Gaussian noise matrices can be randomly generated, or a random set of Gaussian noise matrices can be obtained from an echocardiogram noise library.

[0075] According to an embodiment of the present invention, the initial echocardiogram image is input into the image encoder of the optical flow encoding synthesis model for feature extraction processing to obtain initial cardiac spatio-temporal features corresponding to the heart.

[0076] According to an embodiment of the present invention, a Gaussian noise matrix, a left ventricular volume-time curve, and initial cardiac spatio-temporal features are input into the optical flow generation sub-module of the optical flow encoding synthesis model. Based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-module performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatio-temporal features.

[0077] According to an embodiment of the present invention, the distorted cardiac spatio-temporal features can be characterized as the spatio-temporal features of echocardiographic images that are continuous with the initial echocardiographic images within a predetermined time period and are generated based on the left ventricular volume-time curve and the initial cardiac spatio-temporal features.

[0078] According to an embodiment of the present invention, a pre-trained optical flow generation sub-module is used to perform denoising processing on a randomly generated Gaussian noise matrix, and at the same time, the left ventricular volume-time curve and the initial cardiac spatio-temporal features are combined, so as to restore the distorted cardiac spatio-temporal features that neither contain noise nor contain the cardiac activity features in the initial cardiac spatio-temporal features and the left ventricular volume-time curve.

[0079] According to an embodiment of the present invention, multiple distorted cardiac spatio-temporal features are input into the image decoder of the optical flow encoding synthesis model for feature decoding and mapping processing to obtain multiple frames of target echocardiographic images.

[0080] According to an embodiment of the present invention, the optical flow encoding synthesis model includes an image encoder, an optical flow generation sub-module, and an image decoder. By using the self-built optical flow encoding synthesis model to process the features of the Gaussian noise matrix and the initial echocardiographic images, the initial spatio-temporal features of the heart are obtained from the initial echocardiographic images, and then the spatio-temporal features of the subsequent echocardiographic images in the target echocardiographic video to be generated are predicted based on the initial spatio-temporal features of the heart and the previously obtained left ventricular volume-time curve, that is, the distorted cardiac spatio-temporal features. The image decoder is used to decode and restore multiple distorted cardiac spatio-temporal features to restore multiple frames of target echocardiographic images, realizing the accurate estimation of the features of subsequent images by using the optical flow generation sub-module. Furthermore, an accurate target echocardiographic image containing comprehensive cardiac activity features can be generated by using the highly efficient self-built optical flow encoding synthesis model, and the efficiency of generating the target echocardiographic image is high, which can reduce the cost of generating training data and improve the practicability and applicability of the optical flow encoding synthesis model in the clinical environment.

[0081] According to an embodiment of the present invention, based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the method for the optical flow generation sub-module to perform feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatio-temporal features includes the following operations.

[0082] According to an embodiment of the present invention, based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-module denoises the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve.

[0083] According to an embodiment of the present invention, each target optical flow in the target optical flow sequence characterizes the displacement change feature between the pixel points in the initial echocardiogram image and the pixel points in each frame of the target echocardiogram image, and each target occlusion in the target occlusion sequence characterizes the region change feature of the initial echocardiogram image occluded by each frame of the target echocardiogram image.

[0084] According to an embodiment of the present invention, the displacement change feature of the target optical flow sequence includes two channels, that is, the displacement change feature of the pixel points under the horizontal movement between two frames of echocardiogram images and the displacement change feature under the vertical movement, and the target occlusion sequence identifies the occlusion region through a single channel, that is, the region change feature of the previous frame image occluded by the subsequent frame image.

[0085] According to an embodiment of the present invention, the initial cardiac spatio-temporal features are distorted by using the target optical flow sequence and the target occlusion sequence to obtain multiple distorted cardiac spatio-temporal features.

[0086] According to an embodiment of the present invention, by using the optical flow generation sub-module to denoise the Gaussian noise matrix based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve to obtain an estimated accurate target optical flow sequence and target occlusion sequence, and using each target optical flow and each target occlusion in the target optical flow sequence and the target occlusion sequence to distort the initial cardiac spatio-temporal features, thereby obtaining multiple distorted cardiac spatio-temporal features corresponding to the target echocardiogram images to be generated, it realizes the prediction of the spatio-temporal features of the subsequent images from the optical flow dimension of the vertical and horizontal movements of the pixel points and the occlusion dimension of the overlap between the pixel points, so as to use the accurate target optical flow sequence and target occlusion sequence to distort the initial cardiac spatio-temporal features and obtain accurate distorted cardiac spatio-temporal features that can correspond to the cardiac activity features in the left ventricular volume-time curve.

[0087] According to an embodiment of the present invention, the optical flow encoding synthesis model is trained by the following method.

[0088] According to an embodiment of the present invention, an initial model and a training sample data set are obtained.

[0089] According to an embodiment of the present invention, the initial model includes an initial optical flow auto-encoding sub-module and an optical flow generation sub-module to be trained. The initial optical flow auto-encoding sub-module includes a pre-trained image encoder, a pre-trained optical flow prediction sub-module, and a pre-trained image decoder. The training sample data set includes sample echocardiogram videos, sample left ventricular volume-time curves, and sample Gaussian noise.

[0090] According to an embodiment of the present invention, the initial optical flow auto-encoding sub-module is trained by the following method.

[0091] According to an embodiment of the present invention, the initial optical flow auto-encoding sub-module to be trained and an image discriminator to be trained are obtained.

[0092] According to an embodiment of the present invention, the initial optical flow auto-encoding model to be trained includes an image encoder to be trained, an optical flow prediction sub-module to be trained, and an image decoder to be trained.

[0093] According to an embodiment of the present invention, a sample reference echocardiogram image and a sample target echocardiogram image are obtained from the sample echocardiogram video.

[0094] According to an embodiment of the present invention, the sample reference echocardiogram image and the sample target echocardiogram image are two randomly selected sample echocardiogram images from the sample ultrasound video.

[0095] According to an embodiment of the present invention, the sample reference echocardiogram image is input into the image encoder to be trained for feature extraction processing to obtain sample reference cardiac spatio-temporal features.

[0096] According to an embodiment of the present invention, the image encoder to be trained encodes the sample reference echocardiogram image into a sample reference cardiac feature map containing the sample reference cardiac spatio-temporal features.

[0097] According to an embodiment of the present invention, the sample reference echocardiogram image and the sample target echocardiogram image are input into the optical flow prediction sub-module to be trained for optical flow prediction processing to obtain a sample reference optical flow and a sample reference occlusion.

[0098] According to an embodiment of the present invention, the sample reference optical flow represents the displacement change feature between the pixel points in the sample reference echocardiogram image and the pixel points in the sample target echocardiogram image, and the sample reference occlusion represents the region change feature of the sample reference echocardiogram image occluded by the sample target echocardiogram image.

[0099] According to an embodiment of the present invention, the optical flow prediction sub-module to be trained can be used to extract the potential optical flow and occlusion map estimation between two frames of sample echocardiogram images, that is, the sample reference optical flow and the sample reference occlusion.

[0100] According to an embodiment of the present invention, the sample reference cardiac spatio-temporal features are distorted using a sample reference optical flow sequence and a sample reference occlusion sequence to obtain sample distorted cardiac spatio-temporal features.

[0101] According to an embodiment of the present invention, the sample distorted cardiac spatio-temporal features can be calculated according to formula (1), and formula (1) is shown as follows.

[0102] (1);

[0103] Where z t can represent the sample distorted cardiac spatio-temporal features, m can represent the sample reference occlusion, z r can represent the sample reference cardiac spatio-temporal features, f can represent the sample reference optical flow, Warp() can represent a convolution function, can represent a dot product calculation operation.

[0104] According to an embodiment of the present invention, the sample distorted cardiac spatio-temporal features are input into a to-be-trained image decoder for feature decoding mapping processing to obtain a reconstructed echocardiogram image.

[0105] According to an embodiment of the present invention, an image decoding loss value is calculated based on the reconstructed echocardiogram image and the sample target echocardiogram image, and the image decoding loss value can be calculated according to formula (2), and formula (2) is shown as follows.

[0106] (2);

[0107] Where can represent the image decoding loss value, x t can represent the sample target echocardiogram image, G(z t ) can represent the reconstructed echocardiogram image, can represent an expected calculation operation.

[0108] According to an embodiment of the present invention, the reconstructed echocardiogram image is subjected to image adversarial processing using a to-be-trained image discriminator to obtain an adversarial echocardiogram image.

[0109] According to an embodiment of the present invention, the to-be-trained image discriminator can be a discriminator in the form of a PatchGAN (Patch Generative Adversarial Network), and the adversarial processing effect is achieved by classifying patches in the image to obtain an adversarial echocardiogram image.

[0110] According to an embodiment of the present invention, an adversarial loss value is calculated based on the adversarial echocardiogram image, and the adversarial loss value can be calculated according to formula (3), and formula (3) is shown as follows.

[0111] (3);

[0112] Among them, can be characterized as an adversarial loss value, can be characterized as an image obtained by performing image adversarial processing on a sample target echocardiogram image using a to-be-trained image discriminator, can be characterized as an adversarial echocardiogram image, x r can be characterized as a sample reference echocardiogram image.

[0113] According to an embodiment of the present invention, based on an optical flow auto-encoding loss function, an optical flow auto-encoding loss value is obtained according to a reconstructed echocardiogram image, a sample target echocardiogram image, an adversarial echocardiogram image, and a balance parameter.

[0114] According to an embodiment of the present invention, an optical flow auto-encoding loss value is calculated according to an image decoding loss value calculated based on a reconstructed echocardiogram image and a sample target echocardiogram image and an adversarial loss value calculated based on an adversarial echocardiogram image. The optical flow auto-encoding loss value can be calculated according to formula (4), and formula (4) is as follows.

[0115] (4);

[0116] Among them, can be characterized as an optical flow auto-encoding loss value, can be characterized as a balance parameter.

[0117] According to an embodiment of the present invention, according to the optical flow auto-encoding loss value, the module parameters of an initial optical flow auto-encoding sub-module to be trained are adjusted to obtain a trained initial optical flow auto-encoding sub-module.

[0118] According to an embodiment of the present invention, the trained optical flow auto-encoding sub-module includes a trained image encoder , a trained optical flow prediction sub-module, and a trained image decoder .

[0119] According to an embodiment of the present invention, an initial optical flow autoencoder sub-module to be trained and an image discriminator to be trained are obtained. Two sample echocardiogram images are randomly selected from a sample echocardiogram video as a sample reference echocardiogram image and a sample target echocardiogram image for training and verification. The sample reference echocardiogram image is input into the image encoder to be trained for feature extraction processing to obtain sample reference cardiac spatio-temporal features, so that the trained image encoder can extract accurate cardiac spatio-temporal features. Then, the sample reference echocardiogram image and the sample target echocardiogram image are input into the optical flow prediction sub-module to be trained for optical flow prediction processing to obtain a sample reference optical flow and a sample reference occlusion. Then, the sample reference cardiac spatio-temporal features are distorted using the predicted sample reference optical flow and sample reference occlusion to obtain sample distorted cardiac spatio-temporal features, thereby training the ability of the optical flow prediction sub-module to predict optical flow and occlusion based on cardiac spatio-temporal features. At the same time, in combination with an image decoder, the image decoder is trained for the reconstruction ability to reconstruct an image according to image features. After obtaining the reconstructed echocardiogram image, the introduced image discriminator is used to perform image adversarial processing on the reconstructed echocardiogram image to obtain an adversarial echocardiogram image. Based on the optical flow autoencoder loss function, according to the reconstructed echocardiogram image, the sample target echocardiogram image, the adversarial echocardiogram image, and a balance parameter, an optical flow autoencoder loss value is obtained. According to the optical flow autoencoder loss value, the module parameters of the initial optical flow autoencoder sub-module to be trained are adjusted, and the image discriminator is discarded to obtain the trained initial optical flow autoencoder sub-module, realizing the learning and training of the image encoder, the optical flow prediction sub-module, and the image decoder in the initial optical flow autoencoder sub-module, so that the image encoder, the optical flow prediction sub-module, and the image decoder can process information such as images and features, so as to extract or reconstruct accurate feature or image information. Further, by training the optical flow prediction sub-module, accurate optical flow and occlusion information can be predicted, so as to facilitate the training of the subsequent optical flow generation sub-module. Further, by introducing an image discriminator to perform adversarial processing on the reconstructed echocardiogram image and the sample target echocardiogram image, the image decoder can be trained to reconstruct a reconstructed echocardiogram image that is infinitely close to the real image, improving the quality of the decoded image.

[0120] According to an embodiment of the present invention, based on a predetermined frame sequence, multiple sample echocardiogram images are obtained from a sample echocardiogram video.

[0121] According to an embodiment of the present invention, a sample initial echocardiogram image can be first determined. Taking the sample initial echocardiogram image as the initial image, based on a predetermined frame sequence, multiple consecutive sample echocardiogram images including the sample initial echocardiogram image are obtained.

[0122] According to an embodiment of the present invention, multiple frames of sample echocardiogram images are respectively input into a pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatio-temporal features.

[0123] According to an embodiment of the present invention, the multiple sample cardiac spatio-temporal features are respectively input into a pre-trained optical flow prediction sub-module for optical flow prediction processing to obtain sample labels, where the sample labels include a sample optical flow sequence and a sample occlusion sequence.

[0124] According to an embodiment of the present invention, each sample optical flow in the sample optical flow sequence represents the displacement change feature between the same pixel points in every two adjacent frames of sample echocardiogram images, and each sample occlusion in the sample occlusion sequence represents the area change feature of the previous frame of sample echocardiogram image being occluded by the next frame of sample echocardiogram image in every two adjacent frames of sample echocardiogram images.

[0125] According to an embodiment of the present invention, the sample Gaussian noise, the sample optical flow sequence, and the sample occlusion sequence are input into a to-be-trained optical flow generation sub-module for feature distortion processing to obtain sample optical flow occlusion sequence noise features.

[0126] According to an embodiment of the present invention, the method of inputting the sample Gaussian noise, the sample optical flow sequence, and the sample occlusion sequence into a to-be-trained optical flow generation sub-module for feature distortion processing to obtain sample optical flow occlusion sequence noise features includes the following operations.

[0127] According to an embodiment of the present invention, the sample optical flow sequence and the sample occlusion sequence are combined to generate a sample optical flow occlusion sequence.

[0128] According to an embodiment of the present invention, the sample optical flow sequence and the sample occlusion sequence generated by the optical flow prediction sub-module are combined, and the obtained sample optical flow occlusion sequence is used as training data and sample data for training the optical flow generation sub-module. The sample optical flow occlusion sequence can be calculated according to formula (5), and formula (5) is shown as follows.

[0129] (5);

[0130] where s 0 can be represented as the sample optical flow occlusion sequence, F can be represented as the sample optical flow sequence, M can be represented as the sample occlusion sequence, n can be represented as the length of the predetermined frame sequence, h can be represented as the height of the sample echocardiogram image, w can be represented as the width of the sample echocardiogram image, and Cat[] can be represented as the operation of combining the sample optical flow sequence and the sample occlusion sequence.

[0131] According to an embodiment of the present invention, sample Gaussian noise is added to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence.

[0132] According to an embodiment of the present invention, sample Gaussian noise can be added to the sample optical flow occlusion sequence multiple times to obtain an optical flow occlusion noise sequence including optical flow occlusion noise sequences with different numbers of added noise times.

[0133] According to an embodiment of the present invention, the optical flow occlusion noise sequence is input into the optical flow generation sub-module to be trained for feature generation processing, and sample optical flow occlusion sequence noise features are obtained.

[0134] According to an embodiment of the present invention, by performing a merging process on the sample optical flow sequence and the sample occlusion sequence, a sample optical flow occlusion sequence for training and validating the optical flow generation sub-module is generated. Sample Gaussian noise is added to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence. The optical flow occlusion noise sequence is input into the optical flow generation sub-module to be trained for feature generation processing, and sample optical flow occlusion sequence noise features are obtained. Thus, it is realized to use the optical flow sequence and the occlusion sequence predicted by the trained optical flow prediction sub-module as training data samples to train the optical flow generation sub-module, improve the output accuracy of the optical flow generation sub-module, and at the same time improve the correlation between the optical flow generation sub-module and the image encoder and the image decoder, and improve the robustness of the module.

[0135] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, an optical flow occlusion sequence noise loss value is obtained according to the sample left ventricular volume-time curve, the sample optical flow occlusion sequence noise features, and the sample Gaussian noise.

[0136] According to an embodiment of the present invention, the method for obtaining the optical flow occlusion sequence noise loss value based on the optical flow occlusion sequence noise loss function according to the sample left ventricular volume-time curve, the sample optical flow occlusion sequence noise features, and the sample Gaussian noise includes the following operations.

[0137] According to an embodiment of the present invention, multiple left ventricular volume values are obtained from the sample left ventricular volume-time curve.

[0138] According to an embodiment of the present invention, normalization processing is performed on the multiple left ventricular volume values and the first left ventricular volume value among the multiple left ventricular volume values to obtain multiple left ventricular volume ratios.

[0139] According to an embodiment of the present invention, the left ventricular volume ratio can be calculated according to formula (6), and formula (6) is as follows.

[0140] (6);

[0141] Wherein, can be characterized as the left ventricular volume ratio, can be characterized as a plurality of left ventricular volume values, v 0 can be characterized as the first left ventricular volume value.

[0142] According to an embodiment of the present invention, a left ventricular volume ratio curve within a predetermined time period is generated based on a plurality of left ventricular volume ratios.

[0143] According to an embodiment of the present invention, the left ventricular volume ratio curve is input into an enhanced embedding sequence model for extracting time features, and left ventricular volume time features within a predetermined time period are obtained. , where d can be characterized as the feature dimension.

[0144] According to an embodiment of the present invention, the sample cardiac spatio-temporal features corresponding to the sample initial echocardiogram image are represented as , where c can be characterized as the number of features.

[0145] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, an optical flow occlusion sequence noise loss value is obtained according to the sample initial cardiac spatio-temporal features, left ventricular volume time features, sample optical flow occlusion sequence noise features, and sample Gaussian noise.

[0146] According to an embodiment of the present invention, the sample initial cardiac spatio-temporal features are characterized as the first sample cardiac spatio-temporal features among a plurality of sample cardiac spatio-temporal features, that is, the sample cardiac spatio-temporal features corresponding to the sample initial echocardiogram image.

[0147] According to an embodiment of the present invention, the optical flow occlusion sequence noise loss value can be calculated according to formula (7), and formula (7) is shown as follows.

[0148] (7);

[0149] Where can be characterized as the optical flow occlusion sequence noise loss value, can be characterized as the sample optical flow occlusion sequence noise features predicted and generated by the optical flow generation sub-module to be trained, s t can be characterized as the optical flow occlusion noise sequence obtained by adding the sample Gaussian noise t times to the optical flow occlusion noise sequence, and t can be characterized as t times of sample Gaussian noise, z 0 can be characterized as the sample cardiac spatio-temporal features corresponding to the sample initial echocardiogram image, and q can be characterized as the left ventricular volume time features.

[0150] According to an embodiment of the present invention, by obtaining a plurality of left ventricular volume values from a sample left ventricular volume-time curve, normalizing the plurality of left ventricular volume values and the first left ventricular volume value among the plurality of left ventricular volume values to obtain a plurality of left ventricular volume ratios, so as to eliminate the natural differences in the heart size and volume of different individuals, and then inputting the left ventricular volume ratio curve generated by the plurality of left ventricular volume ratios into an enhanced embedding sequential model for time feature extraction to obtain left ventricular volume-time features within a predetermined time period, further extracting the continuous features of the left ventricular volume ratio curve in time, and then based on the optical flow occlusion sequence noise loss function, obtaining an optical flow occlusion sequence noise loss value according to the sample initial cardiac spatio-temporal features, left ventricular volume-time features, sample optical flow occlusion sequence noise features and sample Gaussian noise, so as to obtain the module loss parameter of the optical flow generation sub-module, so as to adjust the module according to the loss parameter of the module to obtain a highly robust optical flow generation sub-module.

[0151] According to an embodiment of the present invention, according to the optical flow occlusion sequence noise loss value, the module parameters of the optical flow generation sub-module to be trained are adjusted to obtain a trained optical flow generation sub-module.

[0152] According to an embodiment of the present invention, a trained optical flow encoding and synthesis model is obtained according to a pre-trained image encoder, a trained optical flow generation sub-module and a pre-trained image decoder.

[0153] According to an embodiment of the present invention, a sample optical flow sequence and a sample occlusion sequence for training the optical flow generation sub-module are obtained by using a pre-trained initial optical flow auto-encoding sub-module, that is, data samples, and then the optical flow generation sub-module is subjected to denoising generation training by using the data samples to obtain an optical flow encoding and synthesis model that can accurately generate a high-fine-grained target echocardiogram video containing important cardiac semantic information such as heart rate and filling speed, so as to use this model to process the initial echocardiogram image and the left ventricular volume-time curve, improve the efficiency of generating the target echocardiogram image and the target echocardiogram video and the training efficiency, reduce the training learning cost, and improve the practicality and applicability of the optical flow encoding and synthesis model in a clinical environment.

[0154] Figure 3 A schematic diagram showing the generation of a target echocardiogram image using an optical flow encoding and synthesis model according to an embodiment of the present invention is shown.

[0155] As Figure 3 shown, Figure 3It shows the generation of a target echocardiogram image using an optical flow encoding synthesis model. The first image on the left is the initial echocardiogram image. The initial echocardiogram image and the left ventricular volume-time curve are input into the optical flow encoding synthesis model. From the black-and-white image, multiple frames of target echocardiogram images generated by the optical flow encoding synthesis model can be seen. The colored image shows an optical flow map containing a target optical flow sequence and a target occlusion sequence. The curve chart at the bottom represents the left ventricular volume-time curve. The optical flow encoding synthesis model generates an optical flow map containing a target optical flow sequence and a target occlusion sequence based on the left ventricular volume-time curve and the initial echocardiogram image, and performs distortion and decoding processing on the image features of the initial echocardiogram image based on the optical flow map containing the target optical flow sequence and the target occlusion sequence to obtain multiple frames of target echocardiogram images.

[0156] Figure 4 It shows a schematic diagram of the working process of the present invention according to an embodiment of the present invention.

[0157] As Figure 4 shown, Figure 4 It shows the process schematic of the present invention. The initial echocardiogram image xr is input into the image encoder to obtain the initial cardiac spatio-temporal features. The initial cardiac spatio-temporal features, the Gaussian noise matrix and the left ventricular volume-time curve are input into the optical flow generation sub-module. I can be represented as the identity matrix. The optical flow generation sub-module predicts and generates a target optical flow sequence and a target occlusion sequence . According to the target optical flow sequence and the target occlusion sequence, feature distortion processing is performed on the initial cardiac spatio-temporal features to obtain multiple distorted cardiac spatio-temporal features, and then the multiple distorted cardiac spatio-temporal features are input into the image decoder G(z t ) to output multiple frames of target echocardiogram images , and a target echocardiogram video is generated based on the multiple frames of target echocardiogram images.

[0158] Figure 5 It shows a schematic diagram of the effect comparison between the target echocardiogram video generated by the method of the present invention and the real video according to an embodiment of the present invention.

[0159] As Figure 5 shown, Figure 5The figure shows the effect comparison between the target echocardiogram video generated by the method of the present invention and the real video. From top to bottom, they are the echogram of the real video, the echogram of the target echocardiogram video, the heat map of the left ventricular region of the real video, the heat map of the left ventricular region of the target echocardiogram video, and the left ventricular volume-time curve graph. By comparing the echogram of the real video and the echogram of the target echocardiogram video, it can be seen that the error between the target echocardiogram video generated by the method of the present invention and the real video is extremely small. By comparing the heat map of the real video and the heat map of the target echocardiogram video, it can be seen that the error between the left ventricular region in the target echocardiogram video generated by the method of the present invention and the left ventricular region in the real video is extremely small. In the left ventricular volume-time curve graph, the blue curve is the real left ventricular volume-time curve, and the orange curve is the left ventricular volume-time curve generated by the left ventricular depth segmentation model of the present invention. It can be seen from this figure that the error between the left ventricular volume-time curve generated by the left ventricular depth segmentation model of the present invention and the real left ventricular volume-time curve is extremely small.

[0160] Figure 6a The figure shows a schematic diagram of the waveform correlation of the left ventricular volume-time curves of the target echocardiogram video and the real video generated by the method of the present invention for 1277 echocardiogram videos according to the embodiments of the present invention at different times. Figure 6b The figure shows a schematic diagram of the mean square error of the left ventricular volume-time curves of the target echocardiogram video and the real video generated by the method of the present invention for 1277 echocardiogram videos according to the embodiments of the present invention at different times. Figure 6c The figure shows a schematic diagram of the waveform correlation of the left ventricular volume-time curves of the target echocardiogram video and the real video generated by the method of the present invention for 1277 echocardiogram videos according to the embodiments of the present invention at different frame rates. Figure 6d The figure shows a schematic diagram of the mean square error of the left ventricular volume-time curves of the target echocardiogram video and the real video generated by the method of the present invention for 1277 echocardiogram videos according to the embodiments of the present invention at different frame rates.

[0161] As Figures 6a to 6d shown, Figures 6a to 6d The figure shows the waveform correlation of the left ventricular volume-time curves of the target echocardiogram video and the real video generated by the method of the present invention at different times and different frame rates. FPS can be characterized as the frame rate of the target echocardiogram video, and Time can be characterized as the duration of the generated target echocardiogram video. Figure 6a and 6c The abscissa in Figure 6b and 6dThe abscissa therein can be characterized as the mean square error, and the ordinate can be characterized as the distribution function. From Figures 6a to 6d It can be seen that the error between the target echocardiogram video generated by the method of the present invention and the left ventricular volume-time curve of the real video is extremely small.

[0162] Figure 7 Fig. shows a structural block diagram of an apparatus for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention.

[0163] As Figure 7 shown, the apparatus for generating an echocardiogram based on a left ventricular volume curve in this embodiment includes: an acquisition module 710 and a feature warping mapping module 720.

[0164] The acquisition module 710 is configured to acquire an initial echocardiogram image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined time period. The acquisition module 710 can be used to perform the operation S210 described above, which will not be elaborated here.

[0165] The feature warping mapping module 720 is configured to perform feature warping mapping processing on the initial echocardiogram image based on the left ventricular volume-time curve by using an optical flow encoding synthesis model to obtain multiple frames of target echocardiogram images. The optical flow encoding synthesis model is trained on an initial model by using multiple frames of echocardiogram images, a sample left ventricular volume-time curve, and sample Gaussian noise in a sample echocardiogram video according to sample labels corresponding to the multiple frames of echocardiogram images. The multiple frames of target echocardiogram images indicate the temporal change characteristics of multiple cardiac functions of the heart within a predetermined time period, and the multiple frames of target echocardiogram images are used to generate a target echocardiogram video. The feature warping mapping module 720 can be used to perform the operation S220 described above, which will not be elaborated here.

[0166] According to an embodiment of the present invention, the feature warping mapping module 720 includes: a first acquisition sub-module, a first extraction sub-module, a first warping sub-module, and a first decoding sub-module.

[0167] The first acquisition sub-module is configured to acquire a Gaussian noise matrix for generating a target echocardiogram image.

[0168] The first extraction sub-module is configured to input the initial echocardiogram image into an image encoder of the optical flow encoding synthesis model for feature extraction processing to obtain an initial cardiac spatio-temporal feature corresponding to the heart.

[0169] The first distortion sub-module is configured to input a Gaussian noise matrix, a left ventricular volume-time curve, and initial cardiac spatio-temporal features into the optical flow generation sub-module of the optical flow encoding and synthesis model. Based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-model performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatio-temporal features.

[0170] The first decoding sub-module is configured to input multiple distorted cardiac spatio-temporal features into the image decoder of the optical flow encoding and synthesis model for feature decoding and mapping processing to obtain multiple frames of target echocardiogram images.

[0171] According to an embodiment of the present invention, the first distortion sub-module includes: a first denoising unit and a first distortion unit.

[0172] The first denoising unit is configured to, based on the initial cardiac spatio-temporal features and the left ventricular volume-time curve, the optical flow generation sub-module performs denoising processing on the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve. Each target optical flow in the target optical flow sequence characterizes the displacement change feature between the pixel points in the initial echocardiogram image and the pixel points in each frame of the target echocardiogram image, and each target occlusion in the target occlusion sequence characterizes the region change feature of the initial echocardiogram image occluded by each frame of the target echocardiogram image.

[0173] The first distortion unit is configured to use the target optical flow sequence and the target occlusion sequence to perform distortion processing on the initial cardiac spatio-temporal features to obtain multiple distorted cardiac spatio-temporal features.

[0174] According to an embodiment of the present invention, the acquisition module 710 includes: a second acquisition sub-module, a first segmentation sub-module, and a first generation sub-module.

[0175] The second acquisition sub-module is configured to acquire an initial echocardiogram video, where the initial echocardiogram video includes multiple frames of echocardiogram single-modal images.

[0176] The first segmentation sub-module is configured to perform depth estimation processing on the initial echocardiogram video using a left ventricular depth segmentation model to obtain a relative depth estimation value corresponding to each pixel point located in the left ventricular region in each frame of the echocardiogram single-modal image, where the relative depth estimation value characterizes the depth of each pixel point in the left ventricular region.

[0177] The first generation sub-module is configured to generate a left ventricular volume-time curve according to multiple relative depth estimation values in each frame of the echocardiogram single-modal image.

[0178] According to an embodiment of the present invention, the device for generating an echocardiogram video based on electromagnetic signals further includes: a training module.

[0179] According to an embodiment of the present invention, the training module includes: a third acquisition sub-module, a fourth acquisition sub-module, a second extraction sub-module, a first prediction sub-module, a second distortion sub-module, a first obtaining sub-module, a first adjustment sub-module, and a second obtaining sub-module.

[0180] The third acquisition sub-module is configured to acquire an initial model and a training sample data set. The initial model includes an initial optical flow auto-encoder sub-module and an optical flow generation sub-module to be trained. The initial optical flow auto-encoder sub-module includes a pre-trained image encoder, a pre-trained optical flow prediction sub-module, and a pre-trained image decoder. The training sample data set includes sample echocardiogram videos, sample left ventricular volume-time curves, and sample Gaussian noises.

[0181] The fourth acquisition sub-module is configured to acquire multiple frames of sample echocardiogram images from the sample echocardiogram videos based on a predetermined frame sequence.

[0182] The second extraction sub-module is configured to respectively input the multiple frames of sample echocardiogram images into the pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatio-temporal features.

[0183] The first prediction sub-module is configured to respectively input the multiple sample cardiac spatio-temporal features into the pre-trained optical flow prediction sub-module for optical flow prediction processing to obtain sample labels, where the sample labels include sample optical flow sequences and sample occlusion sequences.

[0184] The second distortion sub-module is configured to input the sample Gaussian noises, the sample optical flow sequences, and the sample occlusion sequences into the optical flow generation sub-module to be trained for feature distortion processing to obtain sample optical flow occlusion sequence noise features.

[0185] The first obtaining sub-module is configured to obtain an optical flow occlusion sequence noise loss value based on an optical flow occlusion sequence noise loss function according to the sample left ventricular volume-time curve, the sample optical flow occlusion sequence noise features, and the sample Gaussian noises.

[0186] The first adjustment sub-module is configured to adjust the module parameters of the optical flow generation sub-module to be trained according to the optical flow occlusion sequence noise loss value to obtain a trained optical flow generation sub-module.

[0187] The second obtaining sub-module is configured to obtain an optical flow encoding synthesis model according to the pre-trained image encoder, the trained optical flow generation sub-module, and the pre-trained image decoder.

[0188] According to an embodiment of the present invention, the second distortion sub-module includes: a first merging unit, a first adding unit, and a first obtaining unit.

[0189] The first merging unit is configured to perform a merging process on the sample optical flow sequences and the sample occlusion sequences to generate a sample optical flow occlusion sequence sample.

[0190] A first adding unit for adding sample Gaussian noise to a sample optical flow occlusion sequence sample to obtain an optical flow occlusion noise sequence.

[0191] A first obtaining unit for inputting the optical flow occlusion noise sequence into a to-be-trained optical flow generation sub-module for feature generation processing to obtain sample optical flow occlusion sequence noise features.

[0192] According to an embodiment of the present invention, the first obtaining sub-module includes: a first acquiring unit, a first normalization unit, a first generating unit, a second obtaining unit, and a third obtaining unit.

[0193] The first acquiring unit is used to acquire a plurality of left ventricular volume values from a sample left ventricular volume time curve.

[0194] The first normalization unit is used to perform normalization processing on a plurality of left ventricular volume values and the first left ventricular volume value among the plurality of left ventricular volume values to obtain a plurality of left ventricular volume ratios.

[0195] The first generating unit is used to generate a left ventricular volume ratio curve within a preset time period according to the plurality of left ventricular volume ratios.

[0196] The second obtaining unit is used to input the left ventricular volume ratio curve into an enhanced embedding sequential model for extracting time features to obtain left ventricular volume time features within a preset time period.

[0197] The third obtaining unit is used to obtain an optical flow occlusion sequence noise loss value based on an optical flow occlusion sequence noise loss function according to a sample initial cardiac spatio-temporal feature, a left ventricular volume time feature, a sample optical flow occlusion sequence noise feature, and a sample Gaussian noise, where the sample initial cardiac spatio-temporal feature represents the first sample cardiac spatio-temporal feature among a plurality of sample cardiac spatio-temporal features.

[0198] According to an embodiment of the present invention, the training module further includes: a fifth acquiring sub-module, a sixth acquiring sub-module, a third extracting sub-module, a second predicting sub-module, a third warping sub-module, a first reconstructing sub-module, a first adversarial sub-module, a third obtaining sub-module, and a fourth obtaining sub-module.

[0199] The fifth acquiring sub-module is used to acquire a to-be-trained initial optical flow auto-encoding sub-module and a to-be-trained image discriminator, where the to-be-trained initial optical flow auto-encoding model includes a to-be-trained image encoder, a to-be-trained optical flow prediction sub-module, and a to-be-trained image decoder.

[0200] The sixth acquiring sub-module is used to acquire a sample reference echocardiogram image and a sample target echocardiogram image from a sample echocardiogram video.

[0201] The third extraction sub-module is used to input the sample reference echocardiogram image into the image encoder to be trained for feature extraction processing, and obtain the sample reference cardiac spatio-temporal features.

[0202] The second prediction sub-module is used to input the sample reference echocardiogram image and the sample target echocardiogram image into the optical flow prediction sub-module to be trained for optical flow prediction processing, and obtain the sample reference optical flow and the sample reference occlusion. Among them, the sample reference optical flow represents the displacement change feature between the pixel points in the sample reference echocardiogram image and the pixel points in the sample target echocardiogram image, and the sample reference occlusion represents the region change feature of the sample reference echocardiogram image occluded by the sample target echocardiogram image.

[0203] The third warping sub-module is used to warp the sample reference cardiac spatio-temporal features by using the sample reference optical flow sequence and the sample reference occlusion sequence, and obtain the sample warped cardiac spatio-temporal features.

[0204] The first reconstruction sub-module is used to input the sample warped cardiac spatio-temporal features into the image decoder to be trained for feature decoding and mapping processing, and obtain the reconstructed echocardiogram image.

[0205] The first adversarial sub-module is used to perform image adversarial processing on the reconstructed echocardiogram image by using the image discriminator to be trained, and obtain the adversarial echocardiogram image.

[0206] The third obtaining sub-module is used to obtain the optical flow auto-encoding loss value based on the optical flow auto-encoding loss function, according to the reconstructed echocardiogram image, the sample target echocardiogram image, the adversarial echocardiogram image, and the balance parameter.

[0207] The fourth obtaining sub-module is used to adjust the module parameters of the initial optical flow auto-encoding sub-module to be trained according to the optical flow auto-encoding loss value, and obtain the trained initial optical flow auto-encoding sub-module.

[0208] According to an embodiment of the present invention, any plurality of modules among the acquisition module 710 and the feature distortion mapping module 720 can be combined and implemented in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 710 and the feature distortion mapping module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable means such as hardware or firmware through circuit integration or packaging, or can be implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the acquisition module 710 and the feature distortion mapping module 720 can be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0209] Figure 8 The block diagram of an electronic device for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown.

[0210] As Figure 8 shown, the electronic device according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. The processor 801 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 801 can also include on-board memory for caching purposes. The processor 801 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0211] In the RAM 803, various programs and data required for the operation of the electronic device are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. The processor 801 executes various operations of the method flow according to an embodiment of the present invention by executing the program in the ROM 802 and / or the RAM 803. It should be noted that the program can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also execute various operations of the method flow according to an embodiment of the present invention by executing the program stored in the one or more memories.

[0212] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 805, and the input / output (I / O) interface 805 is also connected to the bus 804. The electronic device may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. The driver 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the driver 810 as needed, so that a computer program read from it can be installed into the storage portion 808 as needed.

[0213] The present invention also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present invention is implemented.

[0214] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the above-described ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803.

[0215] An embodiment of the present invention further includes a computer program product, which includes a computer program, and the computer program includes program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to cause the computer system to implement the method for generating an echocardiogram based on the left ventricular volume curve provided by the embodiments of the present invention.

[0216] When the computer program is executed by the processor 801, the above functions defined in the system / apparatus of the embodiments of the present invention are executed. According to the embodiments of the present invention, the systems, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0217] In one embodiment, the computer program can rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program can also be transmitted and distributed in the form of signals on a network medium, and be downloaded and installed through the communication part 809, and / or be installed from the removable medium 811. The program code included in the computer program can be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0218] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or be installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiments of the present invention are executed. According to the embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0219] Those skilled in the art can understand that the features described in the various embodiments of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A method for generating an echocardiogram based on a left ventricular volume curve, characterized in that: include: Acquiring an initial ultrasound cardiac image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve represents a volume change characteristic of the left ventricle of the heart within a predetermined period of time; Based on the left ventricular volume-time curve, the initial ultrasound image is subjected to feature distortion mapping processing using an optical flow coding synthesis model to obtain a multi-frame target ultrasound image, wherein the optical flow coding synthesis model is obtained by training an initial model using a multi-frame sample ultrasound image, a sample left ventricular volume-time curve and a sample Gaussian noise in a sample ultrasound video, according to sample labels corresponding to the multi-frame sample ultrasound images, the multi-frame target ultrasound image indicates the temporal variation characteristics of multiple cardiac functions of the heart within the predetermined time period, and the multi-frame target ultrasound image is used to generate a target ultrasound video; Among them, the optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder, which obtains a Gaussian noise matrix used to generate the target ultrasound image; the initial ultrasound image is input into the image encoder for feature extraction processing to obtain initial heart spatiotemporal features corresponding to the heart; the Gaussian noise matrix, the left ventricular volume-time curve and the initial heart spatiotemporal features are input into the optical flow generation submodule, and based on the initial heart spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted heart spatiotemporal features; the multiple distorted heart spatiotemporal features are input into the image decoder for feature decoding and mapping processing to obtain the multi-frame target ultrasound image.

2. The method according to claim 1, characterized in that Based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain a plurality of distorted cardiac spatiotemporal features, including: Based on the initial cardiac spatiotemporal characteristics and the left ventricular volume-time curve, the optical flow generation submodule performs denoising on the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve, wherein each target optical flow in the target optical flow sequence represents a displacement change feature between a pixel point in the initial ultrasonic cardiac image and a pixel point in each frame of the target ultrasonic cardiac image, and each target occlusion in the target occlusion sequence represents a change feature of an area of ​​the initial ultrasonic cardiac image occluded by each frame of the target ultrasonic cardiac image; The initial cardiac spatiotemporal features are distorted using the target optical flow sequence and the target occlusion sequence to obtain the multiple distorted cardiac spatiotemporal features.

3. The method according to claim 1, characterized in that Obtain left ventricular volume-time curve, including: Acquiring an initial ultrasound video, wherein the initial ultrasound video includes multiple frames of ultrasound single-mode images; Performing depth estimation processing on the initial ultrasound video using a left ventricle depth segmentation model to obtain a relative depth estimation value corresponding to each pixel located in the left ventricle region in each frame of the ultrasound single-modality image, wherein the relative depth estimation value represents the depth of each pixel located in the left ventricle region; The left ventricular volume time curve is generated according to a plurality of the relative depth estimation values ​​in each frame of the echocardiography single modality image.

4. The method according to claim 1, characterized in that: The optical flow coding synthesis model is trained in the following way: Acquire the initial model and the training sample data set, wherein the initial model includes an initial optical flow autoencoder submodule and an optical flow generation submodule to be trained, the initial optical flow autoencoder submodule includes a pre-trained image encoder, a pre-trained optical flow prediction submodule, and a pre-trained image decoder, and the training sample data set includes the sample ultrasound video, the sample left ventricular volume time curve, and the sample Gaussian noise; Based on a predetermined frame sequence, acquiring a plurality of frames of sample ultrasound images from the sample ultrasound video; Inputting the multiple frames of sample ultrasound cardiac images into the pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatiotemporal features; Inputting the plurality of sample heart spatiotemporal features into the pre-trained optical flow prediction submodule for optical flow prediction processing to obtain the sample label, wherein the sample label includes a sample optical flow sequence and a sample occlusion sequence; Inputting the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise feature of the sample optical flow occlusion sequence; Based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, an optical flow occlusion sequence noise loss value is obtained; According to the noise loss value of the optical flow occlusion sequence, adjusting the module parameters of the optical flow generation submodule to be trained to obtain a trained optical flow generation submodule; The optical flow coding synthesis model is obtained according to the pre-trained image encoder, the trained optical flow generation submodule and the pre-trained image decoder.

5. The method according to claim 4, characterized in that The step of inputting the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise feature of the sample optical flow occlusion sequence comprises: Merging the sample optical flow sequence and the sample occlusion sequence to generate a sample optical flow occlusion sequence; Adding the sample Gaussian noise to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence; The optical flow occlusion noise sequence is input into the optical flow generation submodule to be trained for feature generation processing to obtain the sample optical flow occlusion sequence noise feature.

6. The method according to claim 4, characterized in that The optical flow occlusion sequence noise loss function is based on the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise to obtain the optical flow occlusion sequence noise loss value: Acquiring a plurality of left ventricular volume values ​​from the sample left ventricular volume-time curve; Normalizing the multiple left ventricular volume values ​​and the first left ventricular volume value among the multiple left ventricular volume values ​​to obtain multiple left ventricular volume ratios; generating a left ventricular volume ratio curve within the predetermined period according to the plurality of left ventricular volume ratios; Inputting the left ventricular volume ratio curve into the enhanced embedded sequential model to extract the time feature, and obtaining the left ventricular volume time feature within the predetermined time period; Based on the optical flow occlusion sequence noise loss function, the optical flow occlusion sequence noise loss value is obtained according to the sample initial cardiac spatiotemporal characteristics, the left ventricular volume time characteristics, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, wherein the sample initial cardiac spatiotemporal characteristics represent the first sample cardiac spatiotemporal characteristics among the multiple sample cardiac spatiotemporal characteristics.

7. The method according to claim 4, characterized in that The initial optical flow autoencoder submodule is trained in the following manner, including: Acquire an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained, wherein the initial optical flow autoencoder submodule to be trained includes an image encoder to be trained, an optical flow prediction submodule to be trained, and an image decoder to be trained; acquiring a sample reference echocardiogram and a sample target echocardiogram from the sample echocardiogram video; Inputting the sample reference ultrasound cardiac image into the image encoder to be trained for feature extraction processing to obtain sample reference cardiac spatiotemporal features; Inputting the sample reference ultrasound image and the sample target ultrasound image into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain a sample reference optical flow and a sample reference occlusion, wherein the sample reference optical flow represents a displacement change feature between a pixel point in the sample reference ultrasound image and a pixel point in the sample target ultrasound image, and the sample reference occlusion represents a change feature of an area of ​​the sample reference ultrasound image occluded by the sample target ultrasound image; Using a sample reference optical flow sequence and a sample reference occlusion sequence to distort the sample reference heart spatiotemporal features, to obtain a sample distorted heart spatiotemporal features; Inputting the sample distorted cardiac spatiotemporal features into the image decoder to be trained for feature decoding and mapping processing to obtain a reconstructed ultrasound cardiac image; Using the image discriminator to be trained to perform image adversarial processing on the reconstructed ultrasonic cardiac image to obtain an adversarial ultrasonic cardiac image; Based on the optical flow autoencoder loss function, an optical flow autoencoder loss value is obtained according to the reconstructed ultrasound image, the sample target ultrasound image, the adversarial ultrasound image and a balance parameter; According to the optical flow autoencoder loss value, the module parameters of the initial optical flow autoencoder submodule to be trained are adjusted to obtain the trained initial optical flow autoencoder submodule.

8. A device for generating an echocardiogram based on a left ventricular volume curve, characterized in that: include: An acquisition module, used for acquiring an initial ultrasound cardiac image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve represents a volume change characteristic of the left ventricle of the heart within a predetermined period of time; A feature warping mapping module is used to perform feature warping mapping processing on the initial ultrasound image based on the left ventricular volume time curve using an optical flow coding synthesis model to obtain a multi-frame target ultrasound image, wherein the optical flow coding synthesis model is obtained by training an initial model using a multi-frame ultrasound image in a sample ultrasound video, a sample left ventricular volume time curve and a sample Gaussian noise according to sample labels corresponding to the multi-frame ultrasound images, the multi-frame target ultrasound image indicates the time series variation characteristics of multiple cardiac functions of the heart within the predetermined time period, and the multi-frame target ultrasound image is used to generate a target ultrasound video; Among them, the optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder, which obtains a Gaussian noise matrix used to generate the target ultrasound image; the initial ultrasound image is input into the image encoder for feature extraction processing to obtain initial heart spatiotemporal features corresponding to the heart; the Gaussian noise matrix, the left ventricular volume-time curve and the initial heart spatiotemporal features are input into the optical flow generation submodule, and based on the initial heart spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted heart spatiotemporal features; the multiple distorted heart spatiotemporal features are input into the image decoder for feature decoding and mapping processing to obtain the multi-frame target ultrasound image.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.

Images