Method, device and equipment for generating echocardiogram based on left ventricular volume curve

Abstract

The invention provides a method, a device and equipment for generating an echocardiogram based on a left ventricular volume curve, which can be applied to the technical field of medical data generation. The method for generating the echocardiogram based on the left ventricular volume curve comprises the following steps: acquiring an initial echocardiogram and a left ventricular volume time curve; on the basis of the left ventricular volume time curve, feature distortion mapping processing is conducted on the initial echocardiography image through an optical flow coding synthesis model, and multiple frames of target echocardiography images are obtained, the optical flow coding synthesis model is obtained by training an initial model according to sample labels corresponding to multiple frames of sample echocardiography images by utilizing the multiple frames of sample echocardiography images, a sample left ventricle volume time curve and sample Gaussian noise in a sample echocardiography video; the multiple frames of target echocardiography images indicate time sequence change characteristics of multiple cardiac functions of the heart in a preset time period, and the multiple frames of target echocardiography images are used for generating a target echocardiography video.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data generation, and more specifically to a method, device and equipment for generating an echocardiogram based on a left ventricular volume curve. Background Art

[0002] Ultrasound video is one of the most important biomedical images currently. Relevant professionals can use ultrasound video as intermediate information to make targeted diagnosis of the target. Therefore, relevant professionals need a large amount of clinical ultrasound video learning data as training data for simulation training to improve relevant professional knowledge capabilities.

[0003] In the process of realizing the above-mentioned inventive concept, it was found that the ultrasonic cardiology training video generated by the existing technology could not contain comprehensive and accurate cardiac activity characteristics, resulting in the inability to conduct training based on the generated ultrasonic cardiology training video and there were large errors in the simulation training process, which affected the clinical judgment in practice and had poor applicability. Summary of the invention

[0004] In view of the above problems, the present invention provides a method, device and apparatus for generating an echocardiogram based on a left ventricular volume curve.

[0005] According to a first aspect of the present invention, a method for generating an echocardiogram based on a left ventricular volume curve is provided, comprising: acquiring an initial echocardiogram and a left ventricular volume-time curve, wherein the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined time period; based on the left ventricular volume-time curve, performing feature distortion mapping processing on the initial echocardiogram using an optical flow coding synthesis model to obtain a multi-frame target echocardiogram, wherein the optical flow coding synthesis model is obtained by training the initial model using a multi-frame sample echocardiogram, a sample left ventricular volume-time curve and a sample Gaussian noise in a sample echocardiogram video according to sample labels corresponding to the multi-frame sample echocardiogram, the multi-frame target echocardiogram indicates the temporal change characteristics of multiple cardiac functions of the heart within a predetermined time period, and the multi-frame target echocardiogram is used to generate a target echocardiogram.

[0006] According to an embodiment of the present invention, an optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder. Based on the left ventricular volume-time curve, the optical flow coding synthesis model is used to perform feature distortion mapping processing on the initial ultrasound image to obtain multiple frames of target ultrasound images, including: obtaining a Gaussian noise matrix for generating the target ultrasound image; inputting the initial ultrasound image into the image encoder of the optical flow coding synthesis model for feature extraction processing to obtain initial heart spatiotemporal features corresponding to the heart; inputting the Gaussian noise matrix, the left ventricular volume-time curve and the initial heart spatiotemporal features into the optical flow generation submodule of the optical flow coding synthesis model; based on the initial heart spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted heart spatiotemporal features; inputting the multiple distorted heart spatiotemporal features into the image decoder of the optical flow coding synthesis model for feature decoding and mapping processing to obtain multiple frames of target ultrasound images.

[0007] According to an embodiment of the present invention, based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation sub-model performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatiotemporal features, including: based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation sub-module performs denoising processing on the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve, wherein each target optical flow in the target optical flow sequence represents the displacement change characteristics between the pixel points in the initial ultrasonic cardiac image and the pixel points in each frame of the target ultrasonic cardiac image, and each target occlusion in the target occlusion sequence represents the change characteristics of the area of ​​the initial ultrasonic cardiac image occluded by each frame of the target ultrasonic cardiac image; the initial cardiac spatiotemporal features are distorted using the target optical flow sequence and the target occlusion sequence to obtain multiple distorted cardiac spatiotemporal features.

[0008] According to an embodiment of the present invention, obtaining a left ventricular volume-time curve includes: obtaining an ultrasonic cardiology video, wherein the ultrasonic cardiology video includes multiple frames of ultrasonic cardiology single-modality images; performing depth estimation processing on the ultrasonic cardiology video using a left ventricular depth segmentation model to obtain a relative depth estimation value corresponding to each pixel point located in the left ventricular region in each frame of the ultrasonic cardiology single-modality image, wherein the relative depth estimation value represents the depth of each pixel point located in the left ventricular region; and generating a left ventricular volume-time curve based on multiple relative depth estimation values ​​in each frame of the ultrasonic cardiology single-modality image.

[0009] According to an embodiment of the present invention, an optical flow coding synthesis model is trained in the following manner, including: obtaining an initial model and a training sample data set, wherein the initial model includes an initial optical flow self-encoding submodule and an optical flow generation submodule to be trained, the initial optical flow self-encoding submodule includes a pre-trained image encoder, a pre-trained optical flow prediction submodule, and a pre-trained image decoder, and the training sample data set includes a sample ultrasound video, a sample left ventricular volume time curve, and a sample Gaussian noise; based on a predetermined frame sequence, obtaining multiple frames of sample ultrasound images from the sample ultrasound video; inputting the multiple frames of sample ultrasound images into the pre-trained image encoder for feature extraction processing to obtain multiple sample heart spatiotemporal features; and inputting the multiple sample heart spatiotemporal features into the pre-trained optical flow encoder. The prediction submodule performs optical flow prediction processing to obtain sample labels, wherein the sample labels include sample optical flow sequences and sample occlusion sequences; the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence are input into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise characteristics of the sample optical flow occlusion sequence; based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, the optical flow occlusion sequence noise loss value is obtained; according to the optical flow occlusion sequence noise loss value, the module parameters of the optical flow generation submodule to be trained are adjusted to obtain a trained optical flow generation submodule; according to the pre-trained image encoder, the trained optical flow generation submodule and the pre-trained image decoder, a trained optical flow encoding synthesis model is obtained.

[0010] According to an embodiment of the present invention, sample Gaussian noise, sample optical flow sequence and sample occlusion sequence are input into an optical flow generation submodule to be trained for feature distortion processing to obtain noise features of the sample optical flow occlusion sequence, including: merging the sample optical flow sequence and the sample occlusion sequence to generate a sample optical flow occlusion sequence; adding sample Gaussian noise to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence; inputting the optical flow occlusion noise sequence into the optical flow generation submodule to be trained for feature generation processing to obtain noise features of the sample optical flow occlusion sequence.

[0011] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, an optical flow occlusion sequence noise loss value is obtained, including: obtaining multiple left ventricular volume values ​​from the sample left ventricular volume time curve; normalizing the multiple left ventricular volume values ​​and the first left ventricular volume value among the multiple left ventricular volume values ​​to obtain multiple left ventricular volume ratios; generating a left ventricular volume ratio curve within a predetermined time period according to the multiple left ventricular volume ratios; inputting the left ventricular volume ratio curve into the enhanced embedded sequential model to extract the time characteristics to obtain the left ventricular volume time characteristics within the predetermined time period; based on the optical flow occlusion sequence noise loss function, according to the sample initial heart spatiotemporal characteristics, the left ventricular volume time characteristics, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, an optical flow occlusion sequence noise loss value is obtained, wherein the sample initial heart spatiotemporal characteristics represent the first sample heart spatiotemporal characteristics among the multiple sample heart spatiotemporal characteristics.

[0012] According to an embodiment of the present invention, an initial optical flow autoencoder submodule is trained in the following manner, including: obtaining an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained, wherein the initial optical flow autoencoder model to be trained includes an image encoder to be trained, an optical flow prediction submodule to be trained, and an image decoder to be trained; obtaining a sample reference ultrasound image and a sample target ultrasound image from a sample ultrasound video; inputting the sample reference ultrasound image into the image encoder to be trained for feature extraction processing to obtain sample reference cardiac spatiotemporal features; inputting the sample reference ultrasound image and the sample target ultrasound image into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain sample reference optical flow and sample reference occlusion, wherein the sample reference optical flow represents the relationship between a pixel point in the sample reference ultrasound image and a pixel point in the sample target ultrasound image. The displacement change characteristics between points, the sample reference occlusion characterizes the change characteristics of the area where the sample reference ultrasound cardiogram is occluded by the sample target ultrasound cardiogram; the sample reference heart spatiotemporal characteristics are distorted by using the sample reference optical flow sequence and the sample reference occlusion sequence to obtain the sample distorted heart spatiotemporal characteristics; the sample distorted heart spatiotemporal characteristics are input into the image decoder to be trained for feature decoding and mapping processing to obtain the reconstructed ultrasound cardiogram; the reconstructed ultrasound cardiogram is subjected to image adversarial processing by using the image discriminator to be trained to obtain the adversarial ultrasound cardiogram; based on the optical flow autoencoder loss function, the optical flow autoencoder loss value is obtained according to the reconstructed ultrasound cardiogram, the sample target ultrasound cardiogram, the adversarial ultrasound cardiogram and the balance parameter; according to the optical flow autoencoder loss value, the module parameters of the initial optical flow autoencoder submodule to be trained are adjusted to obtain the trained initial optical flow autoencoder submodule.

[0013] The second aspect of the present invention provides a device for generating an ultrasonic cardiology video based on a left ventricular curve, comprising: an acquisition module, used to acquire an initial ultrasonic cardiology image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve characterizes the volume change characteristics of the left ventricle of the heart within a predetermined time period; a feature distortion mapping module, used to perform feature distortion mapping processing on the initial ultrasonic cardiology image based on the left ventricular volume-time curve using an optical flow coding synthesis model to obtain a multi-frame target ultrasonic cardiology image, wherein the optical flow coding synthesis model is obtained by training the initial model using a multi-frame ultrasonic cardiology image, a sample left ventricular volume-time curve and a sample Gaussian noise in a sample ultrasonic cardiology video, according to sample labels corresponding to the multi-frame ultrasonic cardiology images, the multi-frame target ultrasonic cardiology images indicate the temporal change characteristics of multiple cardiac functions of the heart within a predetermined time period, and the multi-frame target ultrasonic cardiology images are used to generate a target ultrasonic cardiology video.

[0014] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0015] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0016] The fifth aspect of the present invention also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0017] According to the method, device and apparatus for generating an echocardiogram based on a left ventricular volume curve of the present invention, an initial echocardiogram and a left ventricular volume-time curve are obtained, and based on the left ventricular volume-time curve, an optical flow coding synthesis model is used to perform feature distortion mapping processing on the initial echocardiogram to obtain a multi-frame target echocardiogram, so as to facilitate synthesis processing of the multi-frame target echocardiogram to obtain a target echocardiogram video, thereby achieving a highly fine-grained target echocardiogram containing important cardiac semantic information such as heart rate and filling speed within a predetermined time period based on the left ventricular volume-time curve containing multiple cardiac function characteristics, and facilitating communication. A large number of target ultrasound echocardiogram videos that are accurate and contain more comprehensive cardiac activity features can help relevant professionals to conduct a large number of simulation trainings. Furthermore, in the process of generating multi-frame target ultrasound echocardiograms using the optical flow coding synthesis model, the generated target ultrasound echocardiograms can also be adjusted, or heart-related activity features can be added to obtain target ultrasound echocardiograms with rich and realistic cardiac activity features, thereby improving the training environment and complexity of the training process for relevant professionals, thereby helping to improve the accuracy of clinical cardiac function assessment of ultrasound echocardiograms and ultrasound echocardiogram videos actually collected from patients by relevant professionals.

[0018] According to the embodiment of the present invention, further, by utilizing multiple frames of sample ultrasound images, sample left ventricular volume-time curves and sample Gaussian noise in a sample ultrasound video, an optical flow coding synthesis model is obtained after a large amount of model training is performed on the initial model according to sample labels corresponding to the multiple frames of sample ultrasound images, so that the trained optical flow coding synthesis model can be used to process the initial ultrasound images and the left ventricular volume-time curve, thereby improving the efficiency and training efficiency of generating target ultrasound images and target ultrasound videos, reducing the training learning cost, and improving the practicality and applicability of the optical flow coding synthesis model in a clinical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 A diagram showing an application scenario of a method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention;

[0021] Figure 2 A flow chart of a method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown;

[0022] Figure 3 A schematic diagram of generating a target ultrasound cardiac image using an optical flow coding synthesis model according to an embodiment of the present invention is shown;

[0023] Figure 4 A schematic diagram showing the workflow of the present invention according to an embodiment of the present invention is shown;

[0024] Figure 5 A schematic diagram showing the effect comparison between a target ultrasound cardiography video generated by the method of the present invention and a real video according to an embodiment of the present invention is shown;

[0025] Figure 6a A schematic diagram showing waveform correlation between left ventricular volume time curves of target ultrasound cardiography videos generated by the method of the present invention and real videos at different times for 1277 ultrasound videos according to an embodiment of the present invention;

[0026] Figure 6b A schematic diagram showing the mean square error of the left ventricular volume time curve of the target ultrasound cardiography video generated by the method of the present invention and the real video at different times for 1277 ultrasound videos according to an embodiment of the present invention;

[0027] Figure 6c A schematic diagram showing waveform correlation between left ventricular volume time curves of target ultrasound cardiography videos generated by the method of the present invention and real videos at different frame rates for 1277 ultrasound videos according to an embodiment of the present invention;

[0028] Figure 6d A schematic diagram showing the mean square error of the left ventricular volume time curve of the target ultrasound cardiography video generated by the method of the present invention and the real video at different frame rates for 1277 ultrasound videos according to an embodiment of the present invention;

[0029] Figure 7 A structural block diagram of a device for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown;

[0030] Figure 8 A block diagram of an electronic device for a method of generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0031] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] As a non-invasive, safe and effective method, ultrasound video is widely used in medical testing to help with targeted diagnosis as an intermediate result. By observing the propagation of ultrasound waves in the human body, ultrasound video can show the movement of the heart, thereby providing video image information that can assist in the diagnosis of diseases. Since the interpretation of ultrasound video often relies heavily on the professional knowledge of relevant professionals, relevant professionals need to use a large amount of ultrasound video as training data for simulation training from time to time to improve their professional capabilities.

[0037] However, existing ultrasound video for simulation training is usually generated based only on ultrasound images generated by left ventricular ejection fraction, which results in the generated ultrasound video failing to include comprehensive cardiac activity features, such as heart rate, heart rate variability, and ventricular filling rate during diastole. During the research and development process, it was found that ultrasound training videos generated using existing technologies failed to include comprehensive and accurate cardiac activity features, resulting in the inability to conduct training based on the generated ultrasound training videos and large errors in the simulation training process, which affected clinical judgment in practice and had poor applicability.

[0038] In view of this, an embodiment of the present invention provides a method for generating an echocardiogram based on a left ventricular volume curve, comprising: acquiring an initial echocardiogram and a left ventricular volume-time curve, wherein the left ventricular volume-time curve represents the volume change characteristics of the left ventricle of the heart within a predetermined time period; based on the left ventricular volume-time curve, performing feature distortion mapping processing on the initial echocardiogram using an optical flow coding synthesis model to obtain a multi-frame target echocardiogram, wherein the optical flow coding synthesis model is obtained by training the initial model using a multi-frame sample echocardiogram, a sample left ventricular volume-time curve and a sample Gaussian noise in a sample echocardiogram video according to sample labels corresponding to the multi-frame sample echocardiogram, the multi-frame target echocardiogram indicates the temporal change characteristics of multiple cardiac functions of the heart within a predetermined time period, and the multi-frame target echocardiogram is used to generate a target echocardiogram.

[0039] Figure 1 A diagram showing an application scenario of a method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention.

[0040] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a user 102 and a second terminal device 103. The first terminal device 101 is used to send a request signal to the second terminal device 103, and the second terminal device 103 generates training data according to the received request signal.

[0041] The user 102 may use the first terminal device 101 to interact with the second terminal device 103 to receive or send signals, etc.

[0042] The second terminal device 103 may be a processor for receiving a request signal, for example, receiving and processing the request signal sent by the first terminal device 101 (for example only). The second terminal device 103 may analyze and process the received request signal and other data, and feed back the processing result (for example, obtaining or generating relevant training data according to the user's request, etc.) to the first terminal device 101.

[0043] It should be noted that the method for generating an echocardiogram based on the left ventricular volume curve provided in the embodiment of the present invention can generally be executed by the second terminal device 103. Accordingly, the apparatus for generating an echocardiogram based on the left ventricular volume curve provided in the embodiment of the present invention can generally be arranged in the second terminal device 103. The method for generating an echocardiogram based on the left ventricular volume curve provided in the embodiment of the present invention can also be executed by a receiver or a receiver cluster that is different from the second terminal device 103 and can communicate with the first terminal device 101 and / or the second terminal device 103. Accordingly, the apparatus for generating an echocardiogram based on the left ventricular volume curve provided in the embodiment of the present invention can also be arranged in a receiver or a receiver cluster that is different from the second terminal device 103 and can communicate with the first terminal device 101 and / or the second terminal device 103.

[0044] It should be understood that Figure 1 The number of first terminal devices, users, and second terminal devices in the embodiment is only illustrative. Any number of first terminal devices, users, and second terminal devices may be provided according to implementation requirements.

[0045] The following will be based on Figure 1 The scene described by Figure 2~Figure 6d The method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is described in detail.

[0046] Figure 2 A flow chart of a method for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown.

[0047] like Figure 2 As shown, the method for generating an echocardiogram based on a left ventricular volume curve in this embodiment includes operations S210 to S220.

[0048] In operation S210 , an initial echocardiographic image and a left ventricular volume-time curve are acquired.

[0049] According to an embodiment of the present invention, the left ventricular volume-time curve represents a volume change characteristic of the left ventricle of the heart within a predetermined period of time.

[0050] According to an embodiment of the present invention, when simulation training is required, an initial ultrasound image and a left ventricular volume-time curve can be obtained from a database, and the initial ultrasound image can be used as a conditional image, i.e., an initial template for generating a target ultrasound video. Then, based on the changing characteristics of the left ventricle in the left ventricular volume-time curve within a predetermined time period, for example, cardiac activity characteristics such as ventricular filling velocity, heart rate and heart rate variability during diastole of the heart within a predetermined time period, an accurate target ultrasound video with high granularity containing relatively comprehensive cardiac activity characteristics can be generated.

[0051] In operation S220, based on the left ventricular volume-time curve, the initial ultrasound image is subjected to feature warping mapping processing by using an optical flow coding synthesis model to obtain multiple frames of target ultrasound images.

[0052] According to an embodiment of the present invention, the optical flow coding synthesis model is obtained by training an initial model using multiple frames of sample ultrasound images, sample left ventricular volume time curves and sample Gaussian noise in a sample ultrasound video according to sample labels corresponding to the multiple frames of sample ultrasound images.

[0053] According to an embodiment of the present invention, multiple frames of target ultrasound images can be used to generate a target ultrasound video, that is, multiple frames of target ultrasound images, that is, a target video frame sequence, can be synthesized to obtain a target ultrasound video containing multiple cardiac function semantic information.

[0054] According to an embodiment of the present invention, the multi-frame target ultrasound cardiology image indicates the temporal variation characteristics of multiple cardiac functions of the heart within a predetermined period of time.

[0055] According to an embodiment of the present invention, an initial ultrasound echocardiogram is subjected to feature warping mapping processing by utilizing an optical flow coding synthesis model to obtain a target video frame sequence, i.e., a multi-frame target ultrasound echocardiogram. The multi-frame target ultrasound echocardiogram includes a plurality of continuous change features related to cardiac function within a predetermined time period, for example, continuous cardiac activity change features such as ventricular filling velocity, heart rate, and heart rate variability during diastole of the heart within a predetermined time period.

[0056] According to an embodiment of the present invention, the obtained target ultrasound cardiography video can help relevant personnel to perform simulation training of professional knowledge, thereby helping to judge the relevant functions of the heart.

[0057] For example, when simulation training is required, an initial ultrasound image and a left ventricular volume-time curve that can be used to generate a target ultrasound video are obtained, and the initial ultrasound image and the left ventricular volume-time curve are input into an optical flow coding synthesis model. The optical flow coding synthesis model performs feature distortion mapping processing on the initial ultrasound image based on the left ventricular volume-time curve to obtain a target video frame sequence, that is, multiple frames of target ultrasound images, so that the multiple frames of target ultrasound images can be synthesized to obtain a target ultrasound video for training, so that scientists or professionals in related fields can perform simulation training based on the target ultrasound video.

[0058] According to an embodiment of the present invention, by acquiring an initial ultrasound image and a left ventricular volume-time curve, based on the left ventricular volume-time curve, the initial ultrasound image is subjected to feature distortion mapping processing using an optical flow coding synthesis model to obtain a multi-frame target ultrasound image, so as to facilitate synthesis processing of the multi-frame target ultrasound image to obtain a target ultrasound video, and a high-grained target ultrasound video containing important cardiac semantic information such as heart rate and filling speed within a predetermined time period based on the left ventricular volume-time curve containing a variety of cardiac function features is obtained, so that relevant professionals can be helped to perform a large number of simulation trainings through a large number of accurate target ultrasound videos containing more comprehensive cardiac activity features. Furthermore, in the process of generating a multi-frame target ultrasound image using an optical flow coding synthesis model, the generated target ultrasound image can also be adjusted, or activity features related to the heart can be added to obtain a target ultrasound video with rich and real cardiac activity features, thereby improving the training environment and the complexity of the training process of relevant professionals, thereby helping to improve the accuracy of clinical cardiac function evaluation of the ultrasound images and ultrasound videos actually collected from the patient objects by relevant professionals.

[0059] According to the embodiment of the present invention, further, by utilizing multiple frames of sample ultrasound images, sample left ventricular volume-time curves and sample Gaussian noise in a sample ultrasound video, an optical flow coding synthesis model is obtained after a large amount of model training is performed on the initial model according to sample labels corresponding to the multiple frames of sample ultrasound images, so that the trained optical flow coding synthesis model can be used to process the initial ultrasound images and the left ventricular volume-time curve, thereby improving the efficiency and training efficiency of generating target ultrasound images and target ultrasound videos, reducing the training learning cost, and improving the practicality and applicability of the optical flow coding synthesis model in a clinical environment.

[0060] It should be noted that the target ultrasonic cardiogram and target ultrasonic cardiogram video obtained by the present invention are only an intermediate result, and the diagnosis result or health condition cannot be directly derived from the target ultrasonic cardiogram and target ultrasonic cardiogram video obtained by the method of the present invention.

[0061] According to an embodiment of the present invention, a method for obtaining a left ventricular volume-time curve includes the following operations.

[0062] According to an embodiment of the present invention, an initial ultrasound video is acquired, and the initial ultrasound video includes a plurality of frames of ultrasound single-modality images.

[0063] According to an embodiment of the present invention, by constructing a left ventricular depth segmentation label based on the Simpson method, an automated left ventricular depth segmentation model is trained and optimized, so that the historical initial ultrasound cardiography video can be segmented using the left ventricular depth segmentation model to obtain multiple left ventricular volume-time curves.

[0064] According to an embodiment of the present invention, a left ventricle depth segmentation model is used to perform depth estimation processing on an initial ultrasonic cardiology video to obtain a relative depth estimation value corresponding to each pixel point located in the left ventricle region in each frame of ultrasonic cardiology single-modality image.

[0065] According to an embodiment of the present invention, the relative depth estimation value represents the depth of each pixel point in the left ventricle region.

[0066] According to an embodiment of the present invention, each frame of the acquired initial ultrasonic cardiac image in the ultrasonic cardiac video is segmented using a left ventricular depth segmentation model to obtain a depth value corresponding to each pixel in each frame of the initial ultrasonic cardiac image. Based on the depth value corresponding to each pixel, it can be determined whether the pixel is located in the left ventricular region, thereby obtaining a relative depth estimation value corresponding to each pixel located in the left ventricular region.

[0067] According to an embodiment of the present invention, a left ventricular volume-time curve is generated according to a plurality of relative depth estimation values ​​in each frame of an echocardiographic single-modality image.

[0068] According to an embodiment of the present invention, the relative depth estimation value of each pixel point in each frame of the initial ultrasound image is summed to calculate the candidate volume of the left ventricle in each frame of the initial ultrasound image, and a left ventricular volume-time curve within a predetermined time period is generated based on multiple candidate volumes of multiple frames of initial ultrasound images. The volume curve graph includes multiple minimum points, multiple maximum points and other numerical points, wherein the minimum points and the maximum points may correspond to the state of target contraction or target relaxation, for example, the movement state of cardiac contraction or cardiac relaxation, and important cardiac activity characteristics such as ventricular filling velocity during diastole may also be observed based on the entire left ventricular volume-time curve.

[0069] According to an embodiment of the present invention, the left ventricle depth segmentation model is trained by using an initial ultrasound cardiology sample video according to sample diastolic images containing segmentation labels, sample contraction images containing segmentation labels, and sample ultrasound cardiology images without segmentation labels, so as to obtain an intermediate segmentation model of the relative depth estimation value of each pixel point. Then, using the sample left ventricular volume-time curve as a label, the volume-time loss value is calculated for the training left ventricular volume-time curve generated by the relative depth estimation value of the pixel points located in the left ventricle obtained according to the intermediate segmentation model, so as to adjust the model parameters of the intermediate segmentation model and obtain the left ventricle depth segmentation model.

[0070] According to an embodiment of the present invention, an initial ultrasonic cardiology video is obtained, and a left ventricular depth segmentation model is used to perform depth estimation processing on the initial ultrasonic cardiology video to obtain a relative depth estimation value corresponding to each pixel point located in the left ventricular region in each frame of the ultrasonic cardiology single-modality image. A left ventricular volume-time curve is generated according to multiple relative depth estimation values ​​in each frame of the ultrasonic cardiology single-modality image, thereby realizing the extraction and generation of left ventricular volume-time curves for multiple historical initial ultrasonic cardiology videos using a pre-trained left ventricular depth segmentation model. While obtaining an accurate left ventricular volume-time curve, the curve can be used as training data to train an optical flow coding synthesis model, thereby improving the ability and accuracy of the optical flow coding synthesis model to generate a target ultrasonic cardiology video based on the left ventricular volume-time curve.

[0071] According to an embodiment of the present invention, the optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder.

[0072] According to an embodiment of the present invention, based on a left ventricular volume-time curve, a method for obtaining a multi-frame target ultrasound image by performing feature warping mapping processing on an initial ultrasound image using an optical flow coding synthesis model includes the following operations.

[0073] According to an embodiment of the present invention, a Gaussian noise matrix for generating a target echocardiographic image is acquired.

[0074] According to an embodiment of the present invention, a set of Gaussian noise matrices may be randomly generated, or a set of Gaussian noise matrices may be obtained from an ultrasonic cardiac noise library.

[0075] According to an embodiment of the present invention, the initial ultrasound cardiac image is input into an image encoder of an optical flow coding synthesis model for feature extraction processing to obtain initial cardiac spatiotemporal features corresponding to the heart.

[0076] According to an embodiment of the present invention, a Gaussian noise matrix, a left ventricular volume-time curve and initial cardiac spatiotemporal features are input into an optical flow generation submodule of an optical flow coding synthesis model; based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain a plurality of distorted cardiac spatiotemporal features.

[0077] According to an embodiment of the present invention, the distorted cardiac spatiotemporal feature may be characterized as a spatiotemporal feature of an ultrasound image that is generated based on a left ventricular volume time curve and the initial cardiac spatiotemporal feature and is continuous with the initial ultrasound image within a predetermined period of time.

[0078] According to an embodiment of the present invention, a pre-trained optical flow generation submodule is used to denoise the randomly generated Gaussian noise matrix, and the left ventricular volume-time curve and the initial cardiac spatiotemporal features are combined to restore the distorted cardiac spatiotemporal features that contain neither noise nor the initial cardiac spatiotemporal features and the cardiac activity features in the left ventricular volume-time curve.

[0079] According to an embodiment of the present invention, a plurality of distorted cardiac spatiotemporal features are input into an image decoder of an optical flow coding synthesis model for feature decoding and mapping processing to obtain a plurality of target ultrasound cardiac images.

[0080] According to an embodiment of the present invention, an optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder. By using the self-built optical flow coding synthesis model to process the features of the Gaussian noise matrix and the initial ultrasound cardiac image, the initial spatiotemporal features of the heart are obtained according to the initial ultrasound cardiac image, and the spatiotemporal features of subsequent ultrasound cardiac images in the target ultrasound video to be generated are predicted based on the initial spatiotemporal features of the heart and the pre-acquired left ventricular volume time curve, that is, the distorted heart spatiotemporal features. The image decoder is used to decode and restore multiple distorted heart spatiotemporal features to restore multiple frames of target ultrasound cardiac images, thereby realizing accurate estimation of the features of subsequent images using the optical flow generation submodule, and then the efficient self-built optical flow coding synthesis model can be used to generate accurate target ultrasound cardiac images containing comprehensive cardiac activity features, and the generated target ultrasound cardiac images are efficient, which can reduce the cost of generating training data and improve the practicality and applicability of the optical flow coding synthesis model in clinical environments.

[0081] According to an embodiment of the present invention, based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodule performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatiotemporal features, including the following operations.

[0082] According to an embodiment of the present invention, based on the initial cardiac spatiotemporal features and the left ventricular volume time curve, the optical flow generation submodule denoises the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume time curve.

[0083] According to an embodiment of the present invention, each target optical flow in the target optical flow sequence represents the displacement change characteristics between the pixel points in the initial ultrasound image and the pixel points in each frame of the target ultrasound image, and each target occlusion in the target occlusion sequence represents the change characteristics of the area of ​​the initial ultrasound image occluded by each frame of the target ultrasound image.

[0084] According to an embodiment of the present invention, the displacement change characteristics of the target optical flow sequence include two channels, namely, the displacement change characteristics of the pixel points under horizontal movement between two frames of ultrasound cardiac images and the displacement change characteristics under vertical movement. The target occlusion sequence is to identify the occlusion area through a single channel, that is, the change characteristics of the area where the previous frame of the image is occluded by the next frame of the image.

[0085] According to an embodiment of the present invention, the initial cardiac spatiotemporal features are distorted using a target optical flow sequence and a target occlusion sequence to obtain a plurality of distorted cardiac spatiotemporal features.

[0086] According to an embodiment of the present invention, an optical flow generation submodule is used to denoise a Gaussian noise matrix based on initial cardiac spatiotemporal features and a left ventricular volume-time curve to obtain an estimated accurate target optical flow sequence and target occlusion sequence, and each target optical flow and each target occlusion in the target optical flow sequence and the target occlusion sequence are used to distort the initial cardiac spatiotemporal features, thereby obtaining a plurality of distorted cardiac spatiotemporal features corresponding to a plurality of target ultrasound cardiac images to be generated, and predicting the spatiotemporal features of subsequent images from the optical flow dimensions of the vertical and horizontal movements of pixels and the occlusion dimensions of the overlap between pixels based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, so as to distort the initial cardiac spatiotemporal features using the accurate target optical flow sequence and the target occlusion sequence, and obtain accurate distorted cardiac spatiotemporal features corresponding to the cardiac activity features in the left ventricular volume-time curve.

[0087] According to an embodiment of the present invention, the optical flow coding synthesis model is trained in the following manner.

[0088] According to an embodiment of the present invention, an initial model and a training sample data set are obtained.

[0089] According to an embodiment of the present invention, the initial model includes an initial optical flow autoencoder submodule and an optical flow generation submodule to be trained, the initial optical flow autoencoder submodule includes a pre-trained image encoder, a pre-trained optical flow prediction submodule, and a pre-trained image decoder, and the training sample data set includes a sample ultrasound cardiology video, a sample left ventricular volume time curve, and a sample Gaussian noise.

[0090] According to an embodiment of the present invention, the initial optical flow autoencoder submodule is trained in the following manner.

[0091] According to an embodiment of the present invention, an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained are obtained.

[0092] According to an embodiment of the present invention, the initial optical flow autoencoder model to be trained includes an image encoder to be trained, an optical flow prediction submodule to be trained, and an image decoder to be trained.

[0093] According to an embodiment of the present invention, a sample reference ultrasound image and a sample target ultrasound image are acquired from a sample ultrasound video.

[0094] According to an embodiment of the present invention, the sample reference ultrasound image and the sample target ultrasound image are two frames of sample ultrasound images randomly selected from a sample ultrasound video.

[0095] According to an embodiment of the present invention, the sample reference ultrasound cardiac image is input to the image encoder to be trained for feature extraction processing to obtain the sample reference cardiac spatiotemporal features.

[0096] According to an embodiment of the present invention, the image encoder to be trained encodes the sample reference ultrasound cardiac image into a sample reference cardiac feature map containing sample reference cardiac spatiotemporal features.

[0097] According to an embodiment of the present invention, the sample reference ultrasound image and the sample target ultrasound image are input into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain the sample reference optical flow and the sample reference occlusion.

[0098] According to an embodiment of the present invention, the sample reference optical flow characterizes the displacement change characteristics between the pixel points in the sample reference ultrasound cardiogram and the pixel points in the sample target ultrasound cardiogram, and the sample reference occlusion characterizes the change characteristics of the area of ​​the sample reference ultrasound cardiogram occluded by the sample target ultrasound cardiogram.

[0099] According to an embodiment of the present invention, the optical flow prediction submodule to be trained may be used to extract potential optical flow and occlusion map estimation between two frames of sample echocardiographic images, ie, sample reference optical flow and sample reference occlusion.

[0100] According to an embodiment of the present invention, the sample reference heart spatiotemporal features are distorted using the sample reference optical flow sequence and the sample reference occlusion sequence to obtain the sample distorted heart spatiotemporal features.

[0101] According to an embodiment of the present invention, the temporal and spatial characteristics of the sample distorted heart can be calculated according to formula (1), which is as follows.

[0102] (1);

[0103] Among them, z t can characterize the temporal and spatial characteristics of the sample distorted heart, m can be characterized as the sample reference occlusion, z r can be characterized as the sample-referenced cardiac spatiotemporal features, f can be characterized as the sample-referenced optical flow, and Warp() can be characterized as a convolution function. It can be characterized as a dot product operation.

[0104] According to an embodiment of the present invention, the sample distorted heart spatiotemporal features are input into an image decoder to be trained for feature decoding and mapping processing to obtain a reconstructed ultrasound cardiac image.

[0105] According to an embodiment of the present invention, an image decoding loss value is calculated based on the reconstructed ultrasound image and the sample target ultrasound image. The image decoding loss value can be calculated according to formula (2), which is as follows.

[0106] (2);

[0107] in, It can be represented as the image decoding loss value, x t The sample target echocardiogram, G(z t ) can be characterized as a reconstructed echocardiogram, Can be characterized as expected computational operations.

[0108] According to an embodiment of the present invention, an image discriminator to be trained is used to perform image adversarial processing on a reconstructed ultrasound cardiac image to obtain an adversarial ultrasound cardiac image.

[0109] According to an embodiment of the present invention, the image discriminator to be trained may be a discriminator in the form of a block generative adversarial network (PatchGAN), which achieves the effect of adversarial processing by classifying patches in the image to obtain an adversarial ultrasound image.

[0110] According to an embodiment of the present invention, the countermeasure loss value is calculated based on the countermeasure ultrasound cardiogram. The countermeasure loss value can be calculated according to formula (3). Formula (3) is as follows.

[0111] (3);

[0112] in, It can be represented as the adversarial loss value, It can be characterized as an image obtained by performing image adversarial processing on the sample target ultrasound image using the image discriminator to be trained. Can be characterized as a counter echocardiogram, x r A sample reference echocardiogram may be characterized.

[0113] According to an embodiment of the present invention, based on the optical flow autoencoder loss function, an optical flow autoencoder loss value is obtained according to the reconstructed ultrasound image, the sample target ultrasound image, the adversarial ultrasound image and the balance parameter.

[0114] According to an embodiment of the present invention, an optical flow autoencoder loss value is calculated based on an image decoding loss value calculated based on a reconstructed ultrasound image and a sample target ultrasound image and an adversarial loss value calculated based on an adversarial ultrasound image. The optical flow autoencoder loss value can be calculated according to formula (4), which is as follows.

[0115] (4);

[0116] in, It can be represented as the optical flow autoencoder loss value, can be characterized as a balance parameter.

[0117] According to an embodiment of the present invention, the module parameters of the initial optical flow autoencoder submodule to be trained are adjusted according to the optical flow autoencoder loss value to obtain a trained initial optical flow autoencoder submodule.

[0118] According to an embodiment of the present invention, the trained optical flow autoencoder submodule includes a trained image encoder , trained optical flow prediction submodule and trained image decoder .

[0119] According to an embodiment of the present invention, an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained are obtained, two frames of sample ultrasound images are randomly selected from the sample ultrasound video as sample reference ultrasound images and sample target ultrasound images for training and verification, the sample reference ultrasound images are input into the image encoder to be trained for feature extraction processing, and sample reference heart spatiotemporal features are obtained, so that the training image encoder can extract accurate heart spatiotemporal features, and then the sample reference ultrasound images and sample target ultrasound images are input into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain sample reference optical flow and sample reference occlusion, and then the predicted sample reference optical flow and sample reference occlusion are used to distort the sample reference heart spatiotemporal features to obtain sample distorted heart spatiotemporal features, so as to train the optical flow prediction submodule to predict optical flow and occlusion based on the heart's spatiotemporal features, and at the same time, in combination with the image decoder, the image decoder is trained to reconstruct the image according to the image features, and after obtaining the reconstructed ultrasound image, the reconstructed ultrasound image is reconstructed using the introduced image discriminator to be trained. An image adversarial processing is performed to obtain an adversarial ultrasound image. Based on the optical flow autoencoder loss function, the optical flow autoencoder loss value is obtained according to the reconstructed ultrasound image, the sample target ultrasound image, the adversarial ultrasound image and the balance parameter. According to the optical flow autoencoder loss value, the module parameters of the initial optical flow autoencoder submodule to be trained are adjusted, and the image discriminator is discarded to obtain the trained initial optical flow autoencoder submodule. The learning and training of the image encoder, the optical flow prediction submodule and the image decoder in the initial optical flow autoencoder submodule are realized, so that the image encoder, the optical flow prediction submodule and the image decoder can process the image, feature and other information, so as to extract or reconstruct accurate feature or image information. Further, by training the optical flow prediction submodule, accurate optical flow and occlusion information can be predicted, so as to facilitate the training of the subsequent optical flow generation submodule. Further, by introducing the image discriminator, the reconstructed ultrasound image and the sample target ultrasound image are adversarially processed, so that the image decoder can be trained to reconstruct a reconstructed ultrasound image that is infinitely close to the real image, thereby improving the quality of the decoded image.

[0120] According to an embodiment of the present invention, a plurality of frames of sample ultrasound images are acquired from a sample ultrasound video based on a predetermined frame sequence.

[0121] According to an embodiment of the present invention, a sample initial ultrasound image may be determined first, and the sample initial ultrasound image may be used as the initial image. Based on a predetermined frame sequence, multiple frames of continuous sample ultrasound images including the sample initial ultrasound image may be acquired.

[0122] According to an embodiment of the present invention, multiple frames of sample ultrasound cardiac images are respectively input into a pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatiotemporal features.

[0123] According to an embodiment of the present invention, a plurality of sample cardiac spatiotemporal features are respectively input into a pre-trained optical flow prediction submodule for optical flow prediction processing to obtain sample labels, wherein the sample labels include a sample optical flow sequence and a sample occlusion sequence.

[0124] According to an embodiment of the present invention, each sample optical flow in the sample optical flow sequence represents the displacement change characteristics between the same pixel point in each two adjacent frames of sample ultrasound images, and each sample occlusion in the sample occlusion sequence represents the change characteristics of the area where the previous frame of the sample ultrasound image is occluded by the next frame of the sample ultrasound image in each two adjacent frames of sample ultrasound images.

[0125] According to an embodiment of the present invention, the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence are input into the optical flow generation submodule to be trained for feature distortion processing to obtain the sample optical flow occlusion sequence noise feature.

[0126] According to an embodiment of the present invention, a method for inputting sample Gaussian noise, sample optical flow sequence and sample occlusion sequence into an optical flow generation submodule to be trained for feature distortion processing to obtain noise features of a sample optical flow occlusion sequence includes the following operations.

[0127] According to an embodiment of the present invention, a sample optical flow sequence and a sample occlusion sequence are merged to generate a sample optical flow occlusion sequence.

[0128] According to an embodiment of the present invention, the sample optical flow sequence generated by the optical flow prediction submodule and the sample occlusion sequence The sample optical flow occlusion sequence is merged and used as the training data and sample data for training the optical flow generation submodule. The sample optical flow occlusion sequence can be calculated according to formula (5), which is as follows.

[0129] (5);

[0130] Among them, s0 can be represented as a sample optical flow occlusion sequence, F can be represented as a sample optical flow sequence, M can be represented as a sample occlusion sequence, n can be represented as the length of a predetermined frame sequence, h can be represented as the height of a sample ultrasound cardiac image, w can be represented as the width of a sample ultrasound cardiac image, and Cat[] can be represented as an operation of merging a sample optical flow sequence and a sample occlusion sequence.

[0131] According to an embodiment of the present invention, sample Gaussian noise is added to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence.

[0132] According to an embodiment of the present invention, multiple sample Gaussian noises may be added to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence containing noises added at different times.

[0133] According to an embodiment of the present invention, the optical flow occlusion noise sequence is input into the optical flow generation submodule to be trained for feature generation processing to obtain the sample optical flow occlusion sequence noise feature.

[0134] According to an embodiment of the present invention, a sample optical flow sequence and a sample occlusion sequence are merged to generate a sample optical flow occlusion sequence for training and verifying an optical flow generation submodule, sample Gaussian noise is added to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence, and the optical flow occlusion noise sequence is input into the optical flow generation submodule to be trained for feature generation processing to obtain the sample optical flow occlusion sequence noise feature, thereby realizing the training of the optical flow generation submodule using the optical flow sequence and occlusion sequence predicted by the trained optical flow prediction submodule as training data samples, thereby improving the output accuracy of the optical flow generation submodule, improving the correlation between the optical flow generation submodule and the image encoder and image decoder, and improving the robustness of the module.

[0135] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, the optical flow occlusion sequence noise loss value is obtained according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise.

[0136] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, a method for obtaining the optical flow occlusion sequence noise loss value includes the following operations.

[0137] According to an embodiment of the present invention, a plurality of left ventricular volume values ​​are obtained from a sample left ventricular volume time curve.

[0138] According to an embodiment of the present invention, a plurality of left ventricular volume values ​​and a first left ventricular volume value among the plurality of left ventricular volume values ​​are normalized to obtain a plurality of left ventricular volume ratios.

[0139] According to an embodiment of the present invention, the left ventricular volume ratio can be calculated according to formula (6), which is shown as follows.

[0140] (6);

[0141] in, It can be expressed as the ratio of left ventricular volume, It can be represented by multiple left ventricular volume values, and v0 can be represented by the first left ventricular volume value.

[0142] According to an embodiment of the present invention, a left ventricular volume ratio curve within a predetermined period is generated according to a plurality of left ventricular volume ratios.

[0143] According to an embodiment of the present invention, the left ventricular volume ratio curve is input into the enhanced embedded sequential model to extract the time feature, and the left ventricular volume time feature within a predetermined period is obtained. , where d can be represented as the feature dimension.

[0144] According to an embodiment of the present invention, the sample cardiac spatiotemporal features corresponding to the sample initial ultrasound cardiac image are expressed as , where c can be represented as the number of features.

[0145] According to an embodiment of the present invention, based on the optical flow occlusion sequence noise loss function, the optical flow occlusion sequence noise loss value is obtained according to the sample initial heart spatiotemporal characteristics, left ventricular volume time characteristics, sample optical flow occlusion sequence noise characteristics and sample Gaussian noise.

[0146] According to an embodiment of the present invention, the sample initial cardiac spatiotemporal feature characterizes the first sample cardiac spatiotemporal feature among multiple sample cardiac spatiotemporal features, that is, the sample cardiac spatiotemporal feature corresponding to the sample initial ultrasound cardiac image.

[0147] According to an embodiment of the present invention, the noise loss value of the optical flow occlusion sequence can be calculated according to formula (7), and formula (7) is as follows.

[0148] (7);

[0149] in, It can be represented as the noise loss value of the optical flow occlusion sequence. It can be characterized as the noise characteristics of the sample optical flow occlusion sequence predicted by the optical flow generation submodule to be trained, s t It can be represented as the optical flow occlusion noise sequence obtained by adding t times of sample Gaussian noise to the optical flow occlusion noise sequence, t can be represented as t times of sample Gaussian noise, z0 can be represented as the sample heart spatiotemporal characteristics corresponding to the sample initial ultrasound cardiac image, and q can be represented as the left ventricular volume time characteristics.

[0150] According to an embodiment of the present invention, multiple left ventricular volume values ​​are obtained from a sample left ventricular volume time curve, and the multiple left ventricular volume values ​​and the first left ventricular volume value among the multiple left ventricular volume values ​​are normalized to obtain multiple left ventricular volume ratios to eliminate the natural differences in heart size and volume among different individuals. The left ventricular volume ratio curve generated by the multiple left ventricular volume ratios is then input into an enhanced embedded sequential model to extract time features to obtain the left ventricular volume time features within a predetermined period, and further extract the temporal continuous features of the left ventricular volume ratio curve. Then, based on the optical flow occlusion sequence noise loss function, according to the sample initial heart spatiotemporal features, the left ventricular volume time features, the sample optical flow occlusion sequence noise features and the sample Gaussian noise, the optical flow occlusion sequence noise loss value is obtained, thereby obtaining the module loss parameters of the optical flow generation submodule, so as to adjust the module according to the module loss parameters to obtain a highly robust optical flow generation submodule.

[0151] According to an embodiment of the present invention, the module parameters of the optical flow generation submodule to be trained are adjusted according to the noise loss value of the optical flow occlusion sequence to obtain a trained optical flow generation submodule.

[0152] According to an embodiment of the present invention, a trained optical flow coding synthesis model is obtained based on a pre-trained image encoder, a trained optical flow generation submodule and a pre-trained image decoder.

[0153] According to an embodiment of the present invention, a sample optical flow sequence and a sample occlusion sequence, i.e., a data sample, for training an optical flow generation submodule are obtained by using a pre-trained initial optical flow autoencoder submodule, and then the optical flow generation submodule is trained for denoising and generation using the data sample, so as to obtain an optical flow coding synthesis model that can accurately generate a highly fine-grained target ultrasound video containing important cardiac semantic information such as heart rate and filling speed, so that the model can be used to process the initial ultrasound image and the left ventricular volume-time curve, thereby improving the efficiency and training efficiency of the generated target ultrasound image and target ultrasound video, reducing the training and learning cost, and improving the practicality and applicability of the optical flow coding synthesis model in a clinical environment.

[0154] Figure 3 A schematic diagram of generating a target ultrasound cardiac image using an optical flow coding synthesis model according to an embodiment of the present invention is shown.

[0155] like Figure 3 As shown, Figure 3The figure shows the generation of a target ultrasound image using an optical flow coding synthesis model. The first picture on the left is an initial ultrasound image. The initial ultrasound image and the left ventricular volume-time curve are input into the optical flow coding synthesis model. The black and white picture shows multiple frames of target ultrasound images generated by the optical flow coding synthesis model. The color picture shows an optical flow map including a target optical flow sequence and a target occlusion sequence. The bottom curve map represents the left ventricular volume-time curve. The optical flow coding synthesis model generates an optical flow map including a target optical flow sequence and a target occlusion sequence based on the left ventricular volume-time curve and the initial ultrasound image. The image features of the initial ultrasound image are distorted and decoded based on the optical flow map including the target optical flow sequence and the target occlusion sequence to obtain multiple frames of target ultrasound images.

[0156] Figure 4 A schematic diagram showing the workflow of the present invention according to an embodiment of the present invention is shown.

[0157] like Figure 4 As shown, Figure 4 The process of the present invention is shown as follows: the initial ultrasound cardiac image xr is input to the image encoder Get the initial cardiac spatiotemporal features, and transform the initial cardiac spatiotemporal features and Gaussian noise matrix and the left ventricular volume time curve are input to the optical flow generation submodule, I can be represented as a unit matrix, and the optical flow generation submodule predicts and generates the target optical flow sequence and target occlusion sequence , the initial heart spatiotemporal features are distorted according to the target optical flow sequence and the target occlusion sequence to obtain multiple distorted heart spatiotemporal features, and then the multiple distorted heart spatiotemporal features are input into the image decoder G (z t ) Output multiple frames of target ultrasound images , a target ultrasound video is generated according to multiple frames of target ultrasound images.

[0158] Figure 5 A schematic diagram showing the effect comparison between a target ultrasound cardiology video generated by the method of the present invention and a real video according to an embodiment of the present invention is shown.

[0159] like Figure 5 As shown, Figure 5The effect comparison between the target ultrasound video generated by the method of the present invention and the real video is shown, from top to bottom, the ultrasound map of the real video, the ultrasound map of the target ultrasound video, the left ventricular region thermal map of the real video, the left ventricular region thermal map of the target ultrasound video, and the left ventricular volume time curve map. By comparing the ultrasound map of the real video and the ultrasound map of the target ultrasound video, it can be seen that the error between the target ultrasound video generated by the method of the present invention and the real video is extremely small. By comparing the thermal map of the real video and the thermal map of the target ultrasound video, it can be seen that the error between the left ventricular region in the target ultrasound video generated by the method of the present invention and the left ventricular region in the real video is extremely small. In the left ventricular volume time curve map, the blue curve is the real left ventricular volume time curve, and the orange curve is the left ventricular volume time curve generated by the left ventricular deep segmentation model of the present invention. It can be seen from the figure that the error between the left ventricular volume time curve generated by the left ventricular deep segmentation model of the present invention and the real left ventricular volume time curve is extremely small.

[0160] Figure 6a A schematic diagram showing waveform correlation between left ventricular volume time curves of target ultrasound cardiography videos generated by the method of the present invention and real videos at different times for 1277 ultrasound videos according to an embodiment of the present invention. Figure 6b A schematic diagram showing the mean square error of left ventricular volume time curves of 1277 ultrasound videos at different times according to an embodiment of the present invention and the real video using the method of the present invention is shown. Figure 6c A schematic diagram showing waveform correlation between left ventricular volume time curves of target ultrasound cardiography videos generated by the method of the present invention and real videos at different frame rates for 1277 ultrasound videos according to an embodiment of the present invention is shown. Figure 6d A schematic diagram showing the mean square error of the left ventricular volume time curve of 1277 ultrasound videos at different frame rates according to an embodiment of the present invention and the target ultrasound cardiography video generated by the method of the present invention and the real video.

[0161] like Figure 6a~6d As shown, Figure 6a~6d The waveform correlation between the left ventricular volume time curve of the target ultrasound video generated by the method of the present invention and the real video at different times and different frame rates is shown. FPS can be represented as the frame rate of the target ultrasound video, and Time can be represented as the duration of the generated target ultrasound video. Figure 6a and 6c The horizontal axis in can be represented as waveform correlation, and the vertical axis can be represented as distribution function. Figure 6b and 6dThe horizontal axis in can be represented as the mean square error, and the vertical axis can be represented as the distribution function. Figure 6a~6d It can be seen that the error between the left ventricular volume time curve of the target ultrasound cardiography video generated by the method of the present invention and the real video is extremely small.

[0162] Figure 7 A structural block diagram of an apparatus for generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown.

[0163] like Figure 7 As shown, the device for generating an echocardiogram based on a left ventricular volume curve in this embodiment includes: an acquisition module 710 and a feature distortion mapping module 720 .

[0164] The acquisition module 710 is used to acquire an initial ultrasound cardiac image and a left ventricular volume time curve, wherein the left ventricular volume time curve represents the volume change characteristics of the left ventricle of the heart within a predetermined period of time. The acquisition module 710 can be used to perform the operation S210 described above, which will not be described in detail here.

[0165] The feature warping mapping module 720 is used to perform feature warping mapping processing on the initial ultrasound image based on the left ventricular volume time curve using the optical flow coding synthesis model to obtain a multi-frame target ultrasound image, wherein the optical flow coding synthesis model is obtained by training the initial model using the multi-frame ultrasound images, the sample left ventricular volume time curve and the sample Gaussian noise in the sample ultrasound video, according to the sample labels corresponding to the multi-frame ultrasound images, the multi-frame target ultrasound images indicate the temporal change characteristics of multiple cardiac functions of the heart within a predetermined period of time, and the multi-frame target ultrasound images are used to generate the target ultrasound video. The feature warping mapping module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0166] According to an embodiment of the present invention, the feature warp mapping module 720 includes: a first acquisition submodule, a first extraction submodule, a first warp submodule and a first decoding submodule.

[0167] The first acquisition submodule is used to acquire a Gaussian noise matrix for generating a target ultrasound cardiac image.

[0168] The first extraction submodule is used to input the initial ultrasound cardiac image into the image encoder of the optical flow coding synthesis model for feature extraction processing to obtain initial heart spatiotemporal features corresponding to the heart.

[0169] The first distortion submodule is used to input the Gaussian noise matrix, the left ventricular volume-time curve and the initial cardiac spatiotemporal features into the optical flow generation submodule of the optical flow encoding synthesis model. Based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation submodel performs feature distortion processing on the Gaussian noise matrix to obtain multiple distorted cardiac spatiotemporal features.

[0170] The first decoding submodule is used to input multiple distorted cardiac spatiotemporal features into an image decoder of an optical flow coding synthesis model for feature decoding and mapping processing to obtain multiple frames of target ultrasound cardiac images.

[0171] According to an embodiment of the present invention, the first warping submodule includes: a first denoising unit and a first warping unit.

[0172] The first denoising unit is used to denoise the Gaussian noise matrix based on the initial cardiac spatiotemporal characteristics and the left ventricular volume-time curve, and the optical flow generation submodule to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve, wherein each target optical flow in the target optical flow sequence represents a displacement change feature between a pixel point in the initial ultrasonic cardiac image and a pixel point in each frame of the target ultrasonic cardiac image, and each target occlusion in the target occlusion sequence represents a change feature of an area of ​​the initial ultrasonic cardiac image occluded by each frame of the target ultrasonic cardiac image.

[0173] The first warping unit is used to warp the initial heart spatiotemporal features using the target optical flow sequence and the target occlusion sequence to obtain a plurality of warped heart spatiotemporal features.

[0174] According to an embodiment of the present invention, the acquisition module 710 includes: a second acquisition submodule, a first segmentation submodule and a first generation submodule.

[0175] The second acquisition submodule is used to acquire an initial ultrasonic cardiology video, wherein the initial ultrasonic cardiology video includes multiple frames of ultrasonic cardiology single-mode images.

[0176] The first segmentation submodule is used to perform depth estimation processing on the initial ultrasound cardiology video using the left ventricular depth segmentation model to obtain a relative depth estimation value corresponding to each pixel point located in the left ventricular region in each frame of the ultrasound cardiology single-modality image, wherein the relative depth estimation value represents the depth of each pixel point located in the left ventricular region.

[0177] The first generating submodule is used to generate a left ventricular volume time curve according to a plurality of relative depth estimation values ​​in each frame of the ultrasonic cardiology single-modality image.

[0178] According to an embodiment of the present invention, the device for generating a cardiac video based on an electromagnetic signal further includes: a training module.

[0179] According to an embodiment of the present invention, the training module includes: a third acquisition submodule, a fourth acquisition submodule, a second extraction submodule, a first prediction submodule, a second distortion submodule, a first obtainment submodule, a first adjustment submodule and a second obtainment submodule.

[0180] The third acquisition submodule is used to acquire an initial model and a training sample data set, wherein the initial model includes an initial optical flow autoencoder submodule and an optical flow generation submodule to be trained, the initial optical flow autoencoder submodule includes a pre-trained image encoder, a pre-trained optical flow prediction submodule, and a pre-trained image decoder, and the training sample data set includes a sample ultrasound cardiology video, a sample left ventricular volume time curve, and a sample Gaussian noise.

[0181] The fourth acquisition submodule is used to acquire multiple frames of sample ultrasound images from the sample ultrasound video based on a predetermined frame sequence.

[0182] The second extraction submodule is used to input multiple frames of sample ultrasound cardiac images into the pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatiotemporal features;

[0183] The first prediction submodule is used to input multiple sample heart spatiotemporal features into the pre-trained optical flow prediction submodule for optical flow prediction processing to obtain sample labels, wherein the sample labels include sample optical flow sequences and sample occlusion sequences.

[0184] The second distortion submodule is used to input the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise feature of the sample optical flow occlusion sequence.

[0185] The first submodule is used to obtain the optical flow occlusion sequence noise loss value based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise.

[0186] The first adjustment submodule is used to adjust the module parameters of the optical flow generation submodule to be trained according to the noise loss value of the optical flow occlusion sequence to obtain a trained optical flow generation submodule.

[0187] The second submodule is used to obtain an optical flow coding synthesis model according to the pre-trained image encoder, the trained optical flow generation submodule and the pre-trained image decoder.

[0188] According to an embodiment of the present invention, the second twisting submodule includes: a first merging unit, a first adding unit and a first obtaining unit.

[0189] The first merging unit is used to merge the sample optical flow sequence and the sample occlusion sequence to generate a sample optical flow occlusion sequence sample.

[0190] The first adding unit is used to add sample Gaussian noise to the sample optical flow occlusion sequence samples to obtain an optical flow occlusion noise sequence.

[0191] The first obtaining unit is used to input the optical flow occlusion noise sequence into the optical flow generation submodule to be trained for feature generation processing to obtain the sample optical flow occlusion sequence noise feature.

[0192] According to an embodiment of the present invention, the first obtaining submodule includes: a first obtaining unit, a first normalizing unit, a first generating unit, a second obtaining unit and a third obtaining unit.

[0193] The first acquisition unit is used to acquire a plurality of left ventricular volume values ​​from the sample left ventricular volume time curve.

[0194] The first normalization unit is used to perform normalization processing on the multiple left ventricular volume values ​​and the first left ventricular volume value among the multiple left ventricular volume values ​​to obtain multiple left ventricular volume ratios.

[0195] The first generating unit is used to generate a left ventricular volume ratio curve within a predetermined time period according to a plurality of left ventricular volume ratios.

[0196] The second obtaining unit is used to input the left ventricular volume ratio curve into the enhanced embedded sequential model to extract the time feature, and obtain the left ventricular volume time feature within a predetermined time period.

[0197] The third obtaining unit is used to obtain the optical flow occlusion sequence noise loss value based on the optical flow occlusion sequence noise loss function, according to the sample initial cardiac spatiotemporal characteristics, the left ventricular volume time characteristics, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, wherein the sample initial cardiac spatiotemporal characteristics represent the first sample cardiac spatiotemporal characteristics among multiple sample cardiac spatiotemporal characteristics.

[0198] According to an embodiment of the present invention, the training module also includes: a fifth acquisition submodule, a sixth acquisition submodule, a third extraction submodule, a second prediction submodule, a third distortion submodule, a first reconstruction submodule, a first adversarial submodule, a third obtaining submodule and a fourth obtaining submodule.

[0199] The fifth acquisition submodule is used to acquire an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained, wherein the initial optical flow autoencoder model to be trained includes an image encoder to be trained, an optical flow prediction submodule to be trained, and an image decoder to be trained.

[0200] The sixth acquisition submodule is used to acquire a sample reference ultrasound image and a sample target ultrasound image from the sample ultrasound video.

[0201] The third extraction submodule is used to input the sample reference ultrasound cardiac image into the image encoder to be trained for feature extraction processing to obtain the sample reference cardiac spatiotemporal features.

[0202] The second prediction submodule is used to input the sample reference ultrasound image and the sample target ultrasound image into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain the sample reference optical flow and the sample reference occlusion, wherein the sample reference optical flow represents the displacement change characteristics between the pixel points in the sample reference ultrasound image and the pixel points in the sample target ultrasound image, and the sample reference occlusion represents the change characteristics of the area of ​​the sample reference ultrasound image occluded by the sample target ultrasound image.

[0203] The third distortion submodule is used to distort the sample reference heart spatiotemporal features by using the sample reference optical flow sequence and the sample reference occlusion sequence to obtain the sample distorted heart spatiotemporal features.

[0204] The first reconstruction submodule is used to input the sample distorted heart spatiotemporal features into the image decoder to be trained for feature decoding and mapping processing to obtain a reconstructed ultrasound cardiac image.

[0205] The first confrontation submodule is used to perform image confrontation processing on the reconstructed ultrasonic cardiac image using the image discriminator to be trained to obtain a confrontation ultrasonic cardiac image.

[0206] The third submodule is used to obtain the optical flow autoencoder loss value based on the optical flow autoencoder loss function according to the reconstructed ultrasound image, the sample target ultrasound image, the adversarial ultrasound image and the balance parameter.

[0207] Fourth, a submodule is obtained, which is used to adjust the module parameters of the initial optical flow autoencoder submodule to be trained according to the optical flow autoencoder loss value to obtain the trained initial optical flow autoencoder submodule.

[0208] According to an embodiment of the present invention, any multiple modules in the acquisition module 710 and the feature distortion mapping module 720 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 710 and the feature distortion mapping module 720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 710 and the feature distortion mapping module 720 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0209] Figure 8 A block diagram of an electronic device for a method of generating an echocardiogram based on a left ventricular volume curve according to an embodiment of the present invention is shown.

[0210] like Figure 8 As shown, the electronic device according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0211] In RAM 803, various programs and data required for the operation of the electronic device are stored. The processor 801, ROM 802 and RAM 803 are connected to each other via a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. The processor 801 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0212] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0213] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0214] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0215] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method for generating an echocardiogram based on a left ventricular volume curve provided by the embodiment of the present invention.

[0216] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the processor 801 executes the computer program. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0217] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0218] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0219] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

Claims

1. A method for generating an echocardiogram based on a left ventricular volume curve, characterized in that: include: Acquiring an initial ultrasound cardiac image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve represents a volume change characteristic of the left ventricle of the heart within a predetermined period of time; Based on the left ventricular volume-time curve, the initial ultrasound image is subjected to feature distortion mapping processing using an optical flow coding synthesis model to obtain a multi-frame target ultrasound image, wherein the optical flow coding synthesis model is obtained by training an initial model using a multi-frame sample ultrasound image, a sample left ventricular volume-time curve and a sample Gaussian noise in a sample ultrasound video, according to sample labels corresponding to the multi-frame sample ultrasound images, the multi-frame target ultrasound image indicates the temporal variation characteristics of multiple cardiac functions of the heart within the predetermined time period, and the multi-frame target ultrasound image is used to generate a target ultrasound video.

2. The method according to claim 1, characterized in that in, The optical flow coding synthesis model includes an image encoder, an optical flow generation submodule and an image decoder. The method of performing feature distortion mapping processing on the initial ultrasound cardiac image based on the left ventricular volume-time curve using an optical flow coding synthesis model to obtain multiple frames of target ultrasound cardiac images includes: Acquire a Gaussian noise matrix for generating the target ultrasound cardiac image; Inputting the initial ultrasound cardiac image into the image encoder of the optical flow coding synthesis model for feature extraction processing to obtain initial cardiac spatiotemporal features corresponding to the heart; The Gaussian noise matrix, the left ventricular volume time curve and the initial cardiac spatiotemporal features are input into the optical flow generation submodule of the optical flow coding synthesis model; based on the initial cardiac spatiotemporal features and the left ventricular volume time curve, the optical flow generation submodel performs feature distortion processing on the Gaussian noise matrix to obtain a plurality of distorted cardiac spatiotemporal features; The multiple distorted cardiac spatiotemporal features are input into the image decoder of the optical flow coding synthesis model for feature decoding and mapping processing to obtain the multiple frames of target ultrasound cardiac images.

3. The method according to claim 2, characterized in that Based on the initial cardiac spatiotemporal features and the left ventricular volume-time curve, the optical flow generation sub-model performs feature distortion processing on the Gaussian noise matrix to obtain a plurality of distorted cardiac spatiotemporal features, including: Based on the initial cardiac spatiotemporal characteristics and the left ventricular volume-time curve, the optical flow generation submodule performs denoising on the Gaussian noise matrix to generate a target optical flow sequence and a target occlusion sequence corresponding to the left ventricular volume-time curve, wherein each target optical flow in the target optical flow sequence represents a displacement change feature between a pixel point in the initial ultrasonic cardiac image and a pixel point in each frame of the target ultrasonic cardiac image, and each target occlusion in the target occlusion sequence represents a change feature of an area of ​​the initial ultrasonic cardiac image occluded by each frame of the target ultrasonic cardiac image; The initial cardiac spatiotemporal features are distorted using the target optical flow sequence and the target occlusion sequence to obtain the multiple distorted cardiac spatiotemporal features.

4. The method according to claim 1, characterized in that: Obtain left ventricular volume-time curve, including: Acquiring an initial ultrasound video, wherein the initial ultrasound video includes multiple frames of ultrasound single-mode images; Performing depth estimation processing on the initial ultrasound video using a left ventricle depth segmentation model to obtain a relative depth estimation value corresponding to each pixel located in the left ventricle region in each frame of the ultrasound single-modality image, wherein the relative depth estimation value represents the depth of each pixel located in the left ventricle region; The left ventricular volume time curve is generated according to a plurality of the relative depth estimation values ​​in each frame of the echocardiography single modality image.

5. The method according to claim 1, characterized in that The optical flow coding synthesis model is trained in the following way: Acquire the initial model and the training sample data set, wherein the initial model includes an initial optical flow autoencoder submodule and an optical flow generation submodule to be trained, the initial optical flow autoencoder submodule includes a pre-trained image encoder, a pre-trained optical flow prediction submodule, and a pre-trained image decoder, and the training sample data set includes the sample ultrasound video, the sample left ventricular volume time curve, and the sample Gaussian noise; Based on a predetermined frame sequence, acquiring a plurality of frames of sample ultrasound images from the sample ultrasound video; Inputting the multiple frames of sample ultrasound cardiac images into the pre-trained image encoder for feature extraction processing to obtain multiple sample cardiac spatiotemporal features; Inputting the plurality of sample heart spatiotemporal features into the pre-trained optical flow prediction submodule for optical flow prediction processing to obtain the sample label, wherein the sample label includes a sample optical flow sequence and a sample occlusion sequence; Inputting the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise feature of the sample optical flow occlusion sequence; Based on the optical flow occlusion sequence noise loss function, according to the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, an optical flow occlusion sequence noise loss value is obtained; According to the noise loss value of the optical flow occlusion sequence, adjusting the module parameters of the optical flow generation submodule to be trained to obtain a trained optical flow generation submodule; The optical flow coding synthesis model is obtained according to the pre-trained image encoder, the trained optical flow generation submodule and the pre-trained image decoder.

6. The method according to claim 5, characterized in that The step of inputting the sample Gaussian noise, the sample optical flow sequence and the sample occlusion sequence into the optical flow generation submodule to be trained for feature distortion processing to obtain the noise feature of the sample optical flow occlusion sequence comprises: Merging the sample optical flow sequence and the sample occlusion sequence to generate a sample optical flow occlusion sequence; Adding the sample Gaussian noise to the sample optical flow occlusion sequence to obtain an optical flow occlusion noise sequence; The optical flow occlusion noise sequence is input into the optical flow generation submodule to be trained for feature generation processing to obtain the sample optical flow occlusion sequence noise feature.

7. The method according to claim 5, characterized in that The optical flow occlusion sequence noise loss function is based on the sample left ventricular volume time curve, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise to obtain the optical flow occlusion sequence noise loss value: Acquiring a plurality of left ventricular volume values ​​from the sample left ventricular volume-time curve; Normalizing the multiple left ventricular volume values ​​and the first left ventricular volume value among the multiple left ventricular volume values ​​to obtain multiple left ventricular volume ratios; generating a left ventricular volume ratio curve within the predetermined period according to the plurality of left ventricular volume ratios; Inputting the left ventricular volume ratio curve into the enhanced embedded sequential model to extract the time feature, and obtaining the left ventricular volume time feature within the predetermined time period; Based on the optical flow occlusion sequence noise loss function, the optical flow occlusion sequence noise loss value is obtained according to the sample initial cardiac spatiotemporal characteristics, the left ventricular volume time characteristics, the sample optical flow occlusion sequence noise characteristics and the sample Gaussian noise, wherein the sample initial cardiac spatiotemporal characteristics represent the first sample cardiac spatiotemporal characteristics among the multiple sample cardiac spatiotemporal characteristics.

8. The method according to claim 5, characterized in that The initial optical flow autoencoder submodule is trained in the following manner, including: Acquire an initial optical flow autoencoder submodule to be trained and an image discriminator to be trained, wherein the initial optical flow autoencoder model to be trained includes an image encoder to be trained, an optical flow prediction submodule to be trained, and an image decoder to be trained; acquiring a sample reference echocardiogram and a sample target echocardiogram from the sample echocardiogram video; Inputting the sample reference ultrasound cardiac image into the image encoder to be trained for feature extraction processing to obtain sample reference cardiac spatiotemporal features; Inputting the sample reference ultrasound image and the sample target ultrasound image into the optical flow prediction submodule to be trained for optical flow prediction processing to obtain a sample reference optical flow and a sample reference occlusion, wherein the sample reference optical flow represents a displacement change feature between a pixel point in the sample reference ultrasound image and a pixel point in the sample target ultrasound image, and the sample reference occlusion represents a change feature of an area of ​​the sample reference ultrasound image occluded by the sample target ultrasound image; Using the sample reference optical flow sequence and the sample reference occlusion sequence to distort the sample reference heart spatiotemporal features, to obtain sample distorted heart spatiotemporal features; Inputting the sample distorted cardiac spatiotemporal features into the image decoder to be trained for feature decoding and mapping processing to obtain a reconstructed ultrasound cardiac image; Using the image discriminator to be trained to perform image adversarial processing on the reconstructed ultrasonic cardiac image to obtain an adversarial ultrasonic cardiac image; Based on the optical flow autoencoder loss function, an optical flow autoencoder loss value is obtained according to the reconstructed ultrasound image, the sample target ultrasound image, the adversarial ultrasound image and a balance parameter; According to the optical flow autoencoder loss value, the module parameters of the initial optical flow autoencoder submodule to be trained are adjusted to obtain the trained initial optical flow autoencoder submodule.

9. A device for generating an echocardiogram based on a left ventricular volume curve, characterized in that: include: An acquisition module, used for acquiring an initial ultrasound cardiac image and a left ventricular volume-time curve, wherein the left ventricular volume-time curve represents a volume change characteristic of the left ventricle of the heart within a predetermined period of time; A feature warping mapping module is used to perform feature warping mapping processing on the initial ultrasound image based on the left ventricular volume-time curve using an optical flow coding synthesis model to obtain a multi-frame target ultrasound image, wherein the optical flow coding synthesis model is obtained by training an initial model using a multi-frame ultrasound image in a sample ultrasound video, a sample left ventricular volume-time curve and a sample Gaussian noise according to sample labels corresponding to the multi-frame ultrasound images, the multi-frame target ultrasound image indicates the temporal variation characteristics of multiple cardiac functions of the heart within the predetermined time period, and the multi-frame target ultrasound image is used to generate a target ultrasound video.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 8.

Images