Fetal image generation method and apparatus, electronic device, and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for generating fetal images. The generation method includes: processing ultrasound data of the current fetus to obtain fetal anatomy information; acquiring fetal image prompt information; and generating a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information based on the fetal anatomy information and the fetal image prompt information using an image generation model. On one hand, the fetal anatomy information in the above solution is obtained based on the ultrasound data of the current fetus; therefore, the first fetal image generated accordingly matches the actual image of the fetus in the pregnant woman's womb, which is beneficial for improving scene adaptability. On the other hand, generating a first fetal image based on fetal image prompt information can meet the personalized viewing needs of pregnant women or their families, which is beneficial for improving the viewing experience of fetal images.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method for generating a fetal image, a device for generating a fetal image, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] With the continuous development of medical imaging technology, ultrasound imaging technology can be used to generate images of the fetus, allowing parents to visually observe the fetus's condition. For example, a 3D ultrasound system can be used to perform ultrasound imaging on the pregnant woman's abdomen to obtain a static image of the fetus. A 4D ultrasound system can also be used to continuously perform ultrasound imaging on the pregnant woman's abdomen to obtain a video of the fetus.

[0003] However, the fetal images and videos obtained through the above methods can only represent the state of the fetus at the time of ultrasound acquisition. The ultrasound images or videos obtained using traditional methods are insufficient to meet the personalized viewing needs of pregnant women or their families, thus hindering the improvement of their viewing experience. Therefore, how to improve the viewing experience of fetal images while ensuring accuracy is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The present invention was proposed in view of the above-mentioned problems. The present invention provides a method for generating a fetal image, an apparatus for generating a fetal image, an electronic device, a storage medium, and a computer program product.

[0005] According to one aspect of the present invention, a method for generating a fetal image is provided, the method comprising:

[0006] The ultrasound data of the current fetus is processed to obtain the fetal morphology information.

[0007] Obtain fetal image prompts;

[0008] Based on fetal physical information and fetal image cues, an image generation model is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image cues.

[0009] For example, fetal facial information is represented using a second fetal image, and the image generation model includes: an encoding module, a denoising module, and a decoding module.

[0010] Based on fetal facial features and fetal image cues, an image generation model is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image cues, including:

[0011] Based on the second fetal image, a first image feature is generated through an encoding module, and random noise is added to the first image feature to obtain the second image feature, wherein the first image feature includes the physical features of the current fetus;

[0012] Based on the fetal image prompt information, the second image features are denoised using a denoising module to obtain the third image features;

[0013] The third image features are decoded using the decoding module to obtain the first fetal image.

[0014] For example, based on fetal image prompt information, a denoising module is used to denoise the second image features to obtain the third image features, including:

[0015] The gender information in the fetal image prompt information is removed. Based on the fetal image prompt information and neutral information after removing the gender information, the second image features are denoised by the denoising module to obtain the third image features that include the semantics of the fetal image prompt information and neutral information.

[0016] For example, fetal facial information is represented using a second fetal image. Based on the fetal facial information and fetal image cues, an image generation model is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image cues, including:

[0017] Fetal reconstruction processing is performed on the second fetal image to obtain a third fetal image, wherein the current fetal morphological information in the third fetal image is more than the current fetal morphological information in the fetal morphological information.

[0018] Based on the third fetal image and fetal image prompts, the first fetal image is generated using an image generation model.

[0019] For example, performing fetal reconstruction processing on a second fetal image to obtain a third fetal image includes:

[0020] The second fetal image is input into the image reconstruction model to obtain the third fetal image;

[0021] The image reconstruction model is trained through the following steps:

[0022] The first training image of the baseline fetus is input into the untrained image reconstruction model to obtain the first predicted image, wherein the first predicted image contains more physical information of the baseline fetus than the first training image.

[0023] Based on the first predicted image, a confidence score for the first predicted image is generated, wherein the confidence score is used to represent the probability that the first predicted image is the original ultrasound image;

[0024] Adjust the model parameters of the untrained image reconstruction model based on the confidence level.

[0025] For example, the image reconstruction model includes a global prediction module and a local prediction module.

[0026] The second fetal image is input into the image reconstruction model to obtain the third fetal image, including:

[0027] The second fetal image is input into the global prediction module and the local prediction module respectively to generate global prediction features and local prediction features respectively. The global prediction features are used to represent the overall visual features of the current fetus, and the local prediction features are used to represent the local visual features of the current fetus.

[0028] The global prediction features and local prediction features are fused to obtain the fused prediction features.

[0029] The fused predictive features are decoded to generate a third fetal image.

[0030] For example, the fetal image prompting information also includes dynamic descriptive information;

[0031] The above generation method also includes:

[0032] Perform a three-dimensional ultrasound scan of the fetus to obtain ultrasound data for the target time period;

[0033] The ultrasound data of the current fetus is processed to obtain the fetal morphological information, including:

[0034] Extract the physical features of the fetus from ultrasound data at at least two moments within the target time period to obtain the current dynamic physical features of the fetus.

[0035] The above generation method also includes:

[0036] Based on the first fetal image, a dynamic video of the current fetus is generated using a video generation model. The video generation model is used to generate the dynamic video based on dynamic physical features and dynamic descriptive information.

[0037] For example, obtaining fetal image prompting information includes:

[0038] Obtain a reference fetal video;

[0039] Extracting dynamic features from reference fetal videos;

[0040] Dynamic descriptive information is generated based on the extracted dynamic features to obtain fetal image prompts.

[0041] For example, fetal facial information is represented using a second fetal image. Based on the fetal facial information and fetal image cues, an image generation model is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image cues, including:

[0042] Obtain the first trace region for the second fetal image;

[0043] Based on the image corresponding to the first trace region in the second fetal image and the fetal image prompt information, a local fetal image is generated through an image generation model;

[0044] The portion of the local fetal image and the second fetal image outside the first trace region are fused to obtain the first fetal image.

[0045] For example, the above generation method further includes:

[0046] Obtain the second trace region for the first fetal image;

[0047] Image processing operations are performed on the image corresponding to the second trace region in the first fetal image to obtain the fourth fetal image;

[0048] The image processing operations include:

[0049] Image enhancement; or

[0050] Image removal and image interpolation.

[0051] For example, obtaining a second trace region for a first fetal image includes:

[0052] Image segmentation is performed on the current fetus in the first fetal image to determine the portion within the outline of the current fetus as the second trace region.

[0053] For example, the fetal physical information includes the fetal three-dimensional physical features; the image generation model is a three-dimensional image generation model;

[0054] Based on fetal facial features and fetal image cues, an image generation model is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image cues, including:

[0055] Based on the fetal three-dimensional physical features and fetal image prompts, a three-dimensional fetal image of the current fetus that conforms to the semantics of the fetal image prompts is generated through a three-dimensional image generation model.

[0056] The three-dimensional image of the fetus is rendered to obtain the first fetal image.

[0057] According to another aspect of the present invention, a fetal image generation apparatus is also provided, the apparatus comprising: an ultrasound image processing module, a prompt information acquisition module, and an image generation module.

[0058] The ultrasound image processing module is used to process the ultrasound data of the current fetus to obtain the fetal morphology information.

[0059] The prompt information acquisition module is used to acquire fetal image prompt information;

[0060] The image generation module is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information, based on fetal appearance information and fetal image prompt information, through an image generation model.

[0061] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory and a processor, wherein: the memory is used to store a computer program; and the processor is used to execute the computer program to implement the above-described method for generating fetal images.

[0062] According to another aspect of the present invention, a storage medium is also provided, which stores a computer program / instructions, which, when executed, are used to perform the above-described method for generating fetal images.

[0063] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions which, when executed, are used to perform the above-described method for generating fetal images.

[0064] According to the above-described solution provided by the present invention, the ultrasound data of the current fetus can be processed to obtain the fetal appearance information. Then, fetal image prompt information can be obtained. Finally, based on the fetal appearance information and the fetal image prompt information, a first fetal image conforming to the semantics of the fetal image prompt information can be generated using an image generation model. On the one hand, the fetal appearance information in the above solution is obtained based on the ultrasound data of the current fetus; therefore, the first fetal image generated accordingly conforms to the actual image of the fetus in the pregnant woman's womb, possessing a certain degree of realism and improving scene adaptability. On the other hand, generating the first fetal image based on the fetal image prompt information can meet the personalized viewing needs of pregnant women or their families, thus enhancing the viewing experience of fetal images. Attached Figure Description

[0065] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0066] Figure 1 A schematic flowchart of a method for generating a fetal image according to an embodiment of the present invention is shown;

[0067] Figure 2 A schematic diagram showing a first fetal image and a second fetal image according to an embodiment of the present invention is illustrated;

[0068] Figure 3 A schematic diagram showing incomplete images of a second and a third fetus according to an embodiment of the present invention is provided.

[0069] Figure 4 A schematic block diagram of an apparatus for generating fetal images according to an embodiment of the present invention is shown; and

[0070] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0072] In related technologies, fetal images or videos obtained from ultrasound body data can only represent the fetus's state at the time of ultrasound acquisition. Ultrasound images or videos obtained using traditional methods may not meet the viewing needs of pregnant women or their families. For example, due to the attenuation characteristics of ultrasound waves, fetal images or videos generated from ultrasound body data have fewer image details and poor realism. Furthermore, the possible fetal positions in fetal images or videos are limited, making it difficult to meet the personalized viewing needs of viewers. Therefore, fetal images and videos in related technologies negatively impact the viewing experience for pregnant women or their families.

[0073] To at least partially solve the above problems, embodiments of the present invention provide a method for generating fetal images. Figure 1 A schematic flowchart illustrating a method for generating a fetal image according to an embodiment of the present invention is shown. Figure 1 As shown, the method may include the following steps S110 to S130.

[0074] In step S110, the ultrasound data of the current fetus is processed to obtain the fetal morphology information of the current fetus.

[0075] Ultrasound waves can be emitted onto the pregnant woman's abdomen using an ultrasound device, and the echoes can be received to generate three-dimensional ultrasound data of the fetus. For example, each voxel in the ultrasound data can correspond to an echo intensity of the ultrasound wave.

[0076] The aforementioned fetal physical appearance information can be used to represent the visual characteristics of the current fetus's appearance. For example, this information can be text descriptions, descriptive data, ultrasound images, etc., and can specifically represent age (or gestational age), sex, skin color, style, and other related information. Specifically, the information may include text descriptions such as "head tilting up," "square face," etc., or text descriptions of measurements such as interocular distance, mouth width, jaw width, and head circumference. Furthermore, point cloud data and mesh contour data of the current fetus's outline can be generated based on ultrasound body data to serve as fetal physical appearance information. Additionally, the aforementioned three-dimensional ultrasound body data can be rendered (e.g., surface rendering, volume rendering, etc.) to obtain a rendered image. This rendered image is based on the current fetal ultrasound body data, and therefore can represent the current fetus's visual characteristics.

[0077] In step S120, fetal image prompt information is obtained.

[0078] The aforementioned fetal image cues can be used to represent information related to the current appearance of the fetus that an observer (such as the pregnant woman or her family mentioned above) expects, i.e., expected physical characteristics. It should be understood that these fetal image cues can be a single word, a short phrase, a combination of multiple words, a combination of multiple short phrases, or a combination of words and short phrases, and can also be represented in the form of images, videos, etc. For example, the aforementioned fetal image cues may include multiple words such as "short hair" and "lying on one's side." As another example, the aforementioned fetal image cues may include multiple short phrases such as "smiling and waving," "short-haired child," "2-year-old child," "cartoon effect," "improve image clarity," and "child growing from 0 to 12 years old"; they can also be reference images or videos uploaded by the user.

[0079] In one example, an input box can be displayed in the user interface, and the observer can then input text using an input device (e.g., keyboard, microphone, etc.) as the aforementioned fetal image prompt. In another example, multiple alternative words or phrases can be displayed in the user interface, and the observer can select an alternative word or phrase using an input device (e.g., mouse, trackball, etc.) as the aforementioned fetal image prompt. In yet another example, the user interface can support image and video input. The observer can input images or videos of other fetuses, infants, or children, extract feature information from the images or videos, and use at least a portion of the extracted feature information as the fetal image prompt. For example, based on the extracted feature information, feature text, feature images, or feature videos can be displayed in the user interface, and the observer can select the desired prompt feature as the fetal image prompt. Furthermore, after extracting feature information from images or videos, feature adjustments can be made using image or video editing tools. For example, based on user input, dynamic effects such as automatic rotation and dynamic textures can be added to images or videos to generate more visually impactful image or video features as fetal image prompts.

[0080] In step S130, based on fetal appearance information and fetal image prompt information, a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information is generated through an image generation model.

[0081] The first fetal image is an image obtained by integrating fetal physical information and fetal image prompt information. It is an image generated based on the integration of real physical information and expected physical information. Therefore, its corresponding appearance can be fetal appearance, infant appearance, child appearance, or even adult appearance.

[0082] The aforementioned image generation model can be a machine learning model in related technologies, such as convolutional neural networks, recurrent neural networks, generators in generative adversarial networks, stable diffusion models, etc. It should be understood that machine learning models typically process feature vectors to obtain the desired results; therefore, fetal physical information can also be textual descriptive features obtained by feature extraction from textual descriptive information, image features obtained by feature extraction from images, data features obtained by feature extraction from point cloud data, grid data, etc.

[0083] The training process of this image generation model may include: acquiring a training dataset; iteratively training the image generation model using the training dataset, and adjusting the model parameters of the image generation model using a loss function until training is complete. The training dataset may include multiple training images. For each training image, the training image may correspond to fetal morphological information and fetal image prompts of a baseline fetus. If the fetal morphological information is textual description information or textual description features, the training image may be an image that conforms to the semantics of the fetal image prompts and the fetal morphological information. If the fetal morphological information is an image, image features, or contour data, the training image may be an image generated based on the image, image features, or contour data that conforms to the semantics of the fetal image prompts. For example, the above training image can be drawn based on the fetal morphological information and fetal image prompts of a baseline fetus. This training image conforms to the semantics of the fetal image prompts and the fetal morphological information. The conditions for completing the training may include the loss value calculated by the loss function stabilizing and the number of iterations reaching a preset number, etc. The aforementioned loss value includes the difference between the predicted image output by the image generation model and the corresponding training image, and may also include the loss value calculated by other loss functions in related techniques, so that the predicted image generated by the image generation model during iterative training becomes closer and closer to the training image. After training is completed, the image generation model can generate a first fetal image that matches the fetal image prompt information based on the current fetal appearance information and the current fetal image prompt information.

[0084] Here, we take an image representing the fetus with its eyes closed, with the image prompt "eyes open," as an example. The trained image generation module can generate and output an image representing the fetus with its eyes open. Except for the eye area, the remaining areas in this image with the fetus open can be similar to or identical to the corresponding areas in the image with the fetus closed.

[0085] For example, after generating the first fetal image, the first fetal image can be displayed on a screen. This screen can be the display screen of an ultrasound device, or it can be the display screen of other devices such as a mobile phone or computer.

[0086] According to the above-described solution provided in the embodiments of the present invention, the ultrasound data of the current fetus can be processed to obtain the fetal appearance information. Then, fetal image prompt information can be obtained. Finally, based on the fetal appearance information and the fetal image prompt information, a first fetal image of the current fetus, conforming to the semantics of the fetal image prompt information, can be generated using an image generation model. On the one hand, the fetal appearance information in the above solution is obtained based on the ultrasound data of the current fetus; therefore, the first fetal image generated accordingly conforms to the actual image of the fetus in the pregnant woman's womb, possessing a certain degree of realism and improving scene adaptability. On the other hand, generating the first fetal image based on the fetal image prompt information can meet the personalized viewing needs of pregnant women or their families, thus enhancing the viewing experience of fetal images.

[0087] For example, fetal physical information is represented using a second fetal image, and the image generation model includes an encoding module, a denoising module, and a decoding module.

[0088] The second fetal image can be a rendered image obtained by rendering ultrasound data. Alternatively, the second fetal image can be a rendered image after undergoing image enhancement processing using related techniques. This image enhancement processing may include contrast adjustment, sharpening, etc.

[0089] Step S130 above, based on fetal appearance information and fetal image prompt information, generates a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information through an image generation model, and may include steps S131a to S133a.

[0090] In step S131a, based on the second fetal image, a first image feature is generated by the encoding module, and random noise is added to the first image feature to obtain the second image feature.

[0091] The aforementioned encoding module may include modules with feature extraction capabilities, such as convolutional feature extraction modules, recurrent feature extraction modules, and image encoders. The encoding module can extract features from the second fetal image to obtain image features of the second fetal image, which are then used as the aforementioned first image features. Since the first image features are obtained by feature extraction from the second fetal image through the encoding module, the first image features include the current fetal physical features. These physical features can be feature vectors used to represent the visual characteristics of the current fetal appearance. Specifically, fetal physical information is represented using the second fetal image; therefore, the first image features of the second fetal image can represent the visual characteristics of the current fetal appearance, i.e., they include the current fetal physical features.

[0092] The aforementioned random noise may include Gaussian noise, Rayleigh noise, etc. For example, the random noise can be superimposed on the first image feature using the add operator in related technologies to obtain the second image feature. In one example, a random noise map can be generated first, and then the random noise map can be processed by the feature extraction module in the encoding module described above to obtain a random noise feature with the same feature size as the first image feature. Superimposing this random noise feature on the first image feature yields the second image feature.

[0093] In step S132a, based on the fetal image prompt information, the second image features are denoised using a denoising module to obtain the third image features.

[0094] The aforementioned denoising module can be a denoising module for a stable diffusion model. The denoising module removes noise from the second image features based on the text features of the fetal image prompt information, thereby obtaining a third image feature that conforms to the semantics of the fetal image prompt information. Specifically, for example, the fetal image prompt information can be input into a pre-trained text encoder to extract features from the fetal image prompt information, thus obtaining the text features of the fetal image prompt information. The denoising module can set diffusion stages to indicate which stage of the diffusion process is currently in, removing noise stage by stage. In each diffusion stage, the denoising module performs denoising operations on the noisy second image features or the image features output from the previous diffusion stage. The image features output from the last diffusion stage can then be used as the third image feature. Specifically, in each diffusion stage, the cross-attention mechanism within the denoising module allows it to automatically focus on the most relevant parts of the text features of the fetal image prompt information and the second image features. For each diffusion stage, the denoising module can determine the predicted noise based on the input image features and the text features of the fetal image prompt information, and remove the predicted noise from the image features to obtain the image features output for that diffusion stage. As the diffusion stages increase, the noise in the output image features decreases, so the image features output in the last diffusion stage can be used as the third image features.

[0095] In step S133a, the third image features are decoded by the decoding module to obtain the first fetal image.

[0096] The decoding module can be a decoder in a machine learning model in related technologies, used to decode the features of the third image (or perform a decoding operation), and use the result of the decoding process as the first fetal image. Specifically, for example, the decoding module can be a transposed convolution module, which can adjust the feature size of the third image features to the image size of the first fetal image to obtain the first fetal image.

[0097] Since the third image features conform to the semantics of the current fetal image prompt information, the first fetal image obtained by the third image feature decoding process is an image that conforms to the semantics of the current fetal image prompt information.

[0098] See Figure 2 , Figure 2 A schematic diagram of a first fetal image and a second fetal image according to an embodiment of the present invention is shown. The first fetal image generated in conjunction with the embodiments of the present invention has a stronger sense of realism than the second fetal image, which is beneficial to the viewing experience of the observer.

[0099] According to the above-described scheme provided by the embodiments of the present invention, the second fetal image can be encoded by an encoding module to generate a first image feature. Random noise is then added to the first image feature to obtain a second image feature. Subsequently, based on fetal image prompt information, a denoising module can be used to denoise the second image feature to obtain a third image feature. Finally, a decoding module can be used to decode the third image feature to obtain the first fetal image. The data filtered out by the denoising operation in the above scheme includes not only the aforementioned random noise but also the original noise in the first image feature, which allows the image generation model to better focus on the visual characteristics of the current fetus, thus improving the realism of the first fetal image generated by the image generation model.

[0100] For example, step S132a may include: removing gender information from the fetal image prompt information, and based on the fetal image prompt information and neutral information after removing gender information, performing a denoising operation on the second image features through a denoising module to obtain a third image feature that includes the semantics of the fetal image prompt information and neutral information.

[0101] The system can detect key information in fetal image prompts, which may include information preset by the developer and intended to be masked. For example, the key text may include gender information. If gender information is present in the fetal image prompt, it can be removed to obtain a fetal image prompt without gender information. Then, the fetal image prompt without gender information, along with neutral information indicating the fetal gender is neutral, can be input into the text encoder mentioned above to obtain text features. The denoising module can remove noise from the second image features based on these text features to obtain a third image feature that includes the semantics of the fetal image prompt and the neutral information. For example, if the fetal image prompt only includes "girl" and "eyes open," the word "girl" (i.e., the gender-related text mentioned above) can be removed. In this example, the fetal image prompt after removing the gender information only includes "eyes open." "Eyes open" and "gender neutral" (neutral information in this example) are input into the text encoder, which generates and outputs the text features of the fetal image prompt and the text features of the neutral information. By concatenating the two text features above and then inputting them into the denoising module, the third image feature output by the denoising module will include both fetal image prompt information and neutral semantic information. Following step S133a above, the third image feature is decoded by the decoding module to obtain the first fetal image, thus yielding a first fetal image that conforms to both the semantics of fetal image prompt information and neutral semantic information.

[0102] According to the above-described solution provided in the embodiments of the present invention, gender information in the fetal image prompt information can be removed. Then, based on the fetal image prompt information after removing gender information and neutral information, a denoising module is used to denoise the second image features. Finally, a third image feature including the semantics of the fetal image prompt information and neutral information is obtained. This solution can reduce the probability of the fetus's gender being revealed in the first fetal image, reducing unnecessary attention or stress that pregnant women may experience due to knowing the fetus's gender in advance, and improving the scene adaptability of the first fetal image.

[0103] For example, fetal facial features can be represented using a second fetal image. The second fetal image can be a rendered image obtained by rendering ultrasound data. Alternatively, the second fetal image can be a rendered image after undergoing image enhancement processing according to related technologies. Such image enhancement processing may include contrast adjustment, sharpening, etc.

[0104] Step S130 above, based on fetal appearance information and fetal image prompt information, generates a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information through an image generation model, and may include steps S131b to S132b.

[0105] In step S131b, the second fetal image is subjected to fetal reconstruction processing to obtain a third fetal image.

[0106] The third fetal image contains more information about the current fetus's physical features than the second fetal image. (See also...) Figure 3 The above, Figure 3 This diagram illustrates incomplete second and third fetal images according to an embodiment of the present invention. In practical scenarios, due to the specific pose of the fetus or the ultrasound acquisition angle, the region of interest of the fetus in the second fetal image may be incomplete. This region of interest can be areas of greater interest to the observer, such as the face or hands. In practical scenarios, the second fetal image may only include the side profile of the fetus, or the fetus's hands, feet, or other organs may not be fully displayed. Fetal reconstruction processing can be performed on the second fetal image using image reconstruction algorithms such as symmetry filling algorithms, image interpolation algorithms, and statistical inpainting algorithms to obtain the third fetal image. For example, during the fetal reconstruction process, skin texture information and detail resolution can be added to the second fetal image to make the fetal profile information in the third fetal image more comprehensive than that in the second fetal image. Specifically, fetal reconstruction processing can be performed using an image reconstruction model that has learned skin texture, detail, and other information to obtain the third fetal image.

[0107] In step S132b, based on the third fetal image and the fetal image prompt information, the first fetal image is generated using an image generation model.

[0108] The fetal image prompt information can be input into the text encoder described above to obtain the text features of the fetal image prompt information. The third fetal image and the text features are then input into the image generation model to obtain the first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information. The specific generation process can be referred to in steps S131a to S133a above; these details are not elaborated upon here.

[0109] According to the above-described scheme of the present invention, a fetal reconstruction process can be performed on a second fetal image to obtain a third fetal image. Then, based on the third fetal image and fetal image prompts, a first fetal image is generated using an image generation model. Since the current fetal facial features in the second fetal image may be insufficient, performing fetal reconstruction processing on the second fetal image can yield a third fetal image with more facial features. Generating the first fetal image based on the third fetal image and fetal image prompts using an image generation model can increase the proportion of the current fetal focus area in the first fetal image, thus improving the generation quality and realism of the first fetal image.

[0110] For example, step S131b above may include step S131b1, inputting the second fetal image into the image reconstruction model to obtain the third fetal image.

[0111] The model architecture of the above image reconstruction model can refer to the model architectures of variational autoencoders, generative adversarial networks, diffusion models, U-Net networks, etc., and the embodiments of the present invention are not limited thereto. The above image reconstruction model can be trained through steps S210 to S230.

[0112] In step S210, the first training image of the baseline fetus is input into the untrained image reconstruction model to obtain the first predicted image.

[0113] The first training image can be a side profile image of a reference fetus, or a partial image of a reference fetus. An untrained image reconstruction model can generate and output a first predicted image based on the first training image of the reference fetus. Specifically, the first predicted image can be an image obtained by performing fetal reconstruction processing on the first training image using the untrained image reconstruction model. For example, if the first training image only includes the side profile of the reference fetus, a first predicted image including the frontal face of the reference fetus can be obtained by performing fetal reconstruction processing on the first training image. It should be understood that the image reconstruction model is not limited to reconstructing the fetal face; it can also reconstruct the fetal body, and this disclosure does not limit it in this regard.

[0114] In step S220, the confidence level of the first predicted image is generated based on the first predicted image.

[0115] The confidence score can be used to represent the probability that the first predicted image is the original ultrasound image. The confidence score of the first predicted image is generated by judging the differences between the image features of the first predicted image and the image features of the original ultrasound image. A higher confidence score indicates a higher probability that the first predicted image is the original ultrasound image, and a lower confidence score indicates a lower probability that the first predicted image is the original ultrasound image.

[0116] Here, we take the image reconstruction model as an example, using the generator in a generative adversarial network (GAN). The aforementioned confidence score can be obtained from the discriminator output in the GAN. The training process of the discriminator may include: acquiring multiple original ultrasound images and multiple non-original ultrasound images. The original ultrasound images may be rendered images obtained from fetal ultrasound body data, and may include a complete reference fetus or a frontal view of the reference fetus. The non-original ultrasound images may be images with added noise or not generated based on ultrasound body data. The label corresponding to the original ultrasound images may be 1, and the label corresponding to the non-original ultrasound images may be 0. The original ultrasound images are used as positive samples, and the non-original ultrasound images are used as negative samples to obtain the discriminator's prediction results for the positive and negative samples. Based on the difference between the prediction results and the labels, the model parameters of the discriminator are adjusted until training is complete. The first predicted image can be input into the trained discriminator to output the confidence score of the first predicted image.

[0117] In step S230, the model parameters of the untrained image reconstruction model are adjusted based on the confidence level.

[0118] A preset confidence level can be set. When the confidence level is greater than the preset level, the model parameters of the untrained image reconstruction model are fixed and used as the trained image reconstruction model. When the confidence level is less than the preset level, the model parameters of the untrained image reconstruction model can be adjusted based on the difference between the confidence level and the preset level. Iterative training of the image reconstruction model using multiple first training images can increase the confidence level of the first predicted image output by the image reconstruction model, making the first predicted image increasingly closer to the original ultrasound image. This, in turn, improves the realism of the third fetal image output by the trained image reconstruction model.

[0119] According to the above-described scheme provided in the embodiments of the present invention, a first training image of a reference fetus can be input into an untrained image reconstruction model to obtain a first predicted image. Then, based on the first predicted image, a confidence score for the first predicted image is generated. Finally, based on the confidence score, the model parameters of the untrained image reconstruction model are adjusted. The image reconstruction model trained in the above manner can improve the physical characteristics of the current fetus in the output third fetus image, which is beneficial to improving the realism of the generated first fetus image.

[0120] For example, the image reconstruction model includes a global prediction module and a local prediction module.

[0121] The global prediction module can be used to predict the overall shape of the current fetus, focusing on predicting its overall outline. Specifically, if the developer's goal is to obtain the current fetus's frontal face as part of the fetal reconstruction process, the global prediction module can focus on predicting the outline of the current fetus's facial features and head.

[0122] The local prediction module can be used to predict local features of the current fetus, focusing on predicting local details to make the third-generation fetal image more detailed and realistic. Specifically, if the developer's goal is to obtain the current fetus's frontal face as the fetal reconstruction process, the local prediction module can focus on predicting specific image details in the facial features and head contours of the current fetus.

[0123] The above step S131b1, which inputs the second fetal image into the image reconstruction model to obtain the third fetal image, may include steps S131b11 to S131b12.

[0124] In step S131b11, the second fetal image is input into the global prediction module and the local prediction module respectively to generate global prediction features and local prediction features respectively.

[0125] The global prediction module performs feature extraction on the second fetal image to obtain global features. Then, based on these global features, global prediction features are generated. These global prediction features can represent the overall visual features of the fetus. For example, they can represent the visual features of the facial features and head contours. The local prediction module performs feature extraction on the second fetal image to obtain local features. Then, based on these local features, local prediction features are generated. These local prediction features can represent the local visual features of the fetus. For example, they can represent specific visual details within the facial features and head contours. It should be understood that the parameters used in the feature extraction processes of the global and local prediction modules can be the same or different; that is, the global features and local features of the second fetal image can be the same or different.

[0126] In step S131b12, the global prediction features and local prediction features are fused to obtain the fused prediction features.

[0127] In one example, the global and local predicted features can be fused using the concat or add operators (feature concatenation operators) from related technologies. In another example, the global and local predicted features can also be fused based on an attention mechanism; this embodiment of the invention does not impose any limitations on this approach.

[0128] During the training phase of the image reconstruction model, a first training image can be input into an untrained global prediction module to obtain first training features. The first training image can also be input into an untrained local prediction module to obtain second training features. Then, the first and second training features are fused using a feature fusion operation in related techniques to obtain fused training features. Decoding these fused training features yields the first predicted image. After the discriminator outputs a confidence score for the first predicted image, the module parameters of both the untrained global and local prediction modules can be comprehensively adjusted based on the difference between this confidence score and the preset confidence score mentioned above. In one example, the first training image can be obtained by cropping a second training image. The second training image may include a complete baseline fetus. In this case, feature extraction can be performed on the entire second training image to obtain global baseline features. Feature extraction can be performed on a local area of ​​the second training image (e.g., the facial region) to obtain local baseline features. The module parameters of the untrained global prediction module can be adjusted based on the difference between the global baseline features and the first training features. The module parameters of the untrained local prediction module can be adjusted based on the difference between the local baseline features and the second training features.

[0129] In step S131b13, the fused prediction features are decoded to generate a third fetal image.

[0130] For example, the transposed convolution module mentioned above can be used to adjust the feature size of the fused prediction features to the image size of the third fetus image to obtain the third fetus image.

[0131] According to the above-described scheme provided by the embodiments of the present invention, the second fetal image can be input into the global prediction module and the local prediction module respectively to generate global prediction features and local prediction features respectively. Then, the global prediction features and the local prediction features are fused to obtain fused prediction features. Finally, the fused prediction features are decoded to generate the third fetal image. The above scheme generates the third fetal image by using a global prediction module that focuses more on overall visual features and a local prediction module that focuses more on local visual features, which can improve the generation quality of the third fetal image and is beneficial to improving the generation quality of the first fetal image generated based on the third fetal image.

[0132] For example, the fetal image prompting information also includes dynamic descriptive information.

[0133] The aforementioned dynamic descriptive information can be descriptive text related to the movements of a fetus, infant, or child, and can be used to describe the dynamic processes of limb movements, facial expressions, and facial movements. Examples include descriptive text such as "opening eyes," "raising left hand," and "raising right hand." This invention does not limit this scope.

[0134] The above generation method may also include step S310.

[0135] In step S310, a three-dimensional ultrasound scan of the current fetus is performed to obtain ultrasound data for the target time period.

[0136] Users of 3D ultrasound scanning equipment can perform 3D ultrasound scans of the fetus in a pregnant woman's womb within a target time period to obtain ultrasound data at various points in time. It should be understood that the target time period can also be part of the duration of the 3D ultrasound scan, and can be manually selected by the user or observer.

[0137] Step S110 above processes the ultrasound data of the current fetus to obtain the fetal morphology information of the current fetus, which may include: extracting morphological features from ultrasound data at least two times within the target time period to obtain the dynamic morphological features of the current fetus.

[0138] Among these features, the current dynamic physical characteristics of the fetus can be seen in its movements such as kicking and rolling over. These dynamic physical characteristics can, to a certain extent, characterize the amplitude and frequency of certain movements performed by the fetus. Therefore, the dynamic video determined based on these physical characteristics can also reflect the amplitude and frequency of the fetus's movements relatively well, and is consistent with the actual image of the fetus in the pregnant woman's womb.

[0139] In one example, at least two moments within the target time period can be determined by random sampling, or at least two moments within the target time period can be manually selected by the user or observer.

[0140] For each moment in the ultrasound data, facial features can be extracted to obtain the current dynamic facial features of the fetus. These dynamic facial features can be represented by text, images, or videos. For example, template matching or model classification can be used to determine the descriptive text related to the current fetal movement in the ultrasound data or the rendered image derived from the ultrasound data. In one example, the descriptive text related to the current fetal movement at each moment can be determined separately, and then the dynamic facial features of the current fetus in the target time period can be determined by combining them. For example, the descriptive text with the highest number of corresponding moments that is related to the current fetal movement, or the descriptive text features of that descriptive text, can be used as the dynamic facial features.

[0141] The above-mentioned method for generating fetal images may further include: generating a dynamic video of the current fetus based on a first fetal image using a video generation model.

[0142] A video generation model is used to generate dynamic videos based on dynamic physical features, dynamic descriptive information, and a first fetal image. This invention provides a training process for the video generation model for reference: acquiring a third training image, training descriptive information corresponding to the third training image, and a training video corresponding to the third training image. The third training image may be a frame from the corresponding training video, and this image may include a baseline fetus. The training descriptive information may include descriptive text related to the actions of the baseline fetus in the third training image. The third training image and the corresponding training descriptive information are input into an untrained video generation model to obtain a predicted video. Based on the differences between the predicted video and the training video (e.g., differences in pixel values ​​between their respective frame images, differences in their overall visual similarity, etc.), the model parameters of the untrained video generation model are adjusted. Through iterative training with different third training images, a video generation model that generates the dynamic video of the current fetus can be obtained. It should be understood that a loss value can also be determined using a loss function from related technologies to adjust the model parameters of the untrained video generation model; this invention does not impose limitations on this.

[0143] For example, after generating the dynamic video, the process may also include post-processing of the generated dynamic video, such as adjusting the video's frame rate, resolution, etc.

[0144] According to the above-described scheme provided by the embodiments of the present invention, a three-dimensional ultrasound scan of the current fetus can be performed to obtain ultrasound body data for a target time period. Then, body features are extracted from the ultrasound body data at at least two moments within the target time period to obtain the dynamic body features of the current fetus. Finally, based on the first fetal image, a dynamic video of the current fetus is generated using a video generation model. This scheme combines the actual actions performed by the current fetus (referencing dynamic body features) with the dynamic descriptive information in the fetal image prompts to generate a dynamic video that conforms to the semantics of both, thus improving the realism of the current fetus in the dynamic video.

[0145] For example, step S120, obtaining fetal image prompt information, may include steps S121 to S123.

[0146] In step S121, a reference fetal video is obtained.

[0147] Ultrasound scans at three dimensions can generate ultrasound data of a baseline fetus at multiple time points. For each time point, the corresponding ultrasound data can be rendered to obtain a rendered image. Stitching together the rendered images from each time point yields a reference fetal video. It should be understood that the reference fetal video can also be post-processed using video editing software to extract dynamic features. Furthermore, the reference fetal video can also be a video of another fetus (also called a baseline fetus), an infant, or a child. For example, if a user wants to generate a video of their current fetus based on the style of a video of fetus A, they can upload a video of fetus A as a reference fetal video.

[0148] In step S122, dynamic features are extracted from the reference fetal video.

[0149] The aforementioned dynamic features can be represented by text. For example, descriptive text related to the baseline fetal movement can be determined for each frame in the reference fetal video at each moment using template matching or model classification. Optionally, descriptive text related to the baseline fetal movement at each moment can be determined separately, and then the dynamic features of the baseline fetus in the reference fetal video can be comprehensively determined. For example, the descriptive text related to the baseline fetal movement with the highest total number of corresponding moments, or the descriptive text features of that descriptive text, can be used as the dynamic features.

[0150] In step S123, dynamic description information is generated based on the extracted dynamic features to obtain fetal image prompt information.

[0151] In one example, if the dynamic description information is a text description, then the text description can be directly used as the dynamic description information. In another example, if the dynamic description information is a feature vector, then the feature vector can be input into the decoder of the language model to obtain the text description corresponding to the feature vector, which can then be used as the dynamic description information.

[0152] Optionally, the reference fetal video can be selected by the observer. Depending on the specific scenario, the observer can choose a reference fetal video from multiple fetal videos. Then, based on the dynamic features extracted from the reference fetal video, a first fetal image similar to the baseline fetal movement in the reference fetal video can be generated. It should be understood that, similar to the baseline fetal movement in the reference fetal video, a dynamic video of the current fetus can also be generated by combining the aforementioned process of generating a dynamic video of the current fetus based on the first fetal image using a video generation model.

[0153] In one example, the obtained dynamic description information can be added to the user interface mentioned above to display multiple alternative words or phrases corresponding to the dynamic description information. The observer can select the target text in the alternative words or phrases using an input device (such as a mouse, trackball, etc.) as part of the above fetal image prompt information.

[0154] According to the above-described solution provided in the embodiments of the present invention, a reference fetal video can be obtained. Then, dynamic features can be extracted from the reference fetal video. Finally, dynamic descriptive information can be generated based on the extracted dynamic features to obtain fetal image prompt information. The above solution can extract dynamic features from the reference fetal video, resulting in a wider range of data for the fetal image prompt information, which is beneficial for improving the realism of the first fetal image.

[0155] For example, fetal physical information is represented using a second fetal image.

[0156] The second fetal image can be a rendered image obtained by rendering ultrasound data. Alternatively, the second fetal image can be a rendered image after undergoing image enhancement processing using related techniques. This image enhancement processing may include contrast adjustment, sharpening, etc.

[0157] Step S130, based on fetal appearance information and fetal image prompt information, generates a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information through an image generation model, which may include steps S131c to S133c.

[0158] In step S131c, a first trace region for the second fetal image is obtained.

[0159] The first traced region can be determined by either manual or automatic tracing methods. For example, a tracing interface can be provided to respond to a user's manual selection of a tracing operation, resulting in the first traced region. The first traced region can be an image region in the second fetal image that requires further processing (such as the region corresponding to the face). Alternatively, region detection algorithms or models in related technologies can be used to automatically detect the first traced region in the second fetal image. For instance, a face detection algorithm or model can be used to automatically select the facial region in the second fetal image as the first traced region.

[0160] For example, the methods for obtaining the first tracing region may include frame tracing, eraser tracing, circular tracing, etc. Accordingly, the first tracing region can be a regular region, such as a rectangle or a circle, or it can be an irregular region, such as an irregular shape drawn by the user by moving the mouse.

[0161] In step S132c, based on the image corresponding to the first trace region in the second fetal image and the fetal image prompt information, a local fetal image is generated through an image generation model.

[0162] The image corresponding to the first traced region in the second fetal image, along with the prompt information of the fetal image, is input into the image generation model described above to generate a local fetal image. This local fetal image can be the image corresponding to the first traced region that conforms to the semantics of the fetal image prompt information. For example, the image corresponding to the first traced region is the image in the second fetal image that corresponds to the current fetal face, and the prompt information of the fetal image includes the text description "open eyes". In this case, the local fetal image generated by the image generation model is the current fetal face image, and in this current fetal face image, the current fetal eyes are open.

[0163] In step S133c, the portions outside the first trace region in the local fetal image and the second fetal image are fused to obtain the first fetal image.

[0164] For example, there can be multiple first trace regions. That is, multiple regions of the second fetal image are enhanced, while the remaining regions are not enhanced.

[0165] For example, the portion outside the first traced region can also be processed in the same way as the first traced region to generate a local fetal image. The difference lies in the processing intensity; the processing intensity corresponding to the first traced region can be higher than that of the region outside the first traced region. In this way, while ensuring the realism of the entire area of ​​the obtained first fetal image, computational power is taken into account, and computational efficiency is improved.

[0166] The pixel values ​​of the pixels in the first traced region of the second fetal image can be set to the pixel values ​​of the corresponding pixels in the local fetal image, thereby fusing the portion of the local fetal image and the second fetal image outside the first traced region to obtain the first fetal image. According to the above-described scheme provided by the embodiments of the present invention, a first traced region for the second fetal image can be obtained. Then, based on the image corresponding to the first traced region in the second fetal image and the fetal image prompt information, a local fetal image is generated through an image generation model. Finally, the portion of the local fetal image and the second fetal image outside the first traced region is fused to obtain the first fetal image. The above scheme can generate a local fetal image only for the image corresponding to the first traced region. On the one hand, this reduces the computational power requirement and is beneficial to improving the generation efficiency of the first fetal image; on the other hand, the first traced region is usually the area of ​​interest for the observer, which is beneficial to meeting the actual observation needs of the observer.

[0167] For example, the above generation method may further include steps S410 to S420.

[0168] In step S410, a second trace region for the first fetal image is obtained.

[0169] The second tracing region can be determined through manual or automatic tracing methods. For example, a tracing interface can be provided to respond to a user's manual tracing operation and obtain the second tracing region. The second tracing region can be an image region in the first fetal image that requires further processing (such as the region corresponding to the face). Alternatively, region detection algorithms or models in related technologies can be used to automatically detect the second tracing region in the first fetal image. For example, a face detection algorithm or model can be used to automatically select the facial region in the first fetal image as the second tracing region.

[0170] In step S420, image processing operations are performed on the image corresponding to the second trace region in the first fetal image to obtain the fourth fetal image.

[0171] Image processing operations may include image enhancement, or image removal and image interpolation. For example, when it is necessary to improve the clarity of the current fetal facial region in a first fetal image, the region in the first fetal image that is related to the current fetal facial region can be used as a second traced region. Image enhancement processing is then performed on the second traced region. As another example, when it is necessary to remove images from the first fetal image that are unrelated to the current fetal limb and facial regions, such as hair areas, the region in the first fetal image outside of the current fetal limb and facial regions can be used as a second traced region. Then, image removal processing can be performed on the second traced region to remove pixels located within the second traced region in the first fetal image. After removing pixels located within the second traced region in the first fetal image, image interpolation processing can be performed on the second traced region to avoid the presence of pixels with unfilled pixel values ​​in the second traced region.

[0172] According to the above-described solution provided by the present invention, a second traced region for a first fetal image can be obtained. Then, image processing operations can be performed on the image corresponding to the second traced region in the first fetal image to obtain a fourth fetal image. The above solution can use the second traced region to perform image enhancement, image removal, or image interpolation on the image corresponding to the second traced region in the first fetal image. On the one hand, image enhancement can enhance the appearance details of the current fetus in the first fetal image; on the other hand, image removal and image interpolation can remove non-interested areas in the first fetal image and avoid pixels with unfilled pixel values ​​in the processed first fetal image, thus correcting the generated image and better meeting the actual observation needs of the observer.

[0173] For example, step S410 above also includes step S411.

[0174] In step S411, image segmentation is performed on the current fetus in the first fetal image to determine the portion within the outline of the current fetus as the second trace region.

[0175] The image segmentation described above can be performed using a region detection model in related technologies. For example, this region detection model can determine the fetal region corresponding to the current fetus in the first fetal image, as the image segmentation result of the current fetus. Then, the outline of the fetal region can be used as the outline of the current fetus, and the part within the outline can be determined as the second trace region. The image enhancement processing described above can be performed on the second trace region to enhance the appearance details in the first fetal image. It should be understood that image segmentation can also be performed on different parts of the current fetus, so that the parts within the outline of different parts are respectively used as different second trace regions, for example, the hair area of ​​the current fetus can be used as the second trace region. This embodiment of the invention does not limit this.

[0176] According to the above-described scheme of the present invention, image segmentation can be performed on the current fetus in the first fetal image to determine the portion within the outline of the current fetus as the second trace region. The above scheme can automatically determine the second trace region. Since the second trace region includes the portion within the outline of the current fetus, targeted enhancement of the fetal region in the first fetal image can be achieved, enhancing the appearance details such as skin texture and hair details of the current fetus, thus improving the realism of the first fetal image.

[0177] For example, similar operations can be performed on the second fetal image to generate an image after enhancement or deletion, which helps meet the observer's actual observation needs. Further, after obtaining the ultrasound data of the current fetus, a second fetal image corresponding to the current fetus can be determined, and a third trace region for the second fetal image can be obtained. Image processing operations can be performed on the image corresponding to the third trace region in the second fetal image to obtain a fifth fetal image. These image processing operations include image enhancement, or image removal and image interpolation. Based on the fifth fetal image and fetal image prompt information, a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information is generated using an image generation model.

[0178] For example, fetal facial features may include three-dimensional facial features of the fetus. The image generation model may be a three-dimensional image generation model. The three-dimensional facial features of the fetus may be textual descriptions of stereoscopic measurements such as jawbone width and head circumference, or stereoscopic contour data of the current fetal outline, such as point cloud data or mesh data.

[0179] Step S130, based on fetal appearance information and fetal image prompt information, generates a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information through an image generation model, which may include steps S131d and S132d.

[0180] In step S131d, based on the fetal three-dimensional physical features and fetal image prompts, a three-dimensional fetal image of the current fetus that conforms to the semantics of the fetal image prompts is generated by a three-dimensional image generation model.

[0181] The network architecture of the above-mentioned 3D image generation model can refer to machine learning models in related technologies, such as convolutional neural networks, recurrent neural networks, generators in generative adversarial networks, stable diffusion models, etc. The embodiments of the present invention will not be described in detail here.

[0182] The training process of this 3D image generation model may include: acquiring a training dataset; iteratively training the 3D image generation model using the training dataset, and adjusting the model parameters of the 3D image generation model using a loss function until training is complete. The training dataset includes multiple fourth training images. For each fourth training image, there are corresponding fetal 3D physical features and fetal image prompts. If the 3D physical features are text description information or text description features, then the fourth training image can be a 3D image that conforms to the semantics of the fetal image prompts and the 3D physical features. If the 3D physical features are a 3D image or a 3D image feature, then the fourth training image can be a 3D image generated based on the 3D image or the 3D image feature, conforming to the semantics of the fetal image prompts. The conditions for completing the training may include the loss value calculated by the loss function stabilizing and the number of iterations reaching a preset number. The loss value may include the difference between the predicted image output by the 3D image generation model and the corresponding fourth training image, and may also include the loss value calculated by other loss functions in related technologies, so that the predicted image generated by the model becomes increasingly closer to the fourth training image that conforms to the semantics of the fetal image prompts and the fetal 3D physical features. Based on the above training process, a 3D image generation model can be obtained to generate a 3D image of the current fetus that conforms to the semantics of the fetal image prompt information.

[0183] In step S132d, the three-dimensional image of the fetus is rendered to obtain the first fetal image.

[0184] The three-dimensional fetal image can be processed using surface rendering or volume rendering as described above to obtain a first fetal image. The three-dimensional fetal image offers greater operability; for example, the angle of the image can be adjusted via an interactive interface that responds to the observer's sliding actions. Then, the angle-adjusted three-dimensional fetal image can be rendered to obtain the first fetal image at the specific angle desired by the observer.

[0185] According to the above-described solution provided by the present invention, a three-dimensional fetal image conforming to the semantics of the fetal image prompts can be generated using a three-dimensional image generation model based on the three-dimensional physical features of the fetus and fetal image prompts. Then, the three-dimensional fetal image is rendered to obtain a first fetal image. Compared to first rendering ultrasound data and then generating the first fetal image based on the rendered image, the above solution only requires generating a three-dimensional fetal image once, and then the angle of the three-dimensional fetal image can be adjusted to render first fetal images from different angles, which improves the generation efficiency of the first fetal image.

[0186] This invention also provides a device for generating fetal images. Figure 4 A schematic block diagram of a fetal image generation apparatus 500 according to an embodiment of the present invention is shown. Figure 4 As shown, the generating device 500 may include: an ultrasound image processing module 510, a prompt information acquisition module 520, and an image generation module 530.

[0187] The ultrasound image processing module 510 processes the ultrasound data of the current fetus to obtain fetal morphological information. The prompt information acquisition module 520 acquires fetal image prompt information. The image generation module 530, based on the fetal morphological information and the fetal image prompt information, generates a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information through an image generation model.

[0188] For example, fetal facial information is represented using a second fetal image. The image generation model includes an encoding module, a denoising module, and a decoding module. The image generation module 530 may include a second image feature generation module, a third image feature generation module, and a first generation module.

[0189] The second image feature generation module generates first image features based on the second fetal image using an encoding module, and adds random noise to the first image features to obtain second image features. The first image features include the physical features of the current fetus. The third image feature generation module denoises the second image features based on fetal image prompts using a denoising module to obtain third image features. The first generation module decodes the third image features using a decoding module to obtain the first fetal image.

[0190] For example, the third image feature generation module further includes a third image feature generation submodule.

[0191] The third image feature generation submodule is used to remove gender information from the fetal image prompt information. Based on the fetal image prompt information and neutral information after removing gender information, the second image feature is denoised by the denoising module to obtain the third image feature that includes the semantics of the fetal image prompt information and neutral information.

[0192] For example, fetal facial information is represented using a second fetal image. The image generation module 530 may include a third fetal image generation module and a second generation module.

[0193] The third fetal image generation module performs fetal reconstruction processing on the second fetal image to obtain a third fetal image, wherein the third fetal image contains more fetal facial features than the fetal facial features information in the fetal facial feature information. The second generation module generates a first fetal image based on the third fetal image and fetal image prompts, using an image generation model.

[0194] For example, the third fetal image generation module includes: a third fetal image generation submodule.

[0195] The third fetal image generation submodule is used to input the second fetal image into the image reconstruction model to obtain the third fetal image.

[0196] The third fetal image generation submodule includes: a first predicted image generation module, a confidence generation module, and a model parameter adjustment module.

[0197] The first predicted image generation module is used to input a first training image of the baseline fetus into an untrained image reconstruction model to obtain a first predicted image, wherein the first predicted image contains more facial features of the baseline fetus than the first training image. The confidence score generation module is used to generate a confidence score for the first predicted image based on the first predicted image, wherein the confidence score represents the probability that the first predicted image is the original ultrasound image. The model parameter adjustment module is used to adjust the model parameters of the untrained image reconstruction model based on the confidence score.

[0198] For example, the image reconstruction model described above includes a global prediction module and a local prediction module. The third fetal image generation submodule includes: a prediction feature generation module, a fusion prediction feature generation module, and a first generation submodule.

[0199] The predictive feature generation module inputs the second fetal image into the global prediction module and the local prediction module respectively to generate global predictive features and local predictive features. The global predictive features represent the overall visual features of the fetus, while the local predictive features represent the local visual features of the fetus. The fusion predictive feature generation module performs a feature fusion operation on the global and local predictive features to obtain fused predictive features. The first generation submodule decodes the fused predictive features to generate the third fetal image.

[0200] For example, the fetal image prompting information also includes dynamic descriptive information. The generation device 500 further includes an ultrasound body data generation module and a dynamic video generation module. The ultrasound image processing module 510 includes a dynamic body feature generation module.

[0201] The ultrasound body data generation module performs a 3D ultrasound scan of the current fetus to obtain ultrasound body data for the target time period. The dynamic body feature generation module extracts body features from ultrasound body data at at least two time points within the target time period to obtain the dynamic body features of the current fetus. The dynamic video generation module generates a dynamic video of the current fetus based on the first fetal image using a video generation model. The video generation model generates the dynamic video based on dynamic body features and dynamic descriptive information.

[0202] For example, the prompt information acquisition module 520 includes a video acquisition module, an extraction module, and a prompt information generation module.

[0203] The video acquisition module is used to acquire a reference fetal video. The extraction module is used to extract dynamic features from the reference fetal video. The prompt information generation module is used to generate dynamic description information based on the extracted dynamic features to obtain fetal image prompt information.

[0204] For example, fetal facial information is represented using a second fetal image. The image generation module 530 may include: a first trace region determination module, a local fetal image generation module, and a third generation module.

[0205] The first traced region determination module is used to obtain a first traced region for the second fetal image. The local fetal image generation module is used to generate a local fetal image based on the image corresponding to the first traced region in the second fetal image and the fetal image prompt information, using an image generation model. The third generation module is used to fuse the local fetal image and the portion of the second fetal image outside the first traced region to obtain the first fetal image.

[0206] For example, the generating device 500 further includes a second tracing region determination module and a fourth fetal image generation module.

[0207] The second trace region determination module is used to obtain a second trace region for the first fetal image. The fourth fetal image generation module is used to perform image processing operations on the image corresponding to the second trace region in the first fetal image to obtain a fourth fetal image. The image processing operations include image enhancement or removal and image interpolation.

[0208] For example, the second trace region determination module includes: a second trace region determination submodule.

[0209] The second trace region determination submodule is used to perform image segmentation on the current fetus in the first fetus image, so as to determine the part within the outline of the current fetus as the second trace region.

[0210] For example, the fetal appearance information includes three-dimensional fetal appearance features. The image generation model is a three-dimensional image generation model. The image generation module 530 may include a three-dimensional image generation module and a fourth generation module.

[0211] The 3D image generation module is used to generate a 3D image of the current fetus that conforms to the semantics of the fetal image prompts, based on the fetal 3D physical features and fetal image prompts. The fourth generation module is used to render the 3D fetal image to obtain the first fetal image.

[0212] According to another aspect of the present invention, an electronic device is also provided. Figure 5 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. Figure 5 As shown, the electronic device 600 includes a processor 610 and a memory 620. The memory 620 stores a computer program, and the computer program instructions are executed by the processor 610 to perform the above-described method for generating fetal images.

[0213] Furthermore, according to another aspect of the present invention, a storage medium is provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the above-described method for generating fetal images according to embodiments of the present invention, and is used to implement corresponding modules in the above-described fetal image generation apparatus or electronic device according to embodiments of the present invention. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0214] According to another aspect of the present invention, a computer program product is also provided, comprising computer program instructions that, when executed by a computer or processor, cause the computer or processor to perform corresponding steps of the method for generating the fetal image described above.

[0215] Those skilled in the art can understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the relevant description of the method for generating fetal images, and for the sake of brevity, they will not be described in detail here.

[0216] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0217] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0218] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0219] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0220] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0221] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0222] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0223] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the fetal image generation apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0224] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0225] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating a fetal image, characterized in that, The method includes: The ultrasound data of the current fetus is processed to obtain the fetal morphology information of the current fetus; Obtain fetal image prompts; Based on the fetal physical information and the fetal image prompt information, a first fetal image of the current fetus is generated using an image generation model that conforms to the semantics of the fetal image prompt information.

2. The method as described in claim 1, characterized in that, The fetal physical information is represented using a second fetal image, and the image generation model includes an encoding module, a denoising module, and a decoding module. The step of generating a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information, based on the fetal physical information and the fetal image prompt information, through an image generation model, includes: Based on the second fetal image, the encoding module generates a first image feature and adds random noise to the first image feature to obtain a second image feature, wherein the first image feature includes the physical features of the current fetus; Based on the fetal image prompt information, the second image features are denoised using the denoising module to obtain the third image features; The decoding module decodes the third image features to obtain the first fetal image.

3. The method as described in claim 2, characterized in that, The step of denoising the second image features based on the fetal image prompt information using the denoising module to obtain the third image features includes: The gender information in the fetal image prompt information is removed, and based on the fetal image prompt information and neutral information after removing the gender information, the second image feature is denoised by the denoising module to obtain a third image feature that includes the semantics of the fetal image prompt information and neutral information.

4. The method as described in claim 1, characterized in that, The fetal physical information is represented using a second fetal image. The step of generating a first fetal image of the current fetus based on the fetal physical information and the fetal image prompt information, using an image generation model, and conforming to the semantics of the fetal image prompt information, includes: The second fetal image is subjected to fetal reconstruction processing to obtain a third fetal image, wherein the current fetal appearance information in the third fetal image is more than the current fetal appearance information in the fetal appearance information. Based on the third fetal image and the fetal image prompt information, the first fetal image is generated using the image generation model.

5. The method as described in claim 4, characterized in that, The process of performing fetal reconstruction processing on the second fetal image to obtain the third fetal image includes: The second fetal image is input into the image reconstruction model to obtain the third fetal image; wherein, the image reconstruction model is trained through the following steps: The first training image of the baseline fetus is input into the untrained image reconstruction model to obtain the first predicted image, wherein the body shape information of the baseline fetus in the first predicted image is more than that of the baseline fetus in the first training image. Based on the first predicted image, a confidence score for the first predicted image is generated, wherein the confidence score represents the probability that the first predicted image is the original ultrasound image; Based on the confidence level, the model parameters of the untrained image reconstruction model are adjusted.

6. The method as described in claim 5, characterized in that, The image reconstruction model includes a global prediction module and a local prediction module. The step of inputting the second fetal image into the image reconstruction model to obtain the third fetal image includes: The second fetal image is input into the global prediction module and the local prediction module respectively to generate global prediction features and local prediction features respectively. The global prediction features are used to represent the overall visual features of the current fetus, and the local prediction features are used to represent the local visual features of the current fetus. The global prediction features and the local prediction features are fused to obtain the fused prediction features. The fused predictive features are decoded to generate a third fetal image.

7. The method as described in claim 1, characterized in that, The fetal image prompt information also includes dynamic descriptive information; The method further includes: Perform a three-dimensional ultrasound scan on the current fetus to obtain ultrasound data for the target time period; The process of processing the ultrasound data of the current fetus to obtain the fetal anatomy information includes: The body features of the fetus are extracted from ultrasound data at at least two times within the target time period to obtain the dynamic body features of the current fetus. The method further includes: Based on the first fetal image, a dynamic video of the current fetus is generated using a video generation model, wherein the video generation model is used to generate the dynamic video based on the dynamic physical features and the dynamic descriptive information.

8. The method as described in claim 7, characterized in that, The acquisition of fetal image prompts includes: Obtain a reference fetal video; Extract dynamic features from the reference fetal video; Dynamic descriptive information is generated based on the extracted dynamic features to obtain the fetal image prompt information.

9. The method as described in claim 1, characterized in that, The fetal physical information is represented using a second fetal image. The step of generating a first fetal image of the current fetus based on the fetal physical information and the fetal image prompt information, using an image generation model, and conforming to the semantics of the fetal image prompt information, includes: Obtain the first trace region for the second fetal image; Based on the image corresponding to the first trace region in the second fetal image and the fetal image prompt information, a local fetal image is generated through an image generation model; The portion of the local fetal image and the portion of the second fetal image outside the first trace region are fused together to obtain the first fetal image.

10. The method as described in claim 1, characterized in that, The method further includes: Obtain a second trace region for the first fetal image; Image processing operations are performed on the image corresponding to the second trace region in the first fetal image to obtain a fourth fetal image; The image processing operations include: Image enhancement; or Image removal and image interpolation.

11. The method as described in claim 10, characterized in that, The step of obtaining the second trace region for the first fetal image includes: Image segmentation is performed on the current fetus in the first fetal image to determine the portion within the outline of the current fetus as the second trace region.

12. The method according to any one of claims 1-11, characterized in that, The fetal physical information includes three-dimensional fetal physical features; the image generation model is a three-dimensional image generation model. The step of generating a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information, based on the fetal physical information and the fetal image prompt information, through an image generation model, includes: Based on the fetal three-dimensional physical features and the fetal image prompt information, a fetal three-dimensional image of the current fetus that conforms to the semantics of the fetal image prompt information is generated by the three-dimensional image generation model; The three-dimensional image of the fetus is rendered to obtain the first fetal image.

13. A device for generating fetal images, characterized in that, The device includes: An ultrasound image processing module is used to process the ultrasound data of the current fetus to obtain the fetal morphology information of the current fetus. The prompt information acquisition module is used to acquire fetal image prompt information; The image generation module is used to generate a first fetal image of the current fetus that conforms to the semantics of the fetal image prompt information, based on the fetal appearance information and the fetal image prompt information, through an image generation model.

14. An electronic device, characterized in that, The device includes a memory and a processor, wherein: the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method for generating a fetal image as described in any one of claims 1-12.

15. A storage medium storing a computer program / instructions, characterized in that, The computer program / instructions, when executed, are used to perform the method for generating a fetal image as described in any one of claims 1-12.

16. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the method for generating a fetal image as described in any one of claims 1-12.

Images