Face image prediction method and device based on four-dimensional color Doppler ultrasound, equipment and medium

Abstract

The embodiment of the invention provides a method, device and equipment for predicting a face image based on four-dimensional color Doppler ultrasound and a medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining four-dimensional color Doppler ultrasound image data, and extracting face region image data from the four-dimensional color Doppler ultrasound image data; performing face key point detection on the face region image data to obtain first coordinate information of a plurality of face key points in the face region image data; generating face key point image data based on the first coordinate information; and performing long phase prediction processing on the face key point image data based on a preset face long phase prediction model to obtain a face image prediction result corresponding to the four-dimensional color ultrasound image data. By adopting the technical scheme provided by the embodiment of the invention, end-to-end face image prediction can be realized, so that the stability and the accuracy of predicting the face image through the four-dimensional color Doppler ultrasound are effectively guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for predicting facial images based on four-dimensional color ultrasound. Background Art

[0002] In the related art, methods for predicting a baby's future appearance can be divided into two categories. One prediction method is to extract the facial features of the parents for comparative analysis and fusion. However, in reality, the baby's development in the mother's womb and birth are often affected by multiple factors such as genetics and environment, so the prediction accuracy is low; the other prediction method is to introduce four-dimensional color ultrasound images as prior knowledge. This method processes and predicts the baby's facial features and contours through four-dimensional color ultrasound images, and the accuracy is relatively high. However, the method of predicting the baby's future appearance through four-dimensional color ultrasound images is not stable enough and requires manual intervention during the prediction process, making it difficult to make an end-to-end prediction service. It is also easy for errors during manual intervention to generate strange and deformed results, affecting the user experience. Therefore, how to ensure the stability and accuracy of predicting the baby's future appearance through four-dimensional color ultrasound has become a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment and medium for predicting facial images based on four-dimensional color ultrasound, aiming to ensure the stability and accuracy of predicting the baby's future appearance through four-dimensional color ultrasound.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for predicting a facial image based on four-dimensional color Doppler ultrasound, the method comprising:

[0005] Acquiring four-dimensional color ultrasound image data, and extracting facial region image data from the four-dimensional color ultrasound image data;

[0006] Performing facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data;

[0007] Generate facial key point image data based on the first coordinate information;

[0008] Based on a preset facial appearance prediction model, the facial key point image data is subjected to appearance prediction processing to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0009] Optionally, in some embodiments, after performing appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data, the method further includes:

[0010] Performing a prediction quality test on the facial image prediction result to obtain a test result;

[0011] When the detection result indicates that the facial image prediction result contains a deformed facial image, the step of performing appearance prediction processing on the facial key point image data based on the preset facial appearance prediction model is re-executed to obtain a new facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0012] Optionally, in some embodiments, performing a prediction quality test on the facial image prediction result to obtain a test result includes:

[0013] Inputting the facial image prediction result into a preset quality detection model, and performing quality defect type judgment on the facial image prediction result based on the quality detection model to obtain multiple quality defect type judgment results corresponding to the facial image prediction result; the quality detection model is obtained by training a deep convolutional neural network model for multi-label classification;

[0014] When the confidence scores of the plurality of quality defect type judgment results are all less than a preset confidence threshold, obtaining a first detection result indicating that the facial image prediction result does not contain a deformed facial image; the detection result includes the first detection result;

[0015] When the confidence score of the target judgment result is greater than or equal to a preset confidence threshold, a second detection result is obtained indicating that a deformed facial image exists in the facial image prediction result; the target judgment result is any one of the multiple quality defect type judgment results, and the detection result includes the second detection result.

[0016] Optionally, in some embodiments, the facial appearance prediction model includes an image generation network and a control neural network of the image generation network; the method further includes:

[0017] Constructing a first training sample of the image generation network based on facial image sample data and image description data of the facial image sample data, and constructing a second training sample of the control neural network based on the facial image sample data and facial key point image data corresponding to the facial image sample data;

[0018] The image generation network is fine-tuned based on the first training sample, and the control neural network is trained as an appearance prediction model based on the second training sample to obtain the facial appearance prediction model.

[0019] Optionally, in some embodiments, the facial key point image data includes facial key point connection image data;

[0020] The generating of facial key point image data based on the first coordinate information includes:

[0021] The plurality of facial key points in the facial region image data are sequentially connected based on the first coordinate information to obtain the facial key point connected image data.

[0022] Optionally, in some embodiments, the performing facial key point detection on the facial region image data includes:

[0023] The facial area image is input into a preset facial key point detection model, and based on the deep convolutional neural network model in the facial key point detection model, heat map detection is performed on the facial area image data; and based on the key point detection module in the facial key point detection model, facial key point detection is performed on the detection result output by the heat map detection of the deep convolutional neural network model.

[0024] Optionally, in some embodiments, extracting facial region image data from the four-dimensional color ultrasound image data includes:

[0025] Inputting the four-dimensional color ultrasound image data into a preset face region detection model, performing face region detection on the four-dimensional color ultrasound image data based on the face region detection model, and obtaining second coordinate information of the face region in the four-dimensional color ultrasound image data;

[0026] The four-dimensional color ultrasound image data is cropped based on the second coordinate information to obtain facial area image data in the four-dimensional color ultrasound image data.

[0027] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a device for predicting facial images based on four-dimensional color Doppler ultrasound, the device comprising:

[0028] A color ultrasound image processing module, configured to obtain four-dimensional color ultrasound image data and extract facial region image data from the four-dimensional color ultrasound image data;

[0029] A facial key point detection module is configured to perform facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data; and generate facial key point image data based on the first coordinate information;

[0030] The appearance prediction module is used to perform appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0031] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a device for predicting facial images based on four-dimensional color ultrasound, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method for predicting facial images based on four-dimensional color ultrasound described in the first aspect.

[0032] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for predicting facial images based on four-dimensional color ultrasound described in the first aspect.

[0033] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a computer program product, which includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the method for predicting facial images based on four-dimensional color ultrasound described in the first aspect.

[0034] The embodiments of the present application propose a method, apparatus, device, computer-readable storage medium, and computer program product for predicting facial images based on four-dimensional color ultrasound. The method, apparatus, and computer-readable storage medium acquire four-dimensional color ultrasound image data, extract facial region image data from the four-dimensional color ultrasound image data, perform facial key point detection on the facial region image data, and obtain first coordinate information of multiple facial key points in the facial region image data; generate facial key point image data based on the first coordinate information; and perform appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0035] That is to say, after extracting the facial region image data from the four-dimensional color ultrasound image data, the embodiment of the present application further performs facial key point detection on the facial region image data, and generates facial key point image data based on the first coordinate information of the detected multiple facial key points, and finally performs appearance prediction processing on the facial key point image data through the facial appearance prediction model, thereby obtaining a facial image prediction result corresponding to the four-dimensional color ultrasound image data. In this way, for the user, only the four-dimensional color ultrasound image data needs to be provided to achieve end-to-end facial image prediction and obtain the corresponding facial image prediction result. The user no longer needs to adjust the prediction result and other operations. This can not only improve the overall efficiency of the user's use of four-dimensional color ultrasound image data for facial image prediction, but also avoid errors caused by manual intervention by the user, thereby effectively ensuring the stability and accuracy of facial image prediction through four-dimensional color ultrasound.

[0036] In the embodiment of the present application, the four-dimensional color ultrasound image data is a four-dimensional color ultrasound photo of the baby provided to the user. The technical solution provided by the embodiment of the present application can achieve a stable and accurate prediction of the baby's future appearance, thereby ensuring the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0038] Figure 1 A schematic flow chart of the steps in some embodiments of the method for predicting facial images based on four-dimensional color Doppler ultrasound provided in the embodiments of the present application;

[0039] Figure 2 A schematic flow chart of steps in other embodiments of the method for predicting facial images based on four-dimensional color ultrasound provided in an embodiment of the present application;

[0040] Figure 3 for Figure 1 Schematic diagram of the detailed steps of step S101;

[0041] Figure 4 A schematic flow chart of steps in some further embodiments of the method for predicting facial images based on four-dimensional color ultrasound provided in an embodiment of the present application;

[0042] Figure 5 for Figure 2 Schematic diagram of the detailed steps of step S201;

[0043] Figure 6 This is a flowchart of a method for predicting facial images based on four-dimensional color Doppler ultrasound in a specific embodiment of the present application;

[0044] Figure 7 A schematic diagram of a process for obtaining a four-dimensional color ultrasound key point connection diagram involved in a specific embodiment of the method for predicting a facial image based on four-dimensional color ultrasound provided in an embodiment of the present application;

[0045] Figure 8 An example of baby appearance prediction involved in a specific embodiment of the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application;

[0046] Figure 9 A schematic diagram of the structure of a device for predicting facial images based on four-dimensional color ultrasound provided in an embodiment of the present application;

[0047] Figure 10 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0050] Four-dimensional color ultrasound image data.

[0051] Four-dimensional color ultrasound image data, also referred to as 4D color ultrasound, or alternatively as 4D color ultrasound images or 4D color ultrasound photographs, utilizes four-dimensional (4D) medical color ultrasound imaging technology to provide richer imaging information for clinical ultrasound diagnosis. In obstetrics, 4D ultrasound examinations of the fetus can produce three-dimensional 4D color ultrasound image data that displays the fetus's color, face, and organ development. This 4D color ultrasound image data can even provide insight into the fetus's condition within the mother's womb, effectively assisting in the early diagnosis of fetal developmental abnormalities, cardiovascular malformations, and other conditions.

[0052] Artificial intelligence (AI).

[0053] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0054] Natural language processing (NLP).

[0055] Natural language processing (NLP) uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field between computer science and linguistics, often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and discourse understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0056] Prompt.

[0057] The prompt is a crucial concept in artificial intelligence and natural language processing, particularly in large language models (LLMs) such as ChatGPT and DeepSeek. A prompt is user-entered text that guides the model to generate specific outputs. Prompts provide context and instructions for the model, helping it understand the user's intent and generate relevant responses or content. In some practical applications, the specific content of the prompt can be generated from image descriptions.

[0058] Image Caption.

[0059] Image description generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image's visual scene. For example, image description can transform image retrieval into text retrieval, which can be used to classify images and improve image retrieval results. While people can often describe the details of an image's visual scene with just a quick glance, automatically adding descriptions to images is a comprehensive and challenging computer vision task that requires converting the complex information contained in the image into a natural language description. Compared to common computer vision tasks, image captioning requires not only identifying objects in images but also associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires extracting deep features from images, associating them with semantic features, and converting them for description generation.

[0060] Next, the technical background involved in the embodiments of the present application is briefly explained again to facilitate an accurate understanding of the method for predicting facial images based on four-dimensional color ultrasound provided by the embodiments of the present application.

[0061] Expectant parents are often filled with curiosity and anticipation about their baby's future appearance. Knowing their baby's future appearance can reduce the anxiety associated with pregnancy uncertainty and strengthen the emotional connection between parents and their unborn child. This psychological satisfaction and anticipation is beneficial to their mental health.

[0062] With the development of artificial intelligence algorithms, it has become possible to predict a baby's future appearance using intelligent algorithm models. Currently, baby appearance prediction methods available on the market can be divided into two categories. One type of prediction method extracts the parents' facial features for comparative analysis and fusion. Patents such as "Baby Appearance Prediction Method, Device, and Storage Medium Based on Facial Recognition" disclosed in patent publication number CN110287765A embody this principle: facial features are extracted from the prospective parents' photos using facial recognition technology, followed by comparative analysis. Based on the comparative analysis results, the basic facial features are retained, while auxiliary facial features are retained based on feature statistics. Finally, the basic and auxiliary facial features are fused to create a predicted photo of the baby. However, since a baby's development in the womb and birth are often influenced by multiple factors, such as genetics and the environment, the accuracy of this method in predicting a baby's future appearance is relatively low. Another type of prediction method uses four-dimensional color ultrasound images as prior knowledge. Currently, there are no related patents. This method primarily leverages the image-generating capabilities of AI painting software Midjourney and the open-source AI model Stable Diffusion to make predictions. The entire process is not a one-click process and requires professional fine-tuning based on the four-dimensional color ultrasound image. While using four-dimensional color ultrasound images as prior knowledge to predict a baby's future appearance can achieve relatively high accuracy by processing and predicting the baby's facial features and contours from the four-dimensional color ultrasound image, related techniques for predicting a baby's future appearance using four-dimensional color ultrasound images suffer from insufficient stability, easily resulting in distorted and deformed babies that impact the user experience. End-to-end prediction generation is difficult, creating a significant barrier to entry for users. Furthermore, in actual applications, most users often need to fine-tune and repeatedly predict each prediction result. This also makes it more likely that errors during manual intervention will lead to distorted and deformed results, impacting the user experience.

[0063] In summary, related technologies that only compare, analyze, and integrate the parents' facial features have low accuracy in predicting a baby's future appearance. While prediction methods using four-dimensional color ultrasound images, which incorporate prior knowledge about the baby's facial features and general outline from the four-dimensional color images, have high prediction accuracy, this method currently suffers from unstable prediction results and requires manual intervention, making it difficult to implement as an end-to-end prediction service.

[0064] Based on this, an embodiment of the present application provides a method for predicting facial images based on four-dimensional color ultrasound, aiming to ensure the stability and accuracy of predicting the baby's future appearance through four-dimensional color ultrasound.

[0065] The method, device, equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application for predicting facial images based on four-dimensional color ultrasound are specifically illustrated through the following embodiments. First, an embodiment of the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application is described.

[0066] It should be noted that in the various specific implementation methods of the present application described later, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0067] In addition, the embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology and machine learning / deep learning.

[0068] Furthermore, the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running on a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for predicting facial images based on four-dimensional color ultrasound, etc., but is not limited to the above forms.

[0069] Moreover, the method for predicting facial images based on four-dimensional color ultrasound provided by the embodiments of the present application can also be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] For ease of understanding and explanation, the following text will use the method for predicting facial images based on four-dimensional color ultrasound provided by the embodiments of the present application as an example for detailed description. The implementation of the method for predicting facial images based on four-dimensional color ultrasound provided by the embodiments of the present application by any of the above-mentioned entities can refer to the process of the terminal device applying the method for predicting facial images based on four-dimensional color ultrasound described below.

[0071] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps in some embodiments of the method for predicting facial images based on four-dimensional color Doppler ultrasound provided in the present application. Figure 1 The execution order of some method steps is shown in the figure, but based on the different design requirements of actual applications, the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1The order of the steps in the method shown does not constitute a limitation on the execution logic order of the method for predicting facial images based on four-dimensional color Doppler ultrasound provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the steps of the method shown should be included in the scope of protection of the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application.

[0072] like Figure 1 As shown, in some embodiments, the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application may include but is not limited to steps S101 to S104 as shown below.

[0073] Step S101: Acquire four-dimensional color ultrasound image data, and extract facial region image data from the four-dimensional color ultrasound image data.

[0074] In the process of facial image prediction based on four-dimensional color ultrasound, the terminal device first obtains four-dimensional color ultrasound image data, and performs facial area detection on the four-dimensional color ultrasound image data by adopting a target detection algorithm, thereby extracting facial area image data from the four-dimensional color ultrasound image data.

[0075] Step S102: performing facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data.

[0076] After extracting the facial area image data from the four-dimensional color ultrasound image data, the terminal device further performs facial key point detection on the extracted facial area image data, thereby detecting multiple facial key points such as eyes, nose, mouth, facial contours, etc. in the facial area image data, and then obtains the first coordinate information of each of these multiple facial key points in the facial area image data.

[0077] Step S103: Generate facial key point image data based on the first coordinate information.

[0078] After the terminal device obtains the first coordinate information of multiple facial key points in the facial region image data, it can directly use the first coordinate information to generate facial key point image data. The facial key point image data can at least reflect the positional relationship between the multiple facial key points.

[0079] Step S104: performing appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0080] After generating facial key point image data, the terminal device uses this facial key point image data as input data for a pre-built and trained facial appearance prediction model, which then performs facial appearance prediction on the facial key point image data. After the facial appearance prediction model completes the appearance prediction and outputs the predicted facial image data, the terminal device uses the facial image data output by the facial appearance prediction model as the facial image prediction result corresponding to the 4D color ultrasound image data.

[0081] In steps S101 to S104 shown in the embodiment of the present application, a terminal device acquires four-dimensional color ultrasound image data and uses a target detection algorithm to perform facial region detection on the four-dimensional color ultrasound image data, thereby extracting facial region image data from the four-dimensional color ultrasound image data. Subsequently, the terminal device performs facial key point detection on the extracted facial region image data, thereby detecting multiple facial key points such as eyes, nose, mouth, and facial contours in the facial region image data, and obtaining the first coordinate information of each of these multiple facial key points in the facial region image data. Then, the terminal device directly uses the obtained first coordinate information to draw and generate facial key point image data. Finally, the terminal device uses a facial appearance prediction model to perform appearance prediction on the facial key point image data, and the facial image data output by the facial appearance prediction model is used as the facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0082] In this way, for users, they only need to provide four-dimensional color ultrasound image data to achieve end-to-end facial image prediction and obtain the corresponding facial image prediction results. Users no longer need to adjust the prediction results. This can not only improve the overall efficiency of users using four-dimensional color ultrasound image data for facial image prediction, but also avoid errors caused by user manual intervention, thereby effectively ensuring the stability and accuracy of facial images predicted by four-dimensional color ultrasound.

[0083] In addition, when the four-dimensional color ultrasound image data in the embodiment of the present application is a four-dimensional color ultrasound photo of the baby provided to the user, a stable and accurate prediction of the baby's future appearance can be achieved, thereby ensuring the user experience.

[0084] Please refer to Figure 2 , Figure 2 A schematic flow chart of the steps of the method for predicting facial images based on four-dimensional color ultrasound provided in an embodiment of the present application in other embodiments.

[0085] like Figure 2As shown, in some embodiments, in the above-mentioned step S104: performing appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data, the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiment of the present application may also include but is not limited to steps S201 and S202 as shown below.

[0086] Step S201: performing a prediction quality test on the face image prediction result to obtain a test result.

[0087] After the terminal device uses the facial appearance prediction model to perform appearance prediction on the facial key point image data, thereby obtaining a facial image prediction result corresponding to the four-dimensional color ultrasound image data, it further performs a prediction quality detection operation on the facial image prediction result to determine whether the facial image prediction result predicted by the facial appearance prediction model has quality defects such as deformity, thereby obtaining the corresponding detection result.

[0088] It should be noted that the detection results obtained by the terminal device by further performing a prediction quality test on the facial image prediction result include at least: a detection result indicating that a deformed facial image exists in the facial image prediction result, and a detection result indicating that a deformed facial image does not exist in the facial image prediction result. For example, when the terminal device performs a prediction quality test on the facial image prediction result, and determines that the facial image prediction result has quality defects such as deformity, the detection result indicates that a deformed facial image exists in the facial image prediction result. For another example, when the terminal device performs a prediction quality test on the facial image prediction result, and determines that the facial image prediction result does not have quality defects such as deformity, the detection result indicates that a deformed facial image does not exist in the facial image prediction result.

[0089] Step S202: When the detection result indicates that the facial image prediction result contains a deformed facial image, the step of performing appearance prediction processing on the facial key point image data based on the preset facial appearance prediction model is re-executed to obtain a new facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0090] After the terminal device obtains the corresponding detection result by performing a prediction quality test on the facial image prediction result, if the obtained detection result indicates that the facial image prediction result contains a deformed facial image, then in this case, the terminal device directly and automatically returns to re-execute the above-mentioned step S104, so as to use the facial appearance prediction model to perform appearance prediction on the facial key point image data again, and use the facial image data re-output by the facial appearance prediction model as the new facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0091] In some embodiments, after obtaining a new facial image prediction result, the terminal device still repeats the above steps S201 and S202, and repeats this cycle until the facial key point image data is automatically predicted through the facial appearance prediction model to obtain a facial image prediction result without quality defects such as deformity.

[0092] In some embodiments, after the terminal device obtains the corresponding detection result by performing a prediction quality test on the facial image prediction result, if the detection result indicates that there is no deformed facial image in the facial image prediction result, the terminal device outputs and displays the facial image prediction result to the user.

[0093] In steps S201 and S202 of this embodiment, the terminal device performs a prediction quality test on the facial image prediction result to determine whether the facial image prediction result obtained by the facial appearance prediction model contains quality defects such as deformities, thereby obtaining a corresponding test result. If the test result indicates that the facial image prediction result contains a deformed facial image, the terminal device automatically returns to re-execute step S104 to re-perform facial image prediction on the facial key point image data using the facial appearance prediction model. The facial image data re-output by the facial appearance prediction model is used as the new facial image prediction result corresponding to the four-dimensional color ultrasound image data. This further ensures the stability and accuracy of facial image prediction using the facial appearance prediction model. Furthermore, during the process of re-predicting the facial image using the facial appearance prediction model, manual user intervention is still not required. Instead, the entire process from defect detection to re-prediction of the facial image is completed automatically by the terminal device, ensuring the smooth completion of the end-to-end prediction service and effectively improving the user experience.

[0094] In this way, when the four-dimensional color ultrasound image data is a four-dimensional color ultrasound image of a baby provided by a user, the four-dimensional color ultrasound image is used by the terminal device to predict the baby's future appearance. First, the four-dimensional color ultrasound face area detection model is used to obtain the four-dimensional color ultrasound face area image, and then the image is used to perform facial key point detection to obtain coordinate information. After that, the coordinate information is used to draw a key point connection diagram as a control condition to predict the baby's future appearance. Finally, the defect detection model is used to detect the generated image to ensure the generation quality. In this way, the accuracy and stability of the prediction results of the baby's future appearance predicted using the four-dimensional color ultrasound image can be effectively guaranteed.

[0095] Please refer to Figure 3 , Figure 3 yes Figure 1 Schematic diagram of the detailed process flow of step S101.

[0096] like Figure 3 As shown, in some embodiments, the step of "extracting facial area image data from the four-dimensional color ultrasound image data" in the above-mentioned step S101 may include but is not limited to the following steps S301 and S302.

[0097] Step S301: inputting the four-dimensional color ultrasound image data into a preset face area detection model, performing face area detection on the four-dimensional color ultrasound image data based on the face area detection model, and obtaining second coordinate information of the face area in the four-dimensional color ultrasound image data.

[0098] In the process of performing face area detection on the acquired four-dimensional color ultrasound image data, the terminal device can obtain face area coordinate information by adopting a face area detection model that has been pre-constructed and trained based on the target detection algorithm. That is, the terminal device uses the four-dimensional color ultrasound image data as input data of the face area detection model, and then performs face area detection on the four-dimensional color ultrasound image data based on the face area detection model to obtain the second coordinate information of the face area in the four-dimensional color ultrasound image data.

[0099] Step S302: cropping the four-dimensional color ultrasound image data based on the second coordinate information to obtain facial region image data in the four-dimensional color ultrasound image data.

[0100] After obtaining the second coordinate information of the face area in the four-dimensional color ultrasound image data, the terminal device can further crop the four-dimensional color ultrasound image data based on the second coordinate information, thereby cropping the face area image data in the four-dimensional color ultrasound image data.

[0101] For example, the terminal device can obtain the four-dimensional color ultrasound photo of the baby uploaded by the user through the human-computer interaction interface provided to the user. After that, the terminal device obtains the coordinate information of the baby's face area in the four-dimensional color ultrasound photo by calling the four-dimensional color ultrasound image face area detection model obtained by pre-training the model using the target detection algorithm YOLOX, and then crops the four-dimensional color ultrasound photo according to the coordinate information, thereby obtaining the baby's face area image in the four-dimensional color ultrasound photo.

[0102] It should be noted that the face area detection model can be a four-dimensional color ultrasound face area detection model: target detection algorithm YOLOX model.

[0103] In some embodiments, a 4D color Doppler ultrasound facial region detection model may be established in the following manner:

[0104] Training data annotation: Collect four-dimensional color ultrasound image data, and then manually annotate the coordinate information of the face area in the four-dimensional color ultrasound image.

[0105] Training data preprocessing: The training data is uniformly scaled to 512*512 size, and data enhancement operations such as image rotation, translation, and blur are performed on the data to improve the robustness of the model.

[0106] Model training: Training data is fed into the YOLOX model, which predicts the coordinates of the facial regions. These predicted coordinates often deviate from the manually annotated coordinates. This deviation is used to calculate a loss function. The entire training process aims to minimize this loss function. Once the loss function stabilizes, a 4D color Doppler ultrasound facial region detection model is obtained.

[0107] Among them, the loss function of the YOLOX model can be:

[0108] L=L cls +L obj +λL reg ,

[0109] Among them, L cls is the classification loss, L obj is the confidence loss, L reg is the positioning loss, and λ is the balance coefficient of positioning loss. cls and L obj Both use binary cross entropy loss function (BCE Loss), Among them, A is the intersection of the predicted target box and the real box, B is the union of the predicted target box and the real box, D is the Euclidean distance between the center of the predicted target box and the center of the real box, and Dc is the diagonal distance of the minimum circumscribed rectangle formed by the predicted target box and the real box.

[0110] Based on this, the terminal device can obtain the coordinate information of the face area output by the four-dimensional color ultrasound face area detection model by inputting the four-dimensional color ultrasound image into the trained four-dimensional color ultrasound face area detection model.

[0111] In this embodiment, the terminal device is trained and the four-dimensional color ultrasound face area detection model is used to process the four-dimensional color ultrasound image to filter out the non-face areas therein, which can improve the accuracy of subsequent four-dimensional color ultrasound detection of facial key points, thereby helping to improve the accuracy of appearance prediction.

[0112] In some embodiments, the step of “detecting facial key points on the facial region image data” in step S102 may include but is not limited to the following steps:

[0113] The facial area image is input into a preset facial key point detection model, and based on the deep convolutional neural network model in the facial key point detection model, heat map detection is performed on the facial area image data; and based on the key point detection module in the facial key point detection model, facial key point detection is performed on the detection result output by the heat map detection of the deep convolutional neural network model.

[0114] In the process of detecting facial key points (such as eyes, nose, mouth, facial contours, etc.) on facial area image data, the terminal device uses the facial area image as input data for the constructed and trained facial key point detection model, and then first performs heat map detection on the facial area image data based on the deep convolutional neural network model in the facial key point detection model. Moreover, based on the key point detection module in the facial key point detection model, the detection result of the heat map detection output by the deep convolutional neural network model is further detected for facial key points, so that the first coordinate information of multiple facial key points can be obtained.

[0115] It should be noted that the facial key point detection model can be a four-dimensional color ultrasound image facial key point detection model, which is composed of the following modules: first, a deep convolutional neural network model (such as a ResNet50 network) is used to extract input image features and predict heatmaps. Finally, the detection results (heatmaps) output by the deep convolutional neural network model are input into the key point detection module (such as a dsnt module) to obtain the coordinate positions (first coordinate information) of the facial key points.

[0116] In some embodiments, a 4D color ultrasound facial key point detection model can be established by:

[0117] Training data annotation: Use the facial region coordinate information of the labeled 4D color ultrasound facial region detection model training data set to crop the facial region, and then annotate the coordinate positions of key points such as the eyes, nose, mouth, and facial contour.

[0118] Training data preprocessing: The training data is uniformly scaled to 416*416 size, and image rotation and blurring are performed on the data to improve the robustness of the model.

[0119] Model training: Training data is fed into the 4D color ultrasound facial landmark detection model. The model predicts the coordinates of facial landmarks. The deviation between the predicted coordinates and the labeled coordinates is used to calculate the loss function. The entire training process aims to minimize the loss function. Once the loss function stabilizes, the 4D color ultrasound facial landmark detection model is obtained.

[0120] Among them, the loss function of the four-dimensional color Doppler ultrasound face key point detection model can be:

[0121] L euc (u,p)+λL D (Z,p),

[0122] Among them, λ is the weight of the regularization constraint, L euc (u,p)=||pu||2, u is the predicted coordinate and p is the true coordinate. D(.||.) is the JS divergence (Jensen-Shannon divergence), p(c) is the predicted distribution, is centered at the true coordinate p, and the variance is Gaussian distribution.

[0123] Based on this, by inputting the four-dimensional color ultrasound face area picture (face area image data) into the trained four-dimensional color ultrasound face key point detection model through the terminal device, the first coordinate information of multiple face key points can be obtained.

[0124] In some embodiments, the above-mentioned facial key point image data includes facial key point connection image data. Based on this, the above-mentioned step S103: generating facial key point image data based on the first coordinate information may include but is not limited to the following steps:

[0125] The plurality of facial key points in the facial region image data are sequentially connected based on the first coordinate information to obtain the facial key point connected image data.

[0126] Since each facial key point is represented by only one pixel in the entire facial area image data, its proportion is relatively small. In order to ensure that the facial key point image data can provide more useful information to guide the algorithm to predict facial images, the terminal device can also, after drawing and generating facial key point image data, further use AI painting software and other tools to connect multiple facial key points in the key point map according to relationships such as eyes, nose, mouth, and facial contours, thereby further drawing and generating a key point connection map, and then using the key point connection map as the facial key point image data generated based on the first coordinate information.

[0127] In this embodiment, a terminal device further draws a key point connection diagram as facial key point image data that is subsequently input into the facial appearance prediction model. This key point connection diagram can be used to control the similarity between the prediction result and the 4D color ultrasound image. Specifically, the terminal device first uses the 4D color ultrasound image facial key point detection model to obtain the coordinate information of key points such as the facial contour, mouth, nose, and eyes. The corresponding key points are then connected with lines of different colors to increase the proportion of the key points in the entire image, thereby ultimately enhancing the control effect of the prediction result.

[0128] In some embodiments, the above-mentioned facial appearance prediction model may include an image generation network (Stable Diffusion, SD), and a control neural network of the SD network (such as a ControlNet network).

[0129] Please refer to Figure 4 , Figure 4 A schematic flow chart of the steps of the method for predicting facial images based on four-dimensional color ultrasound provided in the embodiments of the present application in some further embodiments.

[0130] like Figure 4 As shown, in some embodiments, the method for predicting facial images based on four-dimensional color ultrasound provided by the embodiments of the present application may also include steps S401 and S402 as shown below.

[0131] Step S401: constructing a first training sample of the image generation network based on facial image sample data and image description data of the facial image sample data, and constructing a second training sample of the control neural network based on the facial image sample data and facial key point image data corresponding to the facial image sample data.

[0132] Before the terminal device formally adopts the facial appearance prediction model to perform appearance prediction processing on the facial key point image data, it can also first construct a first training sample of the image generation network based on the facial image sample data and the image description data of the facial image sample data, and construct a second training sample of the control neural network based on the facial image sample data and the facial key point image data corresponding to the facial image sample data.

[0133] Exemplarily, the terminal device collects baby pictures aged 0-3 months, describes each picture (face image sample data) to form a prompt word prompt (image description data), and finally forms the prompt word prompt and the baby picture into the training data (first training sample) of the baby face generation model (SD network). In addition, the baby pictures aged 0-3 months are input into the face key point detection model to generate the corresponding key point graph, and then the key points on the graph are connected to form a key point connection graph (face key point image data), and finally form the training data (second training sample) of the ControlNet network.

[0134] It should be noted that the prompt word is mainly used to help the large model understand the input intent and guide the model to generate prediction results.

[0135] Step S402: performing model fine-tuning training on the image generation network based on the first training sample, and performing appearance prediction model training on the control neural network based on the second training sample to obtain the face appearance prediction model.

[0136] After constructing the first training sample and the second training sample, the terminal device uses the first training sample as the input of the general model - SD network to perform model fine-tuning training on the SD network, and uses the second training sample as the input of the control neural network to train the appearance prediction model for the control neural network. In this way, the final facial appearance prediction model can be trained.

[0137] For example, considering that open-source SD networks are usually pre-trained on large-scale, general-purpose datasets that cover a wide range of content and styles, when SD networks are applied to specific fields, the general model may not be able to capture the specific data features and details of the field well. As a result, when used to generate baby face pictures of 0-3 months old, it is often easy to generate deformed faces. Therefore, further fine-tuning training is required to ensure the stability of the generated quality. Therefore, after the terminal device uses the prompt word prompt and the baby picture to form the training data of the baby face generation model (SD network), it uses the prompt as the input of the SD network and the baby picture as the output of the SD model, thereby fine-tuning the SD model to generate baby face pictures of stable quality.

[0138] In addition, after constructing the training data for the ControlNet network, the terminal device uses the key point connection graph as the input of the ControlNet model and the baby's face picture with the corresponding posture as the output, thereby training the ControlNet model so that it can control the baby's facial posture through the key point connection graph.

[0139] Based on this, the terminal device can construct a facial appearance prediction model SD+ControlNet. When using the facial appearance prediction model to predict the baby's future appearance, the terminal device uses the prompt+key point connection diagram (facial key point image data) as the input of SD+ControlNet, thereby obtaining the baby's future appearance picture (facial image prediction result) output by the facial appearance prediction model SD+ControlNet, which is similar to the features of the facial area of ​​the four-dimensional color ultrasound image and consistent with the posture.

[0140] Among them, the value of the positive prompt can be "a cute baby, with closed eyes and closed mouth, a 2-month-old baby, soft movie light, (ultra-detailed photo: 1.2), fair skin, (high resolution: 2.0), centered composition, 8k", which is used to guide the generation direction of the SD network; the value of the negative prompt can be "(blurry: 5.0), (ugly: 1.6), (disfigured: 1.6), poorly drawn face, (bald: 1.40, sick, incomplete), which is used to prohibit the SD network from generating related content.

[0141] In this embodiment, the terminal device uses the key point connection diagram and prompt to predict the baby's future appearance, so as to realize the control model to generate baby face pictures with specified posture and style. That is, the terminal device uses the prompt and the baby's artistic photo style picture to fine-tune the SD model so that the style of the picture generated by it is biased towards artistic photos, thereby improving the beauty of the prediction result. In addition, the key point connection diagram is used as a control condition to input the ControlNet network, so that it can control the baby's facial posture in the generated picture and improve the similarity with the four-dimensional color ultrasound image.

[0142] Please refer to Figure 5 , Figure 5 for Figure 2 Schematic diagram of the detailed steps of step S201.

[0143] like Figure 5 As shown, in some embodiments, the above-mentioned step S201: performing prediction quality detection on the facial image prediction result to obtain a detection result may include but is not limited to steps S501 to S503 shown below.

[0144] Step S501: Input the facial image prediction result into a preset quality detection model, and judge the quality defect type of the facial image prediction result based on the quality detection model to obtain multiple quality defect type judgment results corresponding to the facial image prediction result; the quality detection model is obtained based on multi-label classification model training of a deep convolutional neural network model.

[0145] During the process of detecting the facial image prediction results, the terminal device can use the facial image prediction results as input data for a quality detection model obtained in advance based on multi-label classification model training of a deep convolutional neural network model, thereby performing quality defect type judgment on the facial image prediction results based on the quality detection model, and obtaining multiple quality defect type judgment results corresponding to the facial image prediction results.

[0146] For example, the quality detection model may be a defect detection model. The defect detection model may be established in the following manner:

[0147] Training Data Annotation: We used SD+ControlNet to generate a large number of baby face images. We then selected images with facial deformities for manual annotation. The annotations were divided into eight categories: deformed eyes, deformed noses, deformed mouths, deformed ears, deformed chins, deformed facial contours, deformed hands, and deformed feet.

[0148] Training data preprocessing: The labeled image data is uniformly scaled to 224*224 size, and the data is rotated and blurred to improve the robustness of the model.

[0149] Model training: A ResNet50 model is trained using the training data for multi-label classification. The deviation between the predicted and the labeled actual categories for each category is used to calculate the binary cross-entropy loss function. The entire training process aims to minimize this loss function. Therefore, training is terminated when the loss function stabilizes, resulting in a defect detection model.

[0150] Based on this, the terminal device inputs the generated baby picture (face image prediction result) into the defect detection model, and can obtain the judgment results of the 8 deformity categories output by the defect detection model (quality defect type judgment results).

[0151] Step S502: When the confidence scores of the multiple quality defect type judgment results are all less than a preset confidence threshold, a first detection result is obtained indicating that there is no deformed facial image in the facial image prediction result; the detection result includes the first detection result.

[0152] After the terminal device uses the quality detection model to determine the quality defect type of the facial image prediction result, thereby obtaining multiple quality defect type determination results corresponding to the facial image prediction result, the terminal device further compares the confidence scores of each of the multiple quality defect type determination results with a preset confidence threshold (e.g., 0.5). If the terminal device finds through comparison that the confidence scores of each of the multiple quality defect type determination results are all less than the preset confidence threshold, the terminal device can confirm that a first detection result indicating that the facial image prediction result does not contain a deformed facial image has been obtained, and the first detection result is one of the aforementioned detection results.

[0153] Step S503: When the confidence score of the target judgment result is greater than or equal to a preset confidence threshold, a second detection result is obtained indicating that a deformed facial image exists in the facial image prediction result; the target judgment result is any one of the multiple quality defect type judgment results, and the detection result includes the second detection result.

[0154] When the terminal device compares the confidence scores of multiple quality defect type judgment results with a preset confidence threshold, if it is found that the confidence score of any target judgment result among the multiple quality defect type judgment results is greater than or equal to the preset confidence threshold, then in this case, the terminal device confirms that a second detection result indicating that there is a deformed facial image in the facial image prediction result is obtained, and the second detection result is another one of the aforementioned detection results.

[0155] In this embodiment, the terminal device performs further quality defect detection on the facial image prediction results, which can further ensure the quality of the prediction results and improve the user experience. Specifically, the terminal device divides the quality defects of the facial image prediction results into eight categories (of course, more or fewer categories can be set based on the needs of actual applications) such as deformed eyes, deformed nose, deformed mouth, deformed ears, deformed chin, deformed facial contour, deformed hands, and deformed feet. Then, a deep convolutional neural network model (such as the ResNet50 model) is trained for multi-label classification, so that the type of defect can be accurately identified, thereby ensuring the quality of the prediction results and improving the user experience.

[0156] Next, we propose an embodiment of the present application's method for predicting facial images based on four-dimensional color ultrasound, using the aforementioned algorithm models (four-dimensional color ultrasound face region detection model, four-dimensional color ultrasound face key point detection model, face appearance prediction model, and defect detection model) to predict a baby's future appearance. Embodiment 1 is an embodiment of predicting a baby's future appearance based on four-dimensional color ultrasound, embodiment 2 is an embodiment of obtaining a facial key point connection graph based on four-dimensional color ultrasound, and embodiment 3 is an embodiment of predicting a baby's appearance using a key point connection graph.

[0157] Please refer to Figure 6 , Figure 6 This is an overall flow chart of a specific embodiment of the method for predicting facial images based on four-dimensional color ultrasound provided in an embodiment of the present application.

[0158] like Figure 6 As shown, the overall process of Example 1 mainly includes:

[0159] S1: The user uploads the baby's 4D color ultrasound image through a smart device (such as a mobile phone or other terminal device);

[0160] S2: The smart device calls the 4D color ultrasound face area detection model and then outputs the coordinate information of the four vertices of the face area on the 4D color ultrasound photo;

[0161] S3: The smart device uses the coordinate information of the four vertices of the face area to crop the original 4D color ultrasound image and filter out redundant background;

[0162] S4: The smart device calls the 4D color ultrasound facial key point detection model, performs key point detection on the cropped 4D color ultrasound photo, and outputs the corresponding facial key point coordinate information;

[0163] S5: The smart device uses the facial key point coordinate information to draw a key point connection diagram;

[0164] S6: The smart device inputs the facial key point connection map into the facial appearance prediction model, which outputs a predicted image of the baby's future appearance. The facial appearance prediction model consists of a fine-tuned SD model and a ContorlNet model.

[0165] S7: The smart device calls the defect detection model to perform defect judgment on the predicted baby's future appearance picture. If there is a defect, it returns to S6 for re-prediction. Otherwise, it directly outputs the predicted baby's future appearance picture to the user.

[0166] It should be noted that the image processing flow of detecting key points and drawing the key point connection diagram in the above steps S4 to 5 can be as follows: Figure 7 It should be understood that Figure 7The number of key points shown in the figure is for convenience of explanation only. Based on the different design requirements of actual applications, more key points can certainly be designed and adopted. However, in general, the more key points are selected, the more similar the appearance picture finally predicted by the model will be to the four-dimensional color ultrasound image.

[0167] Please refer to Figure 7 The main process of Example 2 includes:

[0168] S1: Input 4D color ultrasound facial region image;

[0169] S2: Call the 4D color Doppler ultrasound facial region key point detection model to obtain the coordinate information of key points of facial contour, mouth, nose, eyes, etc. This information can be used to guide the baby's future appearance prediction model to generate a face similar to the 4D color Doppler ultrasound image;

[0170] S3: Since key points account for a small proportion of the entire image and contain little information, they are not conducive to model learning and prediction. Therefore, according to the coordinate information of key points such as the facial contour, mouth, nose, and eyes, different colored lines are connected to form a key point connection map.

[0171] In addition, please refer to Figure 8 The main process of Example 3 includes:

[0172] S1: Input positive and negative prompts to the SD model. The positive prompt is used to guide the model to generate baby face images, while the negative prompt is used to prohibit the model from generating specific content. Since the SD model has been fine-tuned using a dataset of artistic baby photos, the generated images are in the style of artistic photos.

[0173] S2: The key point connection diagram is input into ControlNet as a control condition to control the generated baby's facial contour, mouth, nose, eyes and other parts to be similar to the 4D color ultrasound image. In addition, the baby's face orientation in the image can be controlled to be similar to the 4D color ultrasound image. Figure 1 This makes the prediction results more convincing and improves the user experience.

[0174] See also Figure 9 The present application also provides a device for predicting facial images based on four-dimensional color ultrasound, which can implement the above-mentioned method for predicting facial images based on four-dimensional color ultrasound. The device includes:

[0175] A color ultrasound image processing module, configured to obtain four-dimensional color ultrasound image data and extract facial region image data from the four-dimensional color ultrasound image data;

[0176] A facial key point detection module is configured to perform facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data; and generate facial key point image data based on the first coordinate information;

[0177] The appearance prediction module is used to perform appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0178] In some embodiments, the apparatus for predicting facial images based on four-dimensional color Doppler ultrasound provided by the embodiments of the present application further includes:

[0179] A defect detection module is used to perform a prediction quality test on the face image prediction result to obtain a test result;

[0180] The appearance prediction module is also used to re-execute the step of performing appearance prediction processing on the facial key point image data based on the preset facial appearance prediction model when the detection result indicates that the facial image prediction result contains a deformed facial image, so as to obtain a new facial image prediction result corresponding to the four-dimensional color ultrasound image data.

[0181] In some embodiments, the defect detection module is further used to input the facial image prediction result into a preset quality detection model, perform quality defect type judgment on the facial image prediction result based on the quality detection model, and obtain multiple quality defect type judgment results corresponding to the facial image prediction result; the quality detection model is obtained based on multi-label classification model training of a deep convolutional neural network model; when the confidence scores of the multiple quality defect type judgment results are all less than a preset confidence threshold, a first detection result is obtained indicating that there is no deformed facial image in the facial image prediction result; the detection result includes the first detection result; and, when the confidence score of the target judgment result is greater than or equal to the preset confidence threshold, a second detection result is obtained indicating that there is a deformed facial image in the facial image prediction result; the target judgment result is any one of the multiple quality defect type judgment results, and the detection result includes the second detection result.

[0182] In some embodiments, the facial appearance prediction model includes an image generation network and a control neural network of the image generation network; the apparatus for predicting facial images based on four-dimensional color Doppler ultrasound provided in an embodiment of the present application further includes:

[0183] A model training module is used to construct a first training sample of the image generation network based on facial image sample data and image description data of the facial image sample data, and to construct a second training sample of the control neural network based on the facial image sample data and facial key point image data corresponding to the facial image sample data; and to perform model fine-tuning training on the image generation network based on the first training sample, and to perform appearance prediction model training on the control neural network based on the second training sample to obtain the facial appearance prediction model.

[0184] In some embodiments, the facial key point image data includes facial key point connection image data;

[0185] The facial key point detection module is further used to sequentially connect the multiple facial key points in the facial area image data based on the first coordinate information to obtain the facial key point connection image data.

[0186] In some embodiments, the facial key point detection module is further used to input the facial area image into a preset facial key point detection model, perform heat map detection on the facial area image data based on the deep convolutional neural network model in the facial key point detection model, and perform facial key point detection on the detection result output by the heat map detection of the deep convolutional neural network model based on the key point detection module in the facial key point detection model.

[0187] In some embodiments, the color ultrasound image processing module is also used to input the four-dimensional color ultrasound image data into a preset face area detection model, perform face area detection on the four-dimensional color ultrasound image data based on the face area detection model, and obtain second coordinate information of the face area in the four-dimensional color ultrasound image data; and, based on the second coordinate information, crop the four-dimensional color ultrasound image data to obtain face area image data in the four-dimensional color ultrasound image data.

[0188] It should be noted that the specific implementation of the device for predicting facial images based on four-dimensional color ultrasound provided in the embodiment of the present application is basically the same as the specific implementation of the method for predicting facial images based on four-dimensional color ultrasound mentioned above, and will not be repeated here.

[0189] The present application also provides a device for predicting facial images based on four-dimensional color ultrasound, comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the method for predicting facial images based on four-dimensional color ultrasound. It should be noted that the device for predicting facial images based on four-dimensional color ultrasound can be a computer device, specifically any terminal device such as a tablet computer or an in-vehicle computer.

[0190] See also Figure 10 , Figure 10 The hardware structure of a computer device according to an embodiment is shown. The computer device includes:

[0191] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0192] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the method for predicting facial images based on four-dimensional color ultrasound in the embodiments of this application.

[0193] Input / output interface 1003, used to implement information input and output;

[0194] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0195] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0196] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0197] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for predicting facial images based on four-dimensional color ultrasound.

[0198] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the above-mentioned method for predicting facial images based on four-dimensional color ultrasound.

[0199] It should be noted that the above-mentioned memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0200] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0201] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0203] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0204] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0205] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0207] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0209] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0210] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for predicting facial images based on four-dimensional color Doppler ultrasound, characterized in that: The method comprises: Acquiring four-dimensional color ultrasound image data, and extracting facial region image data from the four-dimensional color ultrasound image data; Performing facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data; Generate facial key point image data based on the first coordinate information; Based on a preset facial appearance prediction model, the facial key point image data is subjected to appearance prediction processing to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

2. The method according to claim 1, characterized in that After performing appearance prediction processing on the facial key point image data based on the preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data, the method further includes: Performing a prediction quality test on the facial image prediction result to obtain a test result; When the detection result indicates that the facial image prediction result contains a deformed facial image, the step of performing appearance prediction processing on the facial key point image data based on the preset facial appearance prediction model is re-executed to obtain a new facial image prediction result corresponding to the four-dimensional color ultrasound image data.

3. The method according to claim 2, characterized in that The performing prediction quality detection on the face image prediction result to obtain a detection result includes: Inputting the facial image prediction result into a preset quality detection model, and performing quality defect type judgment on the facial image prediction result based on the quality detection model to obtain multiple quality defect type judgment results corresponding to the facial image prediction result; the quality detection model is obtained by training a deep convolutional neural network model for multi-label classification; When the confidence scores of the plurality of quality defect type judgment results are all less than a preset confidence threshold, obtaining a first detection result indicating that the facial image prediction result does not contain a deformed facial image; the detection result includes the first detection result; When the confidence score of the target judgment result is greater than or equal to a preset confidence threshold, a second detection result is obtained indicating that a deformed facial image exists in the facial image prediction result; the target judgment result is any one of the multiple quality defect type judgment results, and the detection result includes the second detection result.

4. The method according to claim 1, wherein The face appearance prediction model includes an image generation network and a control neural network of the image generation network; the method further includes: Constructing a first training sample of the image generation network based on facial image sample data and image description data of the facial image sample data, and constructing a second training sample of the control neural network based on the facial image sample data and facial key point image data corresponding to the facial image sample data; The image generation network is fine-tuned based on the first training sample, and the control neural network is trained as an appearance prediction model based on the second training sample to obtain the facial appearance prediction model.

5. The method according to claim 1, wherein The facial key point image data includes facial key point connection image data; The generating of facial key point image data based on the first coordinate information includes: The plurality of facial key points in the facial region image data are sequentially connected based on the first coordinate information to obtain the facial key point connected image data.

6. The method according to claim 1, characterized in that The detecting of facial key points on the facial region image data comprises: The facial area image is input into a preset facial key point detection model, and based on the deep convolutional neural network model in the facial key point detection model, heat map detection is performed on the facial area image data; and based on the key point detection module in the facial key point detection model, facial key point detection is performed on the detection result output by the heat map detection of the deep convolutional neural network model.

7. The method according to claim 1, characterized in that The extracting of facial region image data from the four-dimensional color ultrasound image data comprises: Inputting the four-dimensional color ultrasound image data into a preset face region detection model, performing face region detection on the four-dimensional color ultrasound image data based on the face region detection model, and obtaining second coordinate information of the face region in the four-dimensional color ultrasound image data; The four-dimensional color ultrasound image data is cropped based on the second coordinate information to obtain facial area image data in the four-dimensional color ultrasound image data.

8. A device for predicting facial images based on four-dimensional color ultrasound, characterized in that: The device comprises: A color ultrasound image processing module, configured to obtain four-dimensional color ultrasound image data and extract facial region image data from the four-dimensional color ultrasound image data; A facial key point detection module is configured to perform facial key point detection on the facial region image data to obtain first coordinate information of a plurality of facial key points in the facial region image data; and generate facial key point image data based on the first coordinate information; The appearance prediction module is used to perform appearance prediction processing on the facial key point image data based on a preset facial appearance prediction model to obtain a facial image prediction result corresponding to the four-dimensional color ultrasound image data.

9. A device for predicting facial images based on four-dimensional color ultrasound, characterized in that: The device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for predicting facial images based on four-dimensional color ultrasound according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the method for predicting facial images based on four-dimensional color ultrasound according to any one of claims 1 to 7 is implemented.

Images