System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting echo images; means for pre-processing the input echo images into a standard format; generative model means for generating a three dimensional model of a fetus from the pre-processed echo images; means for inputting genetic information; means for analyzing the input genetic information to predict a future appearance; and means for displaying the generated three dimensional model and the predicted future appearance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, it is difficult to accurately understand the specific appearance and future shape of the fetus using only ultrasound images provided during pregnancy. Furthermore, there is a lack of means to predict the fetus's appearance and future characteristics, which often leaves future parents feeling anxious about the unknown. For this reason, new technology is needed to help pregnant women and their families understand the fetus more realistically and spend their pregnancy with peace of mind. [Means for solving the problem]

[0005] This invention solves the above-mentioned problems by providing a system including: means for inputting ultrasound images; means for preprocessing the input ultrasound images and converting them into a standard format; means for generating a 3D model of the fetus from the preprocessed ultrasound images; means for inputting genetic information; means for predicting future appearance by analyzing the input genetic information; and means for displaying the generated 3D model and the predicted future appearance. This system allows pregnant women and their families to concretely see the realistic appearance and future shape of the fetus, providing a moving and meaningful moment and reducing anxiety during pregnancy.

[0006] An "echo image" is an image produced using ultrasound technology to visualize the internal structures of a fetus.

[0007] A "standard format" is a common data format that is applied to unify multiple data formats.

[0008] "Preprocessing" refers to the initial processing of input data to make it easier to analyze.

[0009] A "generative model means" is an algorithm or software that generates new data based on input data.

[0010] "Genetic information" is data about DNA that determines an individual's genetic characteristics.

[0011] "Analysis" is the process of breaking down data to clarify its structure and properties.

[0012] "Prediction" is the estimation of future conditions or results based on current data and information.

[0013] A "3D model" is a data structure that represents an object that has three spatial dimensions.

[0014] "Display" means outputting data or information in a visual form. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information. Specifically, the following process is performed.

[0037] First, a user accesses the system's dedicated application or website using a smartphone or computer, takes or scans an echo image, and uploads it to the system.

[0038] The server accepts uploaded echo images. Since the images may vary in size and format, the server converts them to a standard format, adjusts the resolution, removes noise from the images, and adjusts the contrast to improve visibility.

[0039] Next, based on the preprocessed ultrasound images, the server uses a generative AI model to generate a 3D model of the fetus. The generative AI model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generated 3D model is temporarily stored.

[0040] The user then enters the fetus's genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives the genetic information and converts it into a standard format if necessary.

[0041] The server analyzes the genetic information to extract information about future appearance, such as facial features, hair color, skin color, height, and body type. Based on the analysis results, the server predicts growth patterns and generates a three-dimensional model of the future appearance, utilizing existing growth databases and statistical models.

[0042] Finally, the server renders the generated current and future 3D models to display them visually, and the rendered 3D models are sent to the device in an interactive format (e.g., WebGL), allowing users to view and manipulate these models on their smartphone or computer screen.

[0043] As a concrete example, suppose a user in the 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. At the same time, the server analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see a realistic image of the fetus now, as well as predicted future appearances at age 5 and 10.

[0044] This invention will help expectant mothers and their families understand the specific appearance and future shape of their fetus, allowing them to experience moving and meaningful moments, and will also help reduce anxiety during pregnancy.

[0045] The processing flow will be explained below.

[0046] Step 1:

[0047] A user uses a smartphone or computer to access the system's dedicated application or website.

[0048] Step 2:

[0049] The user takes or scans an echo image and uploads it to the system.

[0050] Step 3:

[0051] The server accepts the uploaded echo images.

[0052] Step 4:

[0053] The server converts the uploaded echo image format to a standard format (e.g., JPEG, PNG), and also performs noise removal, resolution adjustment, and contrast adjustment.

[0054] Step 5:

[0055] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0056] Step 6:

[0057] The server stores the generated 3D model in temporary storage.

[0058] Step 7:

[0059] The user enters the fetus's genetic information into the system, which is typically provided in a data file by a specialized testing laboratory.

[0060] Step 8:

[0061] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0062] Step 9:

[0063] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the genetic information.

[0064] Step 10:

[0065] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0066] Step 11:

[0067] The server renders the generated current and future 3D models and generates the data for visual display. Rendering is done using technologies such as WebGL.

[0068] Step 12:

[0069] The rendered three-dimensional model data is sent to the terminal.

[0070] Step 13:

[0071] The user can view and manipulate a 3D model of the current fetus and a 3D model of its predicted future appearance on a smartphone or computer screen.

[0072] Step 14:

[0073] Users can share the 3D models they create with family and friends.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] Systems that can precisely reconstruct the fetus's appearance from ultrasound images and predict its future appearance based on genetic information require highly accurate image processing and analysis. Previous systems performed the preprocessing of ultrasound images, the generation of 3D models, and the analysis of genetic information separately, making it difficult to provide an integrated system. Furthermore, they lacked an interactive method that allowed users to intuitively manipulate the generated 3D models, leaving a need for an improved user experience.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a three-dimensional model of the fetus from the preprocessed echo image, means for inputting genetic information, means for analyzing the input genetic information and predicting future appearance, means for displaying the generated three-dimensional model and the predicted future appearance, means for rendering the generated three-dimensional model and transmitting it to a terminal in an interactive format, and means for the user to manipulate and confirm the three-dimensional model on the terminal. This integrates highly accurate image processing and analysis, allowing the user to intuitively manipulate and confirm the current and future appearance of the fetus.

[0079] An "echo image" is an image of the inside of a body captured using ultrasound.

[0080] "Preprocessing" refers to the process of converting the input echo image into a format that is easy for the system to analyze, such as by converting the format, removing noise, and adjusting the resolution.

[0081] A "standard format" is a unified data format that improves data compatibility and processing efficiency.

[0082] A "3D model" is a three-dimensional digital representation that can be observed from multiple perspectives.

[0083] A "generative model means" is an algorithm or software that generates new data or information based on input data.

[0084] "Genetic information" is data that records the sequence of DNA or RNA, and is information that determines the characteristics and functions of an individual.

[0085] "Analysis" is the process of examining data using statistical or mathematical methods to extract useful information.

[0086] "Forecasting" is the act of estimating future events or conditions based on current data and trends.

[0087] "Rendering" is the process of displaying three-dimensional models or image data on a flat display.

[0088] An "interactive format" is a two-way display format that changes in real time in response to user operations and inputs.

[0089] A "terminal" is a digital device used by a user to access the system, examples of which include a smartphone or computer.

[0090] "User" refers to the end user who operates the system to input information and check the results.

[0091] This invention is a system that reconstructs a fetus in detail from an ultrasound image and predicts its future appearance by analyzing its genetic information. This system is specifically implemented based on the following procedure and configuration.

[0092] First, the user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system. The echo image can be input using the smartphone's camera or scanner.

[0093] The server receives the uploaded echo images, checks the image format, converts them to a standard format, adjusts the image resolution, removes noise, and adjusts the contrast to improve visibility. This process is often performed using image processing software.

[0094] After preprocessing, the ultrasound images are input to a server running a generative AI model. This model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generative AI model generates a three-dimensional model of the fetus and temporarily stores it.

[0095] The user then enters their genetic information into the system, often in the form of a data file provided by a specialized testing laboratory. The server receives the information, converts it into a standard format if necessary, and analyzes it to extract information about future appearance, such as facial features, hair color, skin tone, and height.

[0096] The server predicts growth patterns based on the genetic analysis, leverages existing growth databases and statistical models, and generates a three-dimensional model of the future, often using growth prediction algorithms.

[0097] Finally, the server renders the generated current and future 3D models. This rendering is done in an interactive format such as WebGL. The rendered 3D models are sent to the device, where users can view and manipulate these models on their smartphone or computer screen. Specifically, users can rotate the 3D models, zoom in and out, and view them from different perspectives.

[0098] As a concrete example, consider a case where a user in the 20th week of pregnancy takes an ultrasound image using their smartphone and uploads it to the system, while also inputting genetic information obtained from a blood test. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It then analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see in detail the realistic appearance of the fetus now, as well as predicted appearances at age 5 and 10.

[0099] By making good use of the features of this invention, expectant mothers and their families will be able to better understand the specific appearance and future shape of their fetus, allowing them to share moving and meaningful moments together, and it is also expected to contribute to reducing anxiety during pregnancy.

[0100] Example prompt sentence:

[0101] "You provide the system with ultrasound images and genetic information to predict what the fetus looks like now and what it will look like in the future."

[0102] "Based on ultrasound images of a fetus at 20 weeks of pregnancy and genetic information, please make a realistic prediction of what the baby will look like at age 5 and 10."

[0103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0104] Step 1:

[0105] A user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads the echo image file to the system, where the echo image as input is sent to the server as output.

[0106] Step 2:

[0107] The server receives the uploaded echo image and first checks the image format. Specifically, the image is converted to a standard format such as JPEG or PNG. Then, resolution adjustments, noise reduction filters, and contrast adjustments are applied to improve visibility. The input echo image is output as a preprocessed image converted to a standard format.

[0108] Step 3:

[0109] The server inputs the preprocessed echo images into a generative AI model. The generative AI model uses a multi-layer neural network to generate a 3D model of the fetus from the echo images. The preprocessed echo images as input are saved as the generated 3D model as output. Specific operations include training and inference of the AI ​​model.

[0110] Step 4:

[0111] The user prepares a genetic information data file provided by a specialized testing institution and proceeds to the genetic information input screen in a dedicated application or website. Next, the user selects the genetic information file and clicks the "Upload" button. The genetic information as input is sent to the server.

[0112] Step 5:

[0113] The server receives the uploaded genetic information, checks the format, converts it to a standard format if necessary, and analyzes the genetic information to extract information about appearance, such as facial features, hair color, skin color, and height. Specifically, an algorithm is used to analyze useful feature information from the genetic data. The input genetic information is output as analyzed feature information.

[0114] Step 6:

[0115] The server predicts growth patterns based on the results of genetic information analysis. It uses existing growth databases and statistical models to simulate fetal growth. Feature information and growth data are input, and predicted growth patterns and a future three-dimensional model are obtained as output. Specific operations include statistical analysis and simulation work.

[0116] Step 7:

[0117] The server renders the generated current and future 3D models. In this process, rendering is performed in an interactive format such as WebGL. The input 3D model is output as a rendered interactive format. Specific operations involve the use of graphics rendering algorithms.

[0118] Step 8:

[0119] The server sends the rendered 3D model to the terminal. The user opens the system's dedicated application or website on the terminal and views and manipulates the 3D model. The input interactive model is output as a manipulable display on the terminal. Specific actions the user can perform include rotating, zooming, and translating the model.

[0120] (Application example 1)

[0121] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] Conventional ultrasound image analysis systems can predict the current and future appearance of a fetus, but they cannot provide users with virtual simulations of its future appearance. This means that users cannot enjoy virtual experiences such as trying on products based on their future appearance. Furthermore, there are limitations to the accuracy of processing and real-time display in the process of analyzing ultrasound images and genetic information.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0124] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a 3D model of a fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for displaying the generated 3D model and the predicted future appearance, and means for inputting and preprocessing a user's facial image and virtually simulating a product based on the user's future appearance, thereby enabling the user to experience a virtual simulation based on the user's future appearance.

[0125] An "echo image" is an image that uses ultrasound to visualize the internal structures of a fetus.

[0126] "Preprocessing" refers to the process of converting the input image into a standard format and performing any necessary corrections or noise removal.

[0127] A "standard format" is a standardized image or data format that can be recognized by systems.

[0128] A "generative model" is an artificial intelligence model that generates three-dimensional models based on echo images and genetic information.

[0129] "Genetic information" means information, including the DNA data of a fetus, that indicates the unique biological characteristics of that individual.

[0130] A "three-dimensional model" is a digital model that realistically reproduces the fetus and its future appearance in three-dimensional space.

[0131] "Display means" refers to a device or software for visually presenting the generated three-dimensional model and prediction results to the user.

[0132] A "face image" is photographic data of a user's face.

[0133] "Virtual simulation" refers to the experience of virtually trying on products, clothing, cosmetics, etc. based on the user's future appearance.

[0134] This invention is a system that generates a three-dimensional model of the fetus's current and future appearance based on ultrasound images and genetic information, and also provides a virtual simulation based on the fetus's future appearance. A specific implementation method of this system is described below.

[0135] Users access the system using a device such as a smartphone, tablet, or computer, and take or scan an echo image using a camera or scanner installed in an autonomous vehicle or logistics center. Once the image is taken or scanned, it can be uploaded to the system via the device.

[0136] The server accepts the uploaded ultrasound images and first performs preprocessing. If the images vary in size or format, they are converted into a standardized format, and the resolution is adjusted and noise is removed. The preprocessed images are then input into a generative AI model, which generates a 3D model of the fetus. This generative AI model uses a multi-layer neural network trained on past ultrasound images and actual post-birth data.

[0137] The user then enters their genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives and analyzes this information, extracting information about the individual's future appearance, including facial features, hair color, skin tone, height, and body type. Using this information, the server predicts growth patterns and generates a three-dimensional model of the individual's future appearance.

[0138] The generated 3D model is then rendered for visual display and sent to the device in an interactive format using technologies such as WebGL, allowing users to view and manipulate the 3D model on their smartphone, tablet, or computer screen.

[0139] Furthermore, the system also has the ability to input and preprocess a user's facial image and virtually simulate products based on their future appearance, allowing users to virtually try on and try on products such as clothing and cosmetics based on their future appearance.

[0140] As a concrete example, consider the case where a user uploads a photo of their face and simulates their hairstyle and makeup from their younger days. In this case, the server uses a generative AI model based on the photo to predict their future appearance and perform a virtual simulation. Examples of prompts include:

[0141] "Upload a photo of yourself from 20 years ago and compare it to how you look now."

[0142] "Enter your DNA information and do a virtual try-on that predicts your future appearance."

[0143] As described above, the present invention enables the generation of three-dimensional models and virtual simulations using echo images and genetic information, providing users with a new experience.

[0144] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0145] Step 1: Upload your ultrasound images

[0146] Users access the system using a smartphone, tablet, or computer to take or scan echo images, and then upload the images to a dedicated application or website. The input is the echo image data, and the output is an image file sent to the server.

[0147] Step 2: Preprocessing of echo images

[0148] The server receives the uploaded echo images and performs preprocessing, specifically converting the image size and format to a standard format, adjusting the resolution, and removing noise. At this stage, the input is echo image data, and the output is preprocessed echo images in a unified format.

[0149] Step 3: Generate a 3D model

[0150] The server inputs the preprocessed ultrasound images into a generative AI model to generate a 3D model of the fetus. This generative AI model is a multi-layer neural network trained on past ultrasound images and actual post-birth data. The input is the preprocessed ultrasound images, and the output is a 3D digital model of the fetus.

[0151] Step 4: Enter genetic information

[0152] Users input genetic information provided by specialized testing laboratories into the system. The input is a data file containing DNA information, and the output is genetic data sent to the server.

[0153] Step 5: Analyzing genetic information

[0154] The server analyzes the input genetic information and extracts information about future appearance, such as facial features, hair color, skin color, height, and body type. This analysis uses a database containing features correlated with various genetic data. The input is genetic data, and the output is appearance feature data derived from the genetic information.

[0155] Step 6: Generate the future 3D model

[0156] The server predicts growth patterns based on the analysis results and generates a 3D model of the future appearance, utilizing existing growth databases and statistical models. The input is appearance feature data and growth prediction data, and the output is a 3D digital model of the future appearance.

[0157] Step 7: Rendering the 3D model

[0158] The server performs rendering to visually display the generated current and future 3D models. Using technologies such as WebGL, it transmits them to the device in an interactive format. The input is the 3D digital model, and the output is the rendered interactive 3D display data.

[0159] Step 8: Conduct a virtual simulation

[0160] The user inputs an image of their face into the system and performs preprocessing. Based on this image, the server uses a generative AI model to predict future appearance and runs a virtual simulation. The input is facial image data and a prediction model, and the output is a virtually tried-on product image. Examples of prompts include "Upload a photo of your face from 20 years ago and compare it with your current self," and "Enter your DNA information to virtually try on products with predicted future appearances."

[0161] Through these steps, the system can provide users with a new experience.

[0162] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0163] This invention combines a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information with an emotion engine that recognizes the user's emotions. Specifically, it performs the following processes:

[0164] First, a user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system.

[0165] The server receives the uploaded ultrasound images and performs preprocessing such as converting the images to a standard format and removing noise, adjusting resolution, and enhancing contrast. Once preprocessing is complete, the server uses a generative AI model to generate a 3D model of the fetus from the ultrasound images. The generated 3D model is temporarily stored in storage.

[0166] Next, the user enters the fetus's genetic information into the system. This information is typically provided in the form of a data file from a specialized testing laboratory. The server receives the genetic information, converts it into a standard format if necessary, and then analyzes it. The analysis results include future appearance, such as facial features, hair color, skin tone, height, and body type. Based on this analysis, the server predicts growth patterns and generates a three-dimensional model of the fetus's future appearance.

[0167] One of the features of the present invention is the incorporation of an emotion engine. While the user is using the system, the terminal is equipped with a camera that detects the user's facial expressions and tone of voice. The server uses the emotion engine to analyze this data and recognize the user's emotions. Based on the recognized emotion information, the server can adjust the displayed 3D model, system interface, and output content. For example, if the user is feeling surprised or moved, it can display more detailed and emotional visuals.

[0168] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It also analyzes the genetic information and predicts the future appearance. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotions, to the device, where the user can view and operate it on their smartphone or computer screen.

[0169] This invention not only helps users understand the specific appearance and future shape of the fetus, but also allows the system to adjust the visuals and interface according to the user's emotions, providing a deeper sense of emotion and security, and contributing to reducing anxiety during pregnancy.

[0170] The processing flow will be explained below.

[0171] Step 1:

[0172] A user uses a smartphone or computer to access the system's dedicated application or website.

[0173] Step 2:

[0174] The user takes or scans an echo image and uploads it to the system.

[0175] Step 3:

[0176] The server accepts the uploaded echo images.

[0177] Step 4:

[0178] The server converts the uploaded echo images into standard formats (e.g., JPEG, PNG), and also performs noise reduction, resolution adjustment, and contrast enhancement.

[0179] Step 5:

[0180] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0181] Step 6:

[0182] The server stores the generated 3D model in temporary storage.

[0183] Step 7:

[0184] The user inputs the fetus's genetic information into the system, which is usually a data file provided by a specialized testing institution.

[0185] Step 8:

[0186] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0187] Step 9:

[0188] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the analyzed genetic information.

[0189] Step 10:

[0190] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0191] Step 11:

[0192] The camera installed on the device detects the user's facial expressions and tone of voice in real time.

[0193] Step 12:

[0194] The server uses an emotion engine to analyze the user's facial expression and voice data sent from the terminal and recognize the user's emotional state.

[0195] Step 13:

[0196] The server adjusts the content of the displayed 3D model and the system interface based on the recognized emotional state. For example, if the user expresses surprise, it displays more detailed and moving visuals.

[0197] Step 14:

[0198] The server renders the generated current and future 3D models and generates the data for visual display, using technologies such as WebGL.

[0199] Step 15:

[0200] The rendered three-dimensional model data is sent to the terminal.

[0201] Step 16:

[0202] The user can view and manipulate a three-dimensional model of the current fetus and a three-dimensional model of its predicted future appearance on a smartphone or computer screen.

[0203] Step 17:

[0204] Users can share the 3D models they create with family and friends.

[0205] In this way, the system generates a three-dimensional model of the fetus based on ultrasound images and genetic information, and then uses an emotion engine to adjust the interface based on the user's emotions, creating a more moving and reassuring experience.

[0206] Example 2

[0207] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0208] In conventional echo image analysis and genetic information analysis systems, preprocessing and noise removal of echo images, and conversion and analysis of genetic information are performed separately, making comprehensive use difficult. Furthermore, the generated 3D models and predicted future appearances are fixed, and dynamic adjustments based on the user's emotions are not possible. As a result, user satisfaction is low and the utility of the system is limited. The purpose of this invention is to solve these problems.

[0209] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0210] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for removing noise from the preprocessed echo image, means for generating a 3D model of the fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for recognizing a user's emotion, and means for adjusting and displaying the generated 3D model and the predicted future appearance based on the recognized emotion information, thereby enabling the generation and display of a dynamic and detailed 3D model in accordance with the user's emotion.

[0211] An "echo image" is an image of internal structures, particularly the fetus, taken using ultrasound technology.

[0212] "Preprocessing" refers to processing of the input echo image, such as conversion to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0213] A "standard format" is an image file format that is specified so that the system can process it appropriately.

[0214] "Noise reduction" is a process that improves image quality by removing unwanted noise and distortion from an image.

[0215] "Generative model means" includes methods and algorithms for generating a three-dimensional model of the fetus from pre-processed echo images.

[0216] "Genetic information" is data describing the genetic characteristics of an individual and is provided by specialized institutions.

[0217] "Genetic information analysis" is a process for predicting future appearance and growth patterns based on input genetic information.

[0218] "Predicting future appearance" means predicting future appearance, including facial features, hair color, skin color, height, and body type, based on the results of analyzing genetic information.

[0219] "User emotion recognition" refers to sensing the user's facial expressions and tone of voice using the device's built-in camera and microphone, and analyzing them to understand the user's emotional state.

[0220] "Adjusted display based on emotion information" refers to dynamically changing the display content of the generated three-dimensional model or interface based on the user's recognized emotion information.

[0221] The present invention combines a system that generates a 3D model of a fetus from ultrasound images, analyzes genetic information to predict its future appearance, and recognizes the user's emotions and adjusts the display content accordingly. To implement this system, the following specific hardware and software are used:

[0222] The user accesses a dedicated application or website using a smartphone or computer. It is recommended that the hardware used has a camera and microphone. The following describes the detailed process at each step and the software used.

[0223] The server receives the echo images uploaded by the user. These images are pre-processed using an image processing library (e.g., Python's Pillow library). Pre-processing includes converting to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0224] After preprocessing, the ultrasound images are used to generate a 3D model of the fetus using a generative AI model (e.g., GANs using TensorFlow), which is then temporarily stored in a database.

[0225] The user then inputs their genetic information. This information is typically provided by a specialized testing institution in the form of a data file in text format. The server receives this genetic information and analyzes it. This analysis involves converting the genetic information into a standard format and extracting specified characteristics (facial features, hair color, skin color, height, body type, etc.). This analysis is performed using the Python Pandas library.

[0226] Based on the analysis results, growth patterns are predicted and a three-dimensional model of the future shape is generated, which is performed using machine learning algorithms (e.g., random forests).

[0227] While the user is using the system, the device's camera and microphone detect the user's facial expressions and tone of voice. Emotion analysis is performed using computer vision libraries (e.g., OpenCV) and facial expression recognition libraries (e.g., DeepFace).

[0228] The server analyzes this data using an emotion engine to recognize the user's emotions. Based on the recognized emotion information, the generated 3D model and system interface are adjusted. Specifically, if the user is feeling surprised or moved, the system increases the level of visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0229] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image to the system and inputs genetic information obtained from a specialized testing institution. The ultrasound image is preprocessed, and a realistic 3D model is generated using a generative AI model. At the same time, the genetic information is analyzed and the future appearance is predicted. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotion, to the device, where the user can view and manipulate it on their smartphone or computer screen.

[0230] Example prompt sentence:

[0231] "I have uploaded an ultrasound image from the 20th week of pregnancy and entered genetic information. Please generate a growth prediction model from this. I would also like you to collect facial expression data from the user in real time and output the data according to their emotions."

[0232] This makes it easier for users to understand the specific appearance and future shape of the fetus, and the system adjusts the visuals and interface according to the user's emotions, allowing them to feel a deeper sense of emotion and security.

[0233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0234] System program processing flow

[0235] Step 1: User uploads an echo image

[0236] Users use a smartphone or computer to access a dedicated application or website to take or scan and upload echo images, and then use the camera or file selection function of the specific device to select and send the image files to the system.

[0237] Input: Echo image file

[0238] Output: Echo image file sent to the server

[0239] Step 2: The server preprocesses the images

[0240] The server preprocesses the received echo images in the following steps:

[0241] Conversion to standard formats: Converting different image formats (e.g. JPEG to PNG) into a format that the system can properly handle.

[0242] Noise reduction: Filters out and removes unwanted noise from the image.

[0243] Resolution Adjustment: Adjust the image to the optimal resolution.

[0244] Contrast Enhancement: Increases the contrast of the image to make it easier to see.

[0245] Input: Echo image file sent by the user

[0246] Output: Preprocessed image files in standard format

[0247] Step 3: The server generates the 3D model

[0248] Based on the preprocessed ultrasound images, the server uses a generative AI model (e.g., GANs using TensorFlow) to generate a 3D model of the fetus. This generated model is temporarily stored in a database. Specifically, features are extracted from the image using a convolutional neural network (CNN), and a 3D model is generated using GANs based on those features.

[0249] Input: Preprocessed image files in standard format

[0250] Output: Generated 3D model of the fetus

[0251] Step 4: User enters genetic information

[0252] Users input genetic information data files provided by professional testing institutions into the system, and use the application's "genetic information upload" function to select and send the data files to the system.

[0253] Input: Gene information file

[0254] Output: Gene information file sent to the server

[0255] Step 5: The server analyzes the genetic information and predicts the future appearance

[0256] The server converts the received genetic information into a standard format and analyzes the information. The analysis extracts facial features, hair color, skin color, height, body type, etc. from the input genetic information, and predicts future appearance based on growth patterns. A machine learning algorithm (e.g., random forest) is used for the prediction.

[0257] Input: Gene information file

[0258] Output: Analyzed genetic information and a predicted 3D model of your future appearance

[0259] Step 6: The device recognizes the user's emotions

[0260] While the user is using the system, the device's camera captures the user's facial expressions and tone of voice and transmits them to the server in real time. The server then uses an emotion engine (e.g., OpenCV and DeepFace) to analyze this data and recognize the user's emotions. For example, the camera captures the user's smiling or surprised expressions and uses them to infer the user's emotions.

[0261] Input: User's facial and voice data

[0262] Output: Parsed emotion information

[0263] Step 7: The server adjusts and displays the 3D model and interface.

[0264] Based on the recognized emotion information, the server adjusts the displayed 3D model and system interface. For example, if the user is surprised or impressed, the server increases the visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0265] Input: Analyzed emotion information, generated 3D model, and predicted future appearance

[0266] Output: emotion-adjusted 3D model and system interface

[0267] This allows users to view and manipulate dynamically adjusted 3D models and interfaces in real time.

[0268] (Application example 2)

[0269] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] Conventional systems for generating 3D fetal models based on ultrasound images tend to display static images without considering the user's emotions. This makes it difficult for users to feel interactivity with the system, preventing the quality of the experience from being maximized. Furthermore, predictions of the fetus's future appearance can sometimes lack persuasiveness. The objective of this invention is not only to provide a 3D fetus model and a prediction of its future appearance from ultrasound images, but also to dynamically adjust the display content using the user's emotional information, providing an interactive and moving experience.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0272] In this invention, the server includes means for preprocessing the ultrasound images and converting them into a standard format, means for generating a 3D model of the fetus from the preprocessed ultrasound images, means for analyzing input genetic information to predict future appearance, means for inputting emotional data, means for analyzing the input emotional data to recognize the user's emotion, and means for dynamically adjusting the display content in accordance with the user's emotion, thereby enabling the provision of an interactive and moving 3D model display of the fetus and a predicted future appearance that are tailored to the user's emotional state.

[0273] An "echo image" is a medical image taken using ultrasound to capture the inside of the body, and is primarily used to check the condition of a fetus.

[0274] "Preprocessing" is the process of converting input image data into a standard format and performing processing such as noise removal, resolution adjustment, and contrast enhancement.

[0275] A "generative model" is an algorithm that uses artificial intelligence or machine learning techniques to generate a three-dimensional model based on specific input data.

[0276] "Genetic information" refers to the data contained in the genetic material of an organism, and is the information recorded in DNA or RNA.

[0277] "Analysis" is the process of examining and breaking down input data in detail to extract meaning and patterns.

[0278] "Future appearance" is the subject's future appearance or appearance predicted based on current data.

[0279] "Emotion data" is data that indicates the user's emotional state, including facial expressions and vocal tones.

[0280] An "emotion engine" is an algorithm or software that recognizes a user's emotional state, primarily by analyzing facial expressions and tone of voice.

[0281] "Dynamic adjustment" is the process of changing the display content and interface in real time based on the situation and data.

[0282] In the system realizing this invention, the server preprocesses the echo image, generates a 3D model of the fetus, analyzes genetic information, and further recognizes the user's emotions to dynamically adjust the display content. This system has the following configuration and processing means.

[0283] The server first receives the echo images uploaded by the user using a tablet or computer, and then preprocesses them using image processing libraries such as OpenCV and scikit-image, including converting them to a standard format, removing noise, adjusting the resolution, and enhancing the contrast.

[0284] Next, the preprocessed echo images are input into a generative AI model using TensorFlow and PyTorch to generate a 3D model of the fetus, which is then temporarily stored in storage.

[0285] The user then inputs genetic information provided by a specialized testing institution into the system. This genetic information is analyzed using libraries such as BioPython and Pandas to predict future appearance and growth patterns. The genetic analysis results include facial features, hair color, skin color, height, and body type, and a three-dimensional model of the future appearance is generated based on these results.

[0286] The system also uses an emotion engine to analyze the user's emotional data. The device's camera detects the user's facial expressions and tone of voice, and the data is analyzed using Google Cloud Vision API and DeepFace. Based on the analyzed emotional data, the server dynamically adjusts the display content. For example, if the user is emotional, it can display more detailed and emotional visuals.

[0287] The three-dimensional model and predicted future appearance generated in this way are displayed to the user via a tablet, computer display, VR headset, etc. The user can manipulate it to experience the growth and future appearance of the fetus in a realistic way.

[0288] As a concrete example, consider a case where a user in the 24th week of pregnancy uploads an ultrasound image and inputs genetic information into the system. In this case, the server preprocesses the ultrasound image and uses a generative AI model to generate a realistic 3D model of the fetus. Meanwhile, the device's camera detects the user's facial expression, and the emotion engine analyzes the user's emotions. Based on the results, the display content is dynamically adjusted. This allows the user to feel more deeply moved and at ease.

[0289] An example prompt is, "Upload an ultrasound image to generate a realistic 3D model. We'll also predict your future appearance based on your genetic information and use emotion recognition to tailor the interface."

[0290] In this way, the present invention utilizes advanced technology to provide the user with an interactive and emotional experience, maximizing comfort and enjoyment during pregnancy.

[0291] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0292] Step 1:

[0293] A user uploads an echo image using a terminal. The echo image file is sent as input to the server, which receives the echo image and converts it into a standard format.

[0294] Step 2:

[0295] The server preprocesses the received echo images using OpenCV and scikit-image libraries to remove noise, adjust resolution, and enhance contrast. The output of the preprocessing is a high-quality image converted to a standard format.

[0296] Step 3:

[0297] The server uses the preprocessed ultrasound images to input them into a generative AI model (TensorFlow or PyTorch) to generate a 3D model of the fetus. This generative AI model is a pre-trained neural network that generates a highly accurate 3D model from the input images. The output is 3D model data of the fetus.

[0298] Step 4:

[0299] The user uploads a genetic information file using a terminal. The genetic information file contains information about future appearance, such as facial features, hair color, skin color, height, and body type. The server receives the genetic information.

[0300] Step 5:

[0301] The server uses BioPython and Pandas to analyze the input genetic information. The analysis results in future appearance prediction data, which includes details about the fetus's growth patterns and predicted appearance. The output is the future appearance prediction data.

[0302] Step 6:

[0303] The server generates a 3D model of the future appearance based on the analysis results. The generated 3D model reflects the fetus's growth pattern, allowing the user to visually confirm the future appearance. The output is the 3D model data of the future appearance.

[0304] Step 7:

[0305] The device's camera detects the user's facial expressions and tone of voice to obtain emotional data, which includes the user's current emotional state. The server receives this emotional data.

[0306] Step 8:

[0307] The server uses an emotion engine (such as Google Cloud Vision API or DeepFace) to analyze the acquired emotion data. The analysis recognizes the user's emotional state. The output is the user's emotional state data.

[0308] Step 9:

[0309] The server dynamically adjusts the display content based on the recognized emotional state data. For example, if the user is emotional, the server changes the display content to more detailed and emotional visuals. The adjusted display content is sent to the user's device and displayed.

[0310] Step 10:

[0311] Based on the adjusted display, users can view a 3D model of the fetus and its future appearance in real time on a tablet, computer screen, or VR headset, with the user's experience optimized according to their emotions.

[0312] Through the above steps, the present invention utilizes echo images and genetic information to provide users with an interactive and moving experience.

[0313] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0314] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0315] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0316] [Second embodiment]

[0317] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0318] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0319] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0320] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0321] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0322] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0323] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0324] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0325] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0326] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0327] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0328] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0329] This invention is a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information. Specifically, the following process is performed.

[0330] First, a user accesses the system's dedicated application or website using a smartphone or computer, takes or scans an echo image, and uploads it to the system.

[0331] The server accepts uploaded echo images. Since the images may vary in size and format, the server converts them to a standard format, adjusts the resolution, removes noise from the images, and adjusts the contrast to improve visibility.

[0332] Next, based on the preprocessed ultrasound images, the server uses a generative AI model to generate a 3D model of the fetus. The generative AI model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generated 3D model is temporarily stored.

[0333] The user then enters the fetus's genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives the genetic information and converts it into a standard format if necessary.

[0334] The server analyzes the genetic information to extract information about future appearance, such as facial features, hair color, skin color, height, and body type. Based on the analysis results, the server predicts growth patterns and generates a three-dimensional model of the future appearance, utilizing existing growth databases and statistical models.

[0335] Finally, the server renders the generated current and future 3D models to display them visually, and the rendered 3D models are sent to the device in an interactive format (e.g., WebGL), allowing users to view and manipulate these models on their smartphone or computer screen.

[0336] As a concrete example, suppose a user in the 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. At the same time, the server analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see a realistic image of the fetus now, as well as predicted future appearances at age 5 and 10.

[0337] This invention will help expectant mothers and their families understand the specific appearance and future shape of their fetus, allowing them to experience moving and meaningful moments, and will also help reduce anxiety during pregnancy.

[0338] The processing flow will be explained below.

[0339] Step 1:

[0340] A user uses a smartphone or computer to access the system's dedicated application or website.

[0341] Step 2:

[0342] The user takes or scans an echo image and uploads it to the system.

[0343] Step 3:

[0344] The server accepts the uploaded echo images.

[0345] Step 4:

[0346] The server converts the uploaded echo image format to a standard format (e.g., JPEG, PNG), and also performs noise removal, resolution adjustment, and contrast adjustment.

[0347] Step 5:

[0348] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0349] Step 6:

[0350] The server stores the generated 3D model in temporary storage.

[0351] Step 7:

[0352] The user enters the fetus's genetic information into the system, which is typically provided in a data file by a specialized testing laboratory.

[0353] Step 8:

[0354] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0355] Step 9:

[0356] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the genetic information.

[0357] Step 10:

[0358] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0359] Step 11:

[0360] The server renders the generated current and future 3D models and generates the data for visual display. Rendering is done using technologies such as WebGL.

[0361] Step 12:

[0362] The rendered three-dimensional model data is sent to the terminal.

[0363] Step 13:

[0364] The user can view and manipulate a 3D model of the current fetus and a 3D model of its predicted future appearance on a smartphone or computer screen.

[0365] Step 14:

[0366] Users can share the 3D models they create with family and friends.

[0367] Example 1

[0368] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0369] Systems that can precisely reconstruct the fetus's appearance from ultrasound images and predict its future appearance based on genetic information require highly accurate image processing and analysis. Previous systems performed the preprocessing of ultrasound images, the generation of 3D models, and the analysis of genetic information separately, making it difficult to provide an integrated system. Furthermore, they lacked an interactive method that allowed users to intuitively manipulate the generated 3D models, leaving a need for an improved user experience.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0371] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a three-dimensional model of the fetus from the preprocessed echo image, means for inputting genetic information, means for analyzing the input genetic information and predicting future appearance, means for displaying the generated three-dimensional model and the predicted future appearance, means for rendering the generated three-dimensional model and transmitting it to a terminal in an interactive format, and means for the user to manipulate and confirm the three-dimensional model on the terminal. This integrates highly accurate image processing and analysis, allowing the user to intuitively manipulate and confirm the current and future appearance of the fetus.

[0372] An "echo image" is an image of the inside of a body captured using ultrasound.

[0373] "Preprocessing" refers to the process of converting the input echo image into a format that is easy for the system to analyze, such as by converting the format, removing noise, and adjusting the resolution.

[0374] A "standard format" is a unified data format that improves data compatibility and processing efficiency.

[0375] A "3D model" is a three-dimensional digital representation that can be observed from multiple perspectives.

[0376] A "generative model means" is an algorithm or software that generates new data or information based on input data.

[0377] "Genetic information" is data that records the sequence of DNA or RNA, and is information that determines the characteristics and functions of an individual.

[0378] "Analysis" is the process of examining data using statistical or mathematical methods to extract useful information.

[0379] "Forecasting" is the act of estimating future events or conditions based on current data and trends.

[0380] "Rendering" is the process of displaying three-dimensional models or image data on a flat display.

[0381] An "interactive format" is a two-way display format that changes in real time in response to user operations and inputs.

[0382] A "terminal" is a digital device used by a user to access the system, examples of which include a smartphone or computer.

[0383] "User" refers to the end user who operates the system to input information and check the results.

[0384] This invention is a system that reconstructs a fetus in detail from an ultrasound image and predicts its future appearance by analyzing its genetic information. This system is specifically implemented based on the following procedure and configuration.

[0385] First, the user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system. The echo image can be input using the smartphone's camera or scanner.

[0386] The server receives the uploaded echo images, checks the image format, converts them to a standard format, adjusts the image resolution, removes noise, and adjusts the contrast to improve visibility. This process is often performed using image processing software.

[0387] After preprocessing, the ultrasound images are input to a server running a generative AI model. This model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generative AI model generates a three-dimensional model of the fetus and temporarily stores it.

[0388] The user then enters their genetic information into the system, often in the form of a data file provided by a specialized testing laboratory. The server receives the information, converts it into a standard format if necessary, and analyzes it to extract information about future appearance, such as facial features, hair color, skin tone, and height.

[0389] The server predicts growth patterns based on the genetic analysis, leverages existing growth databases and statistical models, and generates a three-dimensional model of the future, often using growth prediction algorithms.

[0390] Finally, the server renders the generated current and future 3D models. This rendering is done in an interactive format such as WebGL. The rendered 3D models are sent to the device, where users can view and manipulate these models on their smartphone or computer screen. Specifically, users can rotate the 3D models, zoom in and out, and view them from different perspectives.

[0391] As a concrete example, consider a case where a user in the 20th week of pregnancy takes an ultrasound image using their smartphone and uploads it to the system, while also inputting genetic information obtained from a blood test. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It then analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see in detail the realistic appearance of the fetus now, as well as predicted appearances at age 5 and 10.

[0392] By making good use of the features of this invention, expectant mothers and their families will be able to better understand the specific appearance and future shape of their fetus, allowing them to share moving and meaningful moments together, and it is also expected to contribute to reducing anxiety during pregnancy.

[0393] Example prompt sentence:

[0394] "You provide the system with ultrasound images and genetic information to predict what the fetus looks like now and what it will look like in the future."

[0395] "Based on ultrasound images of a fetus at 20 weeks of pregnancy and genetic information, please make a realistic prediction of what the baby will look like at age 5 and 10."

[0396] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0397] Step 1:

[0398] A user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads the echo image file to the system, where the echo image as input is sent to the server as output.

[0399] Step 2:

[0400] The server receives the uploaded echo image and first checks the image format. Specifically, the image is converted to a standard format such as JPEG or PNG. Then, resolution adjustments, noise reduction filters, and contrast adjustments are applied to improve visibility. The input echo image is output as a preprocessed image converted to a standard format.

[0401] Step 3:

[0402] The server inputs the preprocessed echo images into a generative AI model. The generative AI model uses a multi-layer neural network to generate a 3D model of the fetus from the echo images. The preprocessed echo images as input are saved as the generated 3D model as output. Specific operations include training and inference of the AI ​​model.

[0403] Step 4:

[0404] The user prepares a genetic information data file provided by a specialized testing institution and proceeds to the genetic information input screen in a dedicated application or website. Next, the user selects the genetic information file and clicks the "Upload" button. The genetic information as input is sent to the server.

[0405] Step 5:

[0406] The server receives the uploaded genetic information, checks the format, converts it to a standard format if necessary, and analyzes the genetic information to extract information about appearance, such as facial features, hair color, skin color, and height. Specifically, an algorithm is used to analyze useful feature information from the genetic data. The input genetic information is output as analyzed feature information.

[0407] Step 6:

[0408] The server predicts growth patterns based on the results of genetic information analysis. It uses existing growth databases and statistical models to simulate fetal growth. Feature information and growth data are input, and predicted growth patterns and a future three-dimensional model are obtained as output. Specific operations include statistical analysis and simulation work.

[0409] Step 7:

[0410] The server renders the generated current and future 3D models. In this process, rendering is performed in an interactive format such as WebGL. The input 3D model is output as a rendered interactive format. Specific operations involve the use of graphics rendering algorithms.

[0411] Step 8:

[0412] The server sends the rendered 3D model to the terminal. The user opens the system's dedicated application or website on the terminal and views and manipulates the 3D model. The input interactive model is output as a manipulable display on the terminal. Specific actions the user can perform include rotating, zooming, and translating the model.

[0413] (Application example 1)

[0414] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0415] Conventional ultrasound image analysis systems can predict the current and future appearance of a fetus, but they cannot provide users with virtual simulations of its future appearance. This means that users cannot enjoy virtual experiences such as trying on products based on their future appearance. Furthermore, there are limitations to the accuracy of processing and real-time display in the process of analyzing ultrasound images and genetic information.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0417] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a 3D model of a fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for displaying the generated 3D model and the predicted future appearance, and means for inputting and preprocessing a user's facial image and virtually simulating a product based on the user's future appearance, thereby enabling the user to experience a virtual simulation based on the user's future appearance.

[0418] An "echo image" is an image that uses ultrasound to visualize the internal structures of a fetus.

[0419] "Preprocessing" refers to the process of converting the input image into a standard format and performing any necessary corrections or noise removal.

[0420] A "standard format" is a standardized image or data format that can be recognized by systems.

[0421] A "generative model" is an artificial intelligence model that generates three-dimensional models based on echo images and genetic information.

[0422] "Genetic information" means information, including the DNA data of a fetus, that indicates the unique biological characteristics of that individual.

[0423] A "three-dimensional model" is a digital model that realistically reproduces the fetus and its future appearance in three-dimensional space.

[0424] "Display means" refers to a device or software for visually presenting the generated three-dimensional model and prediction results to the user.

[0425] A "face image" is photographic data of a user's face.

[0426] "Virtual simulation" refers to the experience of virtually trying on products, clothing, cosmetics, etc. based on the user's future appearance.

[0427] This invention is a system that generates a three-dimensional model of the fetus's current and future appearance based on ultrasound images and genetic information, and also provides a virtual simulation based on the fetus's future appearance. A specific implementation method of this system is described below.

[0428] Users access the system using a device such as a smartphone, tablet, or computer, and take or scan an echo image using a camera or scanner installed in an autonomous vehicle or logistics center. Once the image is taken or scanned, it can be uploaded to the system via the device.

[0429] The server accepts the uploaded ultrasound images and first performs preprocessing. If the images vary in size or format, they are converted into a standardized format, and the resolution is adjusted and noise is removed. The preprocessed images are then input into a generative AI model, which generates a 3D model of the fetus. This generative AI model uses a multi-layer neural network trained on past ultrasound images and actual post-birth data.

[0430] The user then enters their genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives and analyzes this information, extracting information about the individual's future appearance, including facial features, hair color, skin tone, height, and body type. Using this information, the server predicts growth patterns and generates a three-dimensional model of the individual's future appearance.

[0431] The generated 3D model is then rendered for visual display and sent to the device in an interactive format using technologies such as WebGL, allowing users to view and manipulate the 3D model on their smartphone, tablet, or computer screen.

[0432] Furthermore, the system also has the ability to input and preprocess a user's facial image and virtually simulate products based on their future appearance, allowing users to virtually try on and try on products such as clothing and cosmetics based on their future appearance.

[0433] As a concrete example, consider the case where a user uploads a photo of their face and simulates their hairstyle and makeup from their younger days. In this case, the server uses a generative AI model based on the photo to predict their future appearance and perform a virtual simulation. Examples of prompts include:

[0434] "Upload a photo of yourself from 20 years ago and compare it to how you look now."

[0435] "Enter your DNA information and do a virtual try-on that predicts your future appearance."

[0436] As described above, the present invention enables the generation of three-dimensional models and virtual simulations using echo images and genetic information, providing users with a new experience.

[0437] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0438] Step 1: Upload your ultrasound images

[0439] Users access the system using a smartphone, tablet, or computer to take or scan echo images, and then upload the images to a dedicated application or website. The input is the echo image data, and the output is an image file sent to the server.

[0440] Step 2: Preprocessing of echo images

[0441] The server receives the uploaded echo images and performs preprocessing, specifically converting the image size and format to a standard format, adjusting the resolution, and removing noise. At this stage, the input is echo image data, and the output is preprocessed echo images in a unified format.

[0442] Step 3: Generate a 3D model

[0443] The server inputs the preprocessed ultrasound images into a generative AI model to generate a 3D model of the fetus. This generative AI model is a multi-layer neural network trained on past ultrasound images and actual post-birth data. The input is the preprocessed ultrasound images, and the output is a 3D digital model of the fetus.

[0444] Step 4: Enter genetic information

[0445] Users input genetic information provided by specialized testing laboratories into the system. The input is a data file containing DNA information, and the output is genetic data sent to the server.

[0446] Step 5: Analyzing genetic information

[0447] The server analyzes the input genetic information and extracts information about future appearance, such as facial features, hair color, skin color, height, and body type. This analysis uses a database containing features correlated with various genetic data. The input is genetic data, and the output is appearance feature data derived from the genetic information.

[0448] Step 6: Generate the future 3D model

[0449] The server predicts growth patterns based on the analysis results and generates a 3D model of the future appearance, utilizing existing growth databases and statistical models. The input is appearance feature data and growth prediction data, and the output is a 3D digital model of the future appearance.

[0450] Step 7: Rendering the 3D model

[0451] The server performs rendering to visually display the generated current and future 3D models. Using technologies such as WebGL, it transmits them to the device in an interactive format. The input is the 3D digital model, and the output is the rendered interactive 3D display data.

[0452] Step 8: Conduct a virtual simulation

[0453] The user inputs an image of their face into the system and performs preprocessing. Based on this image, the server uses a generative AI model to predict future appearance and runs a virtual simulation. The input is facial image data and a prediction model, and the output is a virtually tried-on product image. Examples of prompts include "Upload a photo of your face from 20 years ago and compare it with your current self," and "Enter your DNA information to virtually try on products with predicted future appearances."

[0454] Through these steps, the system can provide users with a new experience.

[0455] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0456] This invention combines a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information with an emotion engine that recognizes the user's emotions. Specifically, it performs the following processes:

[0457] First, a user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system.

[0458] The server receives the uploaded ultrasound images and performs preprocessing such as converting the images to a standard format and removing noise, adjusting resolution, and enhancing contrast. Once preprocessing is complete, the server uses a generative AI model to generate a 3D model of the fetus from the ultrasound images. The generated 3D model is temporarily stored in storage.

[0459] Next, the user enters the fetus's genetic information into the system. This information is typically provided in the form of a data file from a specialized testing laboratory. The server receives the genetic information, converts it into a standard format if necessary, and then analyzes it. The analysis results include future appearance, such as facial features, hair color, skin tone, height, and body type. Based on this analysis, the server predicts growth patterns and generates a three-dimensional model of the fetus's future appearance.

[0460] One of the features of the present invention is the incorporation of an emotion engine. While the user is using the system, the terminal is equipped with a camera that detects the user's facial expressions and tone of voice. The server uses the emotion engine to analyze this data and recognize the user's emotions. Based on the recognized emotion information, the server can adjust the displayed 3D model, system interface, and output content. For example, if the user is feeling surprised or moved, it can display more detailed and emotional visuals.

[0461] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It also analyzes the genetic information and predicts the future appearance. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotions, to the device, where the user can view and operate it on their smartphone or computer screen.

[0462] This invention not only helps users understand the specific appearance and future shape of the fetus, but also allows the system to adjust the visuals and interface according to the user's emotions, providing a deeper sense of emotion and security, and contributing to reducing anxiety during pregnancy.

[0463] The processing flow will be explained below.

[0464] Step 1:

[0465] A user uses a smartphone or computer to access the system's dedicated application or website.

[0466] Step 2:

[0467] The user takes or scans an echo image and uploads it to the system.

[0468] Step 3:

[0469] The server accepts the uploaded echo images.

[0470] Step 4:

[0471] The server converts the uploaded echo images into standard formats (e.g., JPEG, PNG), and also performs noise reduction, resolution adjustment, and contrast enhancement.

[0472] Step 5:

[0473] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0474] Step 6:

[0475] The server stores the generated 3D model in temporary storage.

[0476] Step 7:

[0477] The user inputs the fetus's genetic information into the system, which is usually a data file provided by a specialized testing institution.

[0478] Step 8:

[0479] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0480] Step 9:

[0481] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the analyzed genetic information.

[0482] Step 10:

[0483] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0484] Step 11:

[0485] The camera installed on the device detects the user's facial expressions and tone of voice in real time.

[0486] Step 12:

[0487] The server uses an emotion engine to analyze the user's facial expression and voice data sent from the terminal and recognize the user's emotional state.

[0488] Step 13:

[0489] The server adjusts the content of the displayed 3D model and the system interface based on the recognized emotional state. For example, if the user expresses surprise, it displays more detailed and moving visuals.

[0490] Step 14:

[0491] The server renders the generated current and future 3D models and generates the data for visual display, using technologies such as WebGL.

[0492] Step 15:

[0493] The rendered three-dimensional model data is sent to the terminal.

[0494] Step 16:

[0495] The user can view and manipulate a three-dimensional model of the current fetus and a three-dimensional model of its predicted future appearance on a smartphone or computer screen.

[0496] Step 17:

[0497] Users can share the 3D models they create with family and friends.

[0498] In this way, the system generates a three-dimensional model of the fetus based on ultrasound images and genetic information, and then uses an emotion engine to adjust the interface based on the user's emotions, creating a more moving and reassuring experience.

[0499] Example 2

[0500] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0501] In conventional echo image analysis and genetic information analysis systems, preprocessing and noise removal of echo images, and conversion and analysis of genetic information are performed separately, making comprehensive use difficult. Furthermore, the generated 3D models and predicted future appearances are fixed, and dynamic adjustments based on the user's emotions are not possible. As a result, user satisfaction is low and the utility of the system is limited. The purpose of this invention is to solve these problems.

[0502] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0503] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for removing noise from the preprocessed echo image, means for generating a 3D model of the fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for recognizing a user's emotion, and means for adjusting and displaying the generated 3D model and the predicted future appearance based on the recognized emotion information, thereby enabling the generation and display of a dynamic and detailed 3D model in accordance with the user's emotion.

[0504] An "echo image" is an image of internal structures, particularly the fetus, taken using ultrasound technology.

[0505] "Preprocessing" refers to processing of the input echo image, such as conversion to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0506] A "standard format" is an image file format that is specified so that the system can process it appropriately.

[0507] "Noise reduction" is a process that improves image quality by removing unwanted noise and distortion from an image.

[0508] "Generative model means" includes methods and algorithms for generating a three-dimensional model of the fetus from pre-processed echo images.

[0509] "Genetic information" is data describing the genetic characteristics of an individual and is provided by specialized institutions.

[0510] "Genetic information analysis" is a process for predicting future appearance and growth patterns based on input genetic information.

[0511] "Predicting future appearance" means predicting future appearance, including facial features, hair color, skin color, height, and body type, based on the results of analyzing genetic information.

[0512] "User emotion recognition" refers to sensing the user's facial expressions and tone of voice using the device's built-in camera and microphone, and analyzing them to understand the user's emotional state.

[0513] "Adjusted display based on emotion information" refers to dynamically changing the display content of the generated three-dimensional model or interface based on the user's recognized emotion information.

[0514] The present invention combines a system that generates a 3D model of a fetus from ultrasound images, analyzes genetic information to predict its future appearance, and recognizes the user's emotions and adjusts the display content accordingly. To implement this system, the following specific hardware and software are used:

[0515] The user accesses a dedicated application or website using a smartphone or computer. It is recommended that the hardware used has a camera and microphone. The following describes the detailed process at each step and the software used.

[0516] The server receives the echo images uploaded by the user. These images are pre-processed using an image processing library (e.g., Python's Pillow library). Pre-processing includes converting to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0517] After preprocessing, the ultrasound images are used to generate a 3D model of the fetus using a generative AI model (e.g., GANs using TensorFlow), which is then temporarily stored in a database.

[0518] The user then inputs their genetic information. This information is typically provided by a specialized testing institution in the form of a data file in text format. The server receives this genetic information and analyzes it. This analysis involves converting the genetic information into a standard format and extracting specified characteristics (facial features, hair color, skin color, height, body type, etc.). This analysis is performed using the Python Pandas library.

[0519] Based on the analysis results, growth patterns are predicted and a three-dimensional model of the future shape is generated, which is performed using machine learning algorithms (e.g., random forests).

[0520] While the user is using the system, the device's camera and microphone detect the user's facial expressions and tone of voice. Emotion analysis is performed using computer vision libraries (e.g., OpenCV) and facial expression recognition libraries (e.g., DeepFace).

[0521] The server analyzes this data using an emotion engine to recognize the user's emotions. Based on the recognized emotion information, the generated 3D model and system interface are adjusted. Specifically, if the user is feeling surprised or moved, the system increases the level of visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0522] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image to the system and inputs genetic information obtained from a specialized testing institution. The ultrasound image is preprocessed, and a realistic 3D model is generated using a generative AI model. At the same time, the genetic information is analyzed and the future appearance is predicted. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotion, to the device, where the user can view and manipulate it on their smartphone or computer screen.

[0523] Example prompt sentence:

[0524] "I have uploaded an ultrasound image from the 20th week of pregnancy and entered genetic information. Please generate a growth prediction model from this. I would also like you to collect facial expression data from the user in real time and output the data according to their emotions."

[0525] This makes it easier for users to understand the specific appearance and future shape of the fetus, and the system adjusts the visuals and interface according to the user's emotions, allowing them to feel a deeper sense of emotion and security.

[0526] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0527] System program processing flow

[0528] Step 1: User uploads an echo image

[0529] Users use a smartphone or computer to access a dedicated application or website to take or scan and upload echo images, and then use the camera or file selection function of the specific device to select and send the image files to the system.

[0530] Input: Echo image file

[0531] Output: Echo image file sent to the server

[0532] Step 2: The server preprocesses the images

[0533] The server preprocesses the received echo images in the following steps:

[0534] Conversion to standard formats: Converting different image formats (e.g. JPEG to PNG) into a format that the system can properly handle.

[0535] Noise reduction: Filters out and removes unwanted noise from the image.

[0536] Resolution Adjustment: Adjust the image to the optimal resolution.

[0537] Contrast Enhancement: Increases the contrast of the image to make it easier to see.

[0538] Input: Echo image file sent by the user

[0539] Output: Preprocessed image files in standard format

[0540] Step 3: The server generates the 3D model

[0541] Based on the preprocessed ultrasound images, the server uses a generative AI model (e.g., GANs using TensorFlow) to generate a 3D model of the fetus. This generated model is temporarily stored in a database. Specifically, features are extracted from the image using a convolutional neural network (CNN), and a 3D model is generated using GANs based on those features.

[0542] Input: Preprocessed image files in standard format

[0543] Output: Generated 3D model of the fetus

[0544] Step 4: User enters genetic information

[0545] Users input genetic information data files provided by professional testing institutions into the system, and use the application's "genetic information upload" function to select and send the data files to the system.

[0546] Input: Gene information file

[0547] Output: Gene information file sent to the server

[0548] Step 5: The server analyzes the genetic information and predicts the future appearance

[0549] The server converts the received genetic information into a standard format and analyzes the information. The analysis extracts facial features, hair color, skin color, height, body type, etc. from the input genetic information, and predicts future appearance based on growth patterns. A machine learning algorithm (e.g., random forest) is used for the prediction.

[0550] Input: Gene information file

[0551] Output: Analyzed genetic information and a predicted 3D model of your future appearance

[0552] Step 6: The device recognizes the user's emotions

[0553] While the user is using the system, the device's camera captures the user's facial expressions and tone of voice and transmits them to the server in real time. The server then uses an emotion engine (e.g., OpenCV and DeepFace) to analyze this data and recognize the user's emotions. For example, the camera captures the user's smiling or surprised expressions and uses them to infer the user's emotions.

[0554] Input: User's facial and voice data

[0555] Output: Parsed emotion information

[0556] Step 7: The server adjusts and displays the 3D model and interface.

[0557] Based on the recognized emotion information, the server adjusts the displayed 3D model and system interface. For example, if the user is surprised or impressed, the server increases the visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0558] Input: Analyzed emotion information, generated 3D model, and predicted future appearance

[0559] Output: emotion-adjusted 3D model and system interface

[0560] This allows users to view and manipulate dynamically adjusted 3D models and interfaces in real time.

[0561] (Application example 2)

[0562] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0563] Conventional systems for generating 3D fetal models based on ultrasound images tend to display static images without considering the user's emotions. This makes it difficult for users to feel interactivity with the system, preventing the quality of the experience from being maximized. Furthermore, predictions of the fetus's future appearance can sometimes lack persuasiveness. The objective of this invention is not only to provide a 3D fetus model and a prediction of its future appearance from ultrasound images, but also to dynamically adjust the display content using the user's emotional information, providing an interactive and moving experience.

[0564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0565] In this invention, the server includes means for preprocessing the ultrasound images and converting them into a standard format, means for generating a 3D model of the fetus from the preprocessed ultrasound images, means for analyzing input genetic information to predict future appearance, means for inputting emotional data, means for analyzing the input emotional data to recognize the user's emotion, and means for dynamically adjusting the display content in accordance with the user's emotion, thereby enabling the provision of an interactive and moving 3D model display of the fetus and a predicted future appearance that are tailored to the user's emotional state.

[0566] An "echo image" is a medical image taken using ultrasound to capture the inside of the body, and is primarily used to check the condition of a fetus.

[0567] "Preprocessing" is the process of converting input image data into a standard format and performing processing such as noise removal, resolution adjustment, and contrast enhancement.

[0568] A "generative model" is an algorithm that uses artificial intelligence or machine learning techniques to generate a three-dimensional model based on specific input data.

[0569] "Genetic information" refers to the data contained in the genetic material of an organism, and is the information recorded in DNA or RNA.

[0570] "Analysis" is the process of examining and breaking down input data in detail to extract meaning and patterns.

[0571] "Future appearance" is the subject's future appearance or appearance predicted based on current data.

[0572] "Emotion data" is data that indicates the user's emotional state, including facial expressions and vocal tones.

[0573] An "emotion engine" is an algorithm or software that recognizes a user's emotional state, primarily by analyzing facial expressions and tone of voice.

[0574] "Dynamic adjustment" is the process of changing the display content and interface in real time based on the situation and data.

[0575] In the system realizing this invention, the server preprocesses the echo image, generates a 3D model of the fetus, analyzes genetic information, and further recognizes the user's emotions to dynamically adjust the display content. This system has the following configuration and processing means.

[0576] The server first receives the echo images uploaded by the user using a tablet or computer, and then preprocesses them using image processing libraries such as OpenCV and scikit-image, including converting them to a standard format, removing noise, adjusting the resolution, and enhancing the contrast.

[0577] Next, the preprocessed echo images are input into a generative AI model using TensorFlow and PyTorch to generate a 3D model of the fetus, which is then temporarily stored in storage.

[0578] The user then inputs genetic information provided by a specialized testing institution into the system. This genetic information is analyzed using libraries such as BioPython and Pandas to predict future appearance and growth patterns. The genetic analysis results include facial features, hair color, skin color, height, and body type, and a three-dimensional model of the future appearance is generated based on these results.

[0579] The system also uses an emotion engine to analyze the user's emotional data. The device's camera detects the user's facial expressions and tone of voice, and the data is analyzed using Google Cloud Vision API and DeepFace. Based on the analyzed emotional data, the server dynamically adjusts the display content. For example, if the user is emotional, it can display more detailed and emotional visuals.

[0580] The three-dimensional model and predicted future appearance generated in this way are displayed to the user via a tablet, computer display, VR headset, etc. The user can manipulate it to experience the growth and future appearance of the fetus in a realistic way.

[0581] As a concrete example, consider a case where a user in the 24th week of pregnancy uploads an ultrasound image and inputs genetic information into the system. In this case, the server preprocesses the ultrasound image and uses a generative AI model to generate a realistic 3D model of the fetus. Meanwhile, the device's camera detects the user's facial expression, and the emotion engine analyzes the user's emotions. Based on the results, the display content is dynamically adjusted. This allows the user to feel more deeply moved and at ease.

[0582] An example prompt is, "Upload an ultrasound image to generate a realistic 3D model. We'll also predict your future appearance based on your genetic information and use emotion recognition to tailor the interface."

[0583] In this way, the present invention utilizes advanced technology to provide the user with an interactive and emotional experience, maximizing comfort and enjoyment during pregnancy.

[0584] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0585] Step 1:

[0586] A user uploads an echo image using a terminal. The echo image file is sent as input to the server, which receives the echo image and converts it into a standard format.

[0587] Step 2:

[0588] The server preprocesses the received echo images using OpenCV and scikit-image libraries to remove noise, adjust resolution, and enhance contrast. The output of the preprocessing is a high-quality image converted to a standard format.

[0589] Step 3:

[0590] The server uses the preprocessed ultrasound images to input them into a generative AI model (TensorFlow or PyTorch) to generate a 3D model of the fetus. This generative AI model is a pre-trained neural network that generates a highly accurate 3D model from the input images. The output is 3D model data of the fetus.

[0591] Step 4:

[0592] The user uploads a genetic information file using a terminal. The genetic information file contains information about future appearance, such as facial features, hair color, skin color, height, and body type. The server receives the genetic information.

[0593] Step 5:

[0594] The server uses BioPython and Pandas to analyze the input genetic information. The analysis results in future appearance prediction data, which includes details about the fetus's growth patterns and predicted appearance. The output is the future appearance prediction data.

[0595] Step 6:

[0596] The server generates a 3D model of the future appearance based on the analysis results. The generated 3D model reflects the fetus's growth pattern, allowing the user to visually confirm the future appearance. The output is the 3D model data of the future appearance.

[0597] Step 7:

[0598] The device's camera detects the user's facial expressions and tone of voice to obtain emotional data, which includes the user's current emotional state. The server receives this emotional data.

[0599] Step 8:

[0600] The server uses an emotion engine (such as Google Cloud Vision API or DeepFace) to analyze the acquired emotion data. The analysis recognizes the user's emotional state. The output is the user's emotional state data.

[0601] Step 9:

[0602] The server dynamically adjusts the display content based on the recognized emotional state data. For example, if the user is emotional, the server changes the display content to more detailed and emotional visuals. The adjusted display content is sent to the user's device and displayed.

[0603] Step 10:

[0604] Based on the adjusted display, users can view a 3D model of the fetus and its future appearance in real time on a tablet, computer screen, or VR headset, with the user's experience optimized according to their emotions.

[0605] Through the above steps, the present invention utilizes echo images and genetic information to provide users with an interactive and moving experience.

[0606] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0607] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0608] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0609] [Third embodiment]

[0610] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0611] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0612] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0613] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0614] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0615] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0616] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0617] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0618] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0619] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0620] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0621] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0622] This invention is a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information. Specifically, the following process is performed.

[0623] First, a user accesses the system's dedicated application or website using a smartphone or computer, takes or scans an echo image, and uploads it to the system.

[0624] The server accepts uploaded echo images. Since the images may vary in size and format, the server converts them to a standard format, adjusts the resolution, removes noise from the images, and adjusts the contrast to improve visibility.

[0625] Next, based on the preprocessed ultrasound images, the server uses a generative AI model to generate a 3D model of the fetus. The generative AI model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generated 3D model is temporarily stored.

[0626] The user then enters the fetus's genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives the genetic information and converts it into a standard format if necessary.

[0627] The server analyzes the genetic information to extract information about future appearance, such as facial features, hair color, skin color, height, and body type. Based on the analysis results, the server predicts growth patterns and generates a three-dimensional model of the future appearance, utilizing existing growth databases and statistical models.

[0628] Finally, the server renders the generated current and future 3D models to display them visually, and the rendered 3D models are sent to the device in an interactive format (e.g., WebGL), allowing users to view and manipulate these models on their smartphone or computer screen.

[0629] As a concrete example, suppose a user in the 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. At the same time, the server analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see a realistic image of the fetus now, as well as predicted future appearances at age 5 and 10.

[0630] This invention will help expectant mothers and their families understand the specific appearance and future shape of their fetus, allowing them to experience moving and meaningful moments, and will also help reduce anxiety during pregnancy.

[0631] The processing flow will be explained below.

[0632] Step 1:

[0633] A user uses a smartphone or computer to access the system's dedicated application or website.

[0634] Step 2:

[0635] The user takes or scans an echo image and uploads it to the system.

[0636] Step 3:

[0637] The server accepts the uploaded echo images.

[0638] Step 4:

[0639] The server converts the uploaded echo image format to a standard format (e.g., JPEG, PNG), and also performs noise removal, resolution adjustment, and contrast adjustment.

[0640] Step 5:

[0641] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0642] Step 6:

[0643] The server stores the generated 3D model in temporary storage.

[0644] Step 7:

[0645] The user enters the fetus's genetic information into the system, which is typically provided in a data file by a specialized testing laboratory.

[0646] Step 8:

[0647] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0648] Step 9:

[0649] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the genetic information.

[0650] Step 10:

[0651] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0652] Step 11:

[0653] The server renders the generated current and future 3D models and generates the data for visual display. Rendering is done using technologies such as WebGL.

[0654] Step 12:

[0655] The rendered three-dimensional model data is sent to the terminal.

[0656] Step 13:

[0657] The user can view and manipulate a 3D model of the current fetus and a 3D model of its predicted future appearance on a smartphone or computer screen.

[0658] Step 14:

[0659] Users can share the 3D models they create with family and friends.

[0660] Example 1

[0661] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0662] Systems that can precisely reconstruct the fetus's appearance from ultrasound images and predict its future appearance based on genetic information require highly accurate image processing and analysis. Previous systems performed the preprocessing of ultrasound images, the generation of 3D models, and the analysis of genetic information separately, making it difficult to provide an integrated system. Furthermore, they lacked an interactive method that allowed users to intuitively manipulate the generated 3D models, leaving a need for an improved user experience.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0664] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a three-dimensional model of the fetus from the preprocessed echo image, means for inputting genetic information, means for analyzing the input genetic information and predicting future appearance, means for displaying the generated three-dimensional model and the predicted future appearance, means for rendering the generated three-dimensional model and transmitting it to a terminal in an interactive format, and means for the user to manipulate and confirm the three-dimensional model on the terminal. This integrates highly accurate image processing and analysis, allowing the user to intuitively manipulate and confirm the current and future appearance of the fetus.

[0665] An "echo image" is an image of the inside of a body captured using ultrasound.

[0666] "Preprocessing" refers to the process of converting the input echo image into a format that is easy for the system to analyze, such as by converting the format, removing noise, and adjusting the resolution.

[0667] A "standard format" is a unified data format that improves data compatibility and processing efficiency.

[0668] A "3D model" is a three-dimensional digital representation that can be observed from multiple perspectives.

[0669] A "generative model means" is an algorithm or software that generates new data or information based on input data.

[0670] "Genetic information" is data that records the sequence of DNA or RNA, and is information that determines the characteristics and functions of an individual.

[0671] "Analysis" is the process of examining data using statistical or mathematical methods to extract useful information.

[0672] "Forecasting" is the act of estimating future events or conditions based on current data and trends.

[0673] "Rendering" is the process of displaying three-dimensional models or image data on a flat display.

[0674] An "interactive format" is a two-way display format that changes in real time in response to user operations and inputs.

[0675] A "terminal" is a digital device used by a user to access the system, examples of which include a smartphone or computer.

[0676] "User" refers to the end user who operates the system to input information and check the results.

[0677] This invention is a system that reconstructs a fetus in detail from an ultrasound image and predicts its future appearance by analyzing its genetic information. This system is specifically implemented based on the following procedure and configuration.

[0678] First, the user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system. The echo image can be input using the smartphone's camera or scanner.

[0679] The server receives the uploaded echo images, checks the image format, converts them to a standard format, adjusts the image resolution, removes noise, and adjusts the contrast to improve visibility. This process is often performed using image processing software.

[0680] After preprocessing, the ultrasound images are input to a server running a generative AI model. This model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generative AI model generates a three-dimensional model of the fetus and temporarily stores it.

[0681] The user then enters their genetic information into the system, often in the form of a data file provided by a specialized testing laboratory. The server receives the information, converts it into a standard format if necessary, and analyzes it to extract information about future appearance, such as facial features, hair color, skin tone, and height.

[0682] The server predicts growth patterns based on the genetic analysis, leverages existing growth databases and statistical models, and generates a three-dimensional model of the future, often using growth prediction algorithms.

[0683] Finally, the server renders the generated current and future 3D models. This rendering is done in an interactive format such as WebGL. The rendered 3D models are sent to the device, where users can view and manipulate these models on their smartphone or computer screen. Specifically, users can rotate the 3D models, zoom in and out, and view them from different perspectives.

[0684] As a concrete example, consider a case where a user in the 20th week of pregnancy takes an ultrasound image using their smartphone and uploads it to the system, while also inputting genetic information obtained from a blood test. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It then analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see in detail the realistic appearance of the fetus now, as well as predicted appearances at age 5 and 10.

[0685] By making good use of the features of this invention, expectant mothers and their families will be able to better understand the specific appearance and future shape of their fetus, allowing them to share moving and meaningful moments together, and it is also expected to contribute to reducing anxiety during pregnancy.

[0686] Example prompt sentence:

[0687] "You provide the system with ultrasound images and genetic information to predict what the fetus looks like now and what it will look like in the future."

[0688] "Based on ultrasound images of a fetus at 20 weeks of pregnancy and genetic information, please make a realistic prediction of what the baby will look like at age 5 and 10."

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1:

[0691] A user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads the echo image file to the system, where the echo image as input is sent to the server as output.

[0692] Step 2:

[0693] The server receives the uploaded echo image and first checks the image format. Specifically, the image is converted to a standard format such as JPEG or PNG. Then, resolution adjustments, noise reduction filters, and contrast adjustments are applied to improve visibility. The input echo image is output as a preprocessed image converted to a standard format.

[0694] Step 3:

[0695] The server inputs the preprocessed echo images into a generative AI model. The generative AI model uses a multi-layer neural network to generate a 3D model of the fetus from the echo images. The preprocessed echo images as input are saved as the generated 3D model as output. Specific operations include training and inference of the AI ​​model.

[0696] Step 4:

[0697] The user prepares a genetic information data file provided by a specialized testing institution and proceeds to the genetic information input screen in a dedicated application or website. Next, the user selects the genetic information file and clicks the "Upload" button. The genetic information as input is sent to the server.

[0698] Step 5:

[0699] The server receives the uploaded genetic information, checks the format, converts it to a standard format if necessary, and analyzes the genetic information to extract information about appearance, such as facial features, hair color, skin color, and height. Specifically, an algorithm is used to analyze useful feature information from the genetic data. The input genetic information is output as analyzed feature information.

[0700] Step 6:

[0701] The server predicts growth patterns based on the results of genetic information analysis. It uses existing growth databases and statistical models to simulate fetal growth. Feature information and growth data are input, and predicted growth patterns and a future three-dimensional model are obtained as output. Specific operations include statistical analysis and simulation work.

[0702] Step 7:

[0703] The server renders the generated current and future 3D models. In this process, rendering is performed in an interactive format such as WebGL. The input 3D model is output as a rendered interactive format. Specific operations involve the use of graphics rendering algorithms.

[0704] Step 8:

[0705] The server sends the rendered 3D model to the terminal. The user opens the system's dedicated application or website on the terminal and views and manipulates the 3D model. The input interactive model is output as a manipulable display on the terminal. Specific actions the user can perform include rotating, zooming, and translating the model.

[0706] (Application example 1)

[0707] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0708] Conventional ultrasound image analysis systems can predict the current and future appearance of a fetus, but they cannot provide users with virtual simulations of its future appearance. This means that users cannot enjoy virtual experiences such as trying on products based on their future appearance. Furthermore, there are limitations to the accuracy of processing and real-time display in the process of analyzing ultrasound images and genetic information.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0710] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a 3D model of a fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for displaying the generated 3D model and the predicted future appearance, and means for inputting and preprocessing a user's facial image and virtually simulating a product based on the user's future appearance, thereby enabling the user to experience a virtual simulation based on the user's future appearance.

[0711] An "echo image" is an image that uses ultrasound to visualize the internal structures of a fetus.

[0712] "Preprocessing" refers to the process of converting the input image into a standard format and performing any necessary corrections or noise removal.

[0713] A "standard format" is a standardized image or data format that can be recognized by systems.

[0714] A "generative model" is an artificial intelligence model that generates three-dimensional models based on echo images and genetic information.

[0715] "Genetic information" means information, including the DNA data of a fetus, that indicates the unique biological characteristics of that individual.

[0716] A "three-dimensional model" is a digital model that realistically reproduces the fetus and its future appearance in three-dimensional space.

[0717] "Display means" refers to a device or software for visually presenting the generated three-dimensional model and prediction results to the user.

[0718] A "face image" is photographic data of a user's face.

[0719] "Virtual simulation" refers to the experience of virtually trying on products, clothing, cosmetics, etc. based on the user's future appearance.

[0720] This invention is a system that generates a three-dimensional model of the fetus's current and future appearance based on ultrasound images and genetic information, and also provides a virtual simulation based on the fetus's future appearance. A specific implementation method of this system is described below.

[0721] Users access the system using a device such as a smartphone, tablet, or computer, and take or scan an echo image using a camera or scanner installed in an autonomous vehicle or logistics center. Once the image is taken or scanned, it can be uploaded to the system via the device.

[0722] The server accepts the uploaded ultrasound images and first performs preprocessing. If the images vary in size or format, they are converted into a standardized format, and the resolution is adjusted and noise is removed. The preprocessed images are then input into a generative AI model, which generates a 3D model of the fetus. This generative AI model uses a multi-layer neural network trained on past ultrasound images and actual post-birth data.

[0723] The user then enters their genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives and analyzes this information, extracting information about the individual's future appearance, including facial features, hair color, skin tone, height, and body type. Using this information, the server predicts growth patterns and generates a three-dimensional model of the individual's future appearance.

[0724] The generated 3D model is then rendered for visual display and sent to the device in an interactive format using technologies such as WebGL, allowing users to view and manipulate the 3D model on their smartphone, tablet, or computer screen.

[0725] Furthermore, the system also has the ability to input and preprocess a user's facial image and virtually simulate products based on their future appearance, allowing users to virtually try on and try on products such as clothing and cosmetics based on their future appearance.

[0726] As a concrete example, consider the case where a user uploads a photo of their face and simulates their hairstyle and makeup from their younger days. In this case, the server uses a generative AI model based on the photo to predict their future appearance and perform a virtual simulation. Examples of prompts include:

[0727] "Upload a photo of yourself from 20 years ago and compare it to how you look now."

[0728] "Enter your DNA information and do a virtual try-on that predicts your future appearance."

[0729] As described above, the present invention enables the generation of three-dimensional models and virtual simulations using echo images and genetic information, providing users with a new experience.

[0730] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0731] Step 1: Upload your ultrasound images

[0732] Users access the system using a smartphone, tablet, or computer to take or scan echo images, and then upload the images to a dedicated application or website. The input is the echo image data, and the output is an image file sent to the server.

[0733] Step 2: Preprocessing of echo images

[0734] The server receives the uploaded echo images and performs preprocessing, specifically converting the image size and format to a standard format, adjusting the resolution, and removing noise. At this stage, the input is echo image data, and the output is preprocessed echo images in a unified format.

[0735] Step 3: Generate a 3D model

[0736] The server inputs the preprocessed ultrasound images into a generative AI model to generate a 3D model of the fetus. This generative AI model is a multi-layer neural network trained on past ultrasound images and actual post-birth data. The input is the preprocessed ultrasound images, and the output is a 3D digital model of the fetus.

[0737] Step 4: Enter genetic information

[0738] Users input genetic information provided by specialized testing laboratories into the system. The input is a data file containing DNA information, and the output is genetic data sent to the server.

[0739] Step 5: Analyzing genetic information

[0740] The server analyzes the input genetic information and extracts information about future appearance, such as facial features, hair color, skin color, height, and body type. This analysis uses a database containing features correlated with various genetic data. The input is genetic data, and the output is appearance feature data derived from the genetic information.

[0741] Step 6: Generate the future 3D model

[0742] The server predicts growth patterns based on the analysis results and generates a 3D model of the future appearance, utilizing existing growth databases and statistical models. The input is appearance feature data and growth prediction data, and the output is a 3D digital model of the future appearance.

[0743] Step 7: Rendering the 3D model

[0744] The server performs rendering to visually display the generated current and future 3D models. Using technologies such as WebGL, it transmits them to the device in an interactive format. The input is the 3D digital model, and the output is the rendered interactive 3D display data.

[0745] Step 8: Conduct a virtual simulation

[0746] The user inputs an image of their face into the system and performs preprocessing. Based on this image, the server uses a generative AI model to predict future appearance and runs a virtual simulation. The input is facial image data and a prediction model, and the output is a virtually tried-on product image. Examples of prompts include "Upload a photo of your face from 20 years ago and compare it with your current self," and "Enter your DNA information to virtually try on products with predicted future appearances."

[0747] Through these steps, the system can provide users with a new experience.

[0748] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0749] This invention combines a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information with an emotion engine that recognizes the user's emotions. Specifically, it performs the following processes:

[0750] First, a user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system.

[0751] The server receives the uploaded ultrasound images and performs preprocessing such as converting the images to a standard format and removing noise, adjusting resolution, and enhancing contrast. Once preprocessing is complete, the server uses a generative AI model to generate a 3D model of the fetus from the ultrasound images. The generated 3D model is temporarily stored in storage.

[0752] Next, the user enters the fetus's genetic information into the system. This information is typically provided in the form of a data file from a specialized testing laboratory. The server receives the genetic information, converts it into a standard format if necessary, and then analyzes it. The analysis results include future appearance, such as facial features, hair color, skin tone, height, and body type. Based on this analysis, the server predicts growth patterns and generates a three-dimensional model of the fetus's future appearance.

[0753] One of the features of the present invention is the incorporation of an emotion engine. While the user is using the system, the terminal is equipped with a camera that detects the user's facial expressions and tone of voice. The server uses the emotion engine to analyze this data and recognize the user's emotions. Based on the recognized emotion information, the server can adjust the displayed 3D model, system interface, and output content. For example, if the user is feeling surprised or moved, it can display more detailed and emotional visuals.

[0754] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It also analyzes the genetic information and predicts the future appearance. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotions, to the device, where the user can view and operate it on their smartphone or computer screen.

[0755] This invention not only helps users understand the specific appearance and future shape of the fetus, but also allows the system to adjust the visuals and interface according to the user's emotions, providing a deeper sense of emotion and security, and contributing to reducing anxiety during pregnancy.

[0756] The processing flow will be explained below.

[0757] Step 1:

[0758] A user uses a smartphone or computer to access the system's dedicated application or website.

[0759] Step 2:

[0760] The user takes or scans an echo image and uploads it to the system.

[0761] Step 3:

[0762] The server accepts the uploaded echo images.

[0763] Step 4:

[0764] The server converts the uploaded echo images into standard formats (e.g., JPEG, PNG), and also performs noise reduction, resolution adjustment, and contrast enhancement.

[0765] Step 5:

[0766] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0767] Step 6:

[0768] The server stores the generated 3D model in temporary storage.

[0769] Step 7:

[0770] The user inputs the fetus's genetic information into the system, which is usually a data file provided by a specialized testing institution.

[0771] Step 8:

[0772] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0773] Step 9:

[0774] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the analyzed genetic information.

[0775] Step 10:

[0776] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0777] Step 11:

[0778] The camera installed on the device detects the user's facial expressions and tone of voice in real time.

[0779] Step 12:

[0780] The server uses an emotion engine to analyze the user's facial expression and voice data sent from the terminal and recognize the user's emotional state.

[0781] Step 13:

[0782] The server adjusts the content of the displayed 3D model and the system interface based on the recognized emotional state. For example, if the user expresses surprise, it displays more detailed and moving visuals.

[0783] Step 14:

[0784] The server renders the generated current and future 3D models and generates the data for visual display, using technologies such as WebGL.

[0785] Step 15:

[0786] The rendered three-dimensional model data is sent to the terminal.

[0787] Step 16:

[0788] The user can view and manipulate a three-dimensional model of the current fetus and a three-dimensional model of its predicted future appearance on a smartphone or computer screen.

[0789] Step 17:

[0790] Users can share the 3D models they create with family and friends.

[0791] In this way, the system generates a three-dimensional model of the fetus based on ultrasound images and genetic information, and then uses an emotion engine to adjust the interface based on the user's emotions, creating a more moving and reassuring experience.

[0792] Example 2

[0793] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0794] In conventional echo image analysis and genetic information analysis systems, preprocessing and noise removal of echo images, and conversion and analysis of genetic information are performed separately, making comprehensive use difficult. Furthermore, the generated 3D models and predicted future appearances are fixed, and dynamic adjustments based on the user's emotions are not possible. As a result, user satisfaction is low and the utility of the system is limited. The purpose of this invention is to solve these problems.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0796] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for removing noise from the preprocessed echo image, means for generating a 3D model of the fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for recognizing a user's emotion, and means for adjusting and displaying the generated 3D model and the predicted future appearance based on the recognized emotion information, thereby enabling the generation and display of a dynamic and detailed 3D model in accordance with the user's emotion.

[0797] An "echo image" is an image of internal structures, particularly the fetus, taken using ultrasound technology.

[0798] "Preprocessing" refers to processing of the input echo image, such as conversion to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0799] A "standard format" is an image file format that is specified so that the system can process it appropriately.

[0800] "Noise reduction" is a process that improves image quality by removing unwanted noise and distortion from an image.

[0801] "Generative model means" includes methods and algorithms for generating a three-dimensional model of the fetus from pre-processed echo images.

[0802] "Genetic information" is data describing the genetic characteristics of an individual and is provided by specialized institutions.

[0803] "Genetic information analysis" is a process for predicting future appearance and growth patterns based on input genetic information.

[0804] "Predicting future appearance" means predicting future appearance, including facial features, hair color, skin color, height, and body type, based on the results of analyzing genetic information.

[0805] "User emotion recognition" refers to sensing the user's facial expressions and tone of voice using the device's built-in camera and microphone, and analyzing them to understand the user's emotional state.

[0806] "Adjusted display based on emotion information" refers to dynamically changing the display content of the generated three-dimensional model or interface based on the user's recognized emotion information.

[0807] The present invention combines a system that generates a 3D model of a fetus from ultrasound images, analyzes genetic information to predict its future appearance, and recognizes the user's emotions and adjusts the display content accordingly. To implement this system, the following specific hardware and software are used:

[0808] The user accesses a dedicated application or website using a smartphone or computer. It is recommended that the hardware used has a camera and microphone. The following describes the detailed process at each step and the software used.

[0809] The server receives the echo images uploaded by the user. These images are pre-processed using an image processing library (e.g., Python's Pillow library). Pre-processing includes converting to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[0810] After preprocessing, the ultrasound images are used to generate a 3D model of the fetus using a generative AI model (e.g., GANs using TensorFlow), which is then temporarily stored in a database.

[0811] The user then inputs their genetic information. This information is typically provided by a specialized testing institution in the form of a data file in text format. The server receives this genetic information and analyzes it. This analysis involves converting the genetic information into a standard format and extracting specified characteristics (facial features, hair color, skin color, height, body type, etc.). This analysis is performed using the Python Pandas library.

[0812] Based on the analysis results, growth patterns are predicted and a three-dimensional model of the future shape is generated, which is performed using machine learning algorithms (e.g., random forests).

[0813] While the user is using the system, the device's camera and microphone detect the user's facial expressions and tone of voice. Emotion analysis is performed using computer vision libraries (e.g., OpenCV) and facial expression recognition libraries (e.g., DeepFace).

[0814] The server analyzes this data using an emotion engine to recognize the user's emotions. Based on the recognized emotion information, the generated 3D model and system interface are adjusted. Specifically, if the user is feeling surprised or moved, the system increases the level of visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0815] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image to the system and inputs genetic information obtained from a specialized testing institution. The ultrasound image is preprocessed, and a realistic 3D model is generated using a generative AI model. At the same time, the genetic information is analyzed and the future appearance is predicted. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotion, to the device, where the user can view and manipulate it on their smartphone or computer screen.

[0816] Example prompt sentence:

[0817] "I have uploaded an ultrasound image from the 20th week of pregnancy and entered genetic information. Please generate a growth prediction model from this. I would also like you to collect facial expression data from the user in real time and output the data according to their emotions."

[0818] This makes it easier for users to understand the specific appearance and future shape of the fetus, and the system adjusts the visuals and interface according to the user's emotions, allowing them to feel a deeper sense of emotion and security.

[0819] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0820] System program processing flow

[0821] Step 1: User uploads an echo image

[0822] Users use a smartphone or computer to access a dedicated application or website to take or scan and upload echo images, and then use the camera or file selection function of the specific device to select and send the image files to the system.

[0823] Input: Echo image file

[0824] Output: Echo image file sent to the server

[0825] Step 2: The server preprocesses the images

[0826] The server preprocesses the received echo images in the following steps:

[0827] Conversion to standard formats: Converting different image formats (e.g. JPEG to PNG) into a format that the system can properly handle.

[0828] Noise reduction: Filters out and removes unwanted noise from the image.

[0829] Resolution Adjustment: Adjust the image to the optimal resolution.

[0830] Contrast Enhancement: Increases the contrast of the image to make it easier to see.

[0831] Input: Echo image file sent by the user

[0832] Output: Preprocessed image files in standard format

[0833] Step 3: The server generates the 3D model

[0834] Based on the preprocessed ultrasound images, the server uses a generative AI model (e.g., GANs using TensorFlow) to generate a 3D model of the fetus. This generated model is temporarily stored in a database. Specifically, features are extracted from the image using a convolutional neural network (CNN), and a 3D model is generated using GANs based on those features.

[0835] Input: Preprocessed image files in standard format

[0836] Output: Generated 3D model of the fetus

[0837] Step 4: User enters genetic information

[0838] Users input genetic information data files provided by professional testing institutions into the system, and use the application's "genetic information upload" function to select and send the data files to the system.

[0839] Input: Gene information file

[0840] Output: Gene information file sent to the server

[0841] Step 5: The server analyzes the genetic information and predicts the future appearance

[0842] The server converts the received genetic information into a standard format and analyzes the information. The analysis extracts facial features, hair color, skin color, height, body type, etc. from the input genetic information, and predicts future appearance based on growth patterns. A machine learning algorithm (e.g., random forest) is used for the prediction.

[0843] Input: Gene information file

[0844] Output: Analyzed genetic information and a predicted 3D model of your future appearance

[0845] Step 6: The device recognizes the user's emotions

[0846] While the user is using the system, the device's camera captures the user's facial expressions and tone of voice and transmits them to the server in real time. The server then uses an emotion engine (e.g., OpenCV and DeepFace) to analyze this data and recognize the user's emotions. For example, the camera captures the user's smiling or surprised expressions and uses them to infer the user's emotions.

[0847] Input: User's facial and voice data

[0848] Output: Parsed emotion information

[0849] Step 7: The server adjusts and displays the 3D model and interface.

[0850] Based on the recognized emotion information, the server adjusts the displayed 3D model and system interface. For example, if the user is surprised or impressed, the server increases the visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[0851] Input: Analyzed emotion information, generated 3D model, and predicted future appearance

[0852] Output: emotion-adjusted 3D model and system interface

[0853] This allows users to view and manipulate dynamically adjusted 3D models and interfaces in real time.

[0854] (Application example 2)

[0855] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0856] Conventional systems for generating 3D fetal models based on ultrasound images tend to display static images without considering the user's emotions. This makes it difficult for users to feel interactivity with the system, preventing the quality of the experience from being maximized. Furthermore, predictions of the fetus's future appearance can sometimes lack persuasiveness. The objective of this invention is not only to provide a 3D fetus model and a prediction of its future appearance from ultrasound images, but also to dynamically adjust the display content using the user's emotional information, providing an interactive and moving experience.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0858] In this invention, the server includes means for preprocessing the ultrasound images and converting them into a standard format, means for generating a 3D model of the fetus from the preprocessed ultrasound images, means for analyzing input genetic information to predict future appearance, means for inputting emotional data, means for analyzing the input emotional data to recognize the user's emotion, and means for dynamically adjusting the display content in accordance with the user's emotion, thereby enabling the provision of an interactive and moving 3D model display of the fetus and a predicted future appearance that are tailored to the user's emotional state.

[0859] An "echo image" is a medical image taken using ultrasound to capture the inside of the body, and is primarily used to check the condition of a fetus.

[0860] "Preprocessing" is the process of converting input image data into a standard format and performing processing such as noise removal, resolution adjustment, and contrast enhancement.

[0861] A "generative model" is an algorithm that uses artificial intelligence or machine learning techniques to generate a three-dimensional model based on specific input data.

[0862] "Genetic information" refers to the data contained in the genetic material of an organism, and is the information recorded in DNA or RNA.

[0863] "Analysis" is the process of examining and breaking down input data in detail to extract meaning and patterns.

[0864] "Future appearance" is the subject's future appearance or appearance predicted based on current data.

[0865] "Emotion data" is data that indicates the user's emotional state, including facial expressions and vocal tones.

[0866] An "emotion engine" is an algorithm or software that recognizes a user's emotional state, primarily by analyzing facial expressions and tone of voice.

[0867] "Dynamic adjustment" is the process of changing the display content and interface in real time based on the situation and data.

[0868] In the system realizing this invention, the server preprocesses the echo image, generates a 3D model of the fetus, analyzes genetic information, and further recognizes the user's emotions to dynamically adjust the display content. This system has the following configuration and processing means.

[0869] The server first receives the echo images uploaded by the user using a tablet or computer, and then preprocesses them using image processing libraries such as OpenCV and scikit-image, including converting them to a standard format, removing noise, adjusting the resolution, and enhancing the contrast.

[0870] Next, the preprocessed echo images are input into a generative AI model using TensorFlow and PyTorch to generate a 3D model of the fetus, which is then temporarily stored in storage.

[0871] The user then inputs genetic information provided by a specialized testing institution into the system. This genetic information is analyzed using libraries such as BioPython and Pandas to predict future appearance and growth patterns. The genetic analysis results include facial features, hair color, skin color, height, and body type, and a three-dimensional model of the future appearance is generated based on these results.

[0872] The system also uses an emotion engine to analyze the user's emotional data. The device's camera detects the user's facial expressions and tone of voice, and the data is analyzed using Google Cloud Vision API and DeepFace. Based on the analyzed emotional data, the server dynamically adjusts the display content. For example, if the user is emotional, it can display more detailed and emotional visuals.

[0873] The three-dimensional model and predicted future appearance generated in this way are displayed to the user via a tablet, computer display, VR headset, etc. The user can manipulate it to experience the growth and future appearance of the fetus in a realistic way.

[0874] As a concrete example, consider a case where a user in the 24th week of pregnancy uploads an ultrasound image and inputs genetic information into the system. In this case, the server preprocesses the ultrasound image and uses a generative AI model to generate a realistic 3D model of the fetus. Meanwhile, the device's camera detects the user's facial expression, and the emotion engine analyzes the user's emotions. Based on the results, the display content is dynamically adjusted. This allows the user to feel more deeply moved and at ease.

[0875] An example prompt is, "Upload an ultrasound image to generate a realistic 3D model. We'll also predict your future appearance based on your genetic information and use emotion recognition to tailor the interface."

[0876] In this way, the present invention utilizes advanced technology to provide the user with an interactive and emotional experience, maximizing comfort and enjoyment during pregnancy.

[0877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0878] Step 1:

[0879] A user uploads an echo image using a terminal. The echo image file is sent as input to the server, which receives the echo image and converts it into a standard format.

[0880] Step 2:

[0881] The server preprocesses the received echo images using OpenCV and scikit-image libraries to remove noise, adjust resolution, and enhance contrast. The output of the preprocessing is a high-quality image converted to a standard format.

[0882] Step 3:

[0883] The server uses the preprocessed ultrasound images to input them into a generative AI model (TensorFlow or PyTorch) to generate a 3D model of the fetus. This generative AI model is a pre-trained neural network that generates a highly accurate 3D model from the input images. The output is 3D model data of the fetus.

[0884] Step 4:

[0885] The user uploads a genetic information file using a terminal. The genetic information file contains information about future appearance, such as facial features, hair color, skin color, height, and body type. The server receives the genetic information.

[0886] Step 5:

[0887] The server uses BioPython and Pandas to analyze the input genetic information. The analysis results in future appearance prediction data, which includes details about the fetus's growth patterns and predicted appearance. The output is the future appearance prediction data.

[0888] Step 6:

[0889] The server generates a 3D model of the future appearance based on the analysis results. The generated 3D model reflects the fetus's growth pattern, allowing the user to visually confirm the future appearance. The output is the 3D model data of the future appearance.

[0890] Step 7:

[0891] The device's camera detects the user's facial expressions and tone of voice to obtain emotional data, which includes the user's current emotional state. The server receives this emotional data.

[0892] Step 8:

[0893] The server uses an emotion engine (such as Google Cloud Vision API or DeepFace) to analyze the acquired emotion data. The analysis recognizes the user's emotional state. The output is the user's emotional state data.

[0894] Step 9:

[0895] The server dynamically adjusts the display content based on the recognized emotional state data. For example, if the user is emotional, the server changes the display content to more detailed and emotional visuals. The adjusted display content is sent to the user's device and displayed.

[0896] Step 10:

[0897] Based on the adjusted display, users can view a 3D model of the fetus and its future appearance in real time on a tablet, computer screen, or VR headset, with the user's experience optimized according to their emotions.

[0898] Through the above steps, the present invention utilizes echo images and genetic information to provide users with an interactive and moving experience.

[0899] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0900] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0901] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0902] [Fourth embodiment]

[0903] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0904] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0905] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0906] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0907] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0908] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0909] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0910] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0911] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0912] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0913] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0914] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0915] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0916] This invention is a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information. Specifically, the following process is performed.

[0917] First, a user accesses the system's dedicated application or website using a smartphone or computer, takes or scans an echo image, and uploads it to the system.

[0918] The server accepts uploaded echo images. Since the images may vary in size and format, the server converts them to a standard format, adjusts the resolution, removes noise from the images, and adjusts the contrast to improve visibility.

[0919] Next, based on the preprocessed ultrasound images, the server uses a generative AI model to generate a 3D model of the fetus. The generative AI model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generated 3D model is temporarily stored.

[0920] The user then enters the fetus's genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives the genetic information and converts it into a standard format if necessary.

[0921] The server analyzes the genetic information to extract information about future appearance, such as facial features, hair color, skin color, height, and body type. Based on the analysis results, the server predicts growth patterns and generates a three-dimensional model of the future appearance, utilizing existing growth databases and statistical models.

[0922] Finally, the server renders the generated current and future 3D models to display them visually, and the rendered 3D models are sent to the device in an interactive format (e.g., WebGL), allowing users to view and manipulate these models on their smartphone or computer screen.

[0923] As a concrete example, suppose a user in the 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. At the same time, the server analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see a realistic image of the fetus now, as well as predicted future appearances at age 5 and 10.

[0924] This invention will help expectant mothers and their families understand the specific appearance and future shape of their fetus, allowing them to experience moving and meaningful moments, and will also help reduce anxiety during pregnancy.

[0925] The processing flow will be explained below.

[0926] Step 1:

[0927] A user uses a smartphone or computer to access the system's dedicated application or website.

[0928] Step 2:

[0929] The user takes or scans an echo image and uploads it to the system.

[0930] Step 3:

[0931] The server accepts the uploaded echo images.

[0932] Step 4:

[0933] The server converts the uploaded echo image format to a standard format (e.g., JPEG, PNG), and also performs noise removal, resolution adjustment, and contrast adjustment.

[0934] Step 5:

[0935] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[0936] Step 6:

[0937] The server stores the generated 3D model in temporary storage.

[0938] Step 7:

[0939] The user enters the fetus's genetic information into the system, which is typically provided in a data file by a specialized testing laboratory.

[0940] Step 8:

[0941] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[0942] Step 9:

[0943] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the genetic information.

[0944] Step 10:

[0945] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[0946] Step 11:

[0947] The server renders the generated current and future 3D models and generates the data for visual display. Rendering is done using technologies such as WebGL.

[0948] Step 12:

[0949] The rendered three-dimensional model data is sent to the terminal.

[0950] Step 13:

[0951] The user can view and manipulate a 3D model of the current fetus and a 3D model of its predicted future appearance on a smartphone or computer screen.

[0952] Step 14:

[0953] Users can share the 3D models they create with family and friends.

[0954] Example 1

[0955] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0956] Systems that can precisely reconstruct the fetus's appearance from ultrasound images and predict its future appearance based on genetic information require highly accurate image processing and analysis. Previous systems performed the preprocessing of ultrasound images, the generation of 3D models, and the analysis of genetic information separately, making it difficult to provide an integrated system. Furthermore, they lacked an interactive method that allowed users to intuitively manipulate the generated 3D models, leaving a need for an improved user experience.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0958] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a three-dimensional model of the fetus from the preprocessed echo image, means for inputting genetic information, means for analyzing the input genetic information and predicting future appearance, means for displaying the generated three-dimensional model and the predicted future appearance, means for rendering the generated three-dimensional model and transmitting it to a terminal in an interactive format, and means for the user to manipulate and confirm the three-dimensional model on the terminal. This integrates highly accurate image processing and analysis, allowing the user to intuitively manipulate and confirm the current and future appearance of the fetus.

[0959] An "echo image" is an image of the inside of a body captured using ultrasound.

[0960] "Preprocessing" refers to the process of converting the input echo image into a format that is easy for the system to analyze, such as by converting the format, removing noise, and adjusting the resolution.

[0961] A "standard format" is a unified data format that improves data compatibility and processing efficiency.

[0962] A "3D model" is a three-dimensional digital representation that can be observed from multiple perspectives.

[0963] A "generative model means" is an algorithm or software that generates new data or information based on input data.

[0964] "Genetic information" is data that records the sequence of DNA or RNA, and is information that determines the characteristics and functions of an individual.

[0965] "Analysis" is the process of examining data using statistical or mathematical methods to extract useful information.

[0966] "Forecasting" is the act of estimating future events or conditions based on current data and trends.

[0967] "Rendering" is the process of displaying three-dimensional models or image data on a flat display.

[0968] An "interactive format" is a two-way display format that changes in real time in response to user operations and inputs.

[0969] A "terminal" is a digital device used by a user to access the system, examples of which include a smartphone or computer.

[0970] "User" refers to the end user who operates the system to input information and check the results.

[0971] This invention is a system that reconstructs a fetus in detail from an ultrasound image and predicts its future appearance by analyzing its genetic information. This system is specifically implemented based on the following procedure and configuration.

[0972] First, the user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system. The echo image can be input using the smartphone's camera or scanner.

[0973] The server receives the uploaded echo images, checks the image format, converts them to a standard format, adjusts the image resolution, removes noise, and adjusts the contrast to improve visibility. This process is often performed using image processing software.

[0974] After preprocessing, the ultrasound images are input to a server running a generative AI model. This model uses a multi-layer neural network trained on past ultrasound images and data from actual births. The generative AI model generates a three-dimensional model of the fetus and temporarily stores it.

[0975] The user then enters their genetic information into the system, often in the form of a data file provided by a specialized testing laboratory. The server receives the information, converts it into a standard format if necessary, and analyzes it to extract information about future appearance, such as facial features, hair color, skin tone, and height.

[0976] The server predicts growth patterns based on the genetic analysis, leverages existing growth databases and statistical models, and generates a three-dimensional model of the future, often using growth prediction algorithms.

[0977] Finally, the server renders the generated current and future 3D models. This rendering is done in an interactive format such as WebGL. The rendered 3D models are sent to the device, where users can view and manipulate these models on their smartphone or computer screen. Specifically, users can rotate the 3D models, zoom in and out, and view them from different perspectives.

[0978] As a concrete example, consider a case where a user in the 20th week of pregnancy takes an ultrasound image using their smartphone and uploads it to the system, while also inputting genetic information obtained from a blood test. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It then analyzes the genetic information and performs a growth simulation to predict future appearance. As a result, the user can see in detail the realistic appearance of the fetus now, as well as predicted appearances at age 5 and 10.

[0979] By making good use of the features of this invention, expectant mothers and their families will be able to better understand the specific appearance and future shape of their fetus, allowing them to share moving and meaningful moments together, and it is also expected to contribute to reducing anxiety during pregnancy.

[0980] Example prompt sentence:

[0981] "You provide the system with ultrasound images and genetic information to predict what the fetus looks like now and what it will look like in the future."

[0982] "Based on ultrasound images of a fetus at 20 weeks of pregnancy and genetic information, please make a realistic prediction of what the baby will look like at age 5 and 10."

[0983] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0984] Step 1:

[0985] A user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads the echo image file to the system, where the echo image as input is sent to the server as output.

[0986] Step 2:

[0987] The server receives the uploaded echo image and first checks the image format. Specifically, the image is converted to a standard format such as JPEG or PNG. Then, resolution adjustments, noise reduction filters, and contrast adjustments are applied to improve visibility. The input echo image is output as a preprocessed image converted to a standard format.

[0988] Step 3:

[0989] The server inputs the preprocessed echo images into a generative AI model. The generative AI model uses a multi-layer neural network to generate a 3D model of the fetus from the echo images. The preprocessed echo images as input are saved as the generated 3D model as output. Specific operations include training and inference of the AI ​​model.

[0990] Step 4:

[0991] The user prepares a genetic information data file provided by a specialized testing institution and proceeds to the genetic information input screen in a dedicated application or website. Next, the user selects the genetic information file and clicks the "Upload" button. The genetic information as input is sent to the server.

[0992] Step 5:

[0993] The server receives the uploaded genetic information, checks the format, converts it to a standard format if necessary, and analyzes the genetic information to extract information about appearance, such as facial features, hair color, skin color, and height. Specifically, an algorithm is used to analyze useful feature information from the genetic data. The input genetic information is output as analyzed feature information.

[0994] Step 6:

[0995] The server predicts growth patterns based on the results of genetic information analysis. It uses existing growth databases and statistical models to simulate fetal growth. Feature information and growth data are input, and predicted growth patterns and a future three-dimensional model are obtained as output. Specific operations include statistical analysis and simulation work.

[0996] Step 7:

[0997] The server renders the generated current and future 3D models. In this process, rendering is performed in an interactive format such as WebGL. The input 3D model is output as a rendered interactive format. Specific operations involve the use of graphics rendering algorithms.

[0998] Step 8:

[0999] The server sends the rendered 3D model to the terminal. The user opens the system's dedicated application or website on the terminal and views and manipulates the 3D model. The input interactive model is output as a manipulable display on the terminal. Specific actions the user can perform include rotating, zooming, and translating the model.

[1000] (Application example 1)

[1001] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1002] Conventional ultrasound image analysis systems can predict the current and future appearance of a fetus, but they cannot provide users with virtual simulations of its future appearance. This means that users cannot enjoy virtual experiences such as trying on products based on their future appearance. Furthermore, there are limitations to the accuracy of processing and real-time display in the process of analyzing ultrasound images and genetic information.

[1003] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1004] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for generating a 3D model of a fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for displaying the generated 3D model and the predicted future appearance, and means for inputting and preprocessing a user's facial image and virtually simulating a product based on the user's future appearance, thereby enabling the user to experience a virtual simulation based on the user's future appearance.

[1005] An "echo image" is an image that uses ultrasound to visualize the internal structures of a fetus.

[1006] "Preprocessing" refers to the process of converting the input image into a standard format and performing any necessary corrections or noise removal.

[1007] A "standard format" is a standardized image or data format that can be recognized by systems.

[1008] A "generative model" is an artificial intelligence model that generates three-dimensional models based on echo images and genetic information.

[1009] "Genetic information" means information, including the DNA data of a fetus, that indicates the unique biological characteristics of that individual.

[1010] A "three-dimensional model" is a digital model that realistically reproduces the fetus and its future appearance in three-dimensional space.

[1011] "Display means" refers to a device or software for visually presenting the generated three-dimensional model and prediction results to the user.

[1012] A "face image" is photographic data of a user's face.

[1013] "Virtual simulation" refers to the experience of virtually trying on products, clothing, cosmetics, etc. based on the user's future appearance.

[1014] This invention is a system that generates a three-dimensional model of the fetus's current and future appearance based on ultrasound images and genetic information, and also provides a virtual simulation based on the fetus's future appearance. A specific implementation method of this system is described below.

[1015] Users access the system using a device such as a smartphone, tablet, or computer, and take or scan an echo image using a camera or scanner installed in an autonomous vehicle or logistics center. Once the image is taken or scanned, it can be uploaded to the system via the device.

[1016] The server accepts the uploaded ultrasound images and first performs preprocessing. If the images vary in size or format, they are converted into a standardized format, and the resolution is adjusted and noise is removed. The preprocessed images are then input into a generative AI model, which generates a 3D model of the fetus. This generative AI model uses a multi-layer neural network trained on past ultrasound images and actual post-birth data.

[1017] The user then enters their genetic information into the system, typically in the form of a data file provided by a specialized testing laboratory. The server receives and analyzes this information, extracting information about the individual's future appearance, including facial features, hair color, skin tone, height, and body type. Using this information, the server predicts growth patterns and generates a three-dimensional model of the individual's future appearance.

[1018] The generated 3D model is then rendered for visual display and sent to the device in an interactive format using technologies such as WebGL, allowing users to view and manipulate the 3D model on their smartphone, tablet, or computer screen.

[1019] Furthermore, the system also has the ability to input and preprocess a user's facial image and virtually simulate products based on their future appearance, allowing users to virtually try on and try on products such as clothing and cosmetics based on their future appearance.

[1020] As a concrete example, consider the case where a user uploads a photo of their face and simulates their hairstyle and makeup from their younger days. In this case, the server uses a generative AI model based on the photo to predict their future appearance and perform a virtual simulation. Examples of prompts include:

[1021] "Upload a photo of yourself from 20 years ago and compare it to how you look now."

[1022] "Enter your DNA information and do a virtual try-on that predicts your future appearance."

[1023] As described above, the present invention enables the generation of three-dimensional models and virtual simulations using echo images and genetic information, providing users with a new experience.

[1024] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1025] Step 1: Upload your ultrasound images

[1026] Users access the system using a smartphone, tablet, or computer to take or scan echo images, and then upload the images to a dedicated application or website. The input is the echo image data, and the output is an image file sent to the server.

[1027] Step 2: Preprocessing of echo images

[1028] The server receives the uploaded echo images and performs preprocessing, specifically converting the image size and format to a standard format, adjusting the resolution, and removing noise. At this stage, the input is echo image data, and the output is preprocessed echo images in a unified format.

[1029] Step 3: Generate a 3D model

[1030] The server inputs the preprocessed ultrasound images into a generative AI model to generate a 3D model of the fetus. This generative AI model is a multi-layer neural network trained on past ultrasound images and actual post-birth data. The input is the preprocessed ultrasound images, and the output is a 3D digital model of the fetus.

[1031] Step 4: Enter genetic information

[1032] Users input genetic information provided by specialized testing laboratories into the system. The input is a data file containing DNA information, and the output is genetic data sent to the server.

[1033] Step 5: Analyzing genetic information

[1034] The server analyzes the input genetic information and extracts information about future appearance, such as facial features, hair color, skin color, height, and body type. This analysis uses a database containing features correlated with various genetic data. The input is genetic data, and the output is appearance feature data derived from the genetic information.

[1035] Step 6: Generate the future 3D model

[1036] The server predicts growth patterns based on the analysis results and generates a 3D model of the future appearance, utilizing existing growth databases and statistical models. The input is appearance feature data and growth prediction data, and the output is a 3D digital model of the future appearance.

[1037] Step 7: Rendering the 3D model

[1038] The server performs rendering to visually display the generated current and future 3D models. Using technologies such as WebGL, it transmits them to the device in an interactive format. The input is the 3D digital model, and the output is the rendered interactive 3D display data.

[1039] Step 8: Conduct a virtual simulation

[1040] The user inputs an image of their face into the system and performs preprocessing. Based on this image, the server uses a generative AI model to predict future appearance and runs a virtual simulation. The input is facial image data and a prediction model, and the output is a virtually tried-on product image. Examples of prompts include "Upload a photo of your face from 20 years ago and compare it with your current self," and "Enter your DNA information to virtually try on products with predicted future appearances."

[1041] Through these steps, the system can provide users with a new experience.

[1042] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1043] This invention combines a system that realistically recreates a fetus from an ultrasound image and predicts its future appearance by analyzing its genetic information with an emotion engine that recognizes the user's emotions. Specifically, it performs the following processes:

[1044] First, a user accesses the system's dedicated application or website using a smartphone or computer, then takes or scans an echo image and uploads it to the system.

[1045] The server receives the uploaded ultrasound images and performs preprocessing such as converting the images to a standard format and removing noise, adjusting resolution, and enhancing contrast. Once preprocessing is complete, the server uses a generative AI model to generate a 3D model of the fetus from the ultrasound images. The generated 3D model is temporarily stored in storage.

[1046] Next, the user enters the fetus's genetic information into the system. This information is typically provided in the form of a data file from a specialized testing laboratory. The server receives the genetic information, converts it into a standard format if necessary, and then analyzes it. The analysis results include future appearance, such as facial features, hair color, skin tone, height, and body type. Based on this analysis, the server predicts growth patterns and generates a three-dimensional model of the fetus's future appearance.

[1047] One of the features of the present invention is the incorporation of an emotion engine. While the user is using the system, the terminal is equipped with a camera that detects the user's facial expressions and tone of voice. The server uses the emotion engine to analyze this data and recognize the user's emotions. Based on the recognized emotion information, the server can adjust the displayed 3D model, system interface, and output content. For example, if the user is feeling surprised or moved, it can display more detailed and emotional visuals.

[1048] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image and inputs genetic information obtained from a blood test into the system. The server preprocesses the ultrasound image and generates a realistic 3D model using a generative AI model. It also analyzes the genetic information and predicts the future appearance. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotions, to the device, where the user can view and operate it on their smartphone or computer screen.

[1049] This invention not only helps users understand the specific appearance and future shape of the fetus, but also allows the system to adjust the visuals and interface according to the user's emotions, providing a deeper sense of emotion and security, and contributing to reducing anxiety during pregnancy.

[1050] The processing flow will be explained below.

[1051] Step 1:

[1052] A user uses a smartphone or computer to access the system's dedicated application or website.

[1053] Step 2:

[1054] The user takes or scans an echo image and uploads it to the system.

[1055] Step 3:

[1056] The server accepts the uploaded echo images.

[1057] Step 4:

[1058] The server converts the uploaded echo images into standard formats (e.g., JPEG, PNG), and also performs noise reduction, resolution adjustment, and contrast enhancement.

[1059] Step 5:

[1060] The server inputs the preprocessed ultrasound images into a generative AI model, which generates a 3D fetal model. The generative AI model is a multi-layer neural network trained on historical ultrasound images and actual postnatal photographs.

[1061] Step 6:

[1062] The server stores the generated 3D model in temporary storage.

[1063] Step 7:

[1064] The user inputs the fetus's genetic information into the system, which is usually a data file provided by a specialized testing institution.

[1065] Step 8:

[1066] The server accepts the input genetic information and, if necessary, converts this information into a standard format (e.g., CSV, JSON).

[1067] Step 9:

[1068] The server analyzes the genetic information and predicts future appearance, including facial features, hair color, skin color, height, and body type, based on the analyzed genetic information.

[1069] Step 10:

[1070] The server uses the analysis results to predict growth patterns and generate a 3D model of the future shape, using growth databases and statistical models.

[1071] Step 11:

[1072] The camera installed on the device detects the user's facial expressions and tone of voice in real time.

[1073] Step 12:

[1074] The server uses an emotion engine to analyze the user's facial expression and voice data sent from the terminal and recognize the user's emotional state.

[1075] Step 13:

[1076] The server adjusts the content of the displayed 3D model and the system interface based on the recognized emotional state. For example, if the user expresses surprise, it displays more detailed and moving visuals.

[1077] Step 14:

[1078] The server renders the generated current and future 3D models and generates the data for visual display, using technologies such as WebGL.

[1079] Step 15:

[1080] The rendered three-dimensional model data is sent to the terminal.

[1081] Step 16:

[1082] The user can view and manipulate a three-dimensional model of the current fetus and a three-dimensional model of its predicted future appearance on a smartphone or computer screen.

[1083] Step 17:

[1084] Users can share the 3D models they create with family and friends.

[1085] In this way, the system generates a three-dimensional model of the fetus based on ultrasound images and genetic information, and then uses an emotion engine to adjust the interface based on the user's emotions, creating a more moving and reassuring experience.

[1086] Example 2

[1087] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1088] In conventional echo image analysis and genetic information analysis systems, preprocessing and noise removal of echo images, and conversion and analysis of genetic information are performed separately, making comprehensive use difficult. Furthermore, the generated 3D models and predicted future appearances are fixed, and dynamic adjustments based on the user's emotions are not possible. As a result, user satisfaction is low and the utility of the system is limited. The purpose of this invention is to solve these problems.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1090] In this invention, the server includes means for inputting an echo image, means for preprocessing the input echo image and converting it into a standard format, means for removing noise from the preprocessed echo image, means for generating a 3D model of the fetus from the preprocessed echo image, means for inputting genetic information, means for predicting future appearance by analyzing the input genetic information, means for recognizing a user's emotion, and means for adjusting and displaying the generated 3D model and the predicted future appearance based on the recognized emotion information, thereby enabling the generation and display of a dynamic and detailed 3D model in accordance with the user's emotion.

[1091] An "echo image" is an image of internal structures, particularly the fetus, taken using ultrasound technology.

[1092] "Preprocessing" refers to processing of the input echo image, such as conversion to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[1093] A "standard format" is an image file format that is specified so that the system can process it appropriately.

[1094] "Noise reduction" is a process that improves image quality by removing unwanted noise and distortion from an image.

[1095] "Generative model means" includes methods and algorithms for generating a three-dimensional model of the fetus from pre-processed echo images.

[1096] "Genetic information" is data describing the genetic characteristics of an individual and is provided by specialized institutions.

[1097] "Genetic information analysis" is a process for predicting future appearance and growth patterns based on input genetic information.

[1098] "Predicting future appearance" means predicting future appearance, including facial features, hair color, skin color, height, and body type, based on the results of analyzing genetic information.

[1099] "User emotion recognition" refers to sensing the user's facial expressions and tone of voice using the device's built-in camera and microphone, and analyzing them to understand the user's emotional state.

[1100] "Adjusted display based on emotion information" refers to dynamically changing the display content of the generated three-dimensional model or interface based on the user's recognized emotion information.

[1101] The present invention combines a system that generates a 3D model of a fetus from ultrasound images, analyzes genetic information to predict its future appearance, and recognizes the user's emotions and adjusts the display content accordingly. To implement this system, the following specific hardware and software are used:

[1102] The user accesses a dedicated application or website using a smartphone or computer. It is recommended that the hardware used has a camera and microphone. The following describes the detailed process at each step and the software used.

[1103] The server receives the echo images uploaded by the user. These images are pre-processed using an image processing library (e.g., Python's Pillow library). Pre-processing includes converting to a standard format, noise removal, resolution adjustment, and contrast enhancement.

[1104] After preprocessing, the ultrasound images are used to generate a 3D model of the fetus using a generative AI model (e.g., GANs using TensorFlow), which is then temporarily stored in a database.

[1105] The user then inputs their genetic information. This information is typically provided by a specialized testing institution in the form of a data file in text format. The server receives this genetic information and analyzes it. This analysis involves converting the genetic information into a standard format and extracting specified characteristics (facial features, hair color, skin color, height, body type, etc.). This analysis is performed using the Python Pandas library.

[1106] Based on the analysis results, growth patterns are predicted and a three-dimensional model of the future shape is generated, which is performed using machine learning algorithms (e.g., random forests).

[1107] While the user is using the system, the device's camera and microphone detect the user's facial expressions and tone of voice. Emotion analysis is performed using computer vision libraries (e.g., OpenCV) and facial expression recognition libraries (e.g., DeepFace).

[1108] The server analyzes this data using an emotion engine to recognize the user's emotions. Based on the recognized emotion information, the generated 3D model and system interface are adjusted. Specifically, if the user is feeling surprised or moved, the system increases the level of visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[1109] As a concrete example, suppose a user in her 20th week of pregnancy uploads an ultrasound image to the system and inputs genetic information obtained from a specialized testing institution. The ultrasound image is preprocessed, and a realistic 3D model is generated using a generative AI model. At the same time, the genetic information is analyzed and the future appearance is predicted. During this process, the device's camera detects the user's facial expressions, and the emotion engine analyzes the user's emotions. Finally, the server sends the 3D model, adjusted based on the emotion, to the device, where the user can view and manipulate it on their smartphone or computer screen.

[1110] Example prompt sentence:

[1111] "I have uploaded an ultrasound image from the 20th week of pregnancy and entered genetic information. Please generate a growth prediction model from this. I would also like you to collect facial expression data from the user in real time and output the data according to their emotions."

[1112] This makes it easier for users to understand the specific appearance and future shape of the fetus, and the system adjusts the visuals and interface according to the user's emotions, allowing them to feel a deeper sense of emotion and security.

[1113] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1114] System program processing flow

[1115] Step 1: User uploads an echo image

[1116] Users use a smartphone or computer to access a dedicated application or website to take or scan and upload echo images, and then use the camera or file selection function of the specific device to select and send the image files to the system.

[1117] Input: Echo image file

[1118] Output: Echo image file sent to the server

[1119] Step 2: The server preprocesses the images

[1120] The server preprocesses the received echo images in the following steps:

[1121] Conversion to standard formats: Converting different image formats (e.g. JPEG to PNG) into a format that the system can properly handle.

[1122] Noise reduction: Filters out and removes unwanted noise from the image.

[1123] Resolution Adjustment: Adjust the image to the optimal resolution.

[1124] Contrast Enhancement: Increases the contrast of the image to make it easier to see.

[1125] Input: Echo image file sent by the user

[1126] Output: Preprocessed image files in standard format

[1127] Step 3: The server generates the 3D model

[1128] Based on the preprocessed ultrasound images, the server uses a generative AI model (e.g., GANs using TensorFlow) to generate a 3D model of the fetus. This generated model is temporarily stored in a database. Specifically, features are extracted from the image using a convolutional neural network (CNN), and a 3D model is generated using GANs based on those features.

[1129] Input: Preprocessed image files in standard format

[1130] Output: Generated 3D model of the fetus

[1131] Step 4: User enters genetic information

[1132] Users input genetic information data files provided by professional testing institutions into the system, and use the application's "genetic information upload" function to select and send the data files to the system.

[1133] Input: Gene information file

[1134] Output: Gene information file sent to the server

[1135] Step 5: The server analyzes the genetic information and predicts the future appearance

[1136] The server converts the received genetic information into a standard format and analyzes the information. The analysis extracts facial features, hair color, skin color, height, body type, etc. from the input genetic information, and predicts future appearance based on growth patterns. A machine learning algorithm (e.g., random forest) is used for the prediction.

[1137] Input: Gene information file

[1138] Output: Analyzed genetic information and a predicted 3D model of your future appearance

[1139] Step 6: The device recognizes the user's emotions

[1140] While the user is using the system, the device's camera captures the user's facial expressions and tone of voice and transmits them to the server in real time. The server then uses an emotion engine (e.g., OpenCV and DeepFace) to analyze this data and recognize the user's emotions. For example, the camera captures the user's smiling or surprised expressions and uses them to infer the user's emotions.

[1141] Input: User's facial and voice data

[1142] Output: Parsed emotion information

[1143] Step 7: The server adjusts and displays the 3D model and interface.

[1144] Based on the recognized emotion information, the server adjusts the displayed 3D model and system interface. For example, if the user is surprised or impressed, the server increases the visual detail. This adjusted data is delivered to the device in real time via WebSocket.

[1145] Input: Analyzed emotion information, generated 3D model, and predicted future appearance

[1146] Output: emotion-adjusted 3D model and system interface

[1147] This allows users to view and manipulate dynamically adjusted 3D models and interfaces in real time.

[1148] (Application example 2)

[1149] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1150] Conventional systems for generating 3D fetal models based on ultrasound images tend to display static images without considering the user's emotions. This makes it difficult for users to feel interactivity with the system, preventing the quality of the experience from being maximized. Furthermore, predictions of the fetus's future appearance can sometimes lack persuasiveness. The objective of this invention is not only to provide a 3D fetus model and a prediction of its future appearance from ultrasound images, but also to dynamically adjust the display content using the user's emotional information, providing an interactive and moving experience.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1152] In this invention, the server includes means for preprocessing the ultrasound images and converting them into a standard format, means for generating a 3D model of the fetus from the preprocessed ultrasound images, means for analyzing input genetic information to predict future appearance, means for inputting emotional data, means for analyzing the input emotional data to recognize the user's emotion, and means for dynamically adjusting the display content in accordance with the user's emotion, thereby enabling the provision of an interactive and moving 3D model display of the fetus and a predicted future appearance that are tailored to the user's emotional state.

[1153] An "echo image" is a medical image taken using ultrasound to capture the inside of the body, and is primarily used to check the condition of a fetus.

[1154] "Preprocessing" is the process of converting input image data into a standard format and performing processing such as noise removal, resolution adjustment, and contrast enhancement.

[1155] A "generative model" is an algorithm that uses artificial intelligence or machine learning techniques to generate a three-dimensional model based on specific input data.

[1156] "Genetic information" refers to the data contained in the genetic material of an organism, and is the information recorded in DNA or RNA.

[1157] "Analysis" is the process of examining and breaking down input data in detail to extract meaning and patterns.

[1158] "Future appearance" is the subject's future appearance or appearance predicted based on current data.

[1159] "Emotion data" is data that indicates the user's emotional state, including facial expressions and vocal tones.

[1160] An "emotion engine" is an algorithm or software that recognizes a user's emotional state, primarily by analyzing facial expressions and tone of voice.

[1161] "Dynamic adjustment" is the process of changing the display content and interface in real time based on the situation and data.

[1162] In the system realizing this invention, the server preprocesses the echo image, generates a 3D model of the fetus, analyzes genetic information, and further recognizes the user's emotions to dynamically adjust the display content. This system has the following configuration and processing means.

[1163] The server first receives the echo images uploaded by the user using a tablet or computer, and then preprocesses them using image processing libraries such as OpenCV and scikit-image, including converting them to a standard format, removing noise, adjusting the resolution, and enhancing the contrast.

[1164] Next, the preprocessed echo images are input into a generative AI model using TensorFlow and PyTorch to generate a 3D model of the fetus, which is then temporarily stored in storage.

[1165] The user then inputs genetic information provided by a specialized testing institution into the system. This genetic information is analyzed using libraries such as BioPython and Pandas to predict future appearance and growth patterns. The genetic analysis results include facial features, hair color, skin color, height, and body type, and a three-dimensional model of the future appearance is generated based on these results.

[1166] The system also uses an emotion engine to analyze the user's emotional data. The device's camera detects the user's facial expressions and tone of voice, and the data is analyzed using Google Cloud Vision API and DeepFace. Based on the analyzed emotional data, the server dynamically adjusts the display content. For example, if the user is emotional, it can display more detailed and emotional visuals.

[1167] The three-dimensional model and predicted future appearance generated in this way are displayed to the user via a tablet, computer display, VR headset, etc. The user can manipulate it to experience the growth and future appearance of the fetus in a realistic way.

[1168] As a concrete example, consider a case where a user in the 24th week of pregnancy uploads an ultrasound image and inputs genetic information into the system. In this case, the server preprocesses the ultrasound image and uses a generative AI model to generate a realistic 3D model of the fetus. Meanwhile, the device's camera detects the user's facial expression, and the emotion engine analyzes the user's emotions. Based on the results, the display content is dynamically adjusted. This allows the user to feel more deeply moved and at ease.

[1169] An example prompt is, "Upload an ultrasound image to generate a realistic 3D model. We'll also predict your future appearance based on your genetic information and use emotion recognition to tailor the interface."

[1170] In this way, the present invention utilizes advanced technology to provide the user with an interactive and emotional experience, maximizing comfort and enjoyment during pregnancy.

[1171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1172] Step 1:

[1173] A user uploads an echo image using a terminal. The echo image file is sent as input to the server, which receives the echo image and converts it into a standard format.

[1174] Step 2:

[1175] The server preprocesses the received echo images using OpenCV and scikit-image libraries to remove noise, adjust resolution, and enhance contrast. The output of the preprocessing is a high-quality image converted to a standard format.

[1176] Step 3:

[1177] The server uses the preprocessed ultrasound images to input them into a generative AI model (TensorFlow or PyTorch) to generate a 3D model of the fetus. This generative AI model is a pre-trained neural network that generates a highly accurate 3D model from the input images. The output is 3D model data of the fetus.

[1178] Step 4:

[1179] The user uploads a genetic information file using a terminal. The genetic information file contains information about future appearance, such as facial features, hair color, skin color, height, and body type. The server receives the genetic information.

[1180] Step 5:

[1181] The server uses BioPython and Pandas to analyze the input genetic information. The analysis results in future appearance prediction data, which includes details about the fetus's growth patterns and predicted appearance. The output is the future appearance prediction data.

[1182] Step 6:

[1183] The server generates a 3D model of the future appearance based on the analysis results. The generated 3D model reflects the fetus's growth pattern, allowing the user to visually confirm the future appearance. The output is the 3D model data of the future appearance.

[1184] Step 7:

[1185] The device's camera detects the user's facial expressions and tone of voice to obtain emotional data, which includes the user's current emotional state. The server receives this emotional data.

[1186] Step 8:

[1187] The server uses an emotion engine (such as Google Cloud Vision API or DeepFace) to analyze the acquired emotion data. The analysis recognizes the user's emotional state. The output is the user's emotional state data.

[1188] Step 9:

[1189] The server dynamically adjusts the display content based on the recognized emotional state data. For example, if the user is emotional, the server changes the display content to more detailed and emotional visuals. The adjusted display content is sent to the user's device and displayed.

[1190] Step 10:

[1191] Based on the adjusted display, users can view a 3D model of the fetus and its future appearance in real time on a tablet, computer screen, or VR headset, with the user's experience optimized according to their emotions.

[1192] Through the above steps, the present invention utilizes echo images and genetic information to provide users with an interactive and moving experience.

[1193] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1194] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1195] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1196] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1197] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1198] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1199] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1200] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1201] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1202] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1203] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1204] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1205] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1206] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1207] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1208] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1209] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1210] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1211] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1212] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1213] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1214] The following is further disclosed regarding the above embodiment.

[1215] (Claim 1)

[1216] a means for inputting an echo image;

[1217] means for preprocessing the input echo image and converting it into a standard format;

[1218] a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image;

[1219] a means for inputting genetic information;

[1220] A method for predicting future appearance by analyzing input genetic information, and

[1221] The system includes a means for displaying the generated three-dimensional model and the predicted future appearance.

[1222] (Claim 2)

[1223] 10. The system of claim 1, further comprising means for denoising the preprocessed echo image.

[1224] (Claim 3)

[1225] The system of claim 1, further comprising means for predicting a growth pattern based on the results of an analysis of genetic information.

[1226] "Example 1"

[1227] (Claim 1)

[1228] a means for inputting an echo image;

[1229] means for preprocessing the input echo image and converting it into a standard format;

[1230] a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image;

[1231] a means for inputting genetic information;

[1232] A method for predicting future appearance by analyzing input genetic information, and

[1233] means for displaying the generated three-dimensional model and the predicted future appearance;

[1234] means for rendering the generated three-dimensional model and transmitting it to a terminal in an interactive format;

[1235] A means for users to manipulate and check 3D models on their devices

[1236] Including system.

[1237] (Claim 2)

[1238] 10. The system of claim 1, further comprising means for denoising the preprocessed echo image.

[1239] (Claim 3)

[1240] The system of claim 1, further comprising means for predicting a growth pattern based on the results of an analysis of genetic information.

[1241] "Application Example 1"

[1242] (Claim 1)

[1243] a means for inputting an echo image;

[1244] means for preprocessing the input echo image and converting it into a standard format;

[1245] a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image;

[1246] a means for inputting genetic information;

[1247] A method for predicting future appearance by analyzing input genetic information, and

[1248] means for displaying the generated three-dimensional model and the predicted future appearance;

[1249] The system includes a means for inputting and preprocessing a user's facial image and virtually simulating a product based on its future appearance.

[1250] (Claim 2)

[1251] 10. The system of claim 1, further comprising means for denoising the preprocessed echo image.

[1252] (Claim 3)

[1253] The system of claim 1, further comprising means for predicting a growth pattern based on the results of an analysis of genetic information.

[1254] "Example 2: Combining Emotion Engines"

[1255] (Claim 1)

[1256] a means for inputting an echo image;

[1257] means for preprocessing the input echo image and converting it into a standard format;

[1258] means for denoising the preprocessed echo images;

[1259] a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image;

[1260] a means for inputting genetic information;

[1261] A method for predicting future appearance by analyzing input genetic information, and

[1262] means for recognizing a user's emotion;

[1263] The system includes a means for adjusting and displaying the generated three-dimensional model and predicted future appearance based on the recognized emotion information.

[1264] (Claim 2)

[1265] 10. The system of claim 1, further comprising means for converting the input genetic information into a standard format as needed.

[1266] (Claim 3)

[1267] The system of claim 1, further comprising means for predicting a growth pattern based on the results of an analysis of genetic information.

[1268] (Claim 4)

[1269] 10. The system of claim 1, further comprising means for temporarily storing the generated three-dimensional model.

[1270] "Application example 2 when combining emotion engines"

[1271] (Claim 1)

[1272] a means for inputting an echo image;

[1273] means for preprocessing the input echo image and converting it into a standard format;

[1274] a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image;

[1275] a means for inputting genetic information;

[1276] A method for predicting future appearance by analyzing input genetic information, and

[1277] means for displaying the generated three-dimensional model and the predicted future appearance;

[1278] a means for inputting emotion data;

[1279] means for analyzing input emotion data to recognize the emotion of a user;

[1280] A system including means for dynamically adjusting display content in response to a user's emotions.

[1281] (Claim 2)

[1282] 10. The system of claim 1, further comprising means for denoising the preprocessed echo image.

[1283] (Claim 3)

[1284] The system of claim 1, further comprising means for predicting a growth pattern based on the results of an analysis of genetic information. [Explanation of symbols]

[1285] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting an echo image; means for preprocessing the input echo image and converting it into a standard format; a generative modeling means for generating a three-dimensional model of the fetus from the preprocessed echo image; a means for inputting genetic information; A method for predicting future appearance by analyzing input genetic information, and The system includes a means for displaying the generated three-dimensional model and the predicted future appearance.

2. The system of claim 1 further comprising means for denoising the preprocessed echo image.

3. The system according to claim 1, further comprising means for predicting a growth pattern based on the analysis of genetic information.

Images