System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an echo image; means for extracting feature data from the echo image; means for generating voice data based on the feature data; means for collecting user's living environment information, preferences, and needs; means for analyzing the user information; means for suggesting an optimal lifestyle and childcare method based on the analysis result; and means for providing the voice data and the suggestion to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, parents often feel anxious about their future lifestyle and parent-child relationships before their baby is born. Furthermore, there is an abundance of information available about child-rearing, making it difficult to find an appropriate approach. As a result, parents may find it difficult to gain confidence in their child-rearing and may feel stressed and confused in the early stages of child-rearing. To address these issues, parents need a way to foresee their future lifestyle and obtain appropriate countermeasures before their baby is born. [Means for solving the problem]

[0005] The present invention relates to a generative AI system that generates a baby's voice from ultrasound images and suggests optimal lifestyle choices. The system includes a means for receiving ultrasound images and a means for extracting feature data from the ultrasound images. It also includes a means for generating voice data based on the feature data, a means for collecting information about the user's living environment, preferences, and needs, a means for analyzing the user information, a means for suggesting optimal lifestyle and child-rearing methods based on the analysis results, and a means for providing the voice data and suggestions to the user. Furthermore, by applying a machine learning model to generate the voice data and using a questionnaire to collect user information, personalized suggestions can be made. This allows parents to get a more concrete sense of their future lives and parent with confidence using appropriate approaches.

[0006] Below are definitions of important terms included in the claims.

[0007] An "echo image" is an image of a fetus taken using ultrasound.

[0008] "Means for receiving" refers to the hardware and software components for capturing echo images into the system.

[0009] "Feature data" refers to information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0010] The "means for extracting" refers to an algorithm for extracting feature data from an echo image and a device for executing the algorithm.

[0011] "Audio data" is data representing the baby's voice generated based on an echo image.

[0012] The "means for generating" refers to the algorithms and system components that create audio data based on feature data.

[0013] "Living environment information" refers to information such as the user's residential area, household composition, and daily life patterns.

[0014] "Preferences" are information about a user's tastes and preferences.

[0015] "Needs" are information about the needs and demands felt by users.

[0016] "Means for collection" are the interfaces and processes that capture users' living environment information, preferences, and needs.

[0017] "Means for analyzing" are the algorithms and system components that analyze collected user information and extract useful data.

[0018] "Means for making suggestions" refers to the means of specifically showing users the optimal lifestyle and child-rearing methods based on the analysis results.

[0019] The "means for providing" refers to the interface and process for transmitting the generated voice data and suggestions to the user.

[0020] A "machine learning model" is an algorithm that learns patterns from data and makes future predictions and classifications.

[0021] A "survey format" is a method of collecting information from users by asking them to answer questions.

[0022] "Personalization" is the process of optimizing content and services based on a user's individual characteristics and needs. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The system of the present invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. This system involves a series of processes, starting with the user uploading the ultrasound image and ending with providing the generated baby's voice and lifestyle suggestions.

[0045] Program processing

[0046] 1. Uploading ultrasound images

[0047] User: First, the user accesses the system's website or application and uploads their own echo images, which are then sent to the server.

[0048] Terminal: Provides an interface for sending user-selected echo images to the server.

[0049] 2. Analysis of Echo Images

[0050] Server: Upon receiving the echo images, the server applies image pre-processing algorithms to remove noise and adjust contrast, making the echo images ready for analysis. Then, it uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images.

[0051] 3. Baby voice generation

[0052] Server: Runs a voice generation model based on the feature data. This model has learned from past baby voice data and generates voice data based on the given feature data. The generated voice data is saved as an audio file and linked to the user's account.

[0053] 4. Enter user data

[0054] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0055] Terminal: Provides an interface for sending user-entered information to the server.

[0056] 5. User data analysis

[0057] Server: Runs machine learning models to analyze the received user data in detail. The models organize the user input data into categories and scores to gain a detailed understanding of the user's attributes.

[0058] 6. Proposal for the best lifestyle

[0059] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. For example, for users living in urban areas, the server suggests nearby parent-child support facilities and recommended activities. It also suggests lifestyle schedules that match the baby's sleep rhythm.

[0060] Server: Notifies the user of the generated suggestions and provides audio files and lifestyle suggestions.

[0061] Terminal: An interface is provided to display the received proposals to the user, allowing the user to view the proposals in detail.

[0062] Specific examples

[0063] For example, suppose User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information. The server analyzes this information and suggests the use of childcare support services. The server also generates baby voice data and provides it to User A, allowing them to feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence.

[0064] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and provides personalized lifestyle suggestions that meet the needs of each user, providing an environment in which parents can approach child-rearing with confidence.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] User: The user visits the Echo Voice AI website or app, selects their Echo image, and clicks the upload button. The user also enters basic information such as name and email address.

[0068] Step 2:

[0069] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0070] Step 3:

[0071] Terminal: Sends pre-processed echo image data to the server. Data security is ensured by using encryption protocols during transmission.

[0072] Step 4:

[0073] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust the contrast, thereby improving the image quality.

[0074] Step 5:

[0075] Server: Runs machine learning models to extract feature data from preprocessed ultrasound images. Specifically, it detects important features such as the shape, position, and size of the fetus and records them as numerical data.

[0076] Step 6:

[0077] Server: Runs a speech generation model based on the feature data. Based on past baby voice data, generates a voice that is closest to the current feature data. Saves this voice data as an audio file (e.g., MP3).

[0078] Step 7:

[0079] Server: The generated audio file is associated with the user's account, a download link is sent to the user, and the audio file is stored in a secure storage.

[0080] Step 8:

[0081] User: The user clicks on the link provided and downloads the generated baby voice audio file to check it.

[0082] Step 9:

[0083] Device: Provides an interface where users can complete a questionnaire about their living environment, preferences, and needs. For example, questions might include, "What is your family structure?" and "What is your approach to childcare?"

[0084] Step 10:

[0085] User: Answers a questionnaire about their living environment, preferences, and needs, and clicks the submit button.

[0086] Step 11:

[0087] Terminal: The survey data entered by the user is sent to the server. The data is encrypted and sent.

[0088] Step 12:

[0089] Server: Runs a machine learning model to analyze the received survey data. The model classifies user responses into categories and scores, and analyzes lifestyle and parenting characteristics.

[0090] Step 13:

[0091] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's needs. The proposals are generated in the form of text, lists, graphs, etc.

[0092] Step 14:

[0093] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0094] Step 15:

[0095] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0096] Step 16:

[0097] User: Check the suggestions, put them into practice with specific lifestyle and child-rearing methods, and provide feedback as needed.

[0098] Step 17:

[0099] Terminal: Sends the feedback entered by the user to the server. The feedback will be used to improve the service in the future.

[0100] Step 18:

[0101] Server: Receives feedback data and stores it in a database. The stored data is fed into the machine learning model and used to improve the accuracy of future suggestions.

[0102] These are the specific processing steps of the Echo Voice AI system, which allows users to experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional ultrasound image analysis systems do not generate baby voices or provide personalized recommendations for optimal childcare lifestyles, making it difficult for users to obtain childcare advice tailored to their own living environment. Furthermore, conventional systems do not utilize machine learning models to generate voice data, which leads to issues with the realism of the generated voice data and low user satisfaction.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for receiving echo images, means for preprocessing the echo images, means for extracting feature data from the echo images, means for generating voice data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods, and means for providing the voice data and suggestions to the user, thereby enabling the user to realistically experience the baby's voice from the echo images and receive suggestions on a child-rearing lifestyle that is optimal for their own living environment.

[0108] "Echo image" refers to an image taken using ultrasound to check the condition of the fetus.

[0109] "Preprocessing" refers to the initial processing steps required to prepare the image for analysis, such as removing noise from the echo image and adjusting the contrast.

[0110] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0111] "Audio data" refers to digital audio data of a baby's voice created from feature data of an ultrasound image using a generative AI model.

[0112] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of baby voice data and generates realistic voices based on specific feature data.

[0113] "User's living environment information" refers to individual information provided by the user related to childcare, such as the user's living situation and residential environment.

[0114] "Survey format" refers to a method in which users respond to questions by selecting options or writing a statement.

[0115] "Lifestyle" and "parenting methods" refer to behavioral patterns and specific parenting advice and methods that are considered optimal for the user and their baby's lifestyle.

[0116] The system of the present invention involves a series of processes that begin with the user uploading an ultrasound image and ultimately provide a generated baby voice and optimal lifestyle recommendations. The specific hardware and software used in each process step are described in detail below.

[0117] First, the user accesses the system's website or application using a device such as a PC or smartphone. The user clicks the "Upload Echo Image" button on the screen, selects an echo image file from their device, and uploads it. The device provides an interface for reading the echo image file and sending it to the server. It is desirable that a progress bar be displayed to indicate the progress of the upload.

[0118] The server temporarily stores the received echo images. It then applies image pre-processing algorithms to remove noise and adjust the contrast. Specifically, it uses noise removal and contrast adjustment algorithms such as a median filter. It then uses a deep learning model (e.g., U-Net) to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0119] The server runs a speech generation model (e.g., WaveNet) based on the extracted feature data. This speech generation model has learned from past baby voice data and generates a realistic baby voice from the input data. The generated voice data is saved as an audio file (e.g., MP3 format) and linked to the user's account.

[0120] Next, the user answers a questionnaire about their living environment, preferences, and needs. The device provides a questionnaire-style user interface and transmits the information the user has entered to the server, displaying a progress bar while the data is being transmitted.

[0121] The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. This model categorizes the user data and assigns a score to each data point to gain a detailed understanding of the user's living environment and needs. For example, it can organize information such as "dual-income household" and "limited childcare time."

[0122] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, it generates specific suggestions such as "use local childcare support services available on weekends" or "create a daily schedule that matches the baby's sleep rhythm." The server then links the generated suggestions and audio files to the user's account and sends a notification. Users can check this information when they log in.

[0123] The terminal provides an interface that displays the suggestions and the generated baby voice to the user, who can then check the suggestions in detail and download or print them as needed.

[0124] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the server will analyze this information and suggest the use of childcare support services. Furthermore, by generating baby voice data and providing it to User A, User A will be able to feel the baby's presence even before it is born. Furthermore, the server will also suggest childcare methods and specific schedules suitable for dual-income households, allowing User A to make plans with confidence regarding childcare.

[0125] An example of a prompt would be the following text:

[0126] "We are a dual-income household, and the time we can devote to childcare is limited. Please suggest baby voice data and childcare methods that are suitable for our living environment."

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1:

[0129] Uploading ultrasound images

[0130] User: The user visits the system's website or application and clicks the "Upload Echo Image" button. As input, there is an echo image file selected by the user from the device. This file will be uploaded to the system.

[0131] Terminal: The terminal provides an interface to read the selected echo image file and send it to the server. Specifically, it sends the image data to the server using an HTTP request and displays a progress bar to indicate the upload progress to the user. As an output, the image data is sent to the server.

[0132] Step 2:

[0133] Echo image preprocessing

[0134] Server: The server temporarily stores the received echo images. The input is the uploaded echo image file. Then, image pre-processing algorithms are applied to remove noise and adjust contrast. Specifically, a median filter is used to remove noise, and histogram equalization is used to adjust contrast. The output is the processed echo image.

[0135] Step 3:

[0136] Extracting feature data from echo images

[0137] Server: The server extracts feature data from the preprocessed echo images using a deep learning model (e.g., U-Net). The input is the preprocessed echo images. The model extracts specific information from the echo images, such as the shape, size, and position of the fetus. The output is the fetal feature data.

[0138] Step 4:

[0139] Baby voice generation

[0140] Server: The server inputs the extracted feature data into a speech generation model (e.g., WaveNet). The input is the fetus's feature data. This speech generation model learns from past baby voice data to generate a realistic baby voice. The output is an MP3 audio file that is associated with the user's account.

[0141] Step 5:

[0142] Entering User Data

[0143] User: The user answers survey-style questions provided within the system. Input includes information about the user's living environment, preferences, and needs. For example, questions include "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0144] Terminal: The terminal provides a questionnaire-style user interface and sends the information entered by the user to the server. Specifically, an HTTP request is issued to collect form data and send it to the server.

[0145] Step 6:

[0146] User data analysis

[0147] Server: The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. The input is user data collected from surveys. The model classifies the data into categories and assigns a score to each data point to gain a detailed understanding of the user's attributes. The output is the analysis results.

[0148] Step 7:

[0149] Proposal of the best lifestyle

[0150] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that are best suited to the user's living environment. The inputs are the analysis results and user data. For example, it generates specific suggestions such as "use local child-rearing support services available on weekends" and "create a daily schedule that matches your baby's sleep rhythm." The output is a lifestyle suggestion.

[0151] Server: Binds the generated suggestions and audio files to the user's account and sends notifications. When the user logs in, this information is displayed.

[0152] Step 8:

[0153] Display of suggestions and audio

[0154] Terminal: The terminal provides an interface that displays the suggestions and the generated baby voice to the user. The inputs are the suggestions and audio files sent from the server. Specifically, a detail page is generated using HTML / CSS, allowing the user to check the suggestions in detail and download or print them if necessary. The output is the suggestions displayed to the user and the audio playback.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Conventional baby ultrasound image analysis and voice generation systems only focus on helping people feel the baby's presence, and do not address safety and security measures within the home. The purpose of this invention is to provide a means to ensure safety and security within the home while feeling the baby's presence by adding a home security system linking function to a system that generates a baby's voice from an ultrasound image.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods based on the analysis results, means for providing the voice data and the proposals to the user, means for setting the generated voice data in a home security system, and means for detecting abnormalities in conjunction with the set security system and notifying the user. This makes it possible to detect abnormalities in the home using a baby's voice and promptly notify the user.

[0160] An "echo image" is an image of a fetus or internal organs taken using a medical ultrasound device.

[0161] "Feature data" refers to data such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0162] "Voice data" is data of a baby's voice generated based on feature data.

[0163] "User's living environment information" is information about the environment in which the user lives, including, for example, the size and environment of the residence, whether it is a dual-income household, and so on.

[0164] "User Preferences" is information about a user's parenting policies and priorities.

[0165] "User needs" refers to information about the user's requests for help, services, etc. related to childcare.

[0166] "Lifestyle suggestions" are suggestions for optimal lifestyle habits and child-rearing methods created based on the user's living environment, preferences, and needs.

[0167] A "home security system" is a system that detects abnormalities within the home and notifies the user.

[0168] A "generative AI model" is a machine learning model that learns from past data and generates new data.

[0169] A "prompt sentence" is a sentence that prompts input or instructions to the system.

[0170] The following procedure is provided as an embodiment of the present invention.

[0171] First, users use a smartphone application to upload ultrasound images of their baby, which are then sent to a server through the application's interface.

[0172] The server applies image pre-processing algorithms, such as noise reduction and contrast adjustment, to the received echo images to prepare them for analysis, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[0173] The server then runs a speech generation model based on the extracted feature data to generate a baby's voice. The model uses past baby voice data that it has already learned from to generate speech data from the feature data. The generated speech data is saved as an audio file and linked to the user's account.

[0174] Next, users enter information about their living environment, preferences, and needs in a questionnaire format via a smartphone application, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?"

[0175] The server receives the living environment information entered by the user and analyzes it using a machine learning model. This analysis organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes. Based on the analysis results, the system suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, the system suggests nearby parent-child support facilities and recommended activities, as well as a daily schedule that matches the baby's sleep rhythm. The generated suggestions and audio data are notified to the user, and can be viewed through the smartphone application interface.

[0176] Furthermore, the generated voice data is also set up in a home security system. This security system detects abnormalities in the home based on the baby's voice and sends a notification to the user. Sensors used include infrared cameras, voice recognition sensors, and vibration sensors. When an abnormality is detected, a smartphone application sends a notification to the user in real time.

[0177] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the system will suggest the use of a childcare support service. Furthermore, by generating baby voice data and providing it to User A, User A can feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence. In this way, the system of the present invention not only generates baby voices and offers personalized lifestyle suggestions tailored to each user's needs, but also provides safety and security within the home.

[0178] Example prompt:

[0179] "Upload an ultrasound image and we'll automatically generate a baby voice and set it up in your home security system."

[0180] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0181] Step 1:

[0182] Uploading ultrasound images

[0183] The user uploads an ultrasound image of the baby through the interface of a smartphone application. The uploaded ultrasound image is sent from the device to the server. Specifically, the user selects an ultrasound image on the device, and the device sends the data to the server via the Internet. The input is the ultrasound image selected by the user, and the output is the ultrasound image data stored on the server.

[0184] Step 2:

[0185] Echo image analysis

[0186] The server applies image pre-processing algorithms to the received echo images to remove noise and adjust contrast, making the echo images ready for analysis. The server then uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images. The input is the echo image data received by the server, and the output is the extracted feature data.

[0187] Step 3:

[0188] Baby voice generation

[0189] The server runs a speech generation model based on the extracted feature data. This model has been trained on past baby voice data and generates new speech data based on the feature data. This generation process saves the baby's voice as an audio file. The input is the extracted feature data, and the output is the generated speech data (audio file).

[0190] Step 4:

[0191] Entering User Data

[0192] Users use a smartphone application to input information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?" The input is the living environment information, preferences, and needs entered by the user, and the output is user data sent to the server.

[0193] Step 5:

[0194] User data analysis

[0195] The server runs a machine learning model to perform a detailed analysis of the data entered by the user regarding their living environment, preferences, and needs. The model organizes the user data into categories and scores to gain a detailed understanding of the user's attributes. The input is the user data received by the server, and the output is the analysis results (details of the user's attributes).

[0196] Step 6:

[0197] Proposals for optimal lifestyle and child-rearing methods

[0198] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, it suggests nearby parent-child support facilities and recommended activities. It also suggests a daily schedule that matches the baby's sleep rhythm. The input is the analysis results, and the output is the suggested lifestyle and child-rearing methods.

[0199] Step 7:

[0200] Proposal and voice data provided

[0201] The server notifies the user of the generated suggestions and voice data and provides them through the smartphone application interface. The user can view the detailed suggestions and baby voice data. The input is the generated suggestions and voice data, and the output is the information provided to the user.

[0202] Step 8:

[0203] Setting up a home security system

[0204] Based on the generated voice data, the user can configure the home security system. An interface for configuring the system is provided through a smartphone application. The input is the generated voice data, and the output is the configured security system.

[0205] Step 9:

[0206] Anomaly detection and notification

[0207] The security system detects abnormalities based on the baby's voice that has been set. For example, it uses a voice recognition sensor, infrared camera, or vibration sensor to notify the user in real time when an abnormality occurs. The input is the information detected by the sensor, and the output is an abnormality notification to the user.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] The system of the present invention is a generative AI system that generates a baby's voice from ultrasound images and suggests optimal lifestyles, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more accurate personalization. This system involves a series of processes, starting with the user uploading the ultrasound images and ending with dynamically adjusting lifestyle suggestions using the emotion engine.

[0210] Program processing

[0211] 1. Uploading ultrasound images

[0212] User: First, the user accesses the Echo Voice AI website or app, selects and uploads their own Echo image, and the uploaded Echo image is sent to the server.

[0213] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0214] 2. Analysis of Echo Images

[0215] Server: After receiving the echo images, it applies image pre-processing algorithms to remove noise and adjust contrast, then uses machine learning models to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0216] 3. Baby voice generation

[0217] Server: Based on the feature data, a speech generation model is run to generate baby voice data. This voice data is saved as an audio file and linked to the user's account.

[0218] 4. Enter user data

[0219] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0220] Terminal: Sends the information entered by the user to the server.

[0221] 5. User data analysis

[0222] Server: Runs machine learning models to analyze the received user data, organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes.

[0223] 6. Applying the Emotion Engine

[0224] Server: Recognizes the user's emotional state using an emotion engine. This involves extracting emotions from the user's facial expressions, voice, and text input when answering surveys or using the system.

[0225] Server: Reflects the emotion recognition results in the user's lifestyle and childcare suggestions, optimizing them. For example, if the user is feeling stressed, the server adds suggestions for relaxation methods.

[0226] 7. Proposal for the best lifestyle

[0227] Server: Based on the analysis and emotion recognition results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. The proposals are generated in the form of text, lists, graphs, etc.

[0228] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0229] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0230] Specific examples

[0231] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. For example, User B could be offered suggestions such as ways to reduce stress or information about joining a local support group. User B will also be provided with data on the generated baby's voice, which will help User B imagine what it would be like to raise a child in the future.

[0232] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and also provides personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration, providing an environment in which parents can approach child-rearing with confidence.

[0233] The processing flow will be explained below.

[0234] Step 1:

[0235] User: The user visits the Echo Voice AI website or app, logs in to their account, selects an echo image, and clicks the upload button to upload the echo image to the system.

[0236] Step 2:

[0237] Terminal: Preprocessing the uploaded echo images involves optimizing the image resolution, compressing the file size, and adding necessary metadata to the images (e.g., capture date and time, device information, etc.).

[0238] Step 3:

[0239] Terminal: Sends pre-processed echo image data to the server, using an encryption protocol (e.g., TLS) to ensure data security.

[0240] Step 4:

[0241] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust color and contrast, thereby improving image quality.

[0242] Step 5:

[0243] Server: Extracts feature data such as fetal shape, size, and position from preprocessed ultrasound images. This is done using machine learning models, and records interesting features as numerical data.

[0244] Step 6:

[0245] Server: Runs a speech generation model based on the extracted features. This model is based on past baby speech data. A speech generation algorithm is used to create a speech that most closely matches the current features.

[0246] Step 7:

[0247] Server: Saves the generated audio data in an audio file format (e.g., MP3), associates it with the user's account, and notifies the user of the download link for the audio file.

[0248] Step 8:

[0249] User: The user clicks on the link provided to download the generated baby voice audio file and listen to the sound.

[0250] Step 9:

[0251] Device: Provides an interface that allows users to answer a detailed questionnaire about their living environment, preferences, and needs, including questions about their family structure, their attitude toward child-rearing, and their daily life challenges.

[0252] Step 10:

[0253] User: Completes a survey, inputting their preferences and needs regarding lifestyle and childcare methods. Once completed, clicks the submit button.

[0254] Step 11:

[0255] Terminal: The survey data entered by the user is sent to the server. Encryption protocols are used to ensure the security of the information when sending the data.

[0256] Step 12:

[0257] Server: Collects the received survey data and runs a machine learning model for detailed analysis. This model categorizes user responses into categories and scores to understand user attributes and needs.

[0258] Step 13:

[0259] Server: Using the emotion engine, the system analyzes the user's emotional state from their facial expressions and text responses when filling out questionnaires and using the system. This data can categorize the user's emotions and evaluate them in real time.

[0260] Step 14:

[0261] Server: Based on the analysis results of the emotion engine, the server proposes lifestyle and child-rearing methods that take into account the user's psychological state and emotions. For example, if the user is feeling tired or stressed, it will suggest relaxation methods or information on how to join support groups.

[0262] Step 15:

[0263] Server: Integrates analysis and emotion recognition results to generate personalized lifestyle and parenting recommendations in the form of text, visual lists, graphs, etc.

[0264] Step 16:

[0265] Server: Notifies the user of the generated proposal and provides further information via a dedicated dashboard or email.

[0266] Step 17:

[0267] Device: Provides an interface to display the recommendations to the user and allows the user to view detailed recommendations, including, for example, video tutorials of the suggested parenting techniques and links to local support groups.

[0268] Step 18:

[0269] User: Check the suggestions, put into practice specific lifestyle and child-rearing methods, and provide feedback within the interface as needed.

[0270] Step 19:

[0271] Device: Sends user-entered feedback to the server, which uses it to improve the accuracy of future suggestions.

[0272] Step 20:

[0273] Server: Receives feedback data and stores it in a database, which feeds the machine learning model for future improvements.

[0274] These are the specific processing steps of the Echo Voice AI system, which combines an emotion engine. Through this series of processes, users can experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0275] Example 2

[0276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0277] Conventional systems have problems with not being able to fully utilize the information obtained from ultrasound images, and are unable to appropriately reduce the anxiety and stress felt by users. Furthermore, the suggestions provided to users are general and not optimized for each user's living environment or psychological state, resulting in insufficient personalized support. Furthermore, the baby's voice, generated based on the analysis of ultrasound images, is not provided in real time, and users' expectations cannot be met.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0279] In this invention, the server includes means for preprocessing the echo image, means for extracting feature data from the echo image, and means for generating voice data based on the feature data. This makes it possible to effectively extract information from the echo image and provide personalized support based on the user's living environment and psychological state. Furthermore, by generating and providing voice data in real time, it is possible to meet user expectations.

[0280] An "echo image" is a medical image that uses ultrasound technology to visually record the internal structure and movements of a fetus.

[0281] "Preprocessing" is a technique for performing initial processing such as noise removal, contrast adjustment, and size compression to make data easier to analyze.

[0282] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0283] "Voice data" refers to data of a baby's voice generated based on feature data extracted from an echo image.

[0284] "User Information" means data about a user's living environment, preferences, and needs, and is information collected in the form of a questionnaire or similar.

[0285] "Analysis" is the process of analyzing collected data using machine learning models and statistical methods to derive specific trends and patterns.

[0286] "Mental state recognition" is the process of extracting and understanding a user's emotions and psychological state from questionnaire responses, facial expression analysis, etc.

[0287] "Lifestyle suggestions" are advice on optimal lifestyle habits and child-rearing methods provided based on the user's living environment and psychological state.

[0288] "Provision" refers to the process of notifying the user of the generated voice data and lifestyle suggestions in a viewable form.

[0289] A "machine learning model" is an algorithm that learns from large amounts of data and automatically performs specific tasks (such as generating voice data or analyzing user information).

[0290] MODE FOR CARRYING OUT THE INVENTION

[0291] This invention is a system that generates a baby's voice based on an ultrasound image and suggests a personalized lifestyle and child-rearing method based on the user's living environment and psychological state. Each processing step of this system and the necessary hardware and software are described in detail below.

[0292] Uploading ultrasound images

[0293] Users access a dedicated website or application, select and upload echo images, and use a smartphone or computer to do this. Specifically, users select images from their smartphone's camera roll or a folder on their computer and click the "Upload" button. The uploaded images are then sent to the server.

[0294] Echo image preprocessing

[0295] The device preprocesses the received echo images. Specifically, it compresses the image size and adds metadata required for compression (e.g., shooting date and time, file format, etc.). This process uses the OpenCV library. This makes image data transfer efficient and ensures smooth processing on the server.

[0296] Echo image analysis

[0297] The server performs noise reduction and contrast adjustment on the preprocessed images, then uses a machine learning model (e.g., a TensorFlow or PyTorch-based model) to extract feature data such as the shape, size, and position of the fetus from the image, and uses the OpenCV library for filtering and edge detection at this stage.

[0298] Baby voice generation

[0299] The server generates voice data using a voice generation model (e.g., WaveNet or Tacotron) based on the extracted feature data. Specifically, it uses the fetus's shape data as input to synthesize the baby's voice. This voice data is saved as an audio file (e.g., WAV format) and associated with the user's account.

[0300] Entering User Data

[0301] Users enter information about their living environment, preferences, and needs in a questionnaire format, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?" The entered data is sent to the server.

[0302] Preprocessing and analysis of user data

[0303] The device preprocesses the data sent by the user. Specifically, it structures the data in JSON format. The server then uses a machine learning model (e.g., scikit-learn or XGBoost) to classify the received data into categories and scores. For example, "time spent at home" is classified into two categories, "a lot" and "a little," and "interest in childcare methods" is scored.

[0304] Applying emotion engines and recognizing mental states

[0305] The server uses an emotion engine to recognize the user's psychological state. For example, it performs facial expression analysis (using OpenFace or Dlib) and voice analysis (using Praat or DeepSpeech) to understand the user's emotional state.

[0306] Proposal of the best lifestyle

[0307] The server then proposes optimal lifestyle and child-rearing methods based on the results of analyzing user data and recognizing their psychological state. Suggestions are generated in the form of text, lists, graphs, and more. For example, they include specific advice such as "incorporate relaxation time into your daily routine" or "take advantage of local support groups." The generated suggestions are notified to the user via a dedicated dashboard or email.

[0308] The device provides an interface that displays the proposal content, allowing users to check the details of the proposal. Users can view the proposal content on their smartphones or computers and obtain the information they need.

[0309] Examples of concrete examples and prompts

[0310] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. Specifically, it will suggest "methods for reducing stress" and "information on joining local support groups." It will also provide data on the generated baby's voice, which will help User B imagine what it will be like to raise a child in the future.

[0311] An example of a prompt is, "Based on the ultrasound image and information about the user's living environment, generate a baby's voice and suggest the optimal lifestyle. If the user is a single parent and is feeling anxious, please also suggest additional support methods."

[0312] This allows the invention to generate a baby's voice based on the ultrasound image and provide personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration. By receiving suggestions optimized for their own living environment, users can reduce their anxiety about child-rearing and approach child-rearing with confidence.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1: Upload your ultrasound images

[0315] Users access a dedicated website or application, select and upload echo images from their smartphone's camera roll or a folder on their computer. The input is the echo image file, and the output is the echo image data sent to the server. When the user clicks the "Upload" button, the image data is transferred to the server.

[0316] Step 2: Preprocessing of echo images

[0317] The terminal preprocesses the received echo images. The input is the echo image data sent by the user, and the output is the preprocessed image data. This preprocessing includes image size compression, noise removal, and contrast adjustment using the OpenCV library. This adds image metadata and improves transmission efficiency.

[0318] Step 3: Echo image analysis

[0319] The server analyzes the preprocessed echo images. The input is the preprocessed image data, and the output is feature data such as the shape, size, and position of the fetus. The server uses the OpenCV library to perform filtering and edge detection, and then applies TensorFlow or PyTorch-based machine learning models to extract the feature data.

[0320] Step 4: Generating a baby voice

[0321] The server runs a speech generation model based on the extracted feature data. The input is the feature data, and the output is an audio file of the baby's voice. Specifically, it uses WaveNet and Tacotron models to synthesize the baby's voice from the fetal shape data. This voice data is saved as an audio file and associated with the user's account.

[0322] Step 5: Enter user data

[0323] Users enter information about their living environment, preferences, and needs in the form of a questionnaire. The input is the questionnaire data answered by the user, and the output is user data that is structured and sent to the server. Specifically, users answer questions such as "how much time do you spend at home?" and "interests in childcare methods."

[0324] Step 6: Preprocessing and analyzing user data

[0325] The terminal preprocesses the survey data obtained from the user. The input is the survey data, and the output is the preprocessed data. The data is structured in JSON format. The server then analyzes the data using machine learning models such as scikit-learn and XGBoost, classifying user attributes into categories and scores.

[0326] Step 7: Applying the Emotion Engine and Recognizing Mental States

[0327] The server uses an emotion engine to recognize the user's psychological state. The input is the user's questionnaire responses, behavioral data, and facial expression analysis data, and the output is the user's emotional information. Facial expression analysis is performed using OpenFace or Dlib, and speech analysis is performed using Praat or DeepSpeech. The recognized emotional information is obtained.

[0328] Step 8: Proposal for the optimal lifestyle

[0329] The server proposes optimal lifestyle and child-rearing methods based on the analysis results of user data and emotional information. The input is the analysis results and emotional information, and the output is the proposals. Specific proposals are generated in the form of text, lists, and graphs. The generated proposals are notified to the user via a dedicated dashboard or email.

[0330] Step 9: View the proposal

[0331] The terminal provides an interface that displays the proposal content to the user. The input is the proposal content, and the output is the interface displayed to the user. The user can view the detailed proposal content using a smartphone or PC and obtain the necessary information.

[0332] (Application example 2)

[0333] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0334] Conventional childcare support systems based on ultrasound images have difficulty making personalized suggestions that take into account the user's emotional state, leading to a lack of confidence in users' childcare strategies. Another issue is that there is little correlation between the baby's voice generated from ultrasound images and the childcare suggestions, meaning users are not provided with sufficient information. Furthermore, lifestyle and childcare method suggestions are static and cannot be dynamically adjusted to suit the user's situation or emotions, making it difficult to provide efficient suggestions in physical stores.

[0335] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0336] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for using an emotion engine to recognize the user's emotional state, means for optimizing lifestyle and childcare method suggestions based on the emotion recognition results, and means for providing the voice data and the suggestions to the user, thereby enabling dynamic lifestyle and childcare method suggestions based on the user's emotional state.

[0337] An "echo image" is an image showing the condition of a fetus obtained by an ultrasound diagnostic device.

[0338] "Feature data" refers to information such as the shape, size, and position of the fetus extracted from the echo image.

[0339] "Voice data" is digital audio information including a baby's voice generated based on feature data.

[0340] "User's living environment information" refers to information about the user's daily life, including family structure, living environment, occupation, and the like.

[0341] "Preferences" refers to the user's preferences and intentions regarding child-rearing.

[0342] "Needs" refers to the user's requests and demands regarding childcare.

[0343] An "emotion engine" is a software or hardware technology that recognizes a user's emotional state from their text input, facial expressions, and voice.

[0344] "Lifestyle and child-rearing suggestions" refers to recommending optimal lifestyle habits and child-rearing methods for a user based on the user's living environment information, emotional state, and analysis results.

[0345] "Optimize" means tailoring the recommendations to provide more relevant advice based on the user's personalized information.

[0346] This invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. By combining it with an emotion engine that recognizes the user's emotions, it achieves highly accurate personalization. Specific embodiments for implementing this system are described below.

[0347] Users first capture echo images using their smartphones and then upload them using a dedicated application. The uploaded echo images are then sent to a server. To ensure efficient image transfer, the echo images are pre-processed on the device to compress the image size and add required metadata.

[0348] The server applies image pre-processing algorithms to analyze the received echo images, such as noise reduction and contrast adjustment, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[0349] The server then runs a speech generation model on the feature data to generate baby voice data, which is then saved as an audio file and associated with the user's account.

[0350] The user then enters information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as, "Do you spend a lot of time at home?" and "What kind of child-rearing methods are you interested in?" The device then sends this user-entered information to the server.

[0351] The server runs a machine learning model to analyze the received user data, organizing the user's input data into categories and scores to gain a detailed understanding of the user's attributes.

[0352] The server then uses an emotion engine to recognize the user's emotional state. This involves extracting emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The server then reflects the emotion recognition results in the user's lifestyle and childcare recommendations, optimizing them. For example, if the user is feeling stressed, the server will add suggestions for relaxation methods.

[0353] Based on the analysis and emotion recognition results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. The suggestions are generated in the form of text, lists, graphs, etc. The generated suggestions are notified to the user via a dedicated dashboard or email, and can be viewed.

[0354] For example, suppose User A uploads an ultrasound image to the system and enters "both spouses work, so there is little anxiety about childcare" as living environment information. If the emotion engine analyzes User A's input and behavior and recognizes that stress is low, the system will reflect this and make specific suggestions. For example, User A may be offered nutritionally balanced meal menus or local childcare support information. User A will also be provided with data on the generated baby's voice, giving User A more confidence in their future childcare.

[0355] Examples of prompts include:

[0356] "We are developing a system that generates a baby's voice based on ultrasound images and suggests child-rearing methods and lifestyles that are tailored to the user's living environment and psychological state. With this system, users can upload ultrasound images using their smartphone, listen to their baby's voice, and receive optimal lifestyle suggestions. The user's psychological state is analyzed using an emotion engine, and suggestions are dynamically adjusted according to the user's stress level."

[0357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0358] Step 1:

[0359] The user launches a dedicated application on their smartphone to acquire an echo image. The user selects an echo image and uploads it through the application. The input is the echo image, and the output is an image file transferred to the server. The device compresses the image file size, adds required metadata, and efficiently transmits it to the server.

[0360] Step 2:

[0361] The server runs the received echo images through an image pre-processing algorithm. The input is a compressed echo image, and the output is a pre-processed echo image. This pre-processing includes noise removal and contrast adjustment, which improves image quality and allows for accurate subsequent processing.

[0362] Step 3:

[0363] The server runs a machine learning model to extract feature data from the preprocessed echo images. The input is the preprocessed echo images, and the output is feature data. This feature data includes the shape, size, and position of the fetus. The machine learning model efficiently analyzes important information in the echo images.

[0364] Step 4:

[0365] The server runs a speech generation model based on the feature data to generate baby voice data. The input is the feature data, and the output is the baby's voice data. This voice data is saved as an audio file. The speech generation model analyzes the feature data and creates an audio file that imitates the baby's voice.

[0366] Step 5:

[0367] The user enters information about their living environment, preferences, and needs in a questionnaire format. The input is information about the living environment, preferences, and needs, and the output is the questionnaire data sent to the server. The terminal sends the user's input to the server.

[0368] Step 6:

[0369] The server runs a machine learning model to analyze the received user data, such as living environment information, preferences, and needs. The input is the survey data, and the output is the analysis results. The machine learning model organizes this data into categories and scores to gain a detailed understanding of the user's attributes.

[0370] Step 7:

[0371] The server recognizes the user's emotional state using an emotion engine that extracts emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The input is data related to the user's emotions, and the output is the emotion recognition results. The emotion engine analyzes the emotional aspects of the data entered by the user and grasps the user's current psychological state.

[0372] Step 8:

[0373] The server optimizes lifestyle and child-rearing method suggestions based on the emotion recognition results. The inputs are the analysis results and emotion recognition results, and the output is optimized suggestion data. The suggestion content is dynamically adjusted according to the emotion recognition results.

[0374] Step 9:

[0375] The server provides the generated voice data and optimized lifestyle suggestions to the user. The input is the voice data and suggestion data, and the output is the information provided to the user. The information is notified to the user via a dedicated dashboard or email, and can be viewed.

[0376] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0377] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0378] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0379] [Second embodiment]

[0380] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0381] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0382] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0383] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0384] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0385] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0386] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0387] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0388] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0389] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0390] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0391] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0392] The system of the present invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. This system involves a series of processes, starting with the user uploading the ultrasound image and ending with providing the generated baby's voice and lifestyle suggestions.

[0393] Program processing

[0394] 1. Uploading ultrasound images

[0395] User: First, the user accesses the system's website or application and uploads their own echo images, which are then sent to the server.

[0396] Terminal: Provides an interface for sending user-selected echo images to the server.

[0397] 2. Analysis of Echo Images

[0398] Server: Upon receiving the echo images, the server applies image pre-processing algorithms to remove noise and adjust contrast, making the echo images ready for analysis. Then, it uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images.

[0399] 3. Baby voice generation

[0400] Server: Runs a speech generation model based on the feature data. This model has learned from past baby voice data and generates speech data based on the given feature data. The generated speech data is saved as an audio file and linked to the user's account.

[0401] 4. Enter user data

[0402] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0403] Terminal: Provides an interface for sending user-entered information to the server.

[0404] 5. User data analysis

[0405] Server: Runs machine learning models to analyze the received user data in detail. The models organize the user input data into categories and scores to gain a detailed understanding of the user's attributes.

[0406] 6. Proposal for the best lifestyle

[0407] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. For example, for users living in urban areas, the server suggests nearby parent-child support facilities and recommended activities. It also suggests lifestyle schedules that match the baby's sleep rhythm.

[0408] Server: Notifies the user of the generated suggestions and provides audio files and lifestyle suggestions.

[0409] Terminal: An interface is provided to display the received proposals to the user, allowing the user to view the proposals in detail.

[0410] Specific examples

[0411] For example, suppose User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information. The server analyzes this information and suggests the use of childcare support services. The server also generates baby voice data and provides it to User A, allowing them to feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence.

[0412] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and provides personalized lifestyle suggestions that meet the needs of each user, providing an environment in which parents can approach child-rearing with confidence.

[0413] The processing flow will be explained below.

[0414] Step 1:

[0415] User: The user visits the Echo Voice AI website or app, selects their Echo image, and clicks the upload button. The user also enters basic information such as name and email address.

[0416] Step 2:

[0417] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0418] Step 3:

[0419] Terminal: Sends pre-processed echo image data to the server. Data security is ensured by using encryption protocols during transmission.

[0420] Step 4:

[0421] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust the contrast, thereby improving the image quality.

[0422] Step 5:

[0423] Server: Runs machine learning models to extract feature data from preprocessed ultrasound images. Specifically, it detects important features such as the shape, position, and size of the fetus and records them as numerical data.

[0424] Step 6:

[0425] Server: Runs a speech generation model based on the feature data. Based on past baby voice data, generates a voice that is closest to the current feature data. Saves this voice data as an audio file (e.g., MP3).

[0426] Step 7:

[0427] Server: Binds the generated audio file to the user's account, notifies the user of the download link, and stores the audio file in secure storage.

[0428] Step 8:

[0429] User: The user clicks on the link provided and downloads the generated baby voice audio file to check it.

[0430] Step 9:

[0431] Device: Provides an interface where users can complete a questionnaire about their living environment, preferences, and needs. For example, questions might include, "What is your family structure?" and "What is your approach to childcare?"

[0432] Step 10:

[0433] User: Answers a questionnaire about their living environment, preferences, and needs, and clicks the submit button.

[0434] Step 11:

[0435] Terminal: The survey data entered by the user is sent to the server. The data is encrypted and sent.

[0436] Step 12:

[0437] Server: Runs a machine learning model to analyze the received survey data. The model classifies user responses into categories and scores, and analyzes lifestyle and parenting characteristics.

[0438] Step 13:

[0439] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's needs. The proposals are generated in the form of text, lists, graphs, etc.

[0440] Step 14:

[0441] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0442] Step 15:

[0443] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0444] Step 16:

[0445] User: Check the suggestions, put them into practice with specific lifestyle and child-rearing methods, and provide feedback as needed.

[0446] Step 17:

[0447] Terminal: Sends the feedback entered by the user to the server. The feedback will be used to improve the service in the future.

[0448] Step 18:

[0449] Server: Receives feedback data and stores it in a database. The stored data is fed into the machine learning model and used to improve the accuracy of future suggestions.

[0450] These are the specific processing steps of the Echo Voice AI system, which allows users to experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0451] Example 1

[0452] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0453] Conventional ultrasound image analysis systems do not generate baby voices or provide personalized recommendations for optimal childcare lifestyles, making it difficult for users to obtain childcare advice tailored to their own living environment. Furthermore, conventional systems do not utilize machine learning models to generate voice data, which leads to issues with the realism of the generated voice data and low user satisfaction.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0455] In this invention, the server includes means for receiving echo images, means for preprocessing the echo images, means for extracting feature data from the echo images, means for generating voice data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods, and means for providing the voice data and suggestions to the user, thereby enabling the user to realistically experience the baby's voice from the echo images and receive suggestions on a child-rearing lifestyle that is optimal for their own living environment.

[0456] "Echo image" refers to an image taken using ultrasound to check the condition of the fetus.

[0457] "Preprocessing" refers to the initial processing steps required to prepare the image for analysis, such as removing noise from the echo image and adjusting the contrast.

[0458] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0459] "Audio data" refers to digital audio data of a baby's voice created from feature data of an ultrasound image using a generative AI model.

[0460] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of baby voice data and generates realistic voices based on specific feature data.

[0461] "User's living environment information" refers to individual information provided by the user related to childcare, such as the user's living situation and residential environment.

[0462] "Survey format" refers to a method in which users respond to questions by selecting options or writing a statement.

[0463] "Lifestyle" and "parenting methods" refer to behavioral patterns and specific parenting advice and methods that are considered optimal for the user and their baby's lifestyle.

[0464] The system of the present invention involves a series of processes that begin with the user uploading an ultrasound image and ultimately provide a generated baby voice and optimal lifestyle recommendations. The specific hardware and software used in each process step are described in detail below.

[0465] First, the user accesses the system's website or application using a device such as a PC or smartphone. The user clicks the "Upload Echo Image" button on the screen, selects an echo image file from their device, and uploads it. The device provides an interface for reading the echo image file and sending it to the server. It is desirable that a progress bar be displayed to indicate the progress of the upload.

[0466] The server temporarily stores the received echo images. It then applies image pre-processing algorithms to remove noise and adjust the contrast. Specifically, it uses noise removal and contrast adjustment algorithms such as a median filter. It then uses a deep learning model (e.g., U-Net) to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0467] The server runs a speech generation model (e.g., WaveNet) based on the extracted feature data. This speech generation model has learned from past baby voice data and generates a realistic baby voice from the input data. The generated voice data is saved as an audio file (e.g., MP3 format) and linked to the user's account.

[0468] Next, the user answers a questionnaire about their living environment, preferences, and needs. The device provides a questionnaire-style user interface and transmits the information the user has entered to the server, displaying a progress bar while the data is being transmitted.

[0469] The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. This model categorizes the user data and assigns a score to each data point to gain a detailed understanding of the user's living environment and needs. For example, it can organize information such as "dual-income household" and "limited childcare time."

[0470] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, it generates specific suggestions such as "use local childcare support services available on weekends" or "create a daily schedule that matches the baby's sleep rhythm." The server then links the generated suggestions and audio files to the user's account and sends a notification. Users can check this information when they log in.

[0471] The terminal provides an interface that displays the suggestions and the generated baby voice to the user, who can then check the suggestions in detail and download or print them as needed.

[0472] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the server will analyze this information and suggest the use of childcare support services. Furthermore, by generating baby voice data and providing it to User A, User A will be able to feel the baby's presence even before it is born. Furthermore, the server will also suggest childcare methods and specific schedules suitable for dual-income households, allowing User A to make plans with confidence regarding childcare.

[0473] An example of a prompt would be the following text:

[0474] "We are a dual-income household, and the time we can devote to childcare is limited. Please suggest baby voice data and childcare methods that are suitable for our living environment."

[0475] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0476] Step 1:

[0477] Uploading ultrasound images

[0478] User: The user visits the system's website or application and clicks the "Upload Echo Image" button. As input, there is an echo image file selected by the user from the device. This file will be uploaded to the system.

[0479] Terminal: The terminal provides an interface to read the selected echo image file and send it to the server. Specifically, it sends the image data to the server using an HTTP request and displays a progress bar to indicate the upload progress to the user. As an output, the image data is sent to the server.

[0480] Step 2:

[0481] Echo image preprocessing

[0482] Server: The server temporarily stores the received echo images. The input is the uploaded echo image file. Then, image pre-processing algorithms are applied to remove noise and adjust contrast. Specifically, a median filter is used to remove noise, and histogram equalization is used to adjust contrast. The output is the processed echo image.

[0483] Step 3:

[0484] Extracting feature data from echo images

[0485] Server: The server extracts feature data from the preprocessed echo images using a deep learning model (e.g., U-Net). The input is the preprocessed echo images. The model extracts specific information from the echo images, such as the shape, size, and position of the fetus. The output is the fetal feature data.

[0486] Step 4:

[0487] Baby voice generation

[0488] Server: The server inputs the extracted feature data into a speech generation model (e.g., WaveNet). The input is the fetus's feature data. This speech generation model learns from past baby voice data to generate a realistic baby voice. The output is an MP3 audio file that is associated with the user's account.

[0489] Step 5:

[0490] Entering User Data

[0491] User: The user answers survey-style questions provided within the system. Input includes information about the user's living environment, preferences, and needs. For example, questions include "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0492] Terminal: The terminal provides a questionnaire-style user interface and sends the information entered by the user to the server. Specifically, an HTTP request is issued to collect form data and send it to the server.

[0493] Step 6:

[0494] User data analysis

[0495] Server: The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. The input is user data collected from surveys. The model classifies the data into categories and assigns a score to each data point to gain a detailed understanding of the user's attributes. The output is the analysis results.

[0496] Step 7:

[0497] Proposal of the best lifestyle

[0498] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that are best suited to the user's living environment. The inputs are the analysis results and user data. For example, it generates specific suggestions such as "use local child-rearing support services available on weekends" and "create a daily schedule that matches your baby's sleep rhythm." The output is a lifestyle suggestion.

[0499] Server: Binds the generated suggestions and audio files to the user's account and sends notifications. When the user logs in, this information is displayed.

[0500] Step 8:

[0501] Display of suggestions and audio

[0502] Terminal: The terminal provides an interface that displays the suggestions and the generated baby voice to the user. The inputs are the suggestions and audio files sent from the server. Specifically, a detail page is generated using HTML / CSS, allowing the user to check the suggestions in detail and download or print them if necessary. The output is the suggestions displayed to the user and the audio playback.

[0503] (Application example 1)

[0504] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] Conventional baby ultrasound image analysis and voice generation systems only focus on helping people feel the baby's presence, and do not address safety and security measures within the home. The purpose of this invention is to provide a means to ensure safety and security within the home while feeling the baby's presence by adding a home security system linking function to a system that generates a baby's voice from an ultrasound image.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0507] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods based on the analysis results, means for providing the voice data and the proposals to the user, means for setting the generated voice data in a home security system, and means for detecting abnormalities in conjunction with the set security system and notifying the user. This makes it possible to detect abnormalities in the home using a baby's voice and promptly notify the user.

[0508] An "echo image" is an image of a fetus or internal organs taken using a medical ultrasound device.

[0509] "Feature data" refers to data such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0510] "Voice data" is data of a baby's voice generated based on feature data.

[0511] "User's living environment information" is information about the environment in which the user lives, including, for example, the size and environment of the residence, whether it is a dual-income household, and so on.

[0512] "User Preferences" is information about a user's parenting policies and priorities.

[0513] "User needs" refers to information about the user's requests for help, services, etc. related to childcare.

[0514] "Lifestyle suggestions" are suggestions for optimal lifestyle habits and child-rearing methods created based on the user's living environment, preferences, and needs.

[0515] A "home security system" is a system that detects abnormalities within the home and notifies the user.

[0516] A "generative AI model" is a machine learning model that learns from past data and generates new data.

[0517] A "prompt sentence" is a sentence that prompts input or instructions to the system.

[0518] The following procedure is provided as an embodiment of the present invention.

[0519] First, users use a smartphone application to upload ultrasound images of their baby, which are then sent to a server through the application's interface.

[0520] The server applies image pre-processing algorithms, such as noise reduction and contrast adjustment, to the received echo images to prepare them for analysis, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[0521] The server then runs a speech generation model based on the extracted feature data to generate a baby's voice. The model uses past baby voice data that it has already learned from to generate speech data from the feature data. The generated speech data is saved as an audio file and linked to the user's account.

[0522] Next, users enter information about their living environment, preferences, and needs in a questionnaire format via a smartphone application, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?"

[0523] The server receives the living environment information entered by the user and analyzes it using a machine learning model. This analysis organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes. Based on the analysis results, the system suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, the system suggests nearby parent-child support facilities and recommended activities, as well as a daily schedule that matches the baby's sleep rhythm. The generated suggestions and audio data are notified to the user, and can be viewed through the smartphone application interface.

[0524] Furthermore, the generated voice data is also set up in a home security system. This security system detects abnormalities in the home based on the baby's voice and sends a notification to the user. Sensors used include infrared cameras, voice recognition sensors, and vibration sensors. When an abnormality is detected, a smartphone application sends a notification to the user in real time.

[0525] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the system will suggest the use of a childcare support service. Furthermore, by generating baby voice data and providing it to User A, User A can feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence. In this way, the system of the present invention not only generates baby voices and offers personalized lifestyle suggestions tailored to each user's needs, but also provides safety and security within the home.

[0526] Example prompt:

[0527] "Upload an ultrasound image and we'll automatically generate a baby voice and set it up in your home security system."

[0528] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0529] Step 1:

[0530] Uploading ultrasound images

[0531] The user uploads an ultrasound image of the baby through the interface of a smartphone application. The uploaded ultrasound image is sent from the device to the server. Specifically, the user selects an ultrasound image on the device, and the device sends the data to the server via the Internet. The input is the ultrasound image selected by the user, and the output is the ultrasound image data stored on the server.

[0532] Step 2:

[0533] Echo image analysis

[0534] The server applies image pre-processing algorithms to the received echo images to remove noise and adjust contrast, making the echo images ready for analysis. The server then uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images. The input is the echo image data received by the server, and the output is the extracted feature data.

[0535] Step 3:

[0536] Baby voice generation

[0537] The server runs a speech generation model based on the extracted feature data. This model has been trained on past baby voice data and generates new speech data based on the feature data. This generation process saves the baby's voice as an audio file. The input is the extracted feature data, and the output is the generated speech data (audio file).

[0538] Step 4:

[0539] Entering User Data

[0540] Users use a smartphone application to input information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?" The input is the living environment information, preferences, and needs entered by the user, and the output is user data sent to the server.

[0541] Step 5:

[0542] User data analysis

[0543] The server runs a machine learning model to perform a detailed analysis of the data entered by the user regarding their living environment, preferences, and needs. The model organizes the user data into categories and scores to gain a detailed understanding of the user's attributes. The input is the user data received by the server, and the output is the analysis results (details of the user's attributes).

[0544] Step 6:

[0545] Proposals for optimal lifestyle and child-rearing methods

[0546] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, it suggests nearby parent-child support facilities and recommended activities. It also suggests a daily schedule that matches the baby's sleep rhythm. The input is the analysis results, and the output is the suggested lifestyle and child-rearing methods.

[0547] Step 7:

[0548] Proposal and voice data provided

[0549] The server notifies the user of the generated suggestions and voice data and provides them through the smartphone application interface. The user can view the detailed suggestions and baby voice data. The input is the generated suggestions and voice data, and the output is the information provided to the user.

[0550] Step 8:

[0551] Setting up a home security system

[0552] Based on the generated voice data, the user can configure the home security system. An interface for configuring the system is provided through a smartphone application. The input is the generated voice data, and the output is the configured security system.

[0553] Step 9:

[0554] Anomaly detection and notification

[0555] The security system detects abnormalities based on the baby's voice that has been set. For example, it uses a voice recognition sensor, infrared camera, or vibration sensor to notify the user in real time when an abnormality occurs. The input is the information detected by the sensor, and the output is an abnormality notification to the user.

[0556] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0557] The system of the present invention is a generative AI system that generates a baby's voice from ultrasound images and suggests optimal lifestyles, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more accurate personalization. This system involves a series of processes, starting with the user uploading the ultrasound images and ending with dynamically adjusting lifestyle suggestions using the emotion engine.

[0558] Program processing

[0559] 1. Uploading ultrasound images

[0560] User: First, the user accesses the Echo Voice AI website or app, selects and uploads their own Echo image, and the uploaded Echo image is sent to the server.

[0561] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0562] 2. Analysis of Echo Images

[0563] Server: After receiving the echo images, it applies image pre-processing algorithms to remove noise and adjust contrast, then uses machine learning models to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0564] 3. Baby voice generation

[0565] Server: Based on the feature data, a speech generation model is run to generate baby voice data. This voice data is saved as an audio file and linked to the user's account.

[0566] 4. Enter user data

[0567] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0568] Terminal: Sends the information entered by the user to the server.

[0569] 5. User data analysis

[0570] Server: Runs machine learning models to analyze the received user data, organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes.

[0571] 6. Applying the Emotion Engine

[0572] Server: Recognizes the user's emotional state using an emotion engine. This involves extracting emotions from the user's facial expressions, voice, and text input when answering surveys or using the system.

[0573] Server: Reflects the emotion recognition results in the user's lifestyle and childcare suggestions, optimizing them. For example, if the user is feeling stressed, the server adds suggestions for relaxation methods.

[0574] 7. Proposal for the best lifestyle

[0575] Server: Based on the analysis and emotion recognition results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. The proposals are generated in the form of text, lists, graphs, etc.

[0576] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0577] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0578] Specific examples

[0579] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. For example, User B could be offered suggestions such as ways to reduce stress or information about joining a local support group. User B will also be provided with data on the generated baby's voice, which will help User B imagine what it would be like to raise a child in the future.

[0580] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and also provides personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration, providing an environment in which parents can approach child-rearing with confidence.

[0581] The processing flow will be explained below.

[0582] Step 1:

[0583] User: The user visits the Echo Voice AI website or app, logs in to their account, selects an echo image, and clicks the upload button to upload the echo image to the system.

[0584] Step 2:

[0585] Terminal: Preprocessing the uploaded echo images involves optimizing the image resolution, compressing the file size, and adding necessary metadata to the images (e.g., capture date and time, device information, etc.).

[0586] Step 3:

[0587] Terminal: Sends pre-processed echo image data to the server, using an encryption protocol (e.g., TLS) to ensure data security.

[0588] Step 4:

[0589] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust color and contrast, thereby improving image quality.

[0590] Step 5:

[0591] Server: Extracts feature data such as fetal shape, size, and position from preprocessed ultrasound images. This is done using machine learning models, and records interesting features as numerical data.

[0592] Step 6:

[0593] Server: Runs a speech generation model based on the extracted features. This model is based on past baby speech data. A speech generation algorithm is used to create a speech that most closely matches the current features.

[0594] Step 7:

[0595] Server: Saves the generated audio data in an audio file format (e.g., MP3), associates it with the user's account, and notifies the user of the download link for the audio file.

[0596] Step 8:

[0597] User: The user clicks on the link provided to download the generated baby voice audio file and listen to the sound.

[0598] Step 9:

[0599] Device: Provides an interface that allows users to answer a detailed questionnaire about their living environment, preferences, and needs, including questions about their family structure, their attitude toward child-rearing, and their daily life challenges.

[0600] Step 10:

[0601] User: Completes a survey, inputting their preferences and needs regarding lifestyle and childcare methods. Once completed, clicks the submit button.

[0602] Step 11:

[0603] Terminal: The survey data entered by the user is sent to the server. Encryption protocols are used to ensure the security of the information when sending the data.

[0604] Step 12:

[0605] Server: Collects the received survey data and runs a machine learning model for detailed analysis. This model categorizes user responses into categories and scores to understand user attributes and needs.

[0606] Step 13:

[0607] Server: Using the emotion engine, the system analyzes the user's emotional state from their facial expressions and text responses when filling out questionnaires and using the system. This data can categorize the user's emotions and evaluate them in real time.

[0608] Step 14:

[0609] Server: Based on the analysis results of the emotion engine, the server proposes lifestyle and child-rearing methods that take into account the user's psychological state and emotions. For example, if the user is feeling tired or stressed, it will suggest relaxation methods or information on how to join support groups.

[0610] Step 15:

[0611] Server: Integrates analysis and emotion recognition results to generate personalized lifestyle and parenting recommendations in the form of text, visual lists, graphs, etc.

[0612] Step 16:

[0613] Server: Notifies the user of the generated proposal and provides further information via a dedicated dashboard or email.

[0614] Step 17:

[0615] Device: Provides an interface to display the recommendations to the user and allows the user to view detailed recommendations, including, for example, video tutorials of the suggested parenting techniques and links to local support groups.

[0616] Step 18:

[0617] User: Check the suggestions, put into practice specific lifestyle and child-rearing methods, and provide feedback within the interface as needed.

[0618] Step 19:

[0619] Device: Sends user-entered feedback to the server, which uses it to improve the accuracy of future suggestions.

[0620] Step 20:

[0621] Server: Receives feedback data and stores it in a database, which feeds the machine learning model for future improvements.

[0622] These are the specific processing steps of the Echo Voice AI system, which combines an emotion engine. Through this series of processes, users can experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0623] Example 2

[0624] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0625] Conventional systems have problems with not being able to fully utilize the information obtained from ultrasound images, and are unable to appropriately reduce the anxiety and stress felt by users. Furthermore, the suggestions provided to users are general and not optimized for each user's living environment or psychological state, resulting in insufficient personalized support. Furthermore, the baby's voice, generated based on the analysis of ultrasound images, is not provided in real time, and users' expectations cannot be met.

[0626] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0627] In this invention, the server includes means for preprocessing the echo image, means for extracting feature data from the echo image, and means for generating voice data based on the feature data. This makes it possible to effectively extract information from the echo image and provide personalized support based on the user's living environment and psychological state. Furthermore, by generating and providing voice data in real time, it is possible to meet user expectations.

[0628] An "echo image" is a medical image that uses ultrasound technology to visually record the internal structure and movements of a fetus.

[0629] "Preprocessing" is a technique for performing initial processing such as noise removal, contrast adjustment, and size compression to make data easier to analyze.

[0630] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0631] "Voice data" refers to data of a baby's voice generated based on feature data extracted from an echo image.

[0632] "User Information" means data about a user's living environment, preferences, and needs, and is information collected in the form of a questionnaire or similar.

[0633] "Analysis" is the process of analyzing collected data using machine learning models and statistical methods to derive specific trends and patterns.

[0634] "Mental state recognition" is the process of extracting and understanding a user's emotions and psychological state from questionnaire responses, facial expression analysis, etc.

[0635] "Lifestyle suggestions" are advice on optimal lifestyle habits and child-rearing methods provided based on the user's living environment and psychological state.

[0636] "Provision" refers to the process of notifying the user of the generated voice data and lifestyle suggestions in a viewable form.

[0637] A "machine learning model" is an algorithm that learns from large amounts of data and automatically performs specific tasks (such as generating voice data or analyzing user information).

[0638] MODE FOR CARRYING OUT THE INVENTION

[0639] This invention is a system that generates a baby's voice based on an ultrasound image and suggests a personalized lifestyle and child-rearing method based on the user's living environment and psychological state. Each processing step of this system and the necessary hardware and software are described in detail below.

[0640] Uploading ultrasound images

[0641] Users access a dedicated website or application, select and upload echo images, and use a smartphone or computer to do this. Specifically, users select images from their smartphone's camera roll or a folder on their computer and click the "Upload" button. The uploaded images are then sent to the server.

[0642] Echo image preprocessing

[0643] The device preprocesses the received echo images. Specifically, it compresses the image size and adds metadata required for compression (e.g., shooting date and time, file format, etc.). This process uses the OpenCV library. This makes image data transfer efficient and ensures smooth processing on the server.

[0644] Echo image analysis

[0645] The server performs noise reduction and contrast adjustment on the preprocessed images, then uses a machine learning model (e.g., a TensorFlow or PyTorch-based model) to extract feature data such as the shape, size, and position of the fetus from the image, and uses the OpenCV library for filtering and edge detection at this stage.

[0646] Baby voice generation

[0647] The server generates voice data using a voice generation model (e.g., WaveNet or Tacotron) based on the extracted feature data. Specifically, it uses the fetus's shape data as input to synthesize the baby's voice. This voice data is saved as an audio file (e.g., WAV format) and associated with the user's account.

[0648] Entering User Data

[0649] Users enter information about their living environment, preferences, and needs in a questionnaire format, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?" The entered data is sent to the server.

[0650] Preprocessing and analysis of user data

[0651] The device preprocesses the data sent by the user. Specifically, it structures the data in JSON format. The server then uses a machine learning model (e.g., scikit-learn or XGBoost) to classify the received data into categories and scores. For example, "time spent at home" is classified into two categories, "a lot" and "a little," and "interest in childcare methods" is scored.

[0652] Applying emotion engines and recognizing mental states

[0653] The server uses an emotion engine to recognize the user's psychological state. For example, it performs facial expression analysis (using OpenFace or Dlib) and voice analysis (using Praat or DeepSpeech) to understand the user's emotional state.

[0654] Proposal of the best lifestyle

[0655] The server then proposes optimal lifestyle and child-rearing methods based on the results of analyzing user data and recognizing their psychological state. Suggestions are generated in the form of text, lists, graphs, and more. For example, they include specific advice such as "incorporate relaxation time into your daily routine" or "take advantage of local support groups." The generated suggestions are notified to the user via a dedicated dashboard or email.

[0656] The device provides an interface that displays the proposal content, allowing users to check the details of the proposal. Users can view the proposal content on their smartphones or computers and obtain the necessary information.

[0657] Examples of concrete examples and prompts

[0658] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. Specifically, it will suggest "methods for reducing stress" and "information on joining local support groups." It will also provide data on the generated baby's voice, which will help User B imagine what it will be like to raise a child in the future.

[0659] An example of a prompt is, "Based on the ultrasound image and information about the user's living environment, generate a baby's voice and suggest the optimal lifestyle. If the user is a single parent and is feeling anxious, please also suggest additional support methods."

[0660] This allows the invention to generate a baby's voice based on the ultrasound image and provide personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration. By receiving suggestions optimized for their own living environment, users can reduce their anxiety about child-rearing and approach child-rearing with confidence.

[0661] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0662] Step 1: Upload your ultrasound images

[0663] Users access a dedicated website or application, select and upload echo images from their smartphone's camera roll or a folder on their computer. The input is the echo image file, and the output is the echo image data sent to the server. When the user clicks the "Upload" button, the image data is transferred to the server.

[0664] Step 2: Preprocessing of echo images

[0665] The terminal preprocesses the received echo images. The input is the echo image data sent by the user, and the output is the preprocessed image data. This preprocessing includes image size compression, noise removal, and contrast adjustment using the OpenCV library. This adds image metadata and improves transmission efficiency.

[0666] Step 3: Echo image analysis

[0667] The server analyzes the preprocessed echo images. The input is the preprocessed image data, and the output is feature data such as the shape, size, and position of the fetus. The server uses the OpenCV library to perform filtering and edge detection, and then applies TensorFlow or PyTorch-based machine learning models to extract the feature data.

[0668] Step 4: Generating a baby voice

[0669] The server runs a speech generation model based on the extracted feature data. The input is the feature data, and the output is an audio file of the baby's voice. Specifically, it uses WaveNet and Tacotron models to synthesize the baby's voice from the fetal shape data. This voice data is saved as an audio file and associated with the user's account.

[0670] Step 5: Enter user data

[0671] Users enter information about their living environment, preferences, and needs in the form of a questionnaire. The input is the questionnaire data answered by the user, and the output is user data that is structured and sent to the server. Specifically, users answer questions such as "how much time do you spend at home?" and "interests in childcare methods."

[0672] Step 6: Preprocessing and analyzing user data

[0673] The terminal preprocesses the survey data obtained from the user. The input is the survey data, and the output is the preprocessed data. The data is structured in JSON format. The server then analyzes the data using machine learning models such as scikit-learn and XGBoost, classifying user attributes into categories and scores.

[0674] Step 7: Applying the Emotion Engine and Recognizing Mental States

[0675] The server uses an emotion engine to recognize the user's psychological state. The input is the user's questionnaire responses, behavioral data, and facial expression analysis data, and the output is the user's emotional information. Facial expression analysis is performed using OpenFace or Dlib, and speech analysis is performed using Praat or DeepSpeech. The recognized emotional information is obtained.

[0676] Step 8: Proposal for the optimal lifestyle

[0677] The server proposes optimal lifestyle and child-rearing methods based on the analysis results of user data and emotional information. The input is the analysis results and emotional information, and the output is the proposals. Specific proposals are generated in the form of text, lists, and graphs. The generated proposals are notified to the user via a dedicated dashboard or email.

[0678] Step 9: View the proposal

[0679] The terminal provides an interface that displays the proposal content to the user. The input is the proposal content, and the output is the interface displayed to the user. The user can view the detailed proposal content using a smartphone or PC and obtain the necessary information.

[0680] (Application example 2)

[0681] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] Conventional childcare support systems based on ultrasound images have difficulty making personalized suggestions that take into account the user's emotional state, leading to a lack of confidence in users' childcare strategies. Another issue is that there is little correlation between the baby's voice generated from ultrasound images and the childcare suggestions, meaning users are not provided with sufficient information. Furthermore, lifestyle and childcare method suggestions are static and cannot be dynamically adjusted to suit the user's situation or emotions, making it difficult to provide efficient suggestions in physical stores.

[0683] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0684] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for using an emotion engine to recognize the user's emotional state, means for optimizing lifestyle and childcare method suggestions based on the emotion recognition results, and means for providing the voice data and the suggestions to the user, thereby enabling dynamic lifestyle and childcare method suggestions based on the user's emotional state.

[0685] An "echo image" is an image showing the condition of a fetus obtained by an ultrasound diagnostic device.

[0686] "Feature data" refers to information such as the shape, size, and position of the fetus extracted from the echo image.

[0687] "Voice data" is digital audio information including a baby's voice generated based on feature data.

[0688] "User's living environment information" refers to information about the user's daily life, including family structure, living environment, occupation, and the like.

[0689] "Preferences" refers to the user's preferences and intentions regarding child-rearing.

[0690] "Needs" refers to the user's requests and demands regarding childcare.

[0691] An "emotion engine" is a software or hardware technology that recognizes a user's emotional state from their text input, facial expressions, and voice.

[0692] "Lifestyle and child-rearing suggestions" refers to recommending optimal lifestyle habits and child-rearing methods for a user based on the user's living environment information, emotional state, and analysis results.

[0693] "Optimize" means tailoring the recommendations to provide more relevant advice based on the user's personalized information.

[0694] This invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. By combining it with an emotion engine that recognizes the user's emotions, it achieves highly accurate personalization. Specific embodiments for implementing this system are described below.

[0695] Users first capture echo images using their smartphones and then upload them using a dedicated application. The uploaded echo images are then sent to a server. To ensure efficient image transfer, the echo images are pre-processed on the device to compress the image size and add required metadata.

[0696] The server applies image pre-processing algorithms to analyze the received echo images, such as noise reduction and contrast adjustment, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[0697] The server then runs a speech generation model on the feature data to generate baby voice data, which is then saved as an audio file and associated with the user's account.

[0698] The user then enters information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as, "Do you spend a lot of time at home?" and "What kind of child-rearing methods are you interested in?" The device then sends this user-entered information to the server.

[0699] The server runs a machine learning model to analyze the received user data, organizing the user's input data into categories and scores to gain a detailed understanding of the user's attributes.

[0700] The server then uses an emotion engine to recognize the user's emotional state. This involves extracting emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The server then reflects the emotion recognition results in the user's lifestyle and childcare recommendations, optimizing them. For example, if the user is feeling stressed, the server will add suggestions for relaxation methods.

[0701] Based on the analysis and emotion recognition results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. The suggestions are generated in the form of text, lists, graphs, etc. The generated suggestions are notified to the user via a dedicated dashboard or email, and can be viewed.

[0702] For example, suppose User A uploads an ultrasound image to the system and enters "both spouses work, so there is little anxiety about childcare" as living environment information. If the emotion engine analyzes User A's input and behavior and recognizes that stress is low, the system will reflect this and make specific suggestions. For example, User A may be offered nutritionally balanced meal menus or local childcare support information. User A will also be provided with data on the generated baby's voice, giving User A more confidence in their future childcare.

[0703] Examples of prompts include:

[0704] "We are developing a system that generates a baby's voice based on ultrasound images and suggests child-rearing methods and lifestyles that are tailored to the user's living environment and psychological state. With this system, users can upload ultrasound images using their smartphone, listen to their baby's voice, and receive optimal lifestyle suggestions. The user's psychological state is analyzed using an emotion engine, and suggestions are dynamically adjusted according to the user's stress level."

[0705] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0706] Step 1:

[0707] The user launches a dedicated application on their smartphone to acquire an echo image. The user selects an echo image and uploads it through the application. The input is the echo image, and the output is an image file transferred to the server. The device compresses the image file size, adds required metadata, and efficiently transmits it to the server.

[0708] Step 2:

[0709] The server runs the received echo images through an image pre-processing algorithm. The input is a compressed echo image, and the output is a pre-processed echo image. This pre-processing includes noise removal and contrast adjustment, which improves image quality and allows for accurate subsequent processing.

[0710] Step 3:

[0711] The server runs a machine learning model to extract feature data from the preprocessed echo images. The input is the preprocessed echo images, and the output is feature data. This feature data includes the shape, size, and position of the fetus. The machine learning model efficiently analyzes important information in the echo images.

[0712] Step 4:

[0713] The server runs a speech generation model based on the feature data to generate baby voice data. The input is the feature data, and the output is the baby's voice data. This voice data is saved as an audio file. The speech generation model analyzes the feature data and creates an audio file that imitates the baby's voice.

[0714] Step 5:

[0715] The user enters information about their living environment, preferences, and needs in a questionnaire format. The input is information about the living environment, preferences, and needs, and the output is the questionnaire data sent to the server. The terminal sends the user's input to the server.

[0716] Step 6:

[0717] The server runs a machine learning model to analyze the received user data, such as living environment information, preferences, and needs. The input is the survey data, and the output is the analysis results. The machine learning model organizes this data into categories and scores to gain a detailed understanding of the user's attributes.

[0718] Step 7:

[0719] The server recognizes the user's emotional state using an emotion engine that extracts emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The input is data related to the user's emotions, and the output is the emotion recognition results. The emotion engine analyzes the emotional aspects of the data entered by the user and grasps the user's current psychological state.

[0720] Step 8:

[0721] The server optimizes lifestyle and child-rearing method suggestions based on the emotion recognition results. The inputs are the analysis results and emotion recognition results, and the output is optimized suggestion data. The suggestion content is dynamically adjusted according to the emotion recognition results.

[0722] Step 9:

[0723] The server provides the generated voice data and optimized lifestyle suggestions to the user. The input is the voice data and suggestion data, and the output is the information provided to the user. The information is notified to the user via a dedicated dashboard or email, and can be viewed.

[0724] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0725] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0726] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0727] [Third embodiment]

[0728] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0729] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0730] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0731] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0732] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0733] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0734] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0735] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0736] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0737] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0738] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0739] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0740] The system of the present invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. This system involves a series of processes, starting with the user uploading the ultrasound image and ending with providing the generated baby's voice and lifestyle suggestions.

[0741] Program processing

[0742] 1. Uploading ultrasound images

[0743] User: First, the user accesses the system's website or application and uploads their own echo images, which are then sent to the server.

[0744] Terminal: Provides an interface for sending user-selected echo images to the server.

[0745] 2. Analysis of Echo Images

[0746] Server: Upon receiving the echo images, the server applies image pre-processing algorithms to remove noise and adjust contrast, making the echo images ready for analysis. Then, it uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images.

[0747] 3. Baby voice generation

[0748] Server: Runs a speech generation model based on the feature data. This model has learned from past baby voice data and generates speech data based on the given feature data. The generated speech data is saved as an audio file and linked to the user's account.

[0749] 4. Enter user data

[0750] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0751] Terminal: Provides an interface for sending user-entered information to the server.

[0752] 5. User data analysis

[0753] Server: Runs machine learning models to analyze the received user data in detail. The models organize the user input data into categories and scores to gain a detailed understanding of the user's attributes.

[0754] 6. Proposal for the best lifestyle

[0755] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. For example, for users living in urban areas, the server suggests nearby parent-child support facilities and recommended activities. It also suggests lifestyle schedules that match the baby's sleep rhythm.

[0756] Server: Notifies the user of the generated suggestions and provides audio files and lifestyle suggestions.

[0757] Terminal: An interface is provided to display the received proposals to the user, allowing the user to view the proposals in detail.

[0758] Specific examples

[0759] For example, suppose User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information. The server analyzes this information and suggests the use of childcare support services. The server also generates baby voice data and provides it to User A, allowing them to feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence.

[0760] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and provides personalized lifestyle suggestions that meet the needs of each user, providing an environment in which parents can approach child-rearing with confidence.

[0761] The processing flow will be explained below.

[0762] Step 1:

[0763] User: The user visits the Echo Voice AI website or app, selects their Echo image, and clicks the upload button. The user also enters basic information such as name and email address.

[0764] Step 2:

[0765] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0766] Step 3:

[0767] Terminal: Sends pre-processed echo image data to the server. Data security is ensured by using encryption protocols during transmission.

[0768] Step 4:

[0769] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust the contrast, thereby improving the image quality.

[0770] Step 5:

[0771] Server: Runs machine learning models to extract feature data from preprocessed ultrasound images. Specifically, it detects important features such as the shape, position, and size of the fetus and records them as numerical data.

[0772] Step 6:

[0773] Server: Runs a speech generation model based on the feature data. Based on past baby voice data, generates a voice that is closest to the current feature data. Saves this voice data as an audio file (e.g., MP3).

[0774] Step 7:

[0775] Server: Binds the generated audio file to the user's account, notifies the user of the download link, and stores the audio file in secure storage.

[0776] Step 8:

[0777] User: The user clicks on the link provided and downloads the generated baby voice audio file to check it.

[0778] Step 9:

[0779] Device: Provides an interface where users can complete a questionnaire about their living environment, preferences, and needs. For example, questions might include, "What is your family structure?" and "What is your approach to childcare?"

[0780] Step 10:

[0781] User: Answers a questionnaire about their living environment, preferences, and needs, and clicks the submit button.

[0782] Step 11:

[0783] Terminal: The survey data entered by the user is sent to the server. The data is encrypted and sent.

[0784] Step 12:

[0785] Server: Runs a machine learning model to analyze the received survey data. The model classifies user responses into categories and scores, and analyzes lifestyle and parenting characteristics.

[0786] Step 13:

[0787] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's needs. The proposals are generated in the form of text, lists, graphs, etc.

[0788] Step 14:

[0789] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0790] Step 15:

[0791] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0792] Step 16:

[0793] User: Check the suggestions, put them into practice with specific lifestyle and child-rearing methods, and provide feedback as needed.

[0794] Step 17:

[0795] Terminal: Sends the feedback entered by the user to the server. The feedback will be used to improve the service in the future.

[0796] Step 18:

[0797] Server: Receives feedback data and stores it in a database. The stored data is fed into the machine learning model and used to improve the accuracy of future suggestions.

[0798] These are the specific processing steps of the Echo Voice AI system, which allows users to experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0799] Example 1

[0800] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0801] Conventional ultrasound image analysis systems do not generate baby voices or provide personalized recommendations for optimal childcare lifestyles, making it difficult for users to obtain childcare advice tailored to their own living environment. Furthermore, conventional systems do not utilize machine learning models to generate voice data, which leads to issues with the realism of the generated voice data and low user satisfaction.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0803] In this invention, the server includes means for receiving echo images, means for preprocessing the echo images, means for extracting feature data from the echo images, means for generating voice data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods, and means for providing the voice data and suggestions to the user, thereby enabling the user to realistically experience the baby's voice from the echo images and receive suggestions on a child-rearing lifestyle that is optimal for their own living environment.

[0804] "Echo image" refers to an image taken using ultrasound to check the condition of the fetus.

[0805] "Preprocessing" refers to the initial processing steps required to prepare the image for analysis, such as removing noise from the echo image and adjusting the contrast.

[0806] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0807] "Audio data" refers to digital audio data of a baby's voice created from feature data of an ultrasound image using a generative AI model.

[0808] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of baby voice data and generates realistic voices based on specific feature data.

[0809] "User's living environment information" refers to individual information provided by the user related to childcare, such as the user's living situation and residential environment.

[0810] "Survey format" refers to a method in which users respond to questions by selecting options or writing a statement.

[0811] "Lifestyle" and "parenting methods" refer to behavioral patterns and specific parenting advice and methods that are considered optimal for the user and their baby's lifestyle.

[0812] The system of the present invention involves a series of processes that begin with the user uploading an ultrasound image and ultimately provide a generated baby voice and optimal lifestyle recommendations. The specific hardware and software used in each process step are described in detail below.

[0813] First, the user accesses the system's website or application using a device such as a PC or smartphone. The user clicks the "Upload Echo Image" button on the screen, selects an echo image file from their device, and uploads it. The device provides an interface for reading the echo image file and sending it to the server. It is desirable that a progress bar be displayed to indicate the progress of the upload.

[0814] The server temporarily stores the received echo images. It then applies image pre-processing algorithms to remove noise and adjust the contrast. Specifically, it uses noise removal and contrast adjustment algorithms such as a median filter. It then uses a deep learning model (e.g., U-Net) to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0815] The server runs a speech generation model (e.g., WaveNet) based on the extracted feature data. This speech generation model has learned from past baby voice data and generates a realistic baby voice from the input data. The generated voice data is saved as an audio file (e.g., MP3 format) and linked to the user's account.

[0816] Next, the user answers a questionnaire about their living environment, preferences, and needs. The device provides a questionnaire-style user interface and transmits the information the user has entered to the server, displaying a progress bar while the data is being transmitted.

[0817] The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. This model categorizes the user data and assigns a score to each data point to gain a detailed understanding of the user's living environment and needs. For example, it can organize information such as "dual-income household" and "limited childcare time."

[0818] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, it generates specific suggestions such as "use local childcare support services available on weekends" or "create a daily schedule that matches the baby's sleep rhythm." The server then links the generated suggestions and audio files to the user's account and sends a notification. Users can check this information when they log in.

[0819] The terminal provides an interface that displays the suggestions and the generated baby voice to the user, who can then check the suggestions in detail and download or print them as needed.

[0820] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the server will analyze this information and suggest the use of childcare support services. Furthermore, by generating baby voice data and providing it to User A, User A will be able to feel the baby's presence even before it is born. Furthermore, the server will also suggest childcare methods and specific schedules suitable for dual-income households, allowing User A to make plans with confidence regarding childcare.

[0821] An example of a prompt would be the following text:

[0822] "We are a dual-income household, and the time we can devote to childcare is limited. Please suggest baby voice data and childcare methods that are suitable for our living environment."

[0823] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0824] Step 1:

[0825] Uploading ultrasound images

[0826] User: The user visits the system's website or application and clicks the "Upload Echo Image" button. As input, there is an echo image file selected by the user from the device. This file will be uploaded to the system.

[0827] Terminal: The terminal provides an interface to read the selected echo image file and send it to the server. Specifically, it sends the image data to the server using an HTTP request and displays a progress bar to indicate the upload progress to the user. As an output, the image data is sent to the server.

[0828] Step 2:

[0829] Echo image preprocessing

[0830] Server: The server temporarily stores the received echo images. The input is the uploaded echo image file. Then, image pre-processing algorithms are applied to remove noise and adjust contrast. Specifically, a median filter is used to remove noise, and histogram equalization is used to adjust contrast. The output is the processed echo image.

[0831] Step 3:

[0832] Extracting feature data from echo images

[0833] Server: The server extracts feature data from the preprocessed echo images using a deep learning model (e.g., U-Net). The input is the preprocessed echo images. The model extracts specific information from the echo images, such as the shape, size, and position of the fetus. The output is the fetal feature data.

[0834] Step 4:

[0835] Baby voice generation

[0836] Server: The server inputs the extracted feature data into a speech generation model (e.g., WaveNet). The input is the fetus's feature data. This speech generation model learns from past baby voice data to generate a realistic baby voice. The output is an MP3 audio file that is associated with the user's account.

[0837] Step 5:

[0838] Entering User Data

[0839] User: The user answers survey-style questions provided within the system. Input includes information about the user's living environment, preferences, and needs. For example, questions include "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0840] Terminal: The terminal provides a questionnaire-style user interface and sends the information entered by the user to the server. Specifically, an HTTP request is issued to collect form data and send it to the server.

[0841] Step 6:

[0842] User data analysis

[0843] Server: The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. The input is user data collected from surveys. The model classifies the data into categories and assigns a score to each data point to gain a detailed understanding of the user's attributes. The output is the analysis results.

[0844] Step 7:

[0845] Proposal of the best lifestyle

[0846] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that are best suited to the user's living environment. The inputs are the analysis results and user data. For example, it generates specific suggestions such as "use local child-rearing support services available on weekends" and "create a daily schedule that matches your baby's sleep rhythm." The output is a lifestyle suggestion.

[0847] Server: Binds the generated suggestions and audio files to the user's account and sends notifications. When the user logs in, this information is displayed.

[0848] Step 8:

[0849] Display of suggestions and audio

[0850] Terminal: The terminal provides an interface that displays the suggestions and the generated baby voice to the user. The inputs are the suggestions and audio files sent from the server. Specifically, a detail page is generated using HTML / CSS, allowing the user to check the suggestions in detail and download or print them if necessary. The output is the suggestions displayed to the user and the audio playback.

[0851] (Application example 1)

[0852] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0853] Conventional baby ultrasound image analysis and voice generation systems only focus on helping people feel the baby's presence, and do not address safety and security measures within the home. The purpose of this invention is to provide a means to ensure safety and security within the home while feeling the baby's presence by adding a home security system linking function to a system that generates a baby's voice from an ultrasound image.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0855] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods based on the analysis results, means for providing the voice data and the proposals to the user, means for setting the generated voice data in a home security system, and means for detecting abnormalities in conjunction with the set security system and notifying the user. This makes it possible to detect abnormalities in the home using a baby's voice and promptly notify the user.

[0856] An "echo image" is an image of a fetus or internal organs taken using a medical ultrasound device.

[0857] "Feature data" refers to data such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0858] "Voice data" is data of a baby's voice generated based on feature data.

[0859] "User's living environment information" is information about the environment in which the user lives, including, for example, the size and environment of the residence, whether it is a dual-income household, and so on.

[0860] "User Preferences" is information about a user's parenting policies and priorities.

[0861] "User needs" refers to information about the user's requests for help, services, etc. related to childcare.

[0862] "Lifestyle suggestions" are suggestions for optimal lifestyle habits and child-rearing methods created based on the user's living environment, preferences, and needs.

[0863] A "home security system" is a system that detects abnormalities within the home and notifies the user.

[0864] A "generative AI model" is a machine learning model that learns from past data and generates new data.

[0865] A "prompt sentence" is a sentence that prompts input or instructions to the system.

[0866] The following procedure is provided as an embodiment of the present invention.

[0867] First, users use a smartphone application to upload ultrasound images of their baby, which are then sent to a server through the application's interface.

[0868] The server applies image pre-processing algorithms, such as noise reduction and contrast adjustment, to the received echo images to prepare them for analysis, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[0869] The server then runs a speech generation model based on the extracted feature data to generate a baby's voice. The model uses past baby voice data that it has already learned from to generate speech data from the feature data. The generated speech data is saved as an audio file and linked to the user's account.

[0870] Next, users enter information about their living environment, preferences, and needs in a questionnaire format via a smartphone application, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?"

[0871] The server receives the living environment information entered by the user and analyzes it using a machine learning model. This analysis organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes. Based on the analysis results, the system suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, the system suggests nearby parent-child support facilities and recommended activities, as well as a daily schedule that matches the baby's sleep rhythm. The generated suggestions and audio data are notified to the user, and can be viewed through the smartphone application interface.

[0872] Furthermore, the generated voice data is also set up in a home security system. This security system detects abnormalities in the home based on the baby's voice and sends a notification to the user. Sensors used include infrared cameras, voice recognition sensors, and vibration sensors. When an abnormality is detected, a smartphone application sends a notification to the user in real time.

[0873] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the system will suggest the use of a childcare support service. Furthermore, by generating baby voice data and providing it to User A, User A can feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence. In this way, the system of the present invention not only generates baby voices and offers personalized lifestyle suggestions tailored to each user's needs, but also provides safety and security within the home.

[0874] Example prompt:

[0875] "Upload an ultrasound image and we'll automatically generate a baby voice and set it up in your home security system."

[0876] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0877] Step 1:

[0878] Uploading ultrasound images

[0879] The user uploads an ultrasound image of the baby through the interface of a smartphone application. The uploaded ultrasound image is sent from the device to the server. Specifically, the user selects an ultrasound image on the device, and the device sends the data to the server via the Internet. The input is the ultrasound image selected by the user, and the output is the ultrasound image data stored on the server.

[0880] Step 2:

[0881] Echo image analysis

[0882] The server applies image pre-processing algorithms to the received echo images to remove noise and adjust contrast, making the echo images ready for analysis. The server then uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images. The input is the echo image data received by the server, and the output is the extracted feature data.

[0883] Step 3:

[0884] Baby voice generation

[0885] The server runs a speech generation model based on the extracted feature data. This model has been trained on past baby voice data and generates new speech data based on the feature data. This generation process saves the baby's voice as an audio file. The input is the extracted feature data, and the output is the generated speech data (audio file).

[0886] Step 4:

[0887] Entering User Data

[0888] Users use a smartphone application to input information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?" The input is the living environment information, preferences, and needs entered by the user, and the output is user data sent to the server.

[0889] Step 5:

[0890] User data analysis

[0891] The server runs a machine learning model to perform a detailed analysis of the data entered by the user regarding their living environment, preferences, and needs. The model organizes the user data into categories and scores to gain a detailed understanding of the user's attributes. The input is the user data received by the server, and the output is the analysis results (details of the user's attributes).

[0892] Step 6:

[0893] Proposals for optimal lifestyle and child-rearing methods

[0894] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, it suggests nearby parent-child support facilities and recommended activities. It also suggests a daily schedule that matches the baby's sleep rhythm. The input is the analysis results, and the output is the suggested lifestyle and child-rearing methods.

[0895] Step 7:

[0896] Proposal and voice data provided

[0897] The server notifies the user of the generated suggestions and voice data and provides them through the smartphone application interface. The user can view the detailed suggestions and baby voice data. The input is the generated suggestions and voice data, and the output is the information provided to the user.

[0898] Step 8:

[0899] Setting up a home security system

[0900] Based on the generated voice data, the user can configure the home security system. An interface for configuring the system is provided through a smartphone application. The input is the generated voice data, and the output is the configured security system.

[0901] Step 9:

[0902] Anomaly detection and notification

[0903] The security system detects abnormalities based on the baby's voice that has been set. For example, it uses a voice recognition sensor, infrared camera, or vibration sensor to notify the user in real time when an abnormality occurs. The input is the information detected by the sensor, and the output is an abnormality notification to the user.

[0904] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0905] The system of the present invention is a generative AI system that generates a baby's voice from ultrasound images and suggests optimal lifestyles, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more accurate personalization. This system involves a series of processes, starting with the user uploading the ultrasound images and ending with dynamically adjusting lifestyle suggestions using the emotion engine.

[0906] Program processing

[0907] 1. Uploading ultrasound images

[0908] User: First, the user accesses the Echo Voice AI website or app, selects and uploads their own Echo image, and the uploaded Echo image is sent to the server.

[0909] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[0910] 2. Analysis of Echo Images

[0911] Server: Upon receiving the echo images, it applies image pre-processing algorithms to remove noise and adjust contrast, then uses machine learning models to extract feature data such as the shape, size, and position of the fetus from the echo images.

[0912] 3. Baby voice generation

[0913] Server: Based on the feature data, a speech generation model is run to generate baby voice data. This voice data is saved as an audio file and linked to the user's account.

[0914] 4. Enter user data

[0915] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[0916] Terminal: Sends the information entered by the user to the server.

[0917] 5. User data analysis

[0918] Server: Runs machine learning models to analyze the received user data, organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes.

[0919] 6. Applying the Emotion Engine

[0920] Server: Recognizes the user's emotional state using an emotion engine. This involves extracting emotions from the user's facial expressions, voice, and text input when answering surveys or using the system.

[0921] Server: The emotion recognition results are reflected in the user's lifestyle and childcare suggestions, optimizing them. For example, if the user is feeling stressed, the server adds suggestions for relaxation methods.

[0922] 7. Proposal for the best lifestyle

[0923] Server: Based on the analysis and emotion recognition results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. The proposals are generated in the form of text, lists, graphs, etc.

[0924] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[0925] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[0926] Specific examples

[0927] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. For example, User B could be offered suggestions such as ways to reduce stress or information about joining a local support group. User B will also be provided with data on the generated baby's voice, which will help User B imagine what it would be like to raise a child in the future.

[0928] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and also provides personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration, providing an environment in which parents can approach child-rearing with confidence.

[0929] The processing flow will be explained below.

[0930] Step 1:

[0931] User: The user visits the Echo Voice AI website or app, logs in to their account, selects an echo image, and clicks the upload button to upload the echo image to the system.

[0932] Step 2:

[0933] Terminal: Preprocessing the uploaded echo images involves optimizing the image resolution, compressing the file size, and adding necessary metadata to the images (e.g., capture date and time, device information, etc.).

[0934] Step 3:

[0935] Terminal: Sends preprocessed echo image data to the server, using an encryption protocol (e.g., TLS) to ensure data security.

[0936] Step 4:

[0937] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust color and contrast, thereby improving image quality.

[0938] Step 5:

[0939] Server: Extracts feature data such as fetal shape, size, and position from preprocessed ultrasound images. This is done using machine learning models, and records interesting features as numerical data.

[0940] Step 6:

[0941] Server: Runs a speech generation model based on the extracted features. This model is based on past baby speech data. A speech generation algorithm is used to create a speech that most closely matches the current features.

[0942] Step 7:

[0943] Server: Saves the generated audio data in an audio file format (e.g., MP3), associates it with the user's account, and notifies the user of the download link for the audio file.

[0944] Step 8:

[0945] User: The user clicks on the link provided to download the generated baby voice audio file and listen to the sound.

[0946] Step 9:

[0947] Device: Provides an interface that allows users to answer a detailed questionnaire about their living environment, preferences, and needs, including questions about their family structure, their attitude toward child-rearing, and their daily life challenges.

[0948] Step 10:

[0949] User: Completes a survey, inputting their preferences and needs regarding lifestyle and childcare methods. Once completed, clicks the submit button.

[0950] Step 11:

[0951] Terminal: The survey data entered by the user is sent to the server. Encryption protocols are used to ensure the security of the information when sending the data.

[0952] Step 12:

[0953] Server: Collects the received survey data and runs a machine learning model for detailed analysis. This model categorizes user responses into categories and scores to understand user attributes and needs.

[0954] Step 13:

[0955] Server: Using the emotion engine, the system analyzes the user's emotional state from their facial expressions and text responses when filling out questionnaires and using the system. This data can categorize the user's emotions and evaluate them in real time.

[0956] Step 14:

[0957] Server: Based on the analysis results of the emotion engine, the server proposes lifestyle and child-rearing methods that take into account the user's psychological state and emotions. For example, if the user is feeling tired or stressed, it will suggest relaxation methods or information on how to join support groups.

[0958] Step 15:

[0959] Server: Integrates analysis and emotion recognition results to generate personalized lifestyle and parenting recommendations in the form of text, visual lists, graphs, etc.

[0960] Step 16:

[0961] Server: Notifies the user of the generated proposal and provides further information via a dedicated dashboard or email.

[0962] Step 17:

[0963] Device: Provides an interface to display the recommendations to the user and allows the user to view detailed recommendations, including, for example, video tutorials of the suggested parenting techniques and links to local support groups.

[0964] Step 18:

[0965] User: Check the suggestions, put into practice specific lifestyle and child-rearing methods, and provide feedback within the interface as needed.

[0966] Step 19:

[0967] Device: Sends user-entered feedback to the server, which uses it to improve future suggestions.

[0968] Step 20:

[0969] Server: Receives feedback data and stores it in a database, which feeds the machine learning model for future improvements.

[0970] These are the specific processing steps of the Echo Voice AI system, which combines an emotion engine. Through this series of processes, users can experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[0971] Example 2

[0972] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0973] Conventional systems have problems with not being able to fully utilize the information obtained from ultrasound images, and are unable to appropriately reduce the anxiety and stress felt by users. Furthermore, the suggestions provided to users are general and not optimized for each user's living environment or psychological state, resulting in insufficient personalized support. Furthermore, the baby's voice, generated based on the analysis of ultrasound images, is not provided in real time, and users' expectations cannot be met.

[0974] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0975] In this invention, the server includes means for preprocessing the echo image, means for extracting feature data from the echo image, and means for generating voice data based on the feature data. This makes it possible to effectively extract information from the echo image and provide personalized support based on the user's living environment and psychological state. Furthermore, by generating and providing voice data in real time, it is possible to meet user expectations.

[0976] An "echo image" is a medical image that uses ultrasound technology to visually record the internal structure and movements of a fetus.

[0977] "Preprocessing" is a technique for performing initial processing such as noise removal, contrast adjustment, and size compression to make data easier to analyze.

[0978] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[0979] "Voice data" refers to data of a baby's voice generated based on feature data extracted from an echo image.

[0980] "User Information" means data about a user's living environment, preferences, and needs, and is information collected in the form of a questionnaire or similar.

[0981] "Analysis" is the process of analyzing collected data using machine learning models and statistical methods to derive specific trends and patterns.

[0982] "Mental state recognition" is the process of extracting and understanding a user's emotions and psychological state from questionnaire responses, facial expression analysis, etc.

[0983] "Lifestyle suggestions" are advice on optimal lifestyle habits and child-rearing methods provided based on the user's living environment and psychological state.

[0984] "Provision" refers to the process of notifying the user of the generated voice data and lifestyle suggestions in a viewable form.

[0985] A "machine learning model" is an algorithm that learns from large amounts of data and automatically performs specific tasks (such as generating voice data or analyzing user information).

[0986] MODE FOR CARRYING OUT THE INVENTION

[0987] This invention is a system that generates a baby's voice based on an ultrasound image and suggests a personalized lifestyle and child-rearing method based on the user's living environment and psychological state. Each processing step of this system and the necessary hardware and software are described in detail below.

[0988] Uploading ultrasound images

[0989] Users access a dedicated website or application, select and upload echo images, and use a smartphone or computer to do this. Specifically, users select images from their smartphone's camera roll or a folder on their computer and click the "Upload" button. The uploaded images are then sent to the server.

[0990] Echo image preprocessing

[0991] The device preprocesses the received echo images. Specifically, it compresses the image size and adds metadata required for compression (e.g., shooting date and time, file format, etc.). This process uses the OpenCV library. This makes image data transfer efficient and ensures smooth processing on the server.

[0992] Echo image analysis

[0993] The server performs noise reduction and contrast adjustment on the preprocessed images, then uses a machine learning model (e.g., a TensorFlow or PyTorch-based model) to extract feature data such as the shape, size, and position of the fetus from the image, and uses the OpenCV library for filtering and edge detection at this stage.

[0994] Baby voice generation

[0995] The server generates voice data using a voice generation model (e.g., WaveNet or Tacotron) based on the extracted feature data. Specifically, it uses the fetus's shape data as input to synthesize the baby's voice. This voice data is saved as an audio file (e.g., WAV format) and associated with the user's account.

[0996] Entering User Data

[0997] Users enter information about their living environment, preferences, and needs in a questionnaire format, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?" The entered data is sent to the server.

[0998] Preprocessing and analysis of user data

[0999] The device preprocesses the data sent by the user. Specifically, it structures the data in JSON format. The server then uses a machine learning model (e.g., scikit-learn or XGBoost) to classify the received data into categories and scores. For example, "time spent at home" is classified into two categories, "a lot" and "a little," and "interest in childcare methods" is scored.

[1000] Applying emotion engines and recognizing mental states

[1001] The server uses an emotion engine to recognize the user's psychological state. For example, it performs facial expression analysis (using OpenFace or Dlib) and voice analysis (using Praat or DeepSpeech) to understand the user's emotional state.

[1002] Proposal of the best lifestyle

[1003] The server then proposes optimal lifestyle and child-rearing methods based on the results of analyzing user data and recognizing their psychological state. Suggestions are generated in the form of text, lists, graphs, and more. For example, they include specific advice such as "incorporate relaxation time into your daily routine" or "take advantage of local support groups." The generated suggestions are notified to the user via a dedicated dashboard or email.

[1004] The device provides an interface that displays the proposal content, allowing users to check the details of the proposal. Users can view the proposal content on their smartphones or computers and obtain the necessary information.

[1005] Examples of concrete examples and prompts

[1006] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. Specifically, it will suggest "methods for reducing stress" and "information on joining local support groups." It will also provide data on the generated baby's voice, which will help User B imagine what it will be like to raise a child in the future.

[1007] An example of a prompt is, "Based on the ultrasound image and information about the user's living environment, generate a baby's voice and suggest the optimal lifestyle. If the user is a single parent and is feeling anxious, please also suggest additional support methods."

[1008] This allows the invention to generate a baby's voice based on the ultrasound image and provide personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration. By receiving suggestions optimized for their own living environment, users can reduce their anxiety about child-rearing and approach child-rearing with confidence.

[1009] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1010] Step 1: Upload your ultrasound images

[1011] Users access a dedicated website or application, select and upload echo images from their smartphone's camera roll or a folder on their computer. The input is the echo image file, and the output is the echo image data sent to the server. When the user clicks the "Upload" button, the image data is transferred to the server.

[1012] Step 2: Preprocessing of echo images

[1013] The terminal preprocesses the received echo images. The input is the echo image data sent by the user, and the output is the preprocessed image data. This preprocessing includes image size compression, noise removal, and contrast adjustment using the OpenCV library. This adds image metadata and improves transmission efficiency.

[1014] Step 3: Echo image analysis

[1015] The server analyzes the preprocessed echo images. The input is the preprocessed image data, and the output is feature data such as the shape, size, and position of the fetus. The server uses the OpenCV library to perform filtering and edge detection, and then applies TensorFlow or PyTorch-based machine learning models to extract the feature data.

[1016] Step 4: Generating a baby voice

[1017] The server runs a speech generation model based on the extracted feature data. The input is the feature data, and the output is an audio file of the baby's voice. Specifically, it uses WaveNet and Tacotron models to synthesize the baby's voice from the fetal shape data. This voice data is saved as an audio file and associated with the user's account.

[1018] Step 5: Enter user data

[1019] Users enter information about their living environment, preferences, and needs in the form of a questionnaire. The input is the questionnaire data answered by the user, and the output is user data that is structured and sent to the server. Specifically, users answer questions such as "how much time do you spend at home?" and "interests in childcare methods."

[1020] Step 6: Preprocessing and analyzing user data

[1021] The terminal preprocesses the survey data obtained from the user. The input is the survey data, and the output is the preprocessed data. The data is structured in JSON format. The server then analyzes the data using machine learning models such as scikit-learn and XGBoost, classifying user attributes into categories and scores.

[1022] Step 7: Applying the Emotion Engine and Recognizing Mental States

[1023] The server uses an emotion engine to recognize the user's psychological state. The input is the user's questionnaire responses, behavioral data, and facial expression analysis data, and the output is the user's emotional information. Facial expression analysis is performed using OpenFace or Dlib, and speech analysis is performed using Praat or DeepSpeech. The recognized emotional information is obtained.

[1024] Step 8: Proposal for the optimal lifestyle

[1025] The server proposes optimal lifestyle and child-rearing methods based on the analysis results of user data and emotional information. The input is the analysis results and emotional information, and the output is the proposals. Specific proposals are generated in the form of text, lists, and graphs. The generated proposals are notified to the user via a dedicated dashboard or email.

[1026] Step 9: View the proposal

[1027] The terminal provides an interface that displays the proposal content to the user. The input is the proposal content, and the output is the interface displayed to the user. The user can view the detailed proposal content using a smartphone or PC and obtain the necessary information.

[1028] (Application example 2)

[1029] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1030] Conventional childcare support systems based on ultrasound images have difficulty making personalized suggestions that take into account the user's emotional state, leading to a lack of confidence in users' childcare strategies. Another issue is that there is little correlation between the baby's voice generated from ultrasound images and the childcare suggestions, meaning users are not provided with sufficient information. Furthermore, lifestyle and childcare method suggestions are static and cannot be dynamically adjusted to suit the user's situation or emotions, making it difficult to provide efficient suggestions in physical stores.

[1031] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1032] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for using an emotion engine to recognize the user's emotional state, means for optimizing lifestyle and childcare method suggestions based on the emotion recognition results, and means for providing the voice data and the suggestions to the user, thereby enabling dynamic lifestyle and childcare method suggestions based on the user's emotional state.

[1033] An "echo image" is an image showing the condition of a fetus obtained by an ultrasound diagnostic device.

[1034] "Feature data" refers to information such as the shape, size, and position of the fetus extracted from the echo image.

[1035] "Voice data" is digital audio information including a baby's voice generated based on feature data.

[1036] "User's living environment information" refers to information about the user's daily life, including family structure, living environment, occupation, and the like.

[1037] "Preferences" refers to the user's preferences and intentions regarding child-rearing.

[1038] "Needs" refers to the user's requests and demands regarding childcare.

[1039] An "emotion engine" is a software or hardware technology that recognizes a user's emotional state from their text input, facial expressions, and voice.

[1040] "Lifestyle and child-rearing suggestions" refers to recommending optimal lifestyle habits and child-rearing methods for a user based on the user's living environment information, emotional state, and analysis results.

[1041] "Optimize" means tailoring the recommendations to provide more relevant advice based on the user's personalized information.

[1042] This invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. By combining it with an emotion engine that recognizes the user's emotions, it achieves highly accurate personalization. Specific embodiments for implementing this system are described below.

[1043] Users first capture echo images using their smartphones and then upload them using a dedicated application. The uploaded echo images are then sent to a server. To ensure efficient image transfer, the echo images are pre-processed on the device to compress the image size and add required metadata.

[1044] The server applies image pre-processing algorithms to analyze the received echo images, such as noise reduction and contrast adjustment, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[1045] The server then runs a speech generation model on the feature data to generate baby voice data, which is then saved as an audio file and associated with the user's account.

[1046] The user then enters information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as, "Do you spend a lot of time at home?" and "What kind of child-rearing methods are you interested in?" The device then sends this user-entered information to the server.

[1047] The server runs a machine learning model to analyze the received user data, organizing the user's input data into categories and scores to gain a detailed understanding of the user's attributes.

[1048] The server then uses an emotion engine to recognize the user's emotional state. This involves extracting emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The server then reflects the emotion recognition results in the user's lifestyle and childcare recommendations, optimizing them. For example, if the user is feeling stressed, the server will add suggestions for relaxation methods.

[1049] Based on the analysis and emotion recognition results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. The suggestions are generated in the form of text, lists, graphs, etc. The generated suggestions are notified to the user via a dedicated dashboard or email, and can be viewed.

[1050] For example, suppose User A uploads an ultrasound image to the system and enters "both spouses work, so there is little anxiety about childcare" as living environment information. If the emotion engine analyzes User A's input and behavior and recognizes that stress is low, the system will reflect this and make specific suggestions. For example, User A may be offered nutritionally balanced meal menus or local childcare support information. User A will also be provided with data on the generated baby's voice, giving User A more confidence in their future childcare.

[1051] Examples of prompts include:

[1052] "We are developing a system that generates a baby's voice based on ultrasound images and suggests child-rearing methods and lifestyles that are tailored to the user's living environment and psychological state. With this system, users can upload ultrasound images using their smartphone, listen to their baby's voice, and receive optimal lifestyle suggestions. The user's psychological state is analyzed using an emotion engine, and suggestions are dynamically adjusted according to the user's stress level."

[1053] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1054] Step 1:

[1055] The user launches a dedicated application on their smartphone to acquire an echo image. The user selects an echo image and uploads it through the application. The input is the echo image, and the output is an image file transferred to the server. The device compresses the image file size, adds required metadata, and efficiently transmits it to the server.

[1056] Step 2:

[1057] The server runs the received echo images through an image pre-processing algorithm. The input is a compressed echo image, and the output is a pre-processed echo image. This pre-processing includes noise removal and contrast adjustment, which improves image quality and allows for accurate subsequent processing.

[1058] Step 3:

[1059] The server runs a machine learning model to extract feature data from the preprocessed echo images. The input is the preprocessed echo images, and the output is feature data. This feature data includes the shape, size, and position of the fetus. The machine learning model efficiently analyzes important information in the echo images.

[1060] Step 4:

[1061] The server runs a speech generation model based on the feature data to generate baby voice data. The input is the feature data, and the output is the baby's voice data. This voice data is saved as an audio file. The speech generation model analyzes the feature data and creates an audio file that imitates the baby's voice.

[1062] Step 5:

[1063] The user enters information about their living environment, preferences, and needs in a questionnaire format. The input is information about the living environment, preferences, and needs, and the output is the questionnaire data sent to the server. The terminal sends the user's input to the server.

[1064] Step 6:

[1065] The server runs a machine learning model to analyze the received user data, such as living environment information, preferences, and needs. The input is the survey data, and the output is the analysis results. The machine learning model organizes this data into categories and scores to gain a detailed understanding of the user's attributes.

[1066] Step 7:

[1067] The server recognizes the user's emotional state using an emotion engine that extracts emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The input is data related to the user's emotions, and the output is the emotion recognition results. The emotion engine analyzes the emotional aspects of the data entered by the user and grasps the user's current psychological state.

[1068] Step 8:

[1069] The server optimizes lifestyle and child-rearing method suggestions based on the emotion recognition results. The inputs are the analysis results and emotion recognition results, and the output is optimized suggestion data. The suggestion content is dynamically adjusted according to the emotion recognition results.

[1070] Step 9:

[1071] The server provides the generated voice data and optimized lifestyle suggestions to the user. The input is the voice data and suggestion data, and the output is the information provided to the user. The information is notified to the user via a dedicated dashboard or email, and can be viewed.

[1072] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1073] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1074] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1075] [Fourth embodiment]

[1076] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1077] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1078] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1079] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1080] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1081] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1082] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1083] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1084] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1085] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1086] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1087] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1088] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1089] The system of the present invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. This system involves a series of processes, starting with the user uploading the ultrasound image and ending with providing the generated baby's voice and lifestyle suggestions.

[1090] Program processing

[1091] 1. Uploading ultrasound images

[1092] User: First, the user accesses the system's website or application and uploads their own echo images, which are then sent to the server.

[1093] Terminal: Provides an interface for sending user-selected echo images to the server.

[1094] 2. Analysis of Echo Images

[1095] Server: Upon receiving the echo images, the server applies image pre-processing algorithms to remove noise and adjust contrast, making the echo images ready for analysis. Then, it uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images.

[1096] 3. Baby voice generation

[1097] Server: Runs a speech generation model based on the feature data. This model has learned from past baby voice data and generates speech data based on the given feature data. The generated speech data is saved as an audio file and linked to the user's account.

[1098] 4. Enter user data

[1099] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[1100] Terminal: Provides an interface for sending user-entered information to the server.

[1101] 5. User data analysis

[1102] Server: Runs machine learning models to analyze the received user data in detail. The models organize the user input data into categories and scores to gain a detailed understanding of the user's attributes.

[1103] 6. Proposal for the best lifestyle

[1104] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. For example, for users living in urban areas, the server suggests nearby parent-child support facilities and recommended activities. It also suggests lifestyle schedules that match the baby's sleep rhythm.

[1105] Server: Notifies the user of the generated suggestions and provides audio files and lifestyle suggestions.

[1106] Terminal: An interface is provided to display the received proposals to the user, allowing the user to view the proposals in detail.

[1107] Specific examples

[1108] For example, suppose User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information. The server analyzes this information and suggests the use of childcare support services. The server also generates baby voice data and provides it to User A, allowing them to feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence.

[1109] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and provides personalized lifestyle suggestions that meet the needs of each user, providing an environment in which parents can approach child-rearing with confidence.

[1110] The processing flow will be explained below.

[1111] Step 1:

[1112] User: The user visits the Echo Voice AI website or app, selects their Echo image, and clicks the upload button. The user also enters basic information such as name and email address.

[1113] Step 2:

[1114] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[1115] Step 3:

[1116] Terminal: Sends pre-processed echo image data to the server. Data security is ensured by using encryption protocols during transmission.

[1117] Step 4:

[1118] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust the contrast, thereby improving the image quality.

[1119] Step 5:

[1120] Server: Runs machine learning models to extract feature data from preprocessed ultrasound images. Specifically, it detects important features such as the shape, position, and size of the fetus and records them as numerical data.

[1121] Step 6:

[1122] Server: Runs a speech generation model based on the feature data. Based on past baby voice data, generates a voice that is closest to the current feature data. Saves this voice data as an audio file (e.g., MP3).

[1123] Step 7:

[1124] Server: Binds the generated audio file to the user's account, notifies the user of the download link, and stores the audio file in secure storage.

[1125] Step 8:

[1126] User: The user clicks on the link provided and downloads the generated baby voice audio file to check it.

[1127] Step 9:

[1128] Device: Provides an interface where users can complete a questionnaire about their living environment, preferences, and needs. For example, questions might include, "What is your family structure?" and "What is your approach to childcare?"

[1129] Step 10:

[1130] User: Answers a questionnaire about their living environment, preferences, and needs, and clicks the submit button.

[1131] Step 11:

[1132] Terminal: The survey data entered by the user is sent to the server. The data is encrypted and sent.

[1133] Step 12:

[1134] Server: Runs a machine learning model to analyze the received survey data. The model classifies user responses into categories and scores, and analyzes lifestyle and parenting characteristics.

[1135] Step 13:

[1136] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that best suit the user's needs. The proposals are generated in the form of text, lists, graphs, etc.

[1137] Step 14:

[1138] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[1139] Step 15:

[1140] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[1141] Step 16:

[1142] User: Check the suggestions, put them into practice with specific lifestyle and child-rearing methods, and provide feedback as needed.

[1143] Step 17:

[1144] Terminal: Sends the feedback entered by the user to the server. The feedback will be used to improve the service in the future.

[1145] Step 18:

[1146] Server: Receives feedback data and stores it in a database. The stored data is fed into the machine learning model and used to improve the accuracy of future suggestions.

[1147] These are the specific processing steps of the Echo Voice AI system, which allows users to experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[1148] Example 1

[1149] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1150] Conventional ultrasound image analysis systems do not generate baby voices or provide personalized recommendations for optimal childcare lifestyles, making it difficult for users to obtain childcare advice tailored to their own living environment. Furthermore, conventional systems do not utilize machine learning models to generate voice data, which leads to issues with the realism of the generated voice data and low user satisfaction.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1152] In this invention, the server includes means for receiving echo images, means for preprocessing the echo images, means for extracting feature data from the echo images, means for generating voice data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods, and means for providing the voice data and suggestions to the user, thereby enabling the user to realistically experience the baby's voice from the echo images and receive suggestions on a child-rearing lifestyle that is optimal for their own living environment.

[1153] "Echo image" refers to an image taken using ultrasound to check the condition of the fetus.

[1154] "Preprocessing" refers to the initial processing steps required to prepare the image for analysis, such as removing noise from the echo image and adjusting the contrast.

[1155] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[1156] "Audio data" refers to digital audio data of a baby's voice created from feature data of an ultrasound image using a generative AI model.

[1157] A "generative AI model" refers to an artificial intelligence model that learns from a large amount of baby voice data and generates realistic voices based on specific feature data.

[1158] "User's living environment information" refers to individual information provided by the user related to childcare, such as the user's living situation and residential environment.

[1159] "Survey format" refers to a method in which users respond to questions by selecting options or writing a statement.

[1160] "Lifestyle" and "parenting methods" refer to behavioral patterns and specific parenting advice and methods that are considered optimal for the user and their baby's lifestyle.

[1161] The system of the present invention involves a series of processes that begin with the user uploading an ultrasound image and ultimately provide a generated baby voice and optimal lifestyle recommendations. The specific hardware and software used in each process step are described in detail below.

[1162] First, the user accesses the system's website or application using a device such as a PC or smartphone. The user clicks the "Upload Echo Image" button on the screen, selects an echo image file from their device, and uploads it. The device provides an interface for reading the echo image file and sending it to the server. It is desirable that a progress bar be displayed to indicate the progress of the upload.

[1163] The server temporarily stores the received echo images. It then applies image pre-processing algorithms to remove noise and adjust the contrast. Specifically, it uses noise removal and contrast adjustment algorithms such as a median filter. It then uses a deep learning model (e.g., U-Net) to extract feature data such as the shape, size, and position of the fetus from the echo images.

[1164] The server runs a speech generation model (e.g., WaveNet) based on the extracted feature data. This speech generation model has learned from past baby voice data and generates a realistic baby voice from the input data. The generated voice data is saved as an audio file (e.g., MP3 format) and linked to the user's account.

[1165] Next, the user answers a questionnaire about their living environment, preferences, and needs. The device provides a questionnaire-style user interface and transmits the information the user has entered to the server, displaying a progress bar while the data is being transmitted.

[1166] The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. This model categorizes the user data and assigns a score to each data point to gain a detailed understanding of the user's living environment and needs. For example, it can organize information such as "dual-income household" and "limited childcare time."

[1167] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, it generates specific suggestions such as "use local childcare support services available on weekends" or "create a daily schedule that matches the baby's sleep rhythm." The server then links the generated suggestions and audio files to the user's account and sends a notification. Users can check this information when they log in.

[1168] The terminal provides an interface that displays the suggestions and the generated baby voice to the user, who can then check the suggestions in detail and download or print them as needed.

[1169] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the server will analyze this information and suggest the use of childcare support services. Furthermore, by generating baby voice data and providing it to User A, User A will be able to feel the baby's presence even before it is born. Furthermore, the server will also suggest childcare methods and specific schedules suitable for dual-income households, allowing User A to make plans with confidence regarding childcare.

[1170] An example of a prompt would be the following text:

[1171] "We are a dual-income household, and the time we can devote to childcare is limited. Please suggest baby voice data and childcare methods that are suitable for our living environment."

[1172] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1173] Step 1:

[1174] Uploading ultrasound images

[1175] User: The user visits the system's website or application and clicks the "Upload Echo Image" button. As input, there is an echo image file selected by the user from the device. This file will be uploaded to the system.

[1176] Terminal: The terminal provides an interface to read the selected echo image file and send it to the server. Specifically, it sends the image data to the server using an HTTP request and displays a progress bar to indicate the upload progress to the user. As an output, the image data is sent to the server.

[1177] Step 2:

[1178] Echo image preprocessing

[1179] Server: The server temporarily stores the received echo images. The input is the uploaded echo image file. Then, image pre-processing algorithms are applied to remove noise and adjust contrast. Specifically, a median filter is used to remove noise, and histogram equalization is used to adjust contrast. The output is the processed echo image.

[1180] Step 3:

[1181] Extracting feature data from echo images

[1182] Server: The server extracts feature data from the preprocessed echo images using a deep learning model (e.g., U-Net). The input is the preprocessed echo images. The model extracts specific information from the echo images, such as the shape, size, and position of the fetus. The output is the fetal feature data.

[1183] Step 4:

[1184] Baby voice generation

[1185] Server: The server inputs the extracted feature data into a speech generation model (e.g., WaveNet). The input is the fetus's feature data. This speech generation model learns from past baby voice data to generate a realistic baby voice. The output is an MP3 audio file that is associated with the user's account.

[1186] Step 5:

[1187] Entering User Data

[1188] User: The user answers survey-style questions provided within the system. Input includes information about the user's living environment, preferences, and needs. For example, questions include "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[1189] Terminal: The terminal provides a questionnaire-style user interface and sends the information entered by the user to the server. Specifically, an HTTP request is issued to collect form data and send it to the server.

[1190] Step 6:

[1191] User data analysis

[1192] Server: The server runs a machine learning model (e.g., a random forest classifier) ​​to analyze the received user data. The input is user data collected from surveys. The model classifies the data into categories and assigns a score to each data point to gain a detailed understanding of the user's attributes. The output is the analysis results.

[1193] Step 7:

[1194] Proposal of the best lifestyle

[1195] Server: Based on the analysis results, the server proposes lifestyle and child-rearing methods that are best suited to the user's living environment. The inputs are the analysis results and user data. For example, it generates specific suggestions such as "use local child-rearing support services available on weekends" and "create a daily schedule that matches your baby's sleep rhythm." The output is a lifestyle suggestion.

[1196] Server: Binds the generated suggestions and audio files to the user's account and sends notifications. When the user logs in, this information is displayed.

[1197] Step 8:

[1198] Display of suggestions and audio

[1199] Terminal: The terminal provides an interface that displays the suggestions and the generated baby voice to the user. The inputs are the suggestions and audio files sent from the server. Specifically, a detail page is generated using HTML / CSS, allowing the user to check the suggestions in detail and download or print them if necessary. The output is the suggestions displayed to the user and the audio playback.

[1200] (Application example 1)

[1201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1202] Conventional baby ultrasound image analysis and voice generation systems only focus on helping people feel the baby's presence, and do not address safety and security measures within the home. The purpose of this invention is to provide a means to ensure safety and security within the home while feeling the baby's presence by adding a home security system linking function to a system that generates a baby's voice from an ultrasound image.

[1203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1204] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for proposing optimal lifestyles and child-rearing methods based on the analysis results, means for providing the voice data and the proposals to the user, means for setting the generated voice data in a home security system, and means for detecting abnormalities in conjunction with the set security system and notifying the user. This makes it possible to detect abnormalities in the home using a baby's voice and promptly notify the user.

[1205] An "echo image" is an image of a fetus or internal organs taken using a medical ultrasound device.

[1206] "Feature data" refers to data such as the shape, size, and position of the fetus extracted from the ultrasound image.

[1207] "Voice data" is data of a baby's voice generated based on feature data.

[1208] "User's living environment information" is information about the environment in which the user lives, including, for example, the size and environment of the residence, whether it is a dual-income household, and so on.

[1209] "User Preferences" is information about a user's parenting policies and priorities.

[1210] "User needs" refers to information about the user's requests for help, services, etc. related to childcare.

[1211] "Lifestyle suggestions" are suggestions for optimal lifestyle habits and child-rearing methods created based on the user's living environment, preferences, and needs.

[1212] A "home security system" is a system that detects abnormalities within the home and notifies the user.

[1213] A "generative AI model" is a machine learning model that learns from past data and generates new data.

[1214] A "prompt sentence" is a sentence that prompts input or instructions to the system.

[1215] The following procedure is provided as an embodiment of the present invention.

[1216] First, users use a smartphone application to upload ultrasound images of their baby, which are then sent to a server through the application's interface.

[1217] The server applies image pre-processing algorithms, such as noise reduction and contrast adjustment, to the received echo images to prepare them for analysis, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[1218] The server then runs a speech generation model based on the extracted feature data to generate a baby's voice. The model uses past baby voice data that it has already learned from to generate speech data from the feature data. The generated speech data is saved as an audio file and linked to the user's account.

[1219] Next, users enter information about their living environment, preferences, and needs in a questionnaire format via a smartphone application, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?"

[1220] The server receives the living environment information entered by the user and analyzes it using a machine learning model. This analysis organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes. Based on the analysis results, the system suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, the system suggests nearby parent-child support facilities and recommended activities, as well as a daily schedule that matches the baby's sleep rhythm. The generated suggestions and audio data are notified to the user, and can be viewed through the smartphone application interface.

[1221] Furthermore, the generated voice data is also set up in a home security system. This security system detects abnormalities in the home based on the baby's voice and sends a notification to the user. Sensors used include infrared cameras, voice recognition sensors, and vibration sensors. When an abnormality is detected, a smartphone application sends a notification to the user in real time.

[1222] As a specific example, if User A uploads an ultrasound image to the system and enters "We are a dual-income household and have limited time to devote to childcare" as living environment information, the system will suggest the use of a childcare support service. Furthermore, by generating baby voice data and providing it to User A, User A can feel the baby's presence even before it is born. Furthermore, the system also suggests childcare methods and specific schedules suitable for dual-income households, allowing User A to plan childcare with confidence. In this way, the system of the present invention not only generates baby voices and offers personalized lifestyle suggestions tailored to each user's needs, but also provides safety and security within the home.

[1223] Example prompt:

[1224] "Upload an ultrasound image and we'll automatically generate a baby voice and set it up in your home security system."

[1225] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1226] Step 1:

[1227] Uploading ultrasound images

[1228] The user uploads an ultrasound image of the baby through the interface of a smartphone application. The uploaded ultrasound image is sent from the device to the server. Specifically, the user selects an ultrasound image on the device, and the device sends the data to the server via the Internet. The input is the ultrasound image selected by the user, and the output is the ultrasound image data stored on the server.

[1229] Step 2:

[1230] Echo image analysis

[1231] The server applies image pre-processing algorithms to the received echo images to remove noise and adjust contrast, making the echo images ready for analysis. The server then uses machine learning models to extract feature data, such as the shape, size, and position of the fetus, from the echo images. The input is the echo image data received by the server, and the output is the extracted feature data.

[1232] Step 3:

[1233] Baby voice generation

[1234] The server runs a speech generation model based on the extracted feature data. This model has been trained on past baby voice data and generates new speech data based on the feature data. This generation process saves the baby's voice as an audio file. The input is the extracted feature data, and the output is the generated speech data (audio file).

[1235] Step 4:

[1236] Entering User Data

[1237] Users use a smartphone application to input information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as "Do you spend a lot or a little time at home?" and "What kind of child-rearing methods are you interested in?" The input is the living environment information, preferences, and needs entered by the user, and the output is user data sent to the server.

[1238] Step 5:

[1239] User data analysis

[1240] The server runs a machine learning model to perform a detailed analysis of the data entered by the user regarding their living environment, preferences, and needs. The model organizes the user data into categories and scores to gain a detailed understanding of the user's attributes. The input is the user data received by the server, and the output is the analysis results (details of the user's attributes).

[1241] Step 6:

[1242] Proposals for optimal lifestyle and child-rearing methods

[1243] Based on the analysis results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. For example, for users living in urban areas, it suggests nearby parent-child support facilities and recommended activities. It also suggests a daily schedule that matches the baby's sleep rhythm. The input is the analysis results, and the output is the suggested lifestyle and child-rearing methods.

[1244] Step 7:

[1245] Proposal and voice data provided

[1246] The server notifies the user of the generated suggestions and voice data and provides them through the smartphone application interface. The user can view the detailed suggestions and baby voice data. The input is the generated suggestions and voice data, and the output is the information provided to the user.

[1247] Step 8:

[1248] Setting up a home security system

[1249] Based on the generated voice data, the user can configure the home security system. An interface for configuring the system is provided through a smartphone application. The input is the generated voice data, and the output is the configured security system.

[1250] Step 9:

[1251] Anomaly detection and notification

[1252] The security system detects abnormalities based on the baby's voice that has been set. For example, it uses a voice recognition sensor, infrared camera, or vibration sensor to notify the user in real time when an abnormality occurs. The input is the information detected by the sensor, and the output is an abnormality notification to the user.

[1253] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1254] The system of the present invention is a generative AI system that generates a baby's voice from ultrasound images and suggests optimal lifestyles, and by combining it with an emotion engine that recognizes the user's emotions, it achieves more accurate personalization. This system involves a series of processes, starting with the user uploading the ultrasound images and ending with dynamically adjusting lifestyle suggestions using the emotion engine.

[1255] Program processing

[1256] 1. Uploading ultrasound images

[1257] User: First, the user accesses the Echo Voice AI website or app, selects and uploads their own Echo image, and the uploaded Echo image is sent to the server.

[1258] Terminal: Preprocesses user-selected echo images by compressing the image size and adding required metadata, allowing for efficient image transfer.

[1259] 2. Analysis of Echo Images

[1260] Server: Upon receiving the echo images, it applies image pre-processing algorithms to remove noise and adjust contrast, then uses machine learning models to extract feature data such as the shape, size, and position of the fetus from the echo images.

[1261] 3. Baby voice generation

[1262] Server: Based on the feature data, a speech generation model is run to generate baby voice data. This voice data is saved as an audio file and linked to the user's account.

[1263] 4. Enter user data

[1264] User: Next, the user fills in a questionnaire about their living situation, preferences, and needs, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?"

[1265] Terminal: Sends the information entered by the user to the server.

[1266] 5. User data analysis

[1267] Server: Runs machine learning models to analyze the received user data, organizes the user's input data into categories and scores, and gains a detailed understanding of the user's attributes.

[1268] 6. Applying the Emotion Engine

[1269] Server: Recognizes the user's emotional state using an emotion engine. This involves extracting emotions from the user's facial expressions, voice, and text input when answering surveys or using the system.

[1270] Server: The emotion recognition results are reflected in the user's lifestyle and childcare suggestions, optimizing them. For example, if the user is feeling stressed, the server adds suggestions for relaxation methods.

[1271] 7. Proposal for the best lifestyle

[1272] Server: Based on the analysis and emotion recognition results, the server proposes lifestyle and child-rearing methods that best suit the user's living environment. The proposals are generated in the form of text, lists, graphs, etc.

[1273] Server: Notifies the user of the generated proposals and makes them available for viewing on a dedicated dashboard or via email.

[1274] Terminal: Provides an interface to display the proposal to the user and allows the user to view the detailed proposal.

[1275] Specific examples

[1276] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. For example, User B could be offered suggestions such as ways to reduce stress or information about joining a local support group. User B will also be provided with data on the generated baby's voice, which will help User B imagine what it would be like to raise a child in the future.

[1277] Through these processes, the system of the present invention generates a baby's voice based on the ultrasound image and also provides personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration, providing an environment in which parents can approach child-rearing with confidence.

[1278] The processing flow will be explained below.

[1279] Step 1:

[1280] User: The user visits the Echo Voice AI website or app, logs in to their account, selects an echo image, and clicks the upload button to upload the echo image to the system.

[1281] Step 2:

[1282] Terminal: Preprocessing the uploaded echo images involves optimizing the image resolution, compressing the file size, and adding necessary metadata to the images (e.g., capture date and time, device information, etc.).

[1283] Step 3:

[1284] Terminal: Sends preprocessed echo image data to the server, using an encryption protocol (e.g., TLS) to ensure data security.

[1285] Step 4:

[1286] Server: Analyzes the received echo image data. First, it applies pre-processing algorithms to remove image noise and adjust color and contrast, thereby improving image quality.

[1287] Step 5:

[1288] Server: Extracts feature data such as fetal shape, size, and position from preprocessed ultrasound images. This is done using machine learning models, and records interesting features as numerical data.

[1289] Step 6:

[1290] Server: Runs a speech generation model based on the extracted features. This model is based on past baby speech data. A speech generation algorithm is used to create a speech that most closely matches the current features.

[1291] Step 7:

[1292] Server: Saves the generated audio data in an audio file format (e.g., MP3), associates it with the user's account, and notifies the user of the download link for the audio file.

[1293] Step 8:

[1294] User: The user clicks on the link provided to download the generated baby voice audio file and listen to the sound.

[1295] Step 9:

[1296] Device: Provides an interface that allows users to answer a detailed questionnaire about their living environment, preferences, and needs, including questions about their family structure, their attitude toward child-rearing, and their daily life challenges.

[1297] Step 10:

[1298] User: Completes a survey, inputting their preferences and needs regarding lifestyle and childcare methods. Once completed, clicks the submit button.

[1299] Step 11:

[1300] Terminal: The survey data entered by the user is sent to the server. Encryption protocols are used to ensure the security of the information when sending the data.

[1301] Step 12:

[1302] Server: Collects the received survey data and runs a machine learning model for detailed analysis. This model categorizes user responses into categories and scores to understand user attributes and needs.

[1303] Step 13:

[1304] Server: Using the emotion engine, the system analyzes the user's emotional state from their facial expressions and text responses when filling out questionnaires and using the system. This data can categorize the user's emotions and evaluate them in real time.

[1305] Step 14:

[1306] Server: Based on the analysis results of the emotion engine, the server proposes lifestyle and child-rearing methods that take into account the user's psychological state and emotions. For example, if the user is feeling tired or stressed, it will suggest relaxation methods or information on how to join support groups.

[1307] Step 15:

[1308] Server: Integrates analysis and emotion recognition results to generate personalized lifestyle and parenting recommendations in the form of text, visual lists, graphs, etc.

[1309] Step 16:

[1310] Server: Notifies the user of the generated proposal and provides further information via a dedicated dashboard or email.

[1311] Step 17:

[1312] Device: Provides an interface to display the recommendations to the user and allows the user to view detailed recommendations, including, for example, video tutorials of the suggested parenting techniques and links to local support groups.

[1313] Step 18:

[1314] User: Check the suggestions, put into practice specific lifestyle and child-rearing methods, and provide feedback within the interface as needed.

[1315] Step 19:

[1316] Device: Sends user-entered feedback to the server, which uses it to improve future suggestions.

[1317] Step 20:

[1318] Server: Receives feedback data and stores it in a database, which feeds the machine learning model for future improvements.

[1319] These are the specific processing steps of the Echo Voice AI system, which combines an emotion engine. Through this series of processes, users can experience the baby's voice generated from the ultrasound image and receive personalized lifestyle and childcare suggestions.

[1320] Example 2

[1321] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1322] Conventional systems have problems with not being able to fully utilize the information obtained from ultrasound images, and are unable to appropriately reduce the anxiety and stress felt by users. Furthermore, the suggestions provided to users are general and not optimized for each user's living environment or psychological state, resulting in insufficient personalized support. Furthermore, the baby's voice, generated based on the analysis of ultrasound images, is not provided in real time, and users' expectations cannot be met.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1324] In this invention, the server includes means for preprocessing the echo image, means for extracting feature data from the echo image, and means for generating voice data based on the feature data. This makes it possible to effectively extract information from the echo image and provide personalized support based on the user's living environment and psychological state. Furthermore, by generating and providing voice data in real time, it is possible to meet user expectations.

[1325] An "echo image" is a medical image that uses ultrasound technology to visually record the internal structure and movements of a fetus.

[1326] "Preprocessing" is a technique for performing initial processing such as noise removal, contrast adjustment, and size compression to make data easier to analyze.

[1327] "Feature data" refers to specific information such as the shape, size, and position of the fetus extracted from the ultrasound image.

[1328] "Voice data" refers to data of a baby's voice generated based on feature data extracted from an echo image.

[1329] "User Information" means data about a user's living environment, preferences, and needs, and is information collected in the form of a questionnaire or similar.

[1330] "Analysis" is the process of analyzing collected data using machine learning models and statistical methods to derive specific trends and patterns.

[1331] "Mental state recognition" is the process of extracting and understanding a user's emotions and psychological state from questionnaire responses, facial expression analysis, etc.

[1332] "Lifestyle suggestions" are advice on optimal lifestyle habits and child-rearing methods provided based on the user's living environment and psychological state.

[1333] "Provision" refers to the process of notifying the user of the generated voice data and lifestyle suggestions in a viewable form.

[1334] A "machine learning model" is an algorithm that learns from large amounts of data and automatically performs specific tasks (such as generating voice data or analyzing user information).

[1335] MODE FOR CARRYING OUT THE INVENTION

[1336] This invention is a system that generates a baby's voice based on an ultrasound image and suggests a personalized lifestyle and child-rearing method based on the user's living environment and psychological state. Each processing step of this system and the necessary hardware and software are described in detail below.

[1337] Uploading ultrasound images

[1338] Users access a dedicated website or application, select and upload echo images, and use a smartphone or computer to do this. Specifically, users select images from their smartphone's camera roll or a folder on their computer and click the "Upload" button. The uploaded images are then sent to the server.

[1339] Echo image preprocessing

[1340] The device preprocesses the received echo images. Specifically, it compresses the image size and adds metadata required for compression (e.g., shooting date and time, file format, etc.). This process uses the OpenCV library. This makes image data transfer efficient and ensures smooth processing on the server.

[1341] Echo image analysis

[1342] The server performs noise reduction and contrast adjustment on the preprocessed images, then uses a machine learning model (e.g., a TensorFlow or PyTorch-based model) to extract feature data such as the shape, size, and position of the fetus from the image, and uses the OpenCV library for filtering and edge detection at this stage.

[1343] Baby voice generation

[1344] The server generates voice data using a voice generation model (e.g., WaveNet or Tacotron) based on the extracted feature data. Specifically, it uses the fetus's shape data as input to synthesize the baby's voice. This voice data is saved as an audio file (e.g., WAV format) and associated with the user's account.

[1345] Entering User Data

[1346] Users enter information about their living environment, preferences, and needs in a questionnaire format, answering questions such as "Do you spend a lot or a little time at home?" and "What kind of childcare methods are you interested in?" The entered data is sent to the server.

[1347] Preprocessing and analysis of user data

[1348] The device preprocesses the data sent by the user. Specifically, it structures the data in JSON format. The server then uses a machine learning model (e.g., scikit-learn or XGBoost) to classify the received data into categories and scores. For example, "time spent at home" is classified into two categories, "a lot" and "a little," and "interest in childcare methods" is scored.

[1349] Applying emotion engines and recognizing mental states

[1350] The server uses an emotion engine to recognize the user's psychological state. For example, it performs facial expression analysis (using OpenFace or Dlib) and voice analysis (using Praat or DeepSpeech) to understand the user's emotional state.

[1351] Proposal of the best lifestyle

[1352] The server then proposes optimal lifestyle and child-rearing methods based on the results of analyzing user data and recognizing their psychological state. Suggestions are generated in the form of text, lists, graphs, and more. For example, they include specific advice such as "incorporate relaxation time into your daily routine" or "take advantage of local support groups." The generated suggestions are notified to the user via a dedicated dashboard or email.

[1353] The device provides an interface that displays the proposal content, allowing users to check the details of the proposal. Users can view the proposal content on their smartphones or computers and obtain the necessary information.

[1354] Examples of concrete examples and prompts

[1355] For example, suppose User B uploads an ultrasound image to the system and enters "I'm a single-parent family and I'm very anxious about raising a child" as living environment information. If the emotion engine analyzes User B's input and behavior and recognizes strong feelings of anxiety, the system will reflect this and make specific suggestions. Specifically, it will suggest "methods for reducing stress" and "information on joining local support groups." It will also provide data on the generated baby's voice, which will help User B imagine what it will be like to raise a child in the future.

[1356] An example of a prompt is, "Based on the ultrasound image and information about the user's living environment, generate a baby's voice and suggest the optimal lifestyle. If the user is a single parent and is feeling anxious, please also suggest additional support methods."

[1357] This allows the invention to generate a baby's voice based on the ultrasound image and provide personalized lifestyle and child-rearing suggestions that take the user's emotions into consideration. By receiving suggestions optimized for their own living environment, users can reduce their anxiety about child-rearing and approach child-rearing with confidence.

[1358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1359] Step 1: Upload your ultrasound images

[1360] Users access a dedicated website or application, select and upload echo images from their smartphone's camera roll or a folder on their computer. The input is the echo image file, and the output is the echo image data sent to the server. When the user clicks the "Upload" button, the image data is transferred to the server.

[1361] Step 2: Preprocessing of echo images

[1362] The terminal preprocesses the received echo images. The input is the echo image data sent by the user, and the output is the preprocessed image data. This preprocessing includes image size compression, noise removal, and contrast adjustment using the OpenCV library. This adds image metadata and improves transmission efficiency.

[1363] Step 3: Echo image analysis

[1364] The server analyzes the preprocessed echo images. The input is the preprocessed image data, and the output is feature data such as the shape, size, and position of the fetus. The server uses the OpenCV library to perform filtering and edge detection, and then applies TensorFlow or PyTorch-based machine learning models to extract the feature data.

[1365] Step 4: Generating a baby voice

[1366] The server runs a speech generation model based on the extracted feature data. The input is the feature data, and the output is an audio file of the baby's voice. Specifically, it uses WaveNet and Tacotron models to synthesize the baby's voice from the fetal shape data. This voice data is saved as an audio file and associated with the user's account.

[1367] Step 5: Enter user data

[1368] Users enter information about their living environment, preferences, and needs in the form of a questionnaire. The input is the questionnaire data answered by the user, and the output is user data that is structured and sent to the server. Specifically, users answer questions such as "how much time do you spend at home?" and "interests in childcare methods."

[1369] Step 6: Preprocessing and analyzing user data

[1370] The terminal preprocesses the survey data obtained from the user. The input is the survey data, and the output is the preprocessed data. The data is structured in JSON format. The server then analyzes the data using machine learning models such as scikit-learn and XGBoost, classifying user attributes into categories and scores.

[1371] Step 7: Applying the Emotion Engine and Recognizing Mental States

[1372] The server uses an emotion engine to recognize the user's psychological state. The input is the user's questionnaire responses, behavioral data, and facial expression analysis data, and the output is the user's emotional information. Facial expression analysis is performed using OpenFace or Dlib, and speech analysis is performed using Praat or DeepSpeech. The recognized emotional information is obtained.

[1373] Step 8: Proposal for the optimal lifestyle

[1374] The server proposes optimal lifestyle and child-rearing methods based on the analysis results of user data and emotional information. The input is the analysis results and emotional information, and the output is the proposals. Specific proposals are generated in the form of text, lists, and graphs. The generated proposals are notified to the user via a dedicated dashboard or email.

[1375] Step 9: View the proposal

[1376] The terminal provides an interface that displays the proposal content to the user. The input is the proposal content, and the output is the interface displayed to the user. The user can view the detailed proposal content using a smartphone or PC and obtain the necessary information.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] Conventional childcare support systems based on ultrasound images have difficulty making personalized suggestions that take into account the user's emotional state, leading to a lack of confidence in users' childcare strategies. Another issue is that there is little correlation between the baby's voice generated from ultrasound images and the childcare suggestions, meaning users are not provided with sufficient information. Furthermore, lifestyle and childcare method suggestions are static and cannot be dynamically adjusted to suit the user's situation or emotions, making it difficult to provide efficient suggestions in physical stores.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1381] In this invention, the server includes means for receiving an echo image, means for extracting feature data from the echo image, means for generating voice data based on the feature data, means for collecting information on the user's living environment, preferences, and needs, means for analyzing the user information, means for using an emotion engine to recognize the user's emotional state, means for optimizing lifestyle and childcare method suggestions based on the emotion recognition results, and means for providing the voice data and the suggestions to the user, thereby enabling dynamic lifestyle and childcare method suggestions based on the user's emotional state.

[1382] An "echo image" is an image showing the condition of a fetus obtained by an ultrasound diagnostic device.

[1383] "Feature data" refers to information such as the shape, size, and position of the fetus extracted from the echo image.

[1384] "Voice data" is digital audio information including a baby's voice generated based on feature data.

[1385] "User's living environment information" refers to information about the user's daily life, including family structure, living environment, occupation, and the like.

[1386] "Preferences" refers to the user's preferences and intentions regarding child-rearing.

[1387] "Needs" refers to the user's requests and demands regarding childcare.

[1388] An "emotion engine" is a software or hardware technology that recognizes a user's emotional state from their text input, facial expressions, and voice.

[1389] "Lifestyle and child-rearing suggestions" refers to recommending optimal lifestyle habits and child-rearing methods for a user based on the user's living environment information, emotional state, and analysis results.

[1390] "Optimize" means tailoring the recommendations to provide more relevant advice based on the user's personalized information.

[1391] This invention is a generative AI system that generates a baby's voice from an ultrasound image and suggests an optimal lifestyle. By combining it with an emotion engine that recognizes the user's emotions, it achieves highly accurate personalization. Specific embodiments for implementing this system are described below.

[1392] Users first capture echo images using their smartphones and then upload them using a dedicated application. The uploaded echo images are then sent to a server. To ensure efficient image transfer, the echo images are pre-processed on the device to compress the image size and add required metadata.

[1393] The server applies image pre-processing algorithms to analyze the received echo images, such as noise reduction and contrast adjustment, and then uses machine learning models to extract feature data from the echo images, including the shape, size, and position of the fetus.

[1394] The server then runs a speech generation model on the feature data to generate baby voice data, which is then saved as an audio file and associated with the user's account.

[1395] The user then enters information about their living environment, preferences, and needs in a questionnaire format. For example, they answer questions such as, "Do you spend a lot of time at home?" and "What kind of child-rearing methods are you interested in?" The device then sends this user-entered information to the server.

[1396] The server runs a machine learning model to analyze the received user data, organizing the user's input data into categories and scores to gain a detailed understanding of the user's attributes.

[1397] The server then uses an emotion engine to recognize the user's emotional state. This involves extracting emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The server then reflects the emotion recognition results in the user's lifestyle and childcare recommendations, optimizing them. For example, if the user is feeling stressed, the server will add suggestions for relaxation methods.

[1398] Based on the analysis and emotion recognition results, the server suggests lifestyle and child-rearing methods that are best suited to the user's living environment. The suggestions are generated in the form of text, lists, graphs, etc. The generated suggestions are notified to the user via a dedicated dashboard or email, and can be viewed.

[1399] For example, suppose User A uploads an ultrasound image to the system and enters "both spouses work, so there is little anxiety about childcare" as living environment information. If the emotion engine analyzes User A's input and behavior and recognizes that stress is low, the system will reflect this and make specific suggestions. For example, User A may be offered nutritionally balanced meal menus or local childcare support information. User A will also be provided with data on the generated baby's voice, giving User A more confidence in their future childcare.

[1400] Examples of prompts include:

[1401] "We are developing a system that generates a baby's voice based on ultrasound images and suggests child-rearing methods and lifestyles that are tailored to the user's living environment and psychological state. With this system, users can upload ultrasound images using their smartphone, listen to their baby's voice, and receive optimal lifestyle suggestions. The user's psychological state is analyzed using an emotion engine, and suggestions are dynamically adjusted according to the user's stress level."

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] The user launches a dedicated application on their smartphone to acquire an echo image. The user selects an echo image and uploads it through the application. The input is the echo image, and the output is an image file transferred to the server. The device compresses the image file size, adds required metadata, and efficiently transmits it to the server.

[1405] Step 2:

[1406] The server runs the received echo images through an image pre-processing algorithm. The input is a compressed echo image, and the output is a pre-processed echo image. This pre-processing includes noise removal and contrast adjustment, which improves image quality and allows for accurate subsequent processing.

[1407] Step 3:

[1408] The server runs a machine learning model to extract feature data from the preprocessed echo images. The input is the preprocessed echo images, and the output is feature data. This feature data includes the shape, size, and position of the fetus. The machine learning model efficiently analyzes important information in the echo images.

[1409] Step 4:

[1410] The server runs a speech generation model based on the feature data to generate baby voice data. The input is the feature data, and the output is the baby's voice data. This voice data is saved as an audio file. The speech generation model analyzes the feature data and creates an audio file that imitates the baby's voice.

[1411] Step 5:

[1412] The user enters information about their living environment, preferences, and needs in a questionnaire format. The input is information about the living environment, preferences, and needs, and the output is the questionnaire data sent to the server. The terminal sends the user's input to the server.

[1413] Step 6:

[1414] The server runs a machine learning model to analyze the received user data, such as living environment information, preferences, and needs. The input is the survey data, and the output is the analysis results. The machine learning model organizes this data into categories and scores to gain a detailed understanding of the user's attributes.

[1415] Step 7:

[1416] The server recognizes the user's emotional state using an emotion engine that extracts emotions from the user's facial expressions, voice, and text input when answering questionnaires or using the system. The input is data related to the user's emotions, and the output is the emotion recognition results. The emotion engine analyzes the emotional aspects of the data entered by the user and grasps the user's current psychological state.

[1417] Step 8:

[1418] The server optimizes lifestyle and child-rearing method suggestions based on the emotion recognition results. The inputs are the analysis results and emotion recognition results, and the output is optimized suggestion data. The suggestion content is dynamically adjusted according to the emotion recognition results.

[1419] Step 9:

[1420] The server provides the generated voice data and optimized lifestyle suggestions to the user. The input is the voice data and suggestion data, and the output is the information provided to the user. The information is notified to the user via a dedicated dashboard or email, and can be viewed.

[1421] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1422] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1423] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1424] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1425] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1426] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1427] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1428] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1429] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1430] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1431] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1432] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1433] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1434] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1435] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1436] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1437] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1438] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1439] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1440] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1441] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1442] The following is further disclosed regarding the above embodiment.

[1443] The following is a proposed draft of the claims.

[1444] (Claim 1)

[1445] means for receiving an echo image;

[1446] means for extracting feature data from the echo image;

[1447] means for generating voice data based on the feature data;

[1448] means for collecting information about your living environment, preferences, and needs;

[1449] means for analyzing said user information;

[1450] A means for proposing optimal lifestyle and child-rearing methods based on the analysis results;

[1451] means for providing said audio data and said suggestions to a user;

[1452] A system including:

[1453] (Claim 2)

[1454] The system of claim 1, characterized in that a machine learning model is used to generate the audio data.

[1455] (Claim 3)

[1456] 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire.

[1457] "Example 1"

[1458] (Claim 1)

[1459] means for receiving an echo image;

[1460] means for pre-processing the echo images;

[1461] means for extracting feature data from the echo image;

[1462] means for generating voice data based on the feature data;

[1463] means for collecting information about your living environment, preferences, and needs;

[1464] means for analyzing said user information;

[1465] A means for proposing optimal lifestyle and child-rearing methods based on the analysis results;

[1466] means for providing said audio data and said suggestions to a user;

[1467] A system including:

[1468] (Claim 2)

[1469] The system of claim 1, characterized in that a generative AI model is used to generate the audio data.

[1470] (Claim 3)

[1471] 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire.

[1472] "Application Example 1"

[1473] (Claim 1)

[1474] means for receiving an echo image;

[1475] means for extracting feature data from the echo image;

[1476] means for generating voice data based on the feature data;

[1477] means for collecting information about your living environment, preferences, and needs;

[1478] means for analyzing said user information;

[1479] A means for proposing optimal lifestyle and child-rearing methods based on the analysis results;

[1480] means for providing said audio data and said suggestions to a user;

[1481] means for configuring the generated audio data into a home security system;

[1482] means for detecting an abnormality in conjunction with the configured security system and notifying a user;

[1483] A system including:

[1484] (Claim 2)

[1485] The system of claim 1, characterized in that a machine learning model is used to generate the audio data.

[1486] (Claim 3)

[1487] 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire.

[1488] "Example 2: Combining Emotion Engines"

[1489] (Claim 1)

[1490] means for receiving an echo image;

[1491] means for pre-processing the echo images;

[1492] means for extracting feature data from the echo image;

[1493] means for generating voice data based on the feature data;

[1494] means for collecting information about your living environment, preferences, and needs;

[1495] means for preprocessing the user information;

[1496] means for analyzing said user information;

[1497] means for recognizing a psychological state of a user based on the analysis results;

[1498] means for suggesting an optimal lifestyle and child-rearing method based on the psychological state of the user and the analysis results;

[1499] means for providing said audio data and said suggestions to a user;

[1500] A system including:

[1501] (Claim 2)

[1502] The system of claim 1, characterized in that a machine learning model is used to generate the audio data.

[1503] (Claim 3)

[1504] 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire.

[1505] "Application example 2 when combining emotion engines"

[1506] (Claim 1)

[1507] means for receiving an echo image;

[1508] means for extracting feature data from the echo image;

[1509] means for generating voice data based on the feature data;

[1510] means for collecting information about your living environment, preferences, and needs;

[1511] means for analyzing said user information;

[1512] A means for proposing optimal lifestyle and child-rearing methods based on the analysis results;

[1513] a means for using an emotion engine to recognize an emotional state of a user;

[1514] means for optimizing lifestyle and child-rearing method suggestions based on the emotion recognition results;

[1515] means for providing said audio data and said suggestions to a user;

[1516] A system including:

[1517] (Claim 2)

[1518] The system of claim 1, characterized in that a machine learning model is used to generate the audio data.

[1519] (Claim 3)

[1520] 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire. [Explanation of symbols]

[1521] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving an echo image; means for extracting feature data from the echo image; means for generating voice data based on the feature data; means for collecting information about your living environment, preferences, and needs; means for analyzing said user information; A means for proposing optimal lifestyle and child-rearing methods based on the analysis results; means for providing said audio data and said suggestions to a user; A system including:

2. The system of claim 1, wherein a machine learning model is used to generate the audio data.

3. 2. The system according to claim 1, wherein the user information is collected in the form of a questionnaire.

Images