System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting an image by a user, an artificial intelligence means for analyzing the image input by the user, a feedback generation means for providing customized feedback to the user based on the analysis result, and a management means for performing schedule management and notification of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Pregnant women are prone to anxiety about their health and lifestyle, and a lack of information on diet and appropriate exercise choices, as well as difficulty in finding it, are issues. It is also difficult to schedule regular checkups and keep track of important health information amid busy daily lives. Furthermore, it is necessary to create an environment in which partners can support pregnant women. [Means for solving the problem]

[0005] The present invention is a system that includes a means for users to input images, an artificial intelligence means for analyzing the input images, a feedback generation means for providing customized feedback based on the analysis results, and a management means for managing the user's schedule and sending notifications. This provides comprehensive support for women's health management during pregnancy, providing an environment in which they can spend their pregnancy with peace of mind. The system also includes a suggestion means for proposing appropriate meal menus and exercise plans based on the analysis results, and an acquisition means for acquiring the user's health information from an external database. This enables personalized advice, information sharing, and schedule management, supporting physical condition and mental stability during pregnancy.

[0006] "Users" are pregnant women who use the system to input selfie videos and photos of their meals and receive feedback and advice based on those videos.

[0007] An "image input means" is an interface that allows users to take selfie videos or photos of their meals using their smartphones or other devices and upload them to the system.

[0008] "Artificial intelligence means" refers to machine learning models and algorithms that analyze input images and videos and assess a user's health and nutritional status.

[0009] "Feedback generation means" means a process or system for generating customized health information or advice for a user based on the analysis results of the artificial intelligence means.

[0010] "Management tools" refers to functions and modules that allow users to manage their schedules, remind them of regular checkups, and notify them of important health information.

[0011] "Suggestion means" refers to functions and modules for providing users with appropriate meal menus and exercise plans based on the analysis results.

[0012] "Acquisition means" refers to the interface or protocol for acquiring the user's health information from external databases, maternal and child health handbook data, etc. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] ---

[0035] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0036] Specific Embodiments

[0037] System Configuration

[0038] The system consists of the following main components:

[0039] 1. A means of inputting the user's image (terminal)

[0040] 2. Artificial intelligence means (server) for analyzing images

[0041] 3. Means of generating feedback (server)

[0042] 4. Means for managing user schedules (terminals, servers)

[0043] 5. Means of Providing Appropriate Suggestions (Server)

[0044] 6. Means of obtaining external data (server)

[0045] Program processing flow

[0046] Data entry and upload

[0047] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, users can take a photo of their breakfast and complete the process by tapping the "upload" button in the app.

[0048] Data analysis

[0049] The server analyzes the received data. In the case of selfie videos, artificial intelligence means analyze the video frames to assess complexion, facial expression and skin condition. In the case of food photos, image recognition technology is used to identify ingredients in the photo and assess their nutritional value.

[0050] Feedback Generation

[0051] Based on the analysis results, the server evaluates the user's health status and generates customized feedback, including specific advice such as "You are deficient in vitamin D, so we recommend incorporating fish into your diet," along with an appropriate exercise plan.

[0052] Data Sync and Notifications

[0053] The server generates feedback and sends it to the user's device. It also connects to the maternal and child health handbook database to periodically update health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0054] Displaying feedback to the user

[0055] Users can view the feedback and suggestions they receive within the application, for example, breakfast recommendations or reminders for upcoming checkups, making it easier for users to manage their health.

[0056] Specific examples

[0057] One day, a user wakes up, takes a photo of their breakfast with their smartphone, and uploads it to the server via the app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, it identifies nutrients that the user is lacking (e.g., vitamin D). The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The device displays the received feedback within the app, allowing the user to choose ingredients appropriate for that day's lunch. Furthermore, if a regular medical checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to go for the checkup.

[0058] In this way, the system is designed to be easy for users to use on a daily basis, and provides comprehensive support for health management during pregnancy.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] Users use their smartphones to take selfies or photos of their food, and the app's camera function is used to capture data.

[0062] Step 2:

[0063] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[0064] Step 3:

[0065] The server pre-processes the received video and photo data for analysis, for example by extracting video frames and compressing images.

[0066] Step 4:

[0067] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[0068] Step 5:

[0069] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[0070] Step 6:

[0071] The server combines the results of video and photo analysis to assess the user's current health status, and generates customized feedback based on the assessment results.

[0072] Step 7:

[0073] The server will suggest appropriate meal plans and exercise plans based on the user's health and nutritional status, providing specific advice such as "recommending a lunch that includes fish."

[0074] Step 8:

[0075] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[0076] Step 9:

[0077] The server connects to the maternal and child health handbook database to update the user's health information, checks the dates of regular checkups, and creates necessary reminders.

[0078] Step 10:

[0079] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[0080] Step 11:

[0081] Users open the app and review the feedback and suggestions they have received. The scheduling feature allows users to add the next checkup date to their calendar.

[0082] In this way, the system is structured around an entire flow from user input to processing on the server, generating and notifying feedback, and finally presenting it to the user.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] Proper health management and nutritional balance are extremely important for pregnant women. However, existing health management systems lack the ability to provide personalized feedback, making it difficult for users to obtain specific advice from photos and videos they take. They also lack the ability to recommend appropriate meal plans and exercise plans, and are inconvenient when it comes to managing health information and providing schedule notifications.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results, a management means for managing the user's schedule and notifications, and a means for displaying the feedback and suggestions received by the user. This enables the provision of customized health management and feedback that the user can easily use on a daily basis.

[0088] "Means for users to input images and videos" refers to the function that allows users to take and input images and videos necessary for health management using mobile devices such as smartphones and tablets.

[0089] "Artificial intelligence means" refers to the function of using AI algorithms and machine learning models implemented on the server to analyze input images and videos and evaluate or predict the user's health condition.

[0090] "Feedback generation means" refers to a function that automatically generates customized health advice and suggestions for the user based on the results analyzed by the artificial intelligence means.

[0091] "Management tools" refers to the ability to manage a user's schedule and tasks and provide reminder notifications as needed.

[0092] "Means for displaying feedback and suggestions" refers to a function that displays feedback and suggestions generated by the server on the user's device so that they can be visually confirmed.

[0093] "Means for suggesting meal menus and exercise plans" refers to a function that generates appropriate meal menus and exercise plans based on the user's health condition and analysis results, and presents them as advice.

[0094] "Acquisition means" refers to the function of acquiring user health information from external databases and integrating it into the system.

[0095] MODE FOR CARRYING OUT THE INVENTION

[0096] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0097] System Configuration

[0098] The system consists of the following main components:

[0099] 1. The means (terminal) by which users input images and videos

[0100] 2. Artificial intelligence means (server) for analyzing images and videos

[0101] 3. Means of generating feedback (server)

[0102] 4. User schedule management and notification methods (terminals, servers)

[0103] 5. A means (device) for displaying feedback and suggestions

[0104] 6. Proposal tool (server) that proposes appropriate meal menus and exercise plans

[0105] 7. Means for obtaining health information from external databases (server)

[0106] Data entry and upload

[0107] Users use their smartphone cameras to take selfies or photos of their meals. For example, a user may take photos of bread, eggs, and salad for breakfast. This data is uploaded to a server through an application. For example, when a user taps the "upload" button in an app after taking a photo, the device uploads the data to the server via Wi-Fi or mobile network.

[0108] Data analysis

[0109] The server analyzes the received data. In the case of selfie videos, artificial intelligence analyzes each frame of the video to evaluate the complexion, facial expression, and skin condition. Specifically, AI technologies such as TensorFlow and AWS Rekognition are used. For example, the server scores the user's complexion and facial expression to evaluate their health condition. In the case of a photo of a meal, image recognition technology is used to identify the ingredients contained in the photo and evaluate their nutritional value. Typical technologies include Google Cloud Vision API and OpenCV. The server analyzes the photo and determines that it contains bread, eggs, and salad.

[0110] Feedback Generation

[0111] Based on the analysis results, the server evaluates the user's health status and generates customized feedback. For example, it generates specific advice such as "You are deficient in vitamin D, so we recommend that you include fish in your lunch." It also generates an exercise plan tailored to the user's condition. For example, it suggests "It would be good to do 30 minutes of light walking every day."

[0112] Data Sync and Notifications

[0113] The server sends the generated feedback to the user's device. The device displays this feedback within the app. For example, a reminder such as "Your next health check is in one week" may be displayed. The server also accesses the maternal and child health handbook database to update regular health information. For example, the latest weight and blood pressure records may be added to the maternal and child health handbook. When the next checkup date approaches, the device sends a reminder notification. For example, a push notification such as "Tomorrow is your regular checkup date" may be displayed on the user's smartphone.

[0114] Displaying feedback to the user

[0115] The user opens the application and sees the feedback and suggestions sent by the server. For example, breakfast recommendations or reminders for the next week's medical checkup are displayed within the app. The user can then use this information to plan their meals and exercise for the day.

[0116] Specific examples

[0117] Below are some example prompts to input to a generative AI model:

[0118] "I took a photo of my breakfast this morning to help me manage my health during pregnancy. The photo includes bread, eggs, and salad. I'd like your feedback on health advice for this breakfast and my future meal plan."

[0119] In this way, the system provides users with a customized health management system that can be easily used on a daily basis, enabling comprehensive management of pregnancy health conditions.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Program processing flow

[0122] Step 1: Data entry and upload

[0123] Users take selfie videos or photos of their food with their smartphones.

[0124] Input: The user activates the smartphone camera and takes a selfie video or a photo of a meal (e.g., a photo of bread, eggs, and salad for breakfast) that is necessary for health management.

[0125] Output: Captured image and video data.

[0126] The user taps the "Upload" button in the application to send the captured data to the server.

[0127] Input: Tap the "Upload" button within the app.

[0128] Output: Image and video data sent to the server.

[0129] Step 2: Data analysis

[0130] The server receives the uploaded data and begins analyzing it.

[0131] Input: Image and video data received by the server.

[0132] Output: Data ready for analysis.

[0133] In the case of selfie videos, the server uses AI technology (e.g., TensorFlow or AWS Rekognition) to analyze each frame of the video and evaluate complexion, facial expression, and skin condition.

[0134] Input: Selfie video data.

[0135] Output: Evaluation results of complexion, facial expression, and skin condition.

[0136] For photos of meals, the server uses image recognition technology (e.g., Google Cloud Vision API or OpenCV) to identify ingredients and assess their nutritional value.

[0137] Input: Food photo data.

[0138] Output: Identified ingredients and nutrient evaluation results.

[0139] Step 3: Feedback generation

[0140] The server evaluates the user's health status based on the analysis results.

[0141] Input: Evaluation results of ingredients and nutrients, evaluation results of complexion and facial expression.

[0142] Output: Health status assessment results.

[0143] The server generates customized feedback, such as specific advice like "You're lacking vitamin D, so we recommend you include fish in your lunch."

[0144] Input: Health status assessment results.

[0145] Output: Customized feedback and advice.

[0146] Step 4: Data Sync and Notifications

[0147] The server generates feedback and sends it to the user's device.

[0148] Input: Customized feedback and advice.

[0149] Output: Feedback sent to the user device.

[0150] The server accesses the maternal and child health handbook database and updates the health information periodically.

[0151] Input: Latest health information.

[0152] Output: Updated maternal and child health handbook database.

[0153] When the next checkup date approaches, the server will send a reminder notification to the user's device.

[0154] Input: Schedule information.

[0155] Output: Send reminder notification.

[0156] Step 5: Display feedback to the user

[0157] The user opens the app and reviews the feedback and suggestions they received.

[0158] Input: Feedback sent to the user's device.

[0159] Output: Feedback and suggestions displayed within the application.

[0160] Based on the displayed feedback, users plan their daily health management, food choices, and exercise plans.

[0161] Input: Customized feedback and suggestions.

[0162] Output: Implementation of a specific health management plan.

[0163] (Application example 1)

[0164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0165] Pregnant women need to properly manage their health, and for this, it is important to check their daily diet and evaluate their health. However, doing this manually requires a great deal of effort and time, making it difficult for pregnant women to properly manage their health themselves. Furthermore, even if they receive appropriate dietary suggestions, they lack the means to put them into practice, which makes it difficult for them to actually improve their dietary habits.

[0166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0167] In this invention, the server includes an input means for a user to input an image, an artificial intelligence means for analyzing the image input by the user, a feedback generation means for providing the user with customized feedback based on the analysis results, a management means for managing the user's schedule and notifying them, a means for providing specific meal suggestions, and an ordering means for ordering the suggested meals, thereby enabling pregnant women to easily manage their health status and practice a nutritionally balanced diet.

[0168] "User" means a person who intends to use the system to manage their own health condition.

[0169] "Images" refers to selfie videos, food photos, etc. that users take or obtain through input means.

[0170] "Input means" refers to a device or application that allows a user to take an image and input it into the system.

[0171] "Artificial intelligence means" refers to AI technology that analyzes images input by users and identifies facial complexion, facial expressions, and ingredients.

[0172] The "feedback generation means" is a means for providing customized feedback to the user based on the analysis results obtained by the artificial intelligence means.

[0173] "Management tools" refers to features that manage the user's schedule and provide regular notifications and reminders.

[0174] The "means for providing specific meal suggestions" refers to a means having a function for suggesting a meal menu suitable for the user based on the analysis results.

[0175] "Ordering means" refers to the function that allows the user to order the suggested meal menu as is.

[0176] System Overview

[0177] The present invention is a system for managing the health and nutrition of pregnant women. By using various input means to manage the user's health condition, the system facilitates daily health checks and dietary management.

[0178] Hardware and software used

[0179] Smartphone: A device that allows users to take selfie videos and photos of their meals and upload them to the system.

[0180] Server: Hardware for executing the artificial intelligence means for image analysis, the feedback generation means, the management means, the meal suggestion means, and the ordering means.

[0181] AI analysis software: Artificial intelligence technology (e.g., TensorFlow, PyTorch, etc.) for analyzing user images.

[0182] Database system: A system for managing user health information and feedback data (e.g., MySQL, PostgreSQL, etc.).

[0183] Food Delivery API: An API used to order suggested meals online.

[0184] Data processing and calculation

[0185] 1. Data entry: Users use their smartphones to take selfies and photos of their meals, and then upload this data to the server through the app.

[0186] 2. Image analysis: The server analyzes the uploaded image data using AI analysis software to evaluate the user's health condition (complexion, facial expression, skin condition, etc.) and dietary content (ingredients, nutrients, etc.).

[0187] 3. Feedback Generation: The server generates customized feedback for the user based on the analysis results, such as specific advice such as "You are lacking in vitamin D, so we recommend that you include fish in your diet."

[0188] 4. Schedule Management and Notification: The server manages the user's schedule and notifies them of regular health checkups and meal suggestions, as well as providing reminders for regular checkups during pregnancy.

[0189] 5. Meal Suggestion and Ordering: The server will suggest appropriate meal menus to the user based on the analysis results and allow the suggested meals to be ordered through the food delivery API.

[0190] Specific examples

[0191] Suppose a pregnant woman takes a photo of her breakfast and uploads it to a server via an app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, nutrients that the woman is lacking (e.g., vitamin D) are identified. The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The user can then order the suggested meal directly through the app and use the delivery service. Additionally, if a regular checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to attend.

[0192] Prompt Sentence Examples

[0193] 1. "Consider dietary health management for a woman in her 50s. Using a photo of her breakfast, assess her nutritional status and suggest a meal plan to provide the necessary nutrients."

[0194] 2. "Provide pictures of foods that may be calcium deficient for a pregnant woman. Suggest calcium-rich meals for her."

[0195] 3. "Provide pictures of meals that are deficient in certain vitamins. Suggest foods needed to supplement the vitamins and provide a food delivery service where you can order those foods."

[0196] In this way, the embodiments of the invention enable pregnant women to easily manage their health and practice a nutritionally balanced diet.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] Users use their smartphone cameras to take selfies and photos of their meals.

[0200] Input: Selfie videos and food photos taken with your smartphone.

[0201] How it works: The user launches the dedicated app and uses the camera function to take a photo of their face or the food they're eating.

[0202] Output: Selfie videos and food photo data.

[0203] Step 2:

[0204] Users upload videos and photos they have taken to the server through the app.

[0205] Input: Selfie videos and food photo data taken with a smartphone.

[0206] How it works: The user taps the "upload" button in the app to send the data to the server.

[0207] Output: Video and photo data uploaded to the server.

[0208] Step 3:

[0209] The data received by the server is analyzed using AI analysis software.

[0210] Input: Uploaded selfie videos and food photo data.

[0211] How it works: The server uses AI analysis software (e.g., TensorFlow, PyTorch) to analyze video frames and identify ingredients using image recognition technology.

[0212] Output: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0213] Step 4:

[0214] The server generates customized feedback based on the analysis results.

[0215] Input: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0216] How it works: The server uses the feedback generator to generate specific advice, such as "You are deficient in vitamin D, so we recommend that you include fish in your diet."

[0217] Output: Customized feedback content.

[0218] Step 5:

[0219] The server sends the suggested meal menu to the user's terminal.

[0220] Input: Your customized feedback.

[0221] How it works: The server sends the feedback to the user's device and displays it as a notification within the app on the device.

[0222] Output: Meal suggestions and health advice displayed on the user's device.

[0223] Step 6:

[0224] The user reviews the suggested meal menu and, if desired, places an order within the app.

[0225] Input: Meal menu suggestions displayed on the device.

[0226] How it works: The user selects a suggested menu item and uses the in-app ordering feature to order delivery.

[0227] Output: A meal order from a delivery service.

[0228] Step 7:

[0229] The server manages the user's schedule and notifies them.

[0230] Input: User's schedule information and regular health check information.

[0231] How it works: The server manages the user's health information and sends reminders for the next checkup date and other notifications to the device.

[0232] Output: Health care reminder notification.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] ---

[0235] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0236] Specific Embodiments

[0237] System Configuration

[0238] The system consists of the following main components:

[0239] 1. A means of inputting the user's image (terminal)

[0240] 2. Artificial intelligence means (server) for analyzing images

[0241] 3. Means of generating feedback (server)

[0242] 4. Means for managing user schedules (terminals, servers)

[0243] 5. Means of Providing Appropriate Suggestions (Server)

[0244] 6. Means of obtaining external data (server)

[0245] 7. Emotion engine (server) that recognizes user emotions

[0246] Program processing flow

[0247] Data entry and upload

[0248] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[0249] Data analysis

[0250] The server analyzes the received data. First, for video analysis, artificial intelligence means analyzes video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value.

[0251] emotion recognition

[0252] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This allows the system to understand the user's mental state.

[0253] Feedback Generation

[0254] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[0255] Data Sync and Notifications

[0256] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0257] Displaying feedback to the user

[0258] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[0259] Specific examples

[0260] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis reveals a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "It would be a good idea to include fish in your lunch."

[0261] At the same time, analysis of the selfie video reveals that the user has a tired expression. The emotion engine recognizes this as "fatigue," and the server provides feedback by adding mental support advice such as, "You seem tired today, so we recommend some light stretching."

[0262] In this way, this system simplifies daily use for users and simultaneously provides comprehensive health management and mental support for pregnant women.

[0263] The processing flow will be explained below.

[0264] Step 1:

[0265] The user uses their smartphone to take a selfie video or a photo of their meal using the camera function within the app.

[0266] Step 2:

[0267] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[0268] Step 3:

[0269] The server preprocesses the video and photo data it receives. For video, it extracts frames, and for photos, it compresses them.

[0270] Step 4:

[0271] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[0272] Step 5:

[0273] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[0274] Step 6:

[0275] The server uses an emotion engine to recognize the user's emotions from the selfie video. For example, if the user smiles, it identifies "happiness," and if they frown, it identifies "anxiety."

[0276] Step 7:

[0277] The server combines the results of video analysis, photo analysis, and emotion recognition to assess the user's current health and mental state.

[0278] Step 8:

[0279] The server generates customized feedback based on the evaluation results, such as "You are lacking in vitamin D, so we recommend including fish in your diet" and "You seem tired today, so we recommend some light stretching exercises."

[0280] Step 9:

[0281] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[0282] Step 10:

[0283] The server connects to the maternal and child health handbook database to update the user's health information, and when the next checkup date approaches, a reminder notification is sent to the device.

[0284] Step 11:

[0285] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[0286] Step 12:

[0287] Users can open the app to view the feedback and suggestions they have received, as well as emotional support advice. Users can also add their next check-up date to their calendar through the scheduling feature.

[0288] In this way, the system is structured as a series of steps from user input to processing on the server, feedback generation and notification, and finally presentation to the user.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] Conventional health management systems only assess the user's health status using images and videos, but are unable to provide feedback that takes into account the user's emotional state. As a result, it is difficult to provide comprehensive health management and mental support to pregnant women at the same time. Furthermore, schedule management, appropriate meal menus, and exercise plan suggestions are not integrated, forcing users to use multiple applications.

[0292] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0293] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results and the user's emotion recognition results, and a management means for managing the user's schedule and notifying them. This allows the user to receive feedback that takes into account their emotional state in addition to an evaluation of their health condition using images and videos, and further allows them to receive integrated suggestions for schedule management, meal menus, and exercise plans.

[0294] "Means for user image and video input" refers to devices or software that allow users to take selfie videos or photos of their meals and upload that data to the system.

[0295] "Artificial intelligence means" refers to devices or systems that have programs or algorithms, including machine learning and image recognition technologies, used to analyze images or videos input by users.

[0296] A "feedback generation means" is a device or software that evaluates a user's health condition and emotional state based on the analysis results and the user's emotion recognition results, and provides customized advice and instructions.

[0297] "Management means" refers to devices or software for managing the user's schedule and sending notifications, and is primarily responsible for managing the next checkup date and reminder notifications.

[0298] The "suggestion means" refers to a device or software that suggests appropriate meal menus and exercise plans to users based on the analysis results and emotion recognition results.

[0299] "Acquisition means" refers to a device or software for acquiring the user's health information from an external database.

[0300] "Emotion recognition results" are information about a user's emotional state (e.g., happiness, anxiety, fatigue, etc.) obtained by analyzing the user's selfie video.

[0301] "Customized feedback" refers to advice and instructions created specifically for each user based on analysis and emotion recognition results.

[0302] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0303] System Configuration

[0304] The system consists of the following main components:

[0305] 1. Means (terminal) for inputting user images and videos

[0306] 2. Artificial intelligence means (server) for analyzing images and videos

[0307] 3. Means of generating feedback (server)

[0308] 4. Means for managing user schedules (terminals, servers)

[0309] 5. Means of Providing Appropriate Suggestions (Server)

[0310] 6. Means of obtaining external data (server)

[0311] 7. Emotion engine (server) that recognizes user emotions

[0312] Data entry and upload

[0313] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[0314] Data analysis

[0315] The server analyzes the received data. First, for video analysis, artificial intelligence means analyze video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value. Specific software used includes TensorFlow and OpenCV for image recognition.

[0316] emotion recognition

[0317] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This makes it possible to understand the user's mental state. Specifically, it uses Microsoft Azure Cognitive Services.

[0318] Feedback Generation

[0319] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[0320] Data Sync and Notifications

[0321] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0322] Displaying feedback to the user

[0323] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[0324] Specific examples

[0325] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis results indicate a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "You should include fish in your lunch." At the same time, analysis of the selfie video reveals that the user looks tired. The emotion engine recognizes this as "fatigue," and the server provides additional mental support advice, such as, "You seem tired today, so we recommend some light stretching."

[0326] Prompt Sentence Examples

[0327] "Analyze selfie videos and food photos to provide users with personalized health feedback. Add emotion recognition and include mental support advice."

[0328] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0329] Step 1:

[0330] The user uses the smartphone camera to take a selfie video or a photo of their meal. After taking the photo, the user taps the "upload" button in the app to send the data to the server. Specifically, the smartphone's camera app launches, and includes functions for taking a selfie video for a certain period of time and taking photos of the meal. The data input is the video and photos taken by the user, and the output is the video and photo data uploaded to the server.

[0331] Step 2:

[0332] The server decompresses the received video and photo data and performs initial data processing. Specifically, the server applies a data decompression algorithm and performs preprocessing according to the data format. The input to this step is the received compressed video and photo data, and the output is the decompressed and preprocessed data.

[0333] Step 3:

[0334] The server starts video analysis. Using artificial intelligence tools (e.g., OpenCV or TensorFlow models), it analyzes each frame of the video and evaluates the user's complexion, facial expression, and skin condition. Specifically, the server divides the video into frames, performs facial recognition on each frame, and analyzes facial expression and complexion. The input of this step is the preprocessed video data, and the output is the evaluation results of the user's complexion, facial expression, and skin condition.

[0335] Step 4:

[0336] The server analyzes the photo. It uses image recognition technology (e.g., TensorFlow) to identify the ingredients in the photo and evaluate the nutrients in each. Specifically, the server extracts image features and compares them with a database to identify the ingredients. The input to this step is the preprocessed photo data, and the output is the identification of ingredients and the evaluation of their nutrients.

[0337] Step 5:

[0338] The server runs an emotion recognition engine to recognize the user's emotions from the selfie video. For example, a smile can be identified as "happiness," while a frown can be identified as "anxiety." Specifically, the server uses an emotion recognition algorithm (e.g., Microsoft Azure Cognitive Services) to analyze facial feature points. The input for this step is the video analysis results, and the output is the recognition result of the user's emotional state.

[0339] Step 6:

[0340] The server evaluates the user's health status based on the video and photo analysis results and emotion recognition results, and generates customized feedback. Specifically, the server executes a program that integrates multiple analysis results to perform a comprehensive health assessment and generates feedback based on that assessment. The input for this step is the video and photo analysis results and emotion recognition results, and the output is customized feedback.

[0341] Step 7:

[0342] The server sends the generated feedback to the user's device. Furthermore, the server connects to the maternal and child health handbook database to update the user's health information and send reminder notifications when checkup dates approach. Specifically, the server communicates over the network, sends feedback data, and schedules database updates and reminder notifications. The input of this step is the generated feedback, and the output is the feedback sent to the device and the updated database information.

[0343] Step 8:

[0344] The user checks the feedback and advice through the application. Specifically, the user opens the app and accesses the "Feedback" section to view dietary advice, checkup reminders, and emotional support advice. The input of this step is the feedback data sent to the device, and the output is the feedback information displayed to the user.

[0345] (Application example 2)

[0346] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] Pregnant women experience many physical and mental changes and require appropriate health management and mental support during this process. However, current health care systems are unable to fully address individual needs, and emotional support is particularly lacking. Specific advice on diet and exercise is often not provided, resulting in a lack of efficient and comprehensive support for pregnant women. Therefore, there is a need for a system that allows pregnant women to easily manage their health and receive emotional support.

[0348] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0349] In this invention, the server includes a means for a user to input images, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing the user with customized feedback and meal suggestions based on the analysis results, an emotion engine means for recognizing the user's emotional state, a management means for managing the user's schedule and sending notifications, and a means for providing advice appropriate to the user's emotions and health condition using a generative AI model. This enables pregnant women to comprehensively understand their health condition and receive personalized meal suggestions and mental support.

[0350] "Means for users to input images" refers to a device or function that allows users to take images or videos using a device such as a smartphone and input them into the system.

[0351] "Artificial intelligence means" refers to a computing device that includes algorithms and programs for analyzing input image and video data and determining the user's health condition, the nutritional value of ingredients, and even their emotions.

[0352] The "feedback generation means" is a function or device for generating and providing customized advice or suggestions to the user based on the analyzed data.

[0353] "Emotion engine means" means a software component or algorithm for recognizing and analyzing emotional states from selfie videos or images input by a user.

[0354] "Management means" refers to software and service infrastructure for performing functions such as user schedule management and reminder notifications.

[0355] "Means for providing advice appropriate to emotions and health status using a generative AI model" is a function that utilizes a pre-trained artificial intelligence model to generate optimal advice based on the user's emotional state and health data.

[0356] The system of this invention provides personalized health management and mental support for pregnant women. By linking a server and user terminal, it analyzes image and video data and generates advice based on emotional and health states.

[0357] Hardware and software usage configuration

[0358] Smartphone: Uses a camera and applications to capture and record images and videos.

[0359] Server: A cloud server is used to run the database, analysis engine, and emotion engine. Specifically, cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) is used.

[0360] Machine learning model: Analyze videos and images using OpenCV, TensorFlow, PyTorch, etc.

[0361] Data processing flow

[0362] 1. Data entry and upload

[0363] Users use their smartphone camera to take selfies or photos of their meals and upload them to the application. For example, they can take a photo of their breakfast and tap the "upload" button in the app to send the data to the server.

[0364] 2. Data Analysis

[0365] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a face recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients and evaluate their nutritional value.

[0366] 3. Emotion recognition

[0367] Recognize user emotions from selfie videos. This uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions.

[0368] 4. Feedback Generation

[0369] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. For example, it may provide advice such as, "You're lacking in vitamin D, so we recommend including fish in your diet," or "You seem tired today, so we recommend some light stretching."

[0370] 5. Data Sync and Notifications

[0371] The feedback generated by the server is sent to the user's device, and the system also links with the maternal and child health handbook database to update the user's health information and send reminders for checkup dates.

[0372] 6. Displaying feedback to users

[0373] Users can view feedback and suggestions within the app, such as breakfast tips, check-up reminders, and emotional support advice.

[0374] Specific examples

[0375] If a user takes a photo of their breakfast and uploads the selfie video to the app, the server will analyze the breakfast photo and determine that it contains bread, eggs, and salad. If the server determines that the user is deficient in vitamin D, it will generate feedback suggesting that the user should include fish in their lunch. At the same time, if the analysis of the selfie video reveals that the user looks tired, the emotion engine will recognize this as "fatigue" and add feedback such as, "You seem tired today, so we recommend some light stretching."

[0376] Prompt Sentence Examples

[0377] "Please analyze the breakfast photos uploaded by the user. At the same time, please analyze the user's emotional state based on the selfie videos uploaded by the user. Based on the results, please generate dietary and mental support advice appropriate for the user's health and emotional state, and provide it as feedback."

[0378] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0379] Program processing steps

[0380] Step 1: Data entry and upload

[0381] Users use their smartphone camera to take selfie videos and photos of their meals and upload them to the application. The input data is the selfie video and photos of the meal, which are then sent to a cloud server. Specifically, users take a photo of their breakfast and tap the "upload" button in the app, uploading the data to the server. The server receives this data and prepares it for analysis.

[0382] Step 2: Data analysis

[0383] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a facial recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients in the photo and evaluate their nutritional value. The input data are video frames and food photos, and the output is facial features, a list of ingredients, and the corresponding nutritional information.

[0384] Step 3: Emotion Recognition

[0385] The server recognizes the user's emotions from the selfie video. It uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions. The input is facial features extracted from the video, and the output is the type of emotion (e.g., happiness, fatigue, anxiety, etc.). The server uses this result to understand the user's mental state.

[0386] Step 4: Feedback generation

[0387] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. First, it makes dietary suggestions based on the nutritional assessment results, and then adds mental support advice that takes into account the user's emotional state. For example, it might suggest, "You're lacking in vitamin D, so we recommend including fish in your diet," or, "You seem tired today, so we recommend some light stretching." The input is nutritional and emotional data, and the output is a personalized feedback message.

[0388] Step 5: Data Sync and Notifications

[0389] The feedback generated by the server is sent to the user's device. At the same time, it links with the maternal and child health handbook database to update the user's health information. In addition, reminder notifications are sent to the device when checkup dates approach. The input is the generated feedback message and the maternal and child health handbook database, and the output is the updated health information and notification message.

[0390] Step 6: Show feedback to the user

[0391] The user checks the feedback and suggestions on the device. The app displays breakfast advice, medical checkup reminders, and emotional support advice. The input is the feedback message, and the output is the displayed advice and notifications. The user can use this information to manage their health and mental health.

[0392] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0393] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0394] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0395] [Second embodiment]

[0396] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0397] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0398] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0399] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0400] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0402] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0403] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0404] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0405] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0406] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0407] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0408] ---

[0409] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0410] Specific Embodiments

[0411] System Configuration

[0412] The system consists of the following main components:

[0413] 1. A means of inputting the user's image (terminal)

[0414] 2. Artificial intelligence means (server) for analyzing images

[0415] 3. Means of generating feedback (server)

[0416] 4. Means for managing user schedules (terminals, servers)

[0417] 5. Means of Providing Appropriate Suggestions (Server)

[0418] 6. Means of obtaining external data (server)

[0419] Program processing flow

[0420] Data entry and upload

[0421] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, users can take a photo of their breakfast and complete the process by tapping the "upload" button in the app.

[0422] Data analysis

[0423] The server analyzes the received data. In the case of selfie videos, artificial intelligence means analyze the video frames to assess complexion, facial expression and skin condition. In the case of food photos, image recognition technology is used to identify ingredients in the photo and assess their nutritional value.

[0424] Feedback Generation

[0425] Based on the analysis results, the server evaluates the user's health status and generates customized feedback, including specific advice such as "You are deficient in vitamin D, so we recommend incorporating fish into your diet," along with an appropriate exercise plan.

[0426] Data Sync and Notifications

[0427] The server generates feedback and sends it to the user's device. It also connects to the maternal and child health handbook database to periodically update health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0428] Displaying feedback to the user

[0429] Users can view the feedback and suggestions they receive within the application, for example, breakfast recommendations or reminders for upcoming checkups, making it easier for users to manage their health.

[0430] Specific examples

[0431] One day, a user wakes up, takes a photo of their breakfast with their smartphone, and uploads it to the server via the app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, it identifies nutrients that the user is lacking (e.g., vitamin D). The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The device displays the received feedback within the app, allowing the user to choose ingredients appropriate for that day's lunch. Furthermore, if a regular medical checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to go for the checkup.

[0432] In this way, the system is designed to be easy for users to use on a daily basis, and provides comprehensive support for health management during pregnancy.

[0433] The processing flow will be explained below.

[0434] Step 1:

[0435] Users use their smartphones to take selfies or photos of their food, and the app's camera function is used to capture data.

[0436] Step 2:

[0437] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[0438] Step 3:

[0439] The server pre-processes the received video and photo data for analysis, for example by extracting video frames and compressing images.

[0440] Step 4:

[0441] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[0442] Step 5:

[0443] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[0444] Step 6:

[0445] The server combines the results of video and photo analysis to assess the user's current health status, and generates customized feedback based on the assessment results.

[0446] Step 7:

[0447] The server will suggest appropriate meal plans and exercise plans based on the user's health and nutritional status, providing specific advice such as "recommending a lunch that includes fish."

[0448] Step 8:

[0449] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[0450] Step 9:

[0451] The server connects to the maternal and child health handbook database to update the user's health information, checks the dates of regular checkups, and creates necessary reminders.

[0452] Step 10:

[0453] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[0454] Step 11:

[0455] Users open the app and review the feedback and suggestions they have received. The scheduling feature allows users to add the next checkup date to their calendar.

[0456] In this way, the system is structured around an entire flow from user input to processing on the server, generating and notifying feedback, and finally presenting it to the user.

[0457] Example 1

[0458] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0459] Proper health management and nutritional balance are extremely important for pregnant women. However, existing health management systems lack the ability to provide personalized feedback, making it difficult for users to obtain specific advice from photos and videos they take. They also lack the ability to recommend appropriate meal plans and exercise plans, and are inconvenient when it comes to managing health information and providing schedule notifications.

[0460] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0461] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results, a management means for managing the user's schedule and notifications, and a means for displaying the feedback and suggestions received by the user. This enables the provision of customized health management and feedback that the user can easily use on a daily basis.

[0462] "Means for users to input images and videos" refers to the function that allows users to take and input images and videos necessary for health management using mobile devices such as smartphones and tablets.

[0463] "Artificial intelligence means" refers to the function of using AI algorithms and machine learning models implemented on the server to analyze input images and videos and evaluate or predict the user's health condition.

[0464] "Feedback generation means" refers to a function that automatically generates customized health advice and suggestions for the user based on the results analyzed by the artificial intelligence means.

[0465] "Management tools" refers to the ability to manage a user's schedule and tasks and provide reminder notifications as needed.

[0466] "Means for displaying feedback and suggestions" refers to a function that displays feedback and suggestions generated by the server on the user's device so that they can be visually confirmed.

[0467] "Means for suggesting meal menus and exercise plans" refers to a function that generates appropriate meal menus and exercise plans based on the user's health condition and analysis results, and presents them as advice.

[0468] "Acquisition means" refers to the function of acquiring user health information from external databases and integrating it into the system.

[0469] MODE FOR CARRYING OUT THE INVENTION

[0470] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0471] System Configuration

[0472] The system consists of the following main components:

[0473] 1. The means (terminal) by which users input images and videos

[0474] 2. Artificial intelligence means (server) for analyzing images and videos

[0475] 3. Means of generating feedback (server)

[0476] 4. User schedule management and notification methods (terminals, servers)

[0477] 5. A means (device) for displaying feedback and suggestions

[0478] 6. Proposal tool (server) that proposes appropriate meal menus and exercise plans

[0479] 7. Means for obtaining health information from external databases (server)

[0480] Data entry and upload

[0481] Users use their smartphone cameras to take selfies or photos of their meals. For example, a user may take photos of bread, eggs, and salad for breakfast. This data is uploaded to a server through an application. For example, when a user taps the "upload" button in an app after taking a photo, the device uploads the data to the server via Wi-Fi or mobile network.

[0482] Data analysis

[0483] The server analyzes the received data. In the case of selfie videos, artificial intelligence analyzes each frame of the video to evaluate the complexion, facial expression, and skin condition. Specifically, AI technologies such as TensorFlow and AWS Rekognition are used. For example, the server scores the user's complexion and facial expression to evaluate their health condition. In the case of a photo of a meal, image recognition technology is used to identify the ingredients contained in the photo and evaluate their nutritional value. Typical technologies include Google Cloud Vision API and OpenCV. The server analyzes the photo and determines that it contains bread, eggs, and salad.

[0484] Feedback Generation

[0485] Based on the analysis results, the server evaluates the user's health status and generates customized feedback. For example, it generates specific advice such as "You are deficient in vitamin D, so we recommend that you include fish in your lunch." It also generates an exercise plan tailored to the user's condition. For example, it suggests "It would be good to do 30 minutes of light walking every day."

[0486] Data Sync and Notifications

[0487] The server sends the generated feedback to the user's device. The device displays this feedback within the app. For example, a reminder such as "Your next health check is in one week" may be displayed. The server also accesses the maternal and child health handbook database to update regular health information. For example, the latest weight and blood pressure records may be added to the maternal and child health handbook. When the next checkup date approaches, the device sends a reminder notification. For example, a push notification such as "Tomorrow is your regular checkup date" may be displayed on the user's smartphone.

[0488] Displaying feedback to the user

[0489] The user opens the application and sees the feedback and suggestions sent by the server. For example, breakfast recommendations or reminders for the next week's medical checkup are displayed within the app. The user can then use this information to plan their meals and exercise for the day.

[0490] Specific examples

[0491] Below are some example prompts to input to a generative AI model:

[0492] "I took a photo of my breakfast this morning to help me manage my health during pregnancy. The photo includes bread, eggs, and salad. I'd like your feedback on health advice for this breakfast and my future meal plan."

[0493] In this way, the system provides users with a customized health management system that can be easily used on a daily basis, enabling comprehensive management of pregnancy health conditions.

[0494] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0495] Program processing flow

[0496] Step 1: Data entry and upload

[0497] Users take selfie videos or photos of their food with their smartphones.

[0498] Input: The user activates the smartphone camera and takes a selfie video or a photo of a meal (e.g., a photo of bread, eggs, and salad for breakfast) that is necessary for health management.

[0499] Output: Captured image and video data.

[0500] The user taps the "Upload" button in the application to send the captured data to the server.

[0501] Input: Tap the "Upload" button within the app.

[0502] Output: Image and video data sent to the server.

[0503] Step 2: Data analysis

[0504] The server receives the uploaded data and begins analyzing it.

[0505] Input: Image and video data received by the server.

[0506] Output: Data ready for analysis.

[0507] In the case of selfie videos, the server uses AI technology (e.g., TensorFlow or AWS Rekognition) to analyze each frame of the video and evaluate complexion, facial expression, and skin condition.

[0508] Input: Selfie video data.

[0509] Output: Evaluation results of complexion, facial expression, and skin condition.

[0510] For photos of meals, the server uses image recognition technology (e.g., Google Cloud Vision API or OpenCV) to identify ingredients and assess their nutritional value.

[0511] Input: Food photo data.

[0512] Output: Identified ingredients and nutrient evaluation results.

[0513] Step 3: Feedback generation

[0514] The server evaluates the user's health status based on the analysis results.

[0515] Input: Evaluation results of ingredients and nutrients, evaluation results of complexion and facial expression.

[0516] Output: Health status assessment results.

[0517] The server generates customized feedback, such as specific advice like "You're lacking vitamin D, so we recommend you include fish in your lunch."

[0518] Input: Health status assessment results.

[0519] Output: Customized feedback and advice.

[0520] Step 4: Data Sync and Notifications

[0521] The server generates feedback and sends it to the user's device.

[0522] Input: Customized feedback and advice.

[0523] Output: Feedback sent to the user device.

[0524] The server accesses the maternal and child health handbook database and updates the health information periodically.

[0525] Input: Latest health information.

[0526] Output: Updated maternal and child health handbook database.

[0527] When the next checkup date approaches, the server will send a reminder notification to the user's device.

[0528] Input: Schedule information.

[0529] Output: Send reminder notification.

[0530] Step 5: Display feedback to the user

[0531] The user opens the app and reviews the feedback and suggestions they received.

[0532] Input: Feedback sent to the user's device.

[0533] Output: Feedback and suggestions displayed within the application.

[0534] Based on the displayed feedback, users plan their daily health management, food choices, and exercise plans.

[0535] Input: Customized feedback and suggestions.

[0536] Output: Implementation of a specific health management plan.

[0537] (Application example 1)

[0538] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0539] Pregnant women need to properly manage their health, and for this, it is important to check their daily diet and evaluate their health. However, doing this manually requires a great deal of effort and time, making it difficult for pregnant women to properly manage their health themselves. Furthermore, even if they receive appropriate dietary suggestions, they lack the means to put them into practice, which makes it difficult for them to actually improve their dietary habits.

[0540] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0541] In this invention, the server includes an input means for a user to input an image, an artificial intelligence means for analyzing the image input by the user, a feedback generation means for providing the user with customized feedback based on the analysis results, a management means for managing the user's schedule and notifying them, a means for providing specific meal suggestions, and an ordering means for ordering the suggested meals, thereby enabling pregnant women to easily manage their health status and practice a nutritionally balanced diet.

[0542] "User" means a person who intends to use the system to manage their own health condition.

[0543] "Images" refers to selfie videos, food photos, etc. that users take or obtain through input means.

[0544] "Input means" refers to a device or application that allows a user to take an image and input it into the system.

[0545] "Artificial intelligence means" refers to AI technology that analyzes images input by users and identifies facial complexion, facial expressions, and ingredients.

[0546] The "feedback generation means" is a means for providing customized feedback to the user based on the analysis results obtained by the artificial intelligence means.

[0547] "Management tools" refers to features that manage the user's schedule and provide regular notifications and reminders.

[0548] The "means for providing specific meal suggestions" refers to a means having a function for suggesting a meal menu suitable for the user based on the analysis results.

[0549] "Ordering means" refers to the function that allows the user to order the suggested meal menu as is.

[0550] System Overview

[0551] The present invention is a system for managing the health and nutrition of pregnant women. By using various input means to manage the user's health condition, the system facilitates daily health checks and dietary management.

[0552] Hardware and software used

[0553] Smartphone: A device that allows users to take selfie videos and photos of their meals and upload them to the system.

[0554] Server: Hardware for executing the artificial intelligence means for image analysis, the feedback generation means, the management means, the meal suggestion means, and the ordering means.

[0555] AI analysis software: Artificial intelligence technology (e.g., TensorFlow, PyTorch, etc.) for analyzing user images.

[0556] Database system: A system for managing user health information and feedback data (e.g., MySQL, PostgreSQL, etc.).

[0557] Food Delivery API: An API used to order suggested meals online.

[0558] Data processing and calculation

[0559] 1. Data entry: Users use their smartphones to take selfies and photos of their meals, and then upload this data to the server through the app.

[0560] 2. Image analysis: The server analyzes the uploaded image data using AI analysis software to evaluate the user's health condition (complexion, facial expression, skin condition, etc.) and dietary content (ingredients, nutrients, etc.).

[0561] 3. Feedback Generation: The server generates customized feedback for the user based on the analysis results, such as specific advice such as "You are lacking in vitamin D, so we recommend that you include fish in your diet."

[0562] 4. Schedule Management and Notification: The server manages the user's schedule and notifies them of regular health checkups and meal suggestions, as well as providing reminders for regular checkups during pregnancy.

[0563] 5. Meal Suggestion and Ordering: The server will suggest appropriate meal menus to the user based on the analysis results and allow the suggested meals to be ordered through the food delivery API.

[0564] Specific examples

[0565] Suppose a pregnant woman takes a photo of her breakfast and uploads it to a server via an app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, nutrients that the woman is lacking (e.g., vitamin D) are identified. The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The user can then order the suggested meal directly through the app and use the delivery service. Additionally, if a regular checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to attend.

[0566] Prompt Sentence Examples

[0567] 1. "Consider dietary health management for a woman in her 50s. Using a photo of her breakfast, assess her nutritional status and suggest a meal plan to provide the necessary nutrients."

[0568] 2. "Provide pictures of foods that may be calcium deficient for a pregnant woman. Suggest calcium-rich meals for her."

[0569] 3. "Provide pictures of meals that are deficient in certain vitamins. Suggest foods needed to supplement the vitamins and provide a food delivery service where you can order those foods."

[0570] In this way, the embodiments of the invention enable pregnant women to easily manage their health and practice a nutritionally balanced diet.

[0571] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0572] Step 1:

[0573] Users use their smartphone cameras to take selfies and photos of their meals.

[0574] Input: Selfie videos and food photos taken with your smartphone.

[0575] How it works: The user launches the dedicated app and uses the camera function to take a photo of their face or the food they're eating.

[0576] Output: Selfie videos and food photo data.

[0577] Step 2:

[0578] Users upload videos and photos they have taken to the server through the app.

[0579] Input: Selfie videos and food photo data taken with a smartphone.

[0580] How it works: The user taps the "upload" button in the app to send the data to the server.

[0581] Output: Video and photo data uploaded to the server.

[0582] Step 3:

[0583] The data received by the server is analyzed using AI analysis software.

[0584] Input: Uploaded selfie videos and food photo data.

[0585] How it works: The server uses AI analysis software (e.g., TensorFlow, PyTorch) to analyze video frames and identify ingredients using image recognition technology.

[0586] Output: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0587] Step 4:

[0588] The server generates customized feedback based on the analysis results.

[0589] Input: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0590] How it works: The server uses the feedback generator to generate specific advice, such as "You are deficient in vitamin D, so we recommend that you include fish in your diet."

[0591] Output: Customized feedback content.

[0592] Step 5:

[0593] The server sends the suggested meal menu to the user's terminal.

[0594] Input: Your customized feedback.

[0595] How it works: The server sends the feedback to the user's device and displays it as a notification within the app on the device.

[0596] Output: Meal suggestions and health advice displayed on the user's device.

[0597] Step 6:

[0598] The user reviews the suggested meal menu and, if desired, places an order within the app.

[0599] Input: Meal menu suggestions displayed on the device.

[0600] How it works: The user selects a suggested menu item and uses the in-app ordering feature to order delivery.

[0601] Output: A meal order from a delivery service.

[0602] Step 7:

[0603] The server manages the user's schedule and notifies them.

[0604] Input: User's schedule information and regular health check information.

[0605] How it works: The server manages the user's health information and sends reminders for the next checkup date and other notifications to the device.

[0606] Output: Health care reminder notification.

[0607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0608] ---

[0609] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0610] Specific Embodiments

[0611] System Configuration

[0612] The system consists of the following main components:

[0613] 1. A means of inputting the user's image (terminal)

[0614] 2. Artificial intelligence means (server) for analyzing images

[0615] 3. Means of generating feedback (server)

[0616] 4. Means for managing user schedules (terminals, servers)

[0617] 5. Means of Providing Appropriate Suggestions (Server)

[0618] 6. Means of obtaining external data (server)

[0619] 7. Emotion engine (server) that recognizes user emotions

[0620] Program processing flow

[0621] Data entry and upload

[0622] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[0623] Data analysis

[0624] The server analyzes the received data. First, for video analysis, artificial intelligence means analyzes video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value.

[0625] emotion recognition

[0626] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This allows the system to understand the user's mental state.

[0627] Feedback Generation

[0628] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[0629] Data Sync and Notifications

[0630] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0631] Displaying feedback to the user

[0632] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[0633] Specific examples

[0634] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis reveals a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "It would be a good idea to include fish in your lunch."

[0635] At the same time, analysis of the selfie video reveals that the user has a tired expression. The emotion engine recognizes this as "fatigue," and the server provides feedback by adding mental support advice such as, "You seem tired today, so we recommend some light stretching."

[0636] In this way, this system simplifies daily use for users and simultaneously provides comprehensive health management and mental support for pregnant women.

[0637] The processing flow will be explained below.

[0638] Step 1:

[0639] The user uses their smartphone to take a selfie video or a photo of their meal using the camera function within the app.

[0640] Step 2:

[0641] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[0642] Step 3:

[0643] The server preprocesses the video and photo data it receives. For video, it extracts frames, and for photos, it compresses them.

[0644] Step 4:

[0645] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[0646] Step 5:

[0647] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[0648] Step 6:

[0649] The server uses an emotion engine to recognize the user's emotions from the selfie video. For example, if the user smiles, it identifies "happiness," and if they frown, it identifies "anxiety."

[0650] Step 7:

[0651] The server combines the results of video analysis, photo analysis, and emotion recognition to assess the user's current health and mental state.

[0652] Step 8:

[0653] The server generates customized feedback based on the evaluation results, such as "You are lacking in vitamin D, so we recommend including fish in your diet" and "You seem tired today, so we recommend some light stretching exercises."

[0654] Step 9:

[0655] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[0656] Step 10:

[0657] The server connects to the maternal and child health handbook database to update the user's health information, and when the next checkup date approaches, a reminder notification is sent to the device.

[0658] Step 11:

[0659] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[0660] Step 12:

[0661] Users can open the app to view the feedback and suggestions they have received, as well as emotional support advice. Users can also add their next check-up date to their calendar through the scheduling feature.

[0662] In this way, the system is structured as a series of steps from user input to processing on the server, feedback generation and notification, and finally presentation to the user.

[0663] Example 2

[0664] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] Conventional health management systems only assess the user's health status using images and videos, but are unable to provide feedback that takes into account the user's emotional state. As a result, it is difficult to provide comprehensive health management and mental support to pregnant women at the same time. Furthermore, schedule management, appropriate meal menus, and exercise plan suggestions are not integrated, forcing users to use multiple applications.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0667] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results and the user's emotion recognition results, and a management means for managing the user's schedule and notifying them. This allows the user to receive feedback that takes into account their emotional state in addition to an evaluation of their health condition using images and videos, and further allows them to receive integrated suggestions for schedule management, meal menus, and exercise plans.

[0668] "Means for user image and video input" refers to devices or software that allow users to take selfie videos or photos of their meals and upload that data to the system.

[0669] "Artificial intelligence means" refers to devices or systems that have programs or algorithms, including machine learning and image recognition technologies, used to analyze images or videos input by users.

[0670] A "feedback generation means" is a device or software that evaluates a user's health condition and emotional state based on the analysis results and the user's emotion recognition results, and provides customized advice and instructions.

[0671] "Management means" refers to devices or software for managing the user's schedule and sending notifications, and is primarily responsible for managing the next checkup date and reminder notifications.

[0672] The "suggestion means" refers to a device or software that suggests appropriate meal menus and exercise plans to users based on the analysis results and emotion recognition results.

[0673] "Acquisition means" refers to a device or software for acquiring the user's health information from an external database.

[0674] "Emotion recognition results" are information about a user's emotional state (e.g., happiness, anxiety, fatigue, etc.) obtained by analyzing the user's selfie video.

[0675] "Customized feedback" refers to advice and instructions created specifically for each user based on analysis and emotion recognition results.

[0676] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0677] System Configuration

[0678] The system consists of the following main components:

[0679] 1. Means (terminal) for inputting user images and videos

[0680] 2. Artificial intelligence means (server) for analyzing images and videos

[0681] 3. Means of generating feedback (server)

[0682] 4. Means for managing user schedules (terminals, servers)

[0683] 5. Means of Providing Appropriate Suggestions (Server)

[0684] 6. Means of obtaining external data (server)

[0685] 7. Emotion engine (server) that recognizes user emotions

[0686] Data entry and upload

[0687] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[0688] Data analysis

[0689] The server analyzes the received data. First, for video analysis, artificial intelligence means analyze video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value. Specific software used includes TensorFlow and OpenCV for image recognition.

[0690] emotion recognition

[0691] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This makes it possible to understand the user's mental state. Specifically, it uses Microsoft Azure Cognitive Services.

[0692] Feedback Generation

[0693] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[0694] Data Sync and Notifications

[0695] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0696] Displaying feedback to the user

[0697] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[0698] Specific examples

[0699] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis results indicate a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "You should include fish in your lunch." At the same time, analysis of the selfie video reveals that the user looks tired. The emotion engine recognizes this as "fatigue," and the server provides additional mental support advice, such as, "You seem tired today, so we recommend some light stretching."

[0700] Prompt Sentence Examples

[0701] "Analyze selfie videos and food photos to provide users with personalized health feedback. Add emotion recognition and include mental support advice."

[0702] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0703] Step 1:

[0704] The user uses the smartphone camera to take a selfie video or a photo of their meal. After taking the photo, the user taps the "upload" button in the app to send the data to the server. Specifically, the smartphone's camera app launches, and includes functions for taking a selfie video for a certain period of time and taking photos of the meal. The data input is the video and photos taken by the user, and the output is the video and photo data uploaded to the server.

[0705] Step 2:

[0706] The server decompresses the received video and photo data and performs initial data processing. Specifically, the server applies a data decompression algorithm and performs preprocessing according to the data format. The input to this step is the received compressed video and photo data, and the output is the decompressed and preprocessed data.

[0707] Step 3:

[0708] The server starts video analysis. Using artificial intelligence tools (e.g., OpenCV or TensorFlow models), it analyzes each frame of the video and evaluates the user's complexion, facial expression, and skin condition. Specifically, the server divides the video into frames, performs facial recognition on each frame, and analyzes facial expression and complexion. The input of this step is the preprocessed video data, and the output is the evaluation results of the user's complexion, facial expression, and skin condition.

[0709] Step 4:

[0710] The server analyzes the photo. It uses image recognition technology (e.g., TensorFlow) to identify the ingredients in the photo and evaluate the nutrients in each. Specifically, the server extracts image features and compares them with a database to identify the ingredients. The input to this step is the preprocessed photo data, and the output is the identification of ingredients and the evaluation of their nutrients.

[0711] Step 5:

[0712] The server runs an emotion recognition engine to recognize the user's emotions from the selfie video. For example, a smile can be identified as "happiness," while a frown can be identified as "anxiety." Specifically, the server uses an emotion recognition algorithm (e.g., Microsoft Azure Cognitive Services) to analyze facial feature points. The input for this step is the video analysis results, and the output is the recognition result of the user's emotional state.

[0713] Step 6:

[0714] The server evaluates the user's health status based on the video and photo analysis results and emotion recognition results, and generates customized feedback. Specifically, the server executes a program that integrates multiple analysis results to perform a comprehensive health assessment and generates feedback based on that assessment. The input for this step is the video and photo analysis results and emotion recognition results, and the output is customized feedback.

[0715] Step 7:

[0716] The server sends the generated feedback to the user's device. Furthermore, the server connects to the maternal and child health handbook database to update the user's health information and send reminder notifications when checkup dates approach. Specifically, the server communicates over the network, sends feedback data, and schedules database updates and reminder notifications. The input of this step is the generated feedback, and the output is the feedback sent to the device and the updated database information.

[0717] Step 8:

[0718] The user checks the feedback and advice through the application. Specifically, the user opens the app and accesses the "Feedback" section to view dietary advice, checkup reminders, and emotional support advice. The input of this step is the feedback data sent to the device, and the output is the feedback information displayed to the user.

[0719] (Application example 2)

[0720] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0721] Pregnant women experience many physical and mental changes and require appropriate health management and mental support during this process. However, current health care systems are unable to fully address individual needs, and emotional support is particularly lacking. Specific advice on diet and exercise is often not provided, resulting in a lack of efficient and comprehensive support for pregnant women. Therefore, there is a need for a system that allows pregnant women to easily manage their health and receive emotional support.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0723] In this invention, the server includes a means for a user to input images, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing the user with customized feedback and meal suggestions based on the analysis results, an emotion engine means for recognizing the user's emotional state, a management means for managing the user's schedule and sending notifications, and a means for providing advice appropriate to the user's emotions and health condition using a generative AI model. This enables pregnant women to comprehensively understand their health condition and receive personalized meal suggestions and mental support.

[0724] "Means for users to input images" refers to a device or function that allows users to take images or videos using a device such as a smartphone and input them into the system.

[0725] "Artificial intelligence means" refers to a computing device that includes algorithms and programs for analyzing input image and video data and determining the user's health condition, the nutritional value of ingredients, and even their emotions.

[0726] The "feedback generation means" is a function or device for generating and providing customized advice or suggestions to the user based on the analyzed data.

[0727] "Emotion engine means" means a software component or algorithm for recognizing and analyzing emotional states from selfie videos or images input by a user.

[0728] "Management means" refers to software and service infrastructure for performing functions such as user schedule management and reminder notifications.

[0729] "Means for providing advice appropriate to emotions and health status using a generative AI model" is a function that utilizes a pre-trained artificial intelligence model to generate optimal advice based on the user's emotional state and health data.

[0730] The system of this invention provides personalized health management and mental support for pregnant women. By linking a server and user terminal, it analyzes image and video data and generates advice based on emotional and health states.

[0731] Hardware and software usage configuration

[0732] Smartphone: Uses a camera and applications to capture and record images and videos.

[0733] Server: A cloud server is used to run the database, analysis engine, and emotion engine. Specifically, cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) is used.

[0734] Machine learning model: Analyze videos and images using OpenCV, TensorFlow, PyTorch, etc.

[0735] Data processing flow

[0736] 1. Data entry and upload

[0737] Users use their smartphone camera to take selfies or photos of their meals and upload them to the application. For example, they can take a photo of their breakfast and tap the "upload" button in the app to send the data to the server.

[0738] 2. Data Analysis

[0739] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a face recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients and evaluate their nutritional value.

[0740] 3. Emotion recognition

[0741] Recognize user emotions from selfie videos. This uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions.

[0742] 4. Feedback Generation

[0743] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. For example, it may provide advice such as, "You're lacking in vitamin D, so we recommend including fish in your diet," or "You seem tired today, so we recommend some light stretching."

[0744] 5. Data Sync and Notifications

[0745] The feedback generated by the server is sent to the user's device, and the system also links with the maternal and child health handbook database to update the user's health information and send reminders for checkup dates.

[0746] 6. Displaying feedback to users

[0747] Users can view feedback and suggestions within the app, such as breakfast tips, check-up reminders, and emotional support advice.

[0748] Specific examples

[0749] If a user takes a photo of their breakfast and uploads the selfie video to the app, the server will analyze the breakfast photo and determine that it contains bread, eggs, and salad. If the server determines that the user is deficient in vitamin D, it will generate feedback suggesting that the user should include fish in their lunch. At the same time, if the analysis of the selfie video reveals that the user looks tired, the emotion engine will recognize this as "fatigue" and add feedback such as, "You seem tired today, so we recommend some light stretching."

[0750] Prompt Sentence Examples

[0751] "Please analyze the breakfast photos uploaded by the user. At the same time, please analyze the user's emotional state based on the selfie videos uploaded by the user. Based on the results, please generate dietary and mental support advice appropriate for the user's health and emotional state, and provide it as feedback."

[0752] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0753] Program processing steps

[0754] Step 1: Data entry and upload

[0755] Users use their smartphone camera to take selfie videos and photos of their meals and upload them to the application. The input data is the selfie video and photos of the meal, which are then sent to a cloud server. Specifically, users take a photo of their breakfast and tap the "upload" button in the app, uploading the data to the server. The server receives this data and prepares it for analysis.

[0756] Step 2: Data analysis

[0757] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a facial recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients in the photo and evaluate their nutritional value. The input data are video frames and food photos, and the output is facial features, a list of ingredients, and the corresponding nutritional information.

[0758] Step 3: Emotion Recognition

[0759] The server recognizes the user's emotions from the selfie video. It uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions. The input is facial features extracted from the video, and the output is the type of emotion (e.g., happiness, fatigue, anxiety, etc.). The server uses this result to understand the user's mental state.

[0760] Step 4: Feedback generation

[0761] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. First, it makes dietary suggestions based on the nutritional assessment results, and then adds mental support advice that takes into account the user's emotional state. For example, it might suggest, "You're lacking in vitamin D, so we recommend including fish in your diet," or, "You seem tired today, so we recommend some light stretching." The input is nutritional and emotional data, and the output is a personalized feedback message.

[0762] Step 5: Data Sync and Notifications

[0763] The feedback generated by the server is sent to the user's device. At the same time, it links with the maternal and child health handbook database to update the user's health information. In addition, reminder notifications are sent to the device when checkup dates approach. The input is the generated feedback message and the maternal and child health handbook database, and the output is the updated health information and notification message.

[0764] Step 6: Show feedback to the user

[0765] The user checks the feedback and suggestions on the device. The app displays breakfast advice, medical checkup reminders, and emotional support advice. The input is the feedback message, and the output is the displayed advice and notifications. The user can use this information to manage their health and mental health.

[0766] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0767] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0768] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0769] [Third embodiment]

[0770] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0771] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0772] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0773] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0774] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0775] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0776] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0777] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0778] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0779] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0780] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0781] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0782] ---

[0783] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0784] Specific Embodiments

[0785] System Configuration

[0786] The system consists of the following main components:

[0787] 1. A means of inputting the user's image (terminal)

[0788] 2. Artificial intelligence means (server) for analyzing images

[0789] 3. Means of generating feedback (server)

[0790] 4. Means for managing user schedules (terminals, servers)

[0791] 5. Means of Providing Appropriate Suggestions (Server)

[0792] 6. Means of obtaining external data (server)

[0793] Program processing flow

[0794] Data entry and upload

[0795] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, users can take a photo of their breakfast and complete the process by tapping the "upload" button in the app.

[0796] Data analysis

[0797] The server analyzes the received data. In the case of selfie videos, artificial intelligence means analyze the video frames to assess complexion, facial expression and skin condition. In the case of food photos, image recognition technology is used to identify ingredients in the photo and assess their nutritional value.

[0798] Feedback Generation

[0799] Based on the analysis results, the server evaluates the user's health status and generates customized feedback, including specific advice such as "You are deficient in vitamin D, so we recommend incorporating fish into your diet," along with an appropriate exercise plan.

[0800] Data Sync and Notifications

[0801] The server generates feedback and sends it to the user's device. It also connects to the maternal and child health handbook database to periodically update health information. When the next checkup date approaches, a reminder notification is sent to the device.

[0802] Displaying feedback to the user

[0803] Users can view the feedback and suggestions they receive within the application, for example, breakfast recommendations or reminders for upcoming checkups, making it easier for users to manage their health.

[0804] Specific examples

[0805] One day, a user wakes up, takes a photo of their breakfast with their smartphone, and uploads it to the server via the app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, it identifies nutrients that the user is lacking (e.g., vitamin D). The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The device displays the received feedback within the app, allowing the user to choose ingredients appropriate for that day's lunch. Furthermore, if a regular medical checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to go for the checkup.

[0806] In this way, the system is designed to be easy for users to use on a daily basis, and provides comprehensive support for health management during pregnancy.

[0807] The processing flow will be explained below.

[0808] Step 1:

[0809] Users use their smartphones to take selfies or photos of their food, and the app's camera function is used to capture data.

[0810] Step 2:

[0811] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[0812] Step 3:

[0813] The server pre-processes the received video and photo data for analysis, for example by extracting video frames and compressing images.

[0814] Step 4:

[0815] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[0816] Step 5:

[0817] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[0818] Step 6:

[0819] The server combines the results of video and photo analysis to assess the user's current health status, and generates customized feedback based on the assessment results.

[0820] Step 7:

[0821] The server will suggest appropriate meal plans and exercise plans based on the user's health and nutritional status, providing specific advice such as "recommending a lunch that includes fish."

[0822] Step 8:

[0823] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[0824] Step 9:

[0825] The server connects to the maternal and child health handbook database to update the user's health information, checks the dates of regular checkups, and creates necessary reminders.

[0826] Step 10:

[0827] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[0828] Step 11:

[0829] Users open the app and review the feedback and suggestions they have received. The scheduling feature allows users to add the next checkup date to their calendar.

[0830] In this way, the system is structured around an entire flow from user input to processing on the server, generating and notifying feedback, and finally presenting it to the user.

[0831] Example 1

[0832] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0833] Proper health management and nutritional balance are extremely important for pregnant women. However, existing health management systems lack the ability to provide personalized feedback, making it difficult for users to obtain specific advice from photos and videos they take. They also lack the ability to recommend appropriate meal plans and exercise plans, and are inconvenient when it comes to managing health information and providing schedule notifications.

[0834] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0835] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results, a management means for managing the user's schedule and notifications, and a means for displaying the feedback and suggestions received by the user. This enables the provision of customized health management and feedback that the user can easily use on a daily basis.

[0836] "Means for users to input images and videos" refers to the function that allows users to take and input images and videos necessary for health management using mobile devices such as smartphones and tablets.

[0837] "Artificial intelligence means" refers to the function of using AI algorithms and machine learning models implemented on the server to analyze input images and videos and evaluate or predict the user's health condition.

[0838] "Feedback generation means" refers to a function that automatically generates customized health advice and suggestions for the user based on the results analyzed by the artificial intelligence means.

[0839] "Management tools" refers to the ability to manage a user's schedule and tasks and provide reminder notifications as needed.

[0840] "Means for displaying feedback and suggestions" refers to a function that displays feedback and suggestions generated by the server on the user's device so that they can be visually confirmed.

[0841] "Means for suggesting meal menus and exercise plans" refers to a function that generates appropriate meal menus and exercise plans based on the user's health condition and analysis results, and presents them as advice.

[0842] "Acquisition means" refers to the function of acquiring user health information from external databases and integrating it into the system.

[0843] MODE FOR CARRYING OUT THE INVENTION

[0844] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[0845] System Configuration

[0846] The system consists of the following main components:

[0847] 1. The means (terminal) by which users input images and videos

[0848] 2. Artificial intelligence means (server) for analyzing images and videos

[0849] 3. Means of generating feedback (server)

[0850] 4. User schedule management and notification methods (terminals, servers)

[0851] 5. A means (device) for displaying feedback and suggestions

[0852] 6. Proposal tool (server) that proposes appropriate meal menus and exercise plans

[0853] 7. Means for obtaining health information from external databases (server)

[0854] Data entry and upload

[0855] Users use their smartphone cameras to take selfies or photos of their meals. For example, a user may take photos of bread, eggs, and salad for breakfast. This data is uploaded to a server through an application. For example, when a user taps the "upload" button in an app after taking a photo, the device uploads the data to the server via Wi-Fi or mobile network.

[0856] Data analysis

[0857] The server analyzes the received data. In the case of selfie videos, artificial intelligence analyzes each frame of the video to evaluate the complexion, facial expression, and skin condition. Specifically, AI technologies such as TensorFlow and AWS Rekognition are used. For example, the server scores the user's complexion and facial expression to evaluate their health condition. In the case of a photo of a meal, image recognition technology is used to identify the ingredients contained in the photo and evaluate their nutritional value. Typical technologies include Google Cloud Vision API and OpenCV. The server analyzes the photo and determines that it contains bread, eggs, and salad.

[0858] Feedback Generation

[0859] Based on the analysis results, the server evaluates the user's health status and generates customized feedback. For example, it generates specific advice such as "You are deficient in vitamin D, so we recommend that you include fish in your lunch." It also generates an exercise plan tailored to the user's condition. For example, it suggests "It would be good to do 30 minutes of light walking every day."

[0860] Data Sync and Notifications

[0861] The server sends the generated feedback to the user's device. The device displays this feedback within the app. For example, a reminder such as "Your next health check is in one week" may be displayed. The server also accesses the maternal and child health handbook database to update regular health information. For example, the latest weight and blood pressure records may be added to the maternal and child health handbook. When the next checkup date approaches, the device sends a reminder notification. For example, a push notification such as "Tomorrow is your regular checkup date" may be displayed on the user's smartphone.

[0862] Displaying feedback to the user

[0863] The user opens the application and sees the feedback and suggestions sent by the server. For example, breakfast recommendations or reminders for the next week's medical checkup are displayed within the app. The user can then use this information to plan their meals and exercise for the day.

[0864] Specific examples

[0865] Below are some example prompts to input to a generative AI model:

[0866] "I took a photo of my breakfast this morning to help me manage my health during pregnancy. The photo includes bread, eggs, and salad. I'd like your feedback on health advice for this breakfast and my future meal plan."

[0867] In this way, the system provides users with a customized health management system that can be easily used on a daily basis, enabling comprehensive management of pregnancy health conditions.

[0868] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0869] Program processing flow

[0870] Step 1: Data entry and upload

[0871] Users take selfie videos or photos of their food with their smartphones.

[0872] Input: The user activates the smartphone camera and takes a selfie video or a photo of a meal (e.g., a photo of bread, eggs, and salad for breakfast) that is necessary for health management.

[0873] Output: Captured image and video data.

[0874] The user taps the "Upload" button in the application to send the captured data to the server.

[0875] Input: Tap the "Upload" button within the app.

[0876] Output: Image and video data sent to the server.

[0877] Step 2: Data analysis

[0878] The server receives the uploaded data and begins analyzing it.

[0879] Input: Image and video data received by the server.

[0880] Output: Data ready for analysis.

[0881] In the case of selfie videos, the server uses AI technology (e.g., TensorFlow or AWS Rekognition) to analyze each frame of the video and evaluate complexion, facial expression, and skin condition.

[0882] Input: Selfie video data.

[0883] Output: Evaluation results of complexion, facial expression, and skin condition.

[0884] For photos of meals, the server uses image recognition technology (e.g., Google Cloud Vision API or OpenCV) to identify ingredients and assess their nutritional value.

[0885] Input: Food photo data.

[0886] Output: Identified ingredients and nutrient evaluation results.

[0887] Step 3: Feedback generation

[0888] The server evaluates the user's health status based on the analysis results.

[0889] Input: Evaluation results of ingredients and nutrients, evaluation results of complexion and facial expression.

[0890] Output: Health status assessment results.

[0891] The server generates customized feedback, such as specific advice like "You're lacking vitamin D, so we recommend you include fish in your lunch."

[0892] Input: Health status assessment results.

[0893] Output: Customized feedback and advice.

[0894] Step 4: Data Sync and Notifications

[0895] The server generates feedback and sends it to the user's device.

[0896] Input: Customized feedback and advice.

[0897] Output: Feedback sent to the user device.

[0898] The server accesses the maternal and child health handbook database and updates the health information periodically.

[0899] Input: Latest health information.

[0900] Output: Updated maternal and child health handbook database.

[0901] When the next checkup date approaches, the server will send a reminder notification to the user's device.

[0902] Input: Schedule information.

[0903] Output: Send reminder notification.

[0904] Step 5: Display feedback to the user

[0905] The user opens the app and reviews the feedback and suggestions they received.

[0906] Input: Feedback sent to the user's device.

[0907] Output: Feedback and suggestions displayed within the application.

[0908] Based on the displayed feedback, users plan their daily health management, food choices, and exercise plans.

[0909] Input: Customized feedback and suggestions.

[0910] Output: Implementation of a specific health management plan.

[0911] (Application example 1)

[0912] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0913] Pregnant women need to properly manage their health, and for this, it is important to check their daily diet and evaluate their health. However, doing this manually requires a great deal of effort and time, making it difficult for pregnant women to properly manage their health themselves. Furthermore, even if they receive appropriate dietary suggestions, they lack the means to put them into practice, which makes it difficult for them to actually improve their dietary habits.

[0914] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0915] In this invention, the server includes an input means for a user to input an image, an artificial intelligence means for analyzing the image input by the user, a feedback generation means for providing the user with customized feedback based on the analysis results, a management means for managing the user's schedule and notifying them, a means for providing specific meal suggestions, and an ordering means for ordering the suggested meals, thereby enabling pregnant women to easily manage their health status and practice a nutritionally balanced diet.

[0916] "User" means a person who intends to use the system to manage their own health condition.

[0917] "Images" refers to selfie videos, food photos, etc. that users take or obtain through input means.

[0918] "Input means" refers to a device or application that allows a user to take an image and input it into the system.

[0919] "Artificial intelligence means" refers to AI technology that analyzes images input by users and identifies facial complexion, facial expressions, and ingredients.

[0920] The "feedback generation means" is a means for providing customized feedback to the user based on the analysis results obtained by the artificial intelligence means.

[0921] "Management tools" refers to features that manage the user's schedule and provide regular notifications and reminders.

[0922] The "means for providing specific meal suggestions" refers to a means having a function for suggesting a meal menu suitable for the user based on the analysis results.

[0923] "Ordering means" refers to the function that allows the user to order the suggested meal menu as is.

[0924] System Overview

[0925] The present invention is a system for managing the health and nutrition of pregnant women. By using various input means to manage the user's health condition, the system facilitates daily health checks and dietary management.

[0926] Hardware and software used

[0927] Smartphone: A device that allows users to take selfie videos and photos of their meals and upload them to the system.

[0928] Server: Hardware for executing the artificial intelligence means for image analysis, the feedback generation means, the management means, the meal suggestion means, and the ordering means.

[0929] AI analysis software: Artificial intelligence technology (e.g., TensorFlow, PyTorch, etc.) for analyzing user images.

[0930] Database system: A system for managing user health information and feedback data (e.g., MySQL, PostgreSQL, etc.).

[0931] Food Delivery API: An API used to order suggested meals online.

[0932] Data processing and calculation

[0933] 1. Data entry: Users use their smartphones to take selfies and photos of their meals, and then upload this data to the server through the app.

[0934] 2. Image analysis: The server analyzes the uploaded image data using AI analysis software to evaluate the user's health condition (complexion, facial expression, skin condition, etc.) and dietary content (ingredients, nutrients, etc.).

[0935] 3. Feedback Generation: The server generates customized feedback for the user based on the analysis results, such as specific advice such as "You are lacking in vitamin D, so we recommend that you include fish in your diet."

[0936] 4. Schedule Management and Notification: The server manages the user's schedule and notifies them of regular health checkups and meal suggestions, as well as providing reminders for regular checkups during pregnancy.

[0937] 5. Meal Suggestion and Ordering: The server will suggest appropriate meal menus to the user based on the analysis results and allow the suggested meals to be ordered through the food delivery API.

[0938] Specific examples

[0939] Suppose a pregnant woman takes a photo of her breakfast and uploads it to a server via an app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, nutrients that the woman is lacking (e.g., vitamin D) are identified. The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The user can then order the suggested meal directly through the app and use the delivery service. Additionally, if a regular checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to attend.

[0940] Prompt Sentence Examples

[0941] 1. "Consider dietary health management for a woman in her 50s. Using a photo of her breakfast, assess her nutritional status and suggest a meal plan to provide the necessary nutrients."

[0942] 2. "Provide pictures of foods that may be calcium deficient for a pregnant woman. Suggest calcium-rich meals for her."

[0943] 3. "Provide pictures of meals that are deficient in certain vitamins. Suggest foods needed to supplement the vitamins and provide a food delivery service where you can order those foods."

[0944] In this way, the embodiments of the invention enable pregnant women to easily manage their health and practice a nutritionally balanced diet.

[0945] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0946] Step 1:

[0947] Users use their smartphone cameras to take selfies and photos of their meals.

[0948] Input: Selfie videos and food photos taken with your smartphone.

[0949] How it works: The user launches the dedicated app and uses the camera function to take a photo of their face or the food they're eating.

[0950] Output: Selfie videos and food photo data.

[0951] Step 2:

[0952] Users upload videos and photos they have taken to the server through the app.

[0953] Input: Selfie videos and food photo data taken with a smartphone.

[0954] How it works: The user taps the "upload" button in the app to send the data to the server.

[0955] Output: Video and photo data uploaded to the server.

[0956] Step 3:

[0957] The data received by the server is analyzed using AI analysis software.

[0958] Input: Uploaded selfie videos and food photo data.

[0959] How it works: The server uses AI analysis software (e.g., TensorFlow, PyTorch) to analyze video frames and identify ingredients using image recognition technology.

[0960] Output: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0961] Step 4:

[0962] The server generates customized feedback based on the analysis results.

[0963] Input: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[0964] How it works: The server uses the feedback generator to generate specific advice, such as "You are deficient in vitamin D, so we recommend that you include fish in your diet."

[0965] Output: Customized feedback content.

[0966] Step 5:

[0967] The server sends the suggested meal menu to the user's terminal.

[0968] Input: Your customized feedback.

[0969] How it works: The server sends the feedback to the user's device and displays it as a notification within the app on the device.

[0970] Output: Meal suggestions and health advice displayed on the user's device.

[0971] Step 6:

[0972] The user reviews the suggested meal menu and, if desired, places an order within the app.

[0973] Input: Meal menu suggestions displayed on the device.

[0974] How it works: The user selects a suggested menu item and uses the in-app ordering feature to order delivery.

[0975] Output: A meal order from a delivery service.

[0976] Step 7:

[0977] The server manages the user's schedule and notifies them.

[0978] Input: User's schedule information and regular health check information.

[0979] How it works: The server manages the user's health information and sends reminders for the next checkup date and other notifications to the device.

[0980] Output: Health care reminder notification.

[0981] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0982] ---

[0983] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[0984] Specific Embodiments

[0985] System Configuration

[0986] The system consists of the following main components:

[0987] 1. A means of inputting the user's image (terminal)

[0988] 2. Artificial intelligence means (server) for analyzing images

[0989] 3. Means of generating feedback (server)

[0990] 4. Means for managing user schedules (terminals, servers)

[0991] 5. Means of Providing Appropriate Suggestions (Server)

[0992] 6. Means of obtaining external data (server)

[0993] 7. Emotion engine (server) that recognizes user emotions

[0994] Program processing flow

[0995] Data entry and upload

[0996] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[0997] Data analysis

[0998] The server analyzes the received data. First, for video analysis, artificial intelligence means analyzes video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value.

[0999] emotion recognition

[1000] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This allows the system to understand the user's mental state.

[1001] Feedback Generation

[1002] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[1003] Data Sync and Notifications

[1004] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[1005] Displaying feedback to the user

[1006] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[1007] Specific examples

[1008] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis reveals a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "It would be a good idea to include fish in your lunch."

[1009] At the same time, analysis of the selfie video reveals that the user has a tired expression. The emotion engine recognizes this as "fatigue," and the server provides feedback by adding mental support advice such as, "You seem tired today, so we recommend some light stretching."

[1010] In this way, this system simplifies daily use for users and simultaneously provides comprehensive health management and mental support for pregnant women.

[1011] The processing flow will be explained below.

[1012] Step 1:

[1013] The user uses their smartphone to take a selfie video or a photo of their meal using the camera function within the app.

[1014] Step 2:

[1015] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[1016] Step 3:

[1017] The server preprocesses the video and photo data it receives. For video, it extracts frames, and for photos, it compresses them.

[1018] Step 4:

[1019] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[1020] Step 5:

[1021] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[1022] Step 6:

[1023] The server uses an emotion engine to recognize the user's emotions from the selfie video. For example, if the user smiles, it identifies "happiness," and if they frown, it identifies "anxiety."

[1024] Step 7:

[1025] The server combines the results of video analysis, photo analysis, and emotion recognition to assess the user's current health and mental state.

[1026] Step 8:

[1027] The server generates customized feedback based on the evaluation results, such as "You are lacking in vitamin D, so we recommend including fish in your diet" and "You seem tired today, so we recommend some light stretching exercises."

[1028] Step 9:

[1029] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[1030] Step 10:

[1031] The server connects to the maternal and child health handbook database to update the user's health information, and when the next checkup date approaches, a reminder notification is sent to the device.

[1032] Step 11:

[1033] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[1034] Step 12:

[1035] Users can open the app to view the feedback and suggestions they have received, as well as emotional support advice. Users can also add their next check-up date to their calendar through the scheduling feature.

[1036] In this way, the system is structured as a series of steps from user input to processing on the server, feedback generation and notification, and finally presentation to the user.

[1037] Example 2

[1038] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1039] Conventional health management systems only assess the user's health status using images and videos, but are unable to provide feedback that takes into account the user's emotional state. As a result, it is difficult to provide comprehensive health management and mental support to pregnant women at the same time. Furthermore, schedule management, appropriate meal menus, and exercise plan suggestions are not integrated, forcing users to use multiple applications.

[1040] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1041] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results and the user's emotion recognition results, and a management means for managing the user's schedule and notifying them. This allows the user to receive feedback that takes into account their emotional state in addition to an evaluation of their health condition using images and videos, and further allows them to receive integrated suggestions for schedule management, meal menus, and exercise plans.

[1042] "Means for user image and video input" refers to devices or software that allow users to take selfie videos or photos of their meals and upload that data to the system.

[1043] "Artificial intelligence means" refers to devices or systems that have programs or algorithms, including machine learning and image recognition technologies, used to analyze images or videos input by users.

[1044] A "feedback generation means" is a device or software that evaluates a user's health condition and emotional state based on the analysis results and the user's emotion recognition results, and provides customized advice and instructions.

[1045] "Management means" refers to devices or software for managing the user's schedule and sending notifications, and is primarily responsible for managing the next checkup date and reminder notifications.

[1046] The "suggestion means" refers to a device or software that suggests appropriate meal menus and exercise plans to users based on the analysis results and emotion recognition results.

[1047] "Acquisition means" refers to a device or software for acquiring the user's health information from an external database.

[1048] "Emotion recognition results" are information about a user's emotional state (e.g., happiness, anxiety, fatigue, etc.) obtained by analyzing the user's selfie video.

[1049] "Customized feedback" refers to advice and instructions created specifically for each user based on analysis and emotion recognition results.

[1050] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[1051] System Configuration

[1052] The system consists of the following main components:

[1053] 1. Means (terminal) for inputting user images and videos

[1054] 2. Artificial intelligence means (server) for analyzing images and videos

[1055] 3. Means of generating feedback (server)

[1056] 4. Means for managing user schedules (terminals, servers)

[1057] 5. Means of Providing Appropriate Suggestions (Server)

[1058] 6. Means of obtaining external data (server)

[1059] 7. Emotion engine (server) that recognizes user emotions

[1060] Data entry and upload

[1061] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[1062] Data analysis

[1063] The server analyzes the received data. First, for video analysis, artificial intelligence means analyze video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value. Specific software used includes TensorFlow and OpenCV for image recognition.

[1064] emotion recognition

[1065] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This makes it possible to understand the user's mental state. Specifically, it uses Microsoft Azure Cognitive Services.

[1066] Feedback Generation

[1067] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[1068] Data Sync and Notifications

[1069] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[1070] Displaying feedback to the user

[1071] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[1072] Specific examples

[1073] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis results indicate a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "You should include fish in your lunch." At the same time, analysis of the selfie video reveals that the user looks tired. The emotion engine recognizes this as "fatigue," and the server provides additional mental support advice, such as, "You seem tired today, so we recommend some light stretching."

[1074] Prompt Sentence Examples

[1075] "Analyze selfie videos and food photos to provide users with personalized health feedback. Add emotion recognition and include mental support advice."

[1076] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The user uses the smartphone camera to take a selfie video or a photo of their meal. After taking the photo, the user taps the "upload" button in the app to send the data to the server. Specifically, the smartphone's camera app launches, and includes functions for taking a selfie video for a certain period of time and taking photos of the meal. The data input is the video and photos taken by the user, and the output is the video and photo data uploaded to the server.

[1079] Step 2:

[1080] The server decompresses the received video and photo data and performs initial data processing. Specifically, the server applies a data decompression algorithm and performs preprocessing according to the data format. The input to this step is the received compressed video and photo data, and the output is the decompressed and preprocessed data.

[1081] Step 3:

[1082] The server starts video analysis. Using artificial intelligence tools (e.g., OpenCV or TensorFlow models), it analyzes each frame of the video and evaluates the user's complexion, facial expression, and skin condition. Specifically, the server divides the video into frames, performs facial recognition on each frame, and analyzes facial expression and complexion. The input of this step is the preprocessed video data, and the output is the evaluation results of the user's complexion, facial expression, and skin condition.

[1083] Step 4:

[1084] The server analyzes the photo. It uses image recognition technology (e.g., TensorFlow) to identify the ingredients in the photo and evaluate the nutrients in each. Specifically, the server extracts image features and compares them with a database to identify the ingredients. The input to this step is the preprocessed photo data, and the output is the identification of ingredients and the evaluation of their nutrients.

[1085] Step 5:

[1086] The server runs an emotion recognition engine to recognize the user's emotions from the selfie video. For example, a smile can be identified as "happiness," while a frown can be identified as "anxiety." Specifically, the server uses an emotion recognition algorithm (e.g., Microsoft Azure Cognitive Services) to analyze facial feature points. The input for this step is the video analysis results, and the output is the recognition result of the user's emotional state.

[1087] Step 6:

[1088] The server evaluates the user's health status based on the video and photo analysis results and emotion recognition results, and generates customized feedback. Specifically, the server executes a program that integrates multiple analysis results to perform a comprehensive health assessment and generates feedback based on that assessment. The input for this step is the video and photo analysis results and emotion recognition results, and the output is customized feedback.

[1089] Step 7:

[1090] The server sends the generated feedback to the user's device. Furthermore, the server connects to the maternal and child health handbook database to update the user's health information and send reminder notifications when checkup dates approach. Specifically, the server communicates over the network, sends feedback data, and schedules database updates and reminder notifications. The input of this step is the generated feedback, and the output is the feedback sent to the device and the updated database information.

[1091] Step 8:

[1092] The user checks the feedback and advice through the application. Specifically, the user opens the app and accesses the "Feedback" section to view dietary advice, checkup reminders, and emotional support advice. The input of this step is the feedback data sent to the device, and the output is the feedback information displayed to the user.

[1093] (Application example 2)

[1094] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1095] Pregnant women experience many physical and mental changes and require appropriate health management and mental support during this process. However, current health care systems are unable to fully address individual needs, and emotional support is particularly lacking. Specific advice on diet and exercise is often not provided, resulting in a lack of efficient and comprehensive support for pregnant women. Therefore, there is a need for a system that allows pregnant women to easily manage their health and receive emotional support.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1097] In this invention, the server includes a means for a user to input images, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing the user with customized feedback and meal suggestions based on the analysis results, an emotion engine means for recognizing the user's emotional state, a management means for managing the user's schedule and sending notifications, and a means for providing advice appropriate to the user's emotions and health condition using a generative AI model. This enables pregnant women to comprehensively understand their health condition and receive personalized meal suggestions and mental support.

[1098] "Means for users to input images" refers to a device or function that allows users to take images or videos using a device such as a smartphone and input them into the system.

[1099] "Artificial intelligence means" refers to a computing device that includes algorithms and programs for analyzing input image and video data and determining the user's health condition, the nutritional value of ingredients, and even their emotions.

[1100] The "feedback generation means" is a function or device for generating and providing customized advice or suggestions to the user based on the analyzed data.

[1101] "Emotion engine means" means a software component or algorithm for recognizing and analyzing emotional states from selfie videos or images input by a user.

[1102] "Management means" refers to software and service infrastructure for performing functions such as user schedule management and reminder notifications.

[1103] "Means for providing advice appropriate to emotions and health status using a generative AI model" is a function that utilizes a pre-trained artificial intelligence model to generate optimal advice based on the user's emotional state and health data.

[1104] The system of this invention provides personalized health management and mental support for pregnant women. By linking a server and user terminal, it analyzes image and video data and generates advice based on emotional and health states.

[1105] Hardware and software usage configuration

[1106] Smartphone: Uses a camera and applications to capture and record images and videos.

[1107] Server: A cloud server is used to run the database, analysis engine, and emotion engine. Specifically, cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) is used.

[1108] Machine learning model: Analyze videos and images using OpenCV, TensorFlow, PyTorch, etc.

[1109] Data processing flow

[1110] 1. Data entry and upload

[1111] Users use their smartphone camera to take selfies or photos of their meals and upload them to the application. For example, they can take a photo of their breakfast and tap the "upload" button in the app to send the data to the server.

[1112] 2. Data Analysis

[1113] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a face recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients and evaluate their nutritional value.

[1114] 3. Emotion recognition

[1115] Recognize user emotions from selfie videos. This uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions.

[1116] 4. Feedback Generation

[1117] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. For example, it may provide advice such as, "You're lacking in vitamin D, so we recommend including fish in your diet," or "You seem tired today, so we recommend some light stretching."

[1118] 5. Data Sync and Notifications

[1119] The feedback generated by the server is sent to the user's device, and the system also links with the maternal and child health handbook database to update the user's health information and send reminders for checkup dates.

[1120] 6. Displaying feedback to users

[1121] Users can view feedback and suggestions within the app, such as breakfast tips, check-up reminders, and emotional support advice.

[1122] Specific examples

[1123] If a user takes a photo of their breakfast and uploads the selfie video to the app, the server will analyze the breakfast photo and determine that it contains bread, eggs, and salad. If the server determines that the user is deficient in vitamin D, it will generate feedback suggesting that the user should include fish in their lunch. At the same time, if the analysis of the selfie video reveals that the user looks tired, the emotion engine will recognize this as "fatigue" and add feedback such as, "You seem tired today, so we recommend some light stretching."

[1124] Prompt Sentence Examples

[1125] "Please analyze the breakfast photos uploaded by the user. At the same time, please analyze the user's emotional state based on the selfie videos uploaded by the user. Based on the results, please generate dietary and mental support advice appropriate for the user's health and emotional state, and provide it as feedback."

[1126] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1127] Program processing steps

[1128] Step 1: Data entry and upload

[1129] Users use their smartphone camera to take selfie videos and photos of their meals and upload them to the application. The input data is the selfie video and photos of the meal, which are then sent to a cloud server. Specifically, users take a photo of their breakfast and tap the "upload" button in the app, uploading the data to the server. The server receives this data and prepares it for analysis.

[1130] Step 2: Data analysis

[1131] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a facial recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients in the photo and evaluate their nutritional value. The input data are video frames and food photos, and the output is facial features, a list of ingredients, and the corresponding nutritional information.

[1132] Step 3: Emotion Recognition

[1133] The server recognizes the user's emotions from the selfie video. It uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions. The input is facial features extracted from the video, and the output is the type of emotion (e.g., happiness, fatigue, anxiety, etc.). The server uses this result to understand the user's mental state.

[1134] Step 4: Feedback generation

[1135] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. First, it makes dietary suggestions based on the nutritional assessment results, and then adds mental support advice that takes into account the user's emotional state. For example, it might suggest, "You're lacking in vitamin D, so we recommend including fish in your diet," or, "You seem tired today, so we recommend some light stretching." The input is nutritional and emotional data, and the output is a personalized feedback message.

[1136] Step 5: Data Sync and Notifications

[1137] The feedback generated by the server is sent to the user's device. At the same time, it links with the maternal and child health handbook database to update the user's health information. In addition, reminder notifications are sent to the device when checkup dates approach. The input is the generated feedback message and the maternal and child health handbook database, and the output is the updated health information and notification message.

[1138] Step 6: Show feedback to the user

[1139] The user checks the feedback and suggestions on the device. The app displays breakfast advice, medical checkup reminders, and emotional support advice. The input is the feedback message, and the output is the displayed advice and notifications. The user can use this information to manage their health and mental health.

[1140] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1141] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1142] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1143] [Fourth embodiment]

[1144] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1145] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1146] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1147] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1148] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1149] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1150] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1151] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1152] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1153] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1154] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1155] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1156] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1157] ---

[1158] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[1159] Specific Embodiments

[1160] System Configuration

[1161] The system consists of the following main components:

[1162] 1. A means of inputting the user's image (terminal)

[1163] 2. Artificial intelligence means (server) for analyzing images

[1164] 3. Means of generating feedback (server)

[1165] 4. Means for managing user schedules (terminals, servers)

[1166] 5. Means of Providing Appropriate Suggestions (Server)

[1167] 6. Means of obtaining external data (server)

[1168] Program processing flow

[1169] Data entry and upload

[1170] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, users can take a photo of their breakfast and complete the process by tapping the "upload" button in the app.

[1171] Data analysis

[1172] The server analyzes the received data. In the case of selfie videos, artificial intelligence means analyze the video frames to assess complexion, facial expression and skin condition. In the case of food photos, image recognition technology is used to identify ingredients in the photo and assess their nutritional value.

[1173] Feedback Generation

[1174] Based on the analysis results, the server evaluates the user's health status and generates customized feedback, including specific advice such as "You are deficient in vitamin D, so we recommend incorporating fish into your diet," along with an appropriate exercise plan.

[1175] Data Sync and Notifications

[1176] The server generates feedback and sends it to the user's device. It also connects to the maternal and child health handbook database to periodically update health information. When the next checkup date approaches, a reminder notification is sent to the device.

[1177] Displaying feedback to the user

[1178] Users can view the feedback and suggestions they receive within the application, for example, breakfast recommendations or reminders for upcoming checkups, making it easier for users to manage their health.

[1179] Specific examples

[1180] One day, a user wakes up, takes a photo of their breakfast with their smartphone, and uploads it to the server via the app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, it identifies nutrients that the user is lacking (e.g., vitamin D). The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The device displays the received feedback within the app, allowing the user to choose ingredients appropriate for that day's lunch. Furthermore, if a regular medical checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to go for the checkup.

[1181] In this way, the system is designed to be easy for users to use on a daily basis, and provides comprehensive support for health management during pregnancy.

[1182] The processing flow will be explained below.

[1183] Step 1:

[1184] Users use their smartphones to take selfies or photos of their food, and the app's camera function is used to capture data.

[1185] Step 2:

[1186] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[1187] Step 3:

[1188] The server pre-processes the received video and photo data for analysis, for example by extracting video frames and compressing images.

[1189] Step 4:

[1190] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[1191] Step 5:

[1192] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[1193] Step 6:

[1194] The server combines the results of video and photo analysis to assess the user's current health status, and generates customized feedback based on the assessment results.

[1195] Step 7:

[1196] The server will suggest appropriate meal plans and exercise plans based on the user's health and nutritional status, providing specific advice such as "recommending a lunch that includes fish."

[1197] Step 8:

[1198] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[1199] Step 9:

[1200] The server connects to the maternal and child health handbook database to update the user's health information, checks the dates of regular checkups, and creates necessary reminders.

[1201] Step 10:

[1202] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[1203] Step 11:

[1204] Users open the app and review the feedback and suggestions they have received. The scheduling feature allows users to add the next checkup date to their calendar.

[1205] In this way, the system is structured around an entire flow from user input to processing on the server, generating and notifying feedback, and finally presenting it to the user.

[1206] Example 1

[1207] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] Proper health management and nutritional balance are extremely important for pregnant women. However, existing health management systems lack the ability to provide personalized feedback, making it difficult for users to obtain specific advice from photos and videos they take. They also lack the ability to recommend appropriate meal plans and exercise plans, and are inconvenient when it comes to managing health information and providing schedule notifications.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1210] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results, a management means for managing the user's schedule and notifications, and a means for displaying the feedback and suggestions received by the user. This enables the provision of customized health management and feedback that the user can easily use on a daily basis.

[1211] "Means for users to input images and videos" refers to the function that allows users to take and input images and videos necessary for health management using mobile devices such as smartphones and tablets.

[1212] "Artificial intelligence means" refers to the function of using AI algorithms and machine learning models implemented on the server to analyze input images and videos and evaluate or predict the user's health condition.

[1213] "Feedback generation means" refers to a function that automatically generates customized health advice and suggestions for the user based on the results analyzed by the artificial intelligence means.

[1214] "Management tools" refers to the ability to manage a user's schedule and tasks and provide reminder notifications as needed.

[1215] "Means for displaying feedback and suggestions" refers to a function that displays feedback and suggestions generated by the server on the user's device so that they can be visually confirmed.

[1216] "Means for suggesting meal menus and exercise plans" refers to a function that generates appropriate meal menus and exercise plans based on the user's health condition and analysis results, and presents them as advice.

[1217] "Acquisition means" refers to the function of acquiring user health information from external databases and integrating it into the system.

[1218] MODE FOR CARRYING OUT THE INVENTION

[1219] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and food photos taken by users and provides personalized feedback and advice. The system is primarily composed of the user's device (smartphone) and a server.

[1220] System Configuration

[1221] The system consists of the following main components:

[1222] 1. The means (terminal) by which users input images and videos

[1223] 2. Artificial intelligence means (server) for analyzing images and videos

[1224] 3. Means of generating feedback (server)

[1225] 4. User schedule management and notification methods (terminals, servers)

[1226] 5. A means (device) for displaying feedback and suggestions

[1227] 6. Proposal tool (server) that proposes appropriate meal menus and exercise plans

[1228] 7. Means for obtaining health information from external databases (server)

[1229] Data entry and upload

[1230] Users use their smartphone cameras to take selfies or photos of their meals. For example, a user may take photos of bread, eggs, and salad for breakfast. This data is uploaded to a server through an application. For example, when a user taps the "upload" button in an app after taking a photo, the device uploads the data to the server via Wi-Fi or mobile network.

[1231] Data analysis

[1232] The server analyzes the received data. In the case of selfie videos, artificial intelligence analyzes each frame of the video to evaluate the complexion, facial expression, and skin condition. Specifically, AI technologies such as TensorFlow and AWS Rekognition are used. For example, the server scores the user's complexion and facial expression to evaluate their health condition. In the case of a photo of a meal, image recognition technology is used to identify the ingredients contained in the photo and evaluate their nutritional value. Typical technologies include Google Cloud Vision API and OpenCV. The server analyzes the photo and determines that it contains bread, eggs, and salad.

[1233] Feedback Generation

[1234] Based on the analysis results, the server evaluates the user's health status and generates customized feedback. For example, it generates specific advice such as "You are deficient in vitamin D, so we recommend that you include fish in your lunch." It also generates an exercise plan tailored to the user's condition. For example, it suggests "It would be good to do 30 minutes of light walking every day."

[1235] Data Sync and Notifications

[1236] The server sends the generated feedback to the user's device. The device displays this feedback within the app. For example, a reminder such as "Your next health check is in one week" may be displayed. The server also accesses the maternal and child health handbook database to update regular health information. For example, the latest weight and blood pressure records may be added to the maternal and child health handbook. When the next checkup date approaches, the device sends a reminder notification. For example, a push notification such as "Tomorrow is your regular checkup date" may be displayed on the user's smartphone.

[1237] Displaying feedback to the user

[1238] The user opens the application and sees the feedback and suggestions sent by the server. For example, breakfast recommendations or reminders for the next week's medical checkup are displayed within the app. The user can then use this information to plan their meals and exercise for the day.

[1239] Specific examples

[1240] Below are some example prompts to input to a generative AI model:

[1241] "I took a photo of my breakfast this morning to help me manage my health during pregnancy. The photo includes bread, eggs, and salad. I'd like your feedback on health advice for this breakfast and my future meal plan."

[1242] In this way, the system provides users with a customized health management system that can be easily used on a daily basis, enabling comprehensive management of pregnancy health conditions.

[1243] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1244] Program processing flow

[1245] Step 1: Data entry and upload

[1246] Users take selfie videos or photos of their food with their smartphones.

[1247] Input: The user activates the smartphone camera and takes a selfie video or a photo of a meal (e.g., a photo of bread, eggs, and salad for breakfast) that is necessary for health management.

[1248] Output: Captured image and video data.

[1249] The user taps the "Upload" button in the application to send the captured data to the server.

[1250] Input: Tap the "Upload" button within the app.

[1251] Output: Image and video data sent to the server.

[1252] Step 2: Data analysis

[1253] The server receives the uploaded data and begins analyzing it.

[1254] Input: Image and video data received by the server.

[1255] Output: Data ready for analysis.

[1256] In the case of selfie videos, the server uses AI technology (e.g., TensorFlow or AWS Rekognition) to analyze each frame of the video and evaluate complexion, facial expression, and skin condition.

[1257] Input: Selfie video data.

[1258] Output: Evaluation results of complexion, facial expression, and skin condition.

[1259] For photos of meals, the server uses image recognition technology (e.g., Google Cloud Vision API or OpenCV) to identify ingredients and assess their nutritional value.

[1260] Input: Food photo data.

[1261] Output: Identified ingredients and nutrient evaluation results.

[1262] Step 3: Feedback generation

[1263] The server evaluates the user's health status based on the analysis results.

[1264] Input: Evaluation results of ingredients and nutrients, evaluation results of complexion and facial expression.

[1265] Output: Health status assessment results.

[1266] The server generates customized feedback, such as specific advice like "You're lacking vitamin D, so we recommend you include fish in your lunch."

[1267] Input: Health status assessment results.

[1268] Output: Customized feedback and advice.

[1269] Step 4: Data Sync and Notifications

[1270] The server generates feedback and sends it to the user's device.

[1271] Input: Customized feedback and advice.

[1272] Output: Feedback sent to the user device.

[1273] The server accesses the maternal and child health handbook database and updates the health information periodically.

[1274] Input: Latest health information.

[1275] Output: Updated maternal and child health handbook database.

[1276] When the next checkup date approaches, the server will send a reminder notification to the user's device.

[1277] Input: Schedule information.

[1278] Output: Send reminder notification.

[1279] Step 5: Display feedback to the user

[1280] The user opens the app and reviews the feedback and suggestions they received.

[1281] Input: Feedback sent to the user's device.

[1282] Output: Feedback and suggestions displayed within the application.

[1283] Based on the displayed feedback, users plan their daily health management, food choices, and exercise plans.

[1284] Input: Customized feedback and suggestions.

[1285] Output: Implementation of a specific health management plan.

[1286] (Application example 1)

[1287] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1288] Pregnant women need to properly manage their health, and for this, it is important to check their daily diet and evaluate their health. However, doing this manually requires a great deal of effort and time, making it difficult for pregnant women to properly manage their health themselves. Furthermore, even if they receive appropriate dietary suggestions, they lack the means to put them into practice, which makes it difficult for them to actually improve their dietary habits.

[1289] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1290] In this invention, the server includes an input means for a user to input an image, an artificial intelligence means for analyzing the image input by the user, a feedback generation means for providing the user with customized feedback based on the analysis results, a management means for managing the user's schedule and notifying them, a means for providing specific meal suggestions, and an ordering means for ordering the suggested meals, thereby enabling pregnant women to easily manage their health status and practice a nutritionally balanced diet.

[1291] "User" means a person who intends to use the system to manage their own health condition.

[1292] "Images" refers to selfie videos, food photos, etc. that users take or obtain through input means.

[1293] "Input means" refers to a device or application that allows a user to take an image and input it into the system.

[1294] "Artificial intelligence means" refers to AI technology that analyzes images input by users and identifies facial complexion, facial expressions, and ingredients.

[1295] The "feedback generation means" is a means for providing customized feedback to the user based on the analysis results obtained by the artificial intelligence means.

[1296] "Management tools" refers to features that manage the user's schedule and provide regular notifications and reminders.

[1297] The "means for providing specific meal suggestions" refers to a means having a function for suggesting a meal menu suitable for the user based on the analysis results.

[1298] "Ordering means" refers to the function that allows the user to order the suggested meal menu as is.

[1299] System Overview

[1300] The present invention is a system for managing the health and nutrition of pregnant women. By using various input means to manage the user's health condition, the system facilitates daily health checks and dietary management.

[1301] Hardware and software used

[1302] Smartphone: A device that allows users to take selfie videos and photos of their meals and upload them to the system.

[1303] Server: Hardware for executing the artificial intelligence means for image analysis, the feedback generation means, the management means, the meal suggestion means, and the ordering means.

[1304] AI analysis software: Artificial intelligence technology (e.g., TensorFlow, PyTorch, etc.) for analyzing user images.

[1305] Database system: A system for managing user health information and feedback data (e.g., MySQL, PostgreSQL, etc.).

[1306] Food Delivery API: An API used to order suggested meals online.

[1307] Data processing and calculation

[1308] 1. Data entry: Users use their smartphones to take selfies and photos of their meals, and then upload this data to the server through the app.

[1309] 2. Image analysis: The server analyzes the uploaded image data using AI analysis software to evaluate the user's health condition (complexion, facial expression, skin condition, etc.) and dietary content (ingredients, nutrients, etc.).

[1310] 3. Feedback Generation: The server generates customized feedback for the user based on the analysis results, such as specific advice such as "You are lacking in vitamin D, so we recommend that you include fish in your diet."

[1311] 4. Schedule Management and Notification: The server manages the user's schedule and notifies them of regular health checkups and meal suggestions, as well as providing reminders for regular checkups during pregnancy.

[1312] 5. Meal Suggestion and Ordering: The server will suggest appropriate meal menus to the user based on the analysis results and allow the suggested meals to be ordered through the food delivery API.

[1313] Specific examples

[1314] Suppose a pregnant woman takes a photo of her breakfast and uploads it to a server via an app. The server analyzes the photo and determines that it contains bread, eggs, and salad. As a result of the analysis, nutrients that the woman is lacking (e.g., vitamin D) are identified. The server generates feedback such as "It would be good to include fish in your lunch" and sends it to the user's device. The user can then order the suggested meal directly through the app and use the delivery service. Additionally, if a regular checkup is scheduled for the following week, a reminder notification is sent to the device to help the user remember to attend.

[1315] Prompt Sentence Examples

[1316] 1. "Consider dietary health management for a woman in her 50s. Using a photo of her breakfast, assess her nutritional status and suggest a meal plan to provide the necessary nutrients."

[1317] 2. "Provide pictures of foods that may be calcium deficient for a pregnant woman. Suggest calcium-rich meals for her."

[1318] 3. "Provide pictures of meals that are deficient in certain vitamins. Suggest foods needed to supplement the vitamins and provide a food delivery service where you can order those foods."

[1319] In this way, the embodiments of the invention enable pregnant women to easily manage their health and practice a nutritionally balanced diet.

[1320] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1321] Step 1:

[1322] Users use their smartphone cameras to take selfies and photos of their meals.

[1323] Input: Selfie videos and food photos taken with your smartphone.

[1324] How it works: The user launches the dedicated app and uses the camera function to take a photo of their face or the food they're eating.

[1325] Output: Selfie videos and food photo data.

[1326] Step 2:

[1327] Users upload videos and photos they have taken to the server through the app.

[1328] Input: Selfie videos and food photo data taken with a smartphone.

[1329] How it works: The user taps the "upload" button in the app to send the data to the server.

[1330] Output: Video and photo data uploaded to the server.

[1331] Step 3:

[1332] The data received by the server is analyzed using AI analysis software.

[1333] Input: Uploaded selfie videos and food photo data.

[1334] How it works: The server uses AI analysis software (e.g., TensorFlow, PyTorch) to analyze video frames and identify ingredients using image recognition technology.

[1335] Output: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[1336] Step 4:

[1337] The server generates customized feedback based on the analysis results.

[1338] Input: Analyzed data (e.g., complexion, facial expression, skin condition, types of food and their nutrients).

[1339] How it works: The server uses the feedback generator to generate specific advice, such as "You are deficient in vitamin D, so we recommend that you include fish in your diet."

[1340] Output: Customized feedback content.

[1341] Step 5:

[1342] The server sends the suggested meal menu to the user's terminal.

[1343] Input: Your customized feedback.

[1344] How it works: The server sends the feedback to the user's device and displays it as a notification within the app on the device.

[1345] Output: Meal suggestions and health advice displayed on the user's device.

[1346] Step 6:

[1347] The user reviews the suggested meal menu and, if desired, places an order within the app.

[1348] Input: Meal menu suggestions displayed on the device.

[1349] How it works: The user selects a suggested menu item and uses the in-app ordering feature to order delivery.

[1350] Output: A meal order from a delivery service.

[1351] Step 7:

[1352] The server manages the user's schedule and notifies them.

[1353] Input: User's schedule information and regular health check information.

[1354] How it works: The server manages the user's health information and sends reminders for the next checkup date and other notifications to the device.

[1355] Output: Health care reminder notification.

[1356] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1357] ---

[1358] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[1359] Specific Embodiments

[1360] System Configuration

[1361] The system consists of the following main components:

[1362] 1. A means of inputting the user's image (terminal)

[1363] 2. Artificial intelligence means (server) for analyzing images

[1364] 3. Means of generating feedback (server)

[1365] 4. Means for managing user schedules (terminals, servers)

[1366] 5. Means of Providing Appropriate Suggestions (Server)

[1367] 6. Means of obtaining external data (server)

[1368] 7. Emotion engine (server) that recognizes user emotions

[1369] Program processing flow

[1370] Data entry and upload

[1371] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[1372] Data analysis

[1373] The server analyzes the received data. First, for video analysis, artificial intelligence means analyzes video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value.

[1374] emotion recognition

[1375] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This allows the system to understand the user's mental state.

[1376] Feedback Generation

[1377] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[1378] Data Sync and Notifications

[1379] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[1380] Displaying feedback to the user

[1381] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[1382] Specific examples

[1383] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis reveals a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "It would be a good idea to include fish in your lunch."

[1384] At the same time, analysis of the selfie video reveals that the user has a tired expression. The emotion engine recognizes this as "fatigue," and the server provides feedback by adding mental support advice such as, "You seem tired today, so we recommend some light stretching."

[1385] In this way, this system simplifies daily use for users and simultaneously provides comprehensive health management and mental support for pregnant women.

[1386] The processing flow will be explained below.

[1387] Step 1:

[1388] The user uses their smartphone to take a selfie video or a photo of their meal using the camera function within the app.

[1389] Step 2:

[1390] To upload videos or photos taken by the user to the server through the application, the user taps the "Upload" button in the application. The device retrieves the data from local storage and sends it to the server.

[1391] Step 3:

[1392] The server preprocesses the video and photo data it receives. For video, it extracts frames, and for photos, it compresses them.

[1393] Step 4:

[1394] The server uses artificial intelligence to analyze the video data, which includes analyzing the user's complexion, facial expressions, and skin condition to assess their physical condition and stress level.

[1395] Step 5:

[1396] The server uses artificial intelligence means to analyze the photo data, identify the ingredients in the meal photo, retrieve their nutritional information from a database, and calculate the total nutritional value.

[1397] Step 6:

[1398] The server uses an emotion engine to recognize the user's emotions from the selfie video. For example, if the user smiles, it identifies "happiness," and if they frown, it identifies "anxiety."

[1399] Step 7:

[1400] The server combines the results of video analysis, photo analysis, and emotion recognition to assess the user's current health and mental state.

[1401] Step 8:

[1402] The server generates customized feedback based on the evaluation results, such as "You are lacking in vitamin D, so we recommend including fish in your diet" and "You seem tired today, so we recommend some light stretching exercises."

[1403] Step 9:

[1404] The server generates feedback and suggestions and sends them to the user's device, where the feedback data is formatted for display within the application.

[1405] Step 10:

[1406] The server connects to the maternal and child health handbook database to update the user's health information, and when the next checkup date approaches, a reminder notification is sent to the device.

[1407] Step 11:

[1408] The server sends push notifications to the device informing the user of the next checkup date and other important health information, which the device receives and notifies the user.

[1409] Step 12:

[1410] Users can open the app to view the feedback and suggestions they have received, as well as emotional support advice. Users can also add their next check-up date to their calendar through the scheduling feature.

[1411] In this way, the system is structured as a series of steps from user input to processing on the server, feedback generation and notification, and finally presentation to the user.

[1412] Example 2

[1413] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1414] Conventional health management systems only assess the user's health status using images and videos, but are unable to provide feedback that takes into account the user's emotional state. As a result, it is difficult to provide comprehensive health management and mental support to pregnant women at the same time. Furthermore, schedule management, appropriate meal menus, and exercise plan suggestions are not integrated, forcing users to use multiple applications.

[1415] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1416] In this invention, the server includes a means for the user to input images and videos, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing customized feedback to the user based on the analysis results and the user's emotion recognition results, and a management means for managing the user's schedule and notifying them. This allows the user to receive feedback that takes into account their emotional state in addition to an evaluation of their health condition using images and videos, and further allows them to receive integrated suggestions for schedule management, meal menus, and exercise plans.

[1417] "Means for user image and video input" refers to devices or software that allow users to take selfie videos or photos of their meals and upload that data to the system.

[1418] "Artificial intelligence means" refers to devices or systems that have programs or algorithms, including machine learning and image recognition technologies, used to analyze images or videos input by users.

[1419] A "feedback generation means" is a device or software that evaluates a user's health condition and emotional state based on the analysis results and the user's emotion recognition results, and provides customized advice and instructions.

[1420] "Management means" refers to devices or software for managing the user's schedule and sending notifications, and is primarily responsible for managing the next checkup date and reminder notifications.

[1421] The "suggestion means" refers to a device or software that suggests appropriate meal menus and exercise plans to users based on the analysis results and emotion recognition results.

[1422] "Acquisition means" refers to a device or software for acquiring the user's health information from an external database.

[1423] "Emotion recognition results" are information about a user's emotional state (e.g., happiness, anxiety, fatigue, etc.) obtained by analyzing the user's selfie video.

[1424] "Customized feedback" refers to advice and instructions created specifically for each user based on analysis and emotion recognition results.

[1425] This invention is a health management system for pregnant women that uses AI to analyze selfie videos and meal photos taken by users and provides individual feedback and advice. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized support.

[1426] System Configuration

[1427] The system consists of the following main components:

[1428] 1. Means (terminal) for inputting user images and videos

[1429] 2. Artificial intelligence means (server) for analyzing images and videos

[1430] 3. Means of generating feedback (server)

[1431] 4. Means for managing user schedules (terminals, servers)

[1432] 5. Means of Providing Appropriate Suggestions (Server)

[1433] 6. Means of obtaining external data (server)

[1434] 7. Emotion engine (server) that recognizes user emotions

[1435] Data entry and upload

[1436] Users use their smartphone camera to take selfies and photos of their meals. This data is then uploaded to the server via the application. For example, a user can take a photo of their breakfast and tap the "upload" button in the app to upload it.

[1437] Data analysis

[1438] The server analyzes the received data. First, for video analysis, artificial intelligence means analyze video frames to evaluate the user's complexion, facial expressions, and skin condition. Second, for food photos, image recognition technology is used to identify ingredients in the photo and evaluate their nutritional value. Specific software used includes TensorFlow and OpenCV for image recognition.

[1439] emotion recognition

[1440] The server's emotion engine recognizes the user's emotions from the selfie video. For example, if the user is smiling, it identifies "happiness," and if they are frowning, it identifies "anxiety." This makes it possible to understand the user's mental state. Specifically, it uses Microsoft Azure Cognitive Services.

[1441] Feedback Generation

[1442] The server comprehensively evaluates the user's health status based on the results of video and photo analysis and emotion recognition by the emotion engine. Based on the evaluation results, customized feedback is generated. For example, advice such as "You are lacking in vitamin D, so we recommend including fish in your diet" or "You seem tired today, so we recommend some light stretching exercises" is provided.

[1443] Data Sync and Notifications

[1444] The server generates feedback and sends it to the user's device. The server then connects to the maternal and child health handbook database to update the user's health information. When the next checkup date approaches, a reminder notification is sent to the device.

[1445] Displaying feedback to the user

[1446] Users will receive feedback and suggestions that are then displayed within the app, such as breakfast recommendations, reminders for upcoming check-ups, and even emotional support advice.

[1447] Specific examples

[1448] One day, a user takes a photo of their breakfast and uploads the selfie video to the app. The server analyzes the breakfast photo and determines that it contains bread, eggs, and salad. If the analysis results indicate a nutrient deficiency (e.g., vitamin D), the server generates feedback such as, "You should include fish in your lunch." At the same time, analysis of the selfie video reveals that the user looks tired. The emotion engine recognizes this as "fatigue," and the server provides additional mental support advice, such as, "You seem tired today, so we recommend some light stretching."

[1449] Prompt Sentence Examples

[1450] "Analyze selfie videos and food photos to provide users with personalized health feedback. Add emotion recognition and include mental support advice."

[1451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1452] Step 1:

[1453] The user uses the smartphone camera to take a selfie video or a photo of their meal. After taking the photo, the user taps the "upload" button in the app to send the data to the server. Specifically, the smartphone's camera app launches, and includes functions for taking a selfie video for a certain period of time and taking photos of the meal. The data input is the video and photos taken by the user, and the output is the video and photo data uploaded to the server.

[1454] Step 2:

[1455] The server decompresses the received video and photo data and performs initial data processing. Specifically, the server applies a data decompression algorithm and performs preprocessing according to the data format. The input to this step is the received compressed video and photo data, and the output is the decompressed and preprocessed data.

[1456] Step 3:

[1457] The server starts video analysis. Using artificial intelligence tools (e.g., OpenCV or TensorFlow models), it analyzes each frame of the video and evaluates the user's complexion, facial expression, and skin condition. Specifically, the server divides the video into frames, performs facial recognition on each frame, and analyzes facial expression and complexion. The input of this step is the preprocessed video data, and the output is the evaluation results of the user's complexion, facial expression, and skin condition.

[1458] Step 4:

[1459] The server analyzes the photo. It uses image recognition technology (e.g., TensorFlow) to identify the ingredients in the photo and evaluate the nutrients in each. Specifically, the server extracts image features and compares them with a database to identify the ingredients. The input to this step is the preprocessed photo data, and the output is the identification of ingredients and the evaluation of their nutrients.

[1460] Step 5:

[1461] The server runs an emotion recognition engine to recognize the user's emotions from the selfie video. For example, a smile can be identified as "happiness," while a frown can be identified as "anxiety." Specifically, the server uses an emotion recognition algorithm (e.g., Microsoft Azure Cognitive Services) to analyze facial feature points. The input for this step is the video analysis results, and the output is the recognition result of the user's emotional state.

[1462] Step 6:

[1463] The server evaluates the user's health status based on the video and photo analysis results and emotion recognition results, and generates customized feedback. Specifically, the server executes a program that integrates multiple analysis results to perform a comprehensive health assessment and generates feedback based on that assessment. The input for this step is the video and photo analysis results and emotion recognition results, and the output is customized feedback.

[1464] Step 7:

[1465] The server sends the generated feedback to the user's device. Furthermore, the server connects to the maternal and child health handbook database to update the user's health information and send reminder notifications when checkup dates approach. Specifically, the server communicates over the network, sends feedback data, and schedules database updates and reminder notifications. The input of this step is the generated feedback, and the output is the feedback sent to the device and the updated database information.

[1466] Step 8:

[1467] The user checks the feedback and advice through the application. Specifically, the user opens the app and accesses the "Feedback" section to view dietary advice, checkup reminders, and emotional support advice. The input of this step is the feedback data sent to the device, and the output is the feedback information displayed to the user.

[1468] (Application example 2)

[1469] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1470] Pregnant women experience many physical and mental changes and require appropriate health management and mental support during this process. However, current health care systems are unable to fully address individual needs, and emotional support is particularly lacking. Specific advice on diet and exercise is often not provided, resulting in a lack of efficient and comprehensive support for pregnant women. Therefore, there is a need for a system that allows pregnant women to easily manage their health and receive emotional support.

[1471] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1472] In this invention, the server includes a means for a user to input images, an artificial intelligence means for analyzing the images and videos input by the user, a feedback generation means for providing the user with customized feedback and meal suggestions based on the analysis results, an emotion engine means for recognizing the user's emotional state, a management means for managing the user's schedule and sending notifications, and a means for providing advice appropriate to the user's emotions and health condition using a generative AI model. This enables pregnant women to comprehensively understand their health condition and receive personalized meal suggestions and mental support.

[1473] "Means for users to input images" refers to a device or function that allows users to take images or videos using a device such as a smartphone and input them into the system.

[1474] "Artificial intelligence means" refers to a computing device that includes algorithms and programs for analyzing input image and video data and determining the user's health condition, the nutritional value of ingredients, and even their emotions.

[1475] The "feedback generation means" is a function or device for generating and providing customized advice or suggestions to the user based on the analyzed data.

[1476] "Emotion engine means" means a software component or algorithm for recognizing and analyzing emotional states from selfie videos or images input by a user.

[1477] "Management means" refers to software and service infrastructure for performing functions such as user schedule management and reminder notifications.

[1478] "Means for providing advice appropriate to emotions and health status using a generative AI model" is a function that utilizes a pre-trained artificial intelligence model to generate optimal advice based on the user's emotional state and health data.

[1479] The system of this invention provides personalized health management and mental support for pregnant women. By linking a server and user terminal, it analyzes image and video data and generates advice based on emotional and health states.

[1480] Hardware and software usage configuration

[1481] Smartphone: Uses a camera and applications to capture and record images and videos.

[1482] Server: A cloud server is used to run the database, analysis engine, and emotion engine. Specifically, cloud infrastructure such as Amazon Web Services (AWS) and Google Cloud Platform (GCP) is used.

[1483] Machine learning model: Analyze videos and images using OpenCV, TensorFlow, PyTorch, etc.

[1484] Data processing flow

[1485] 1. Data entry and upload

[1486] Users use their smartphone camera to take selfies or photos of their meals and upload them to the application. For example, they can take a photo of their breakfast and tap the "upload" button in the app to send the data to the server.

[1487] 2. Data Analysis

[1488] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a face recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients and evaluate their nutritional value.

[1489] 3. Emotion recognition

[1490] Recognize user emotions from selfie videos. This uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions.

[1491] 4. Feedback Generation

[1492] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. For example, it may provide advice such as, "You're lacking in vitamin D, so we recommend including fish in your diet," or "You seem tired today, so we recommend some light stretching."

[1493] 5. Data Sync and Notifications

[1494] The feedback generated by the server is sent to the user's device, and the system also links with the maternal and child health handbook database to update the user's health information and send reminders for checkup dates.

[1495] 6. Displaying feedback to users

[1496] Users can view feedback and suggestions within the app, such as breakfast tips, check-up reminders, and emotional support advice.

[1497] Specific examples

[1498] If a user takes a photo of their breakfast and uploads the selfie video to the app, the server will analyze the breakfast photo and determine that it contains bread, eggs, and salad. If the server determines that the user is deficient in vitamin D, it will generate feedback suggesting that the user should include fish in their lunch. At the same time, if the analysis of the selfie video reveals that the user looks tired, the emotion engine will recognize this as "fatigue" and add feedback such as, "You seem tired today, so we recommend some light stretching."

[1499] Prompt Sentence Examples

[1500] "Please analyze the breakfast photos uploaded by the user. At the same time, please analyze the user's emotional state based on the selfie videos uploaded by the user. Based on the results, please generate dietary and mental support advice appropriate for the user's health and emotional state, and provide it as feedback."

[1501] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1502] Program processing steps

[1503] Step 1: Data entry and upload

[1504] Users use their smartphone camera to take selfie videos and photos of their meals and upload them to the application. The input data is the selfie video and photos of the meal, which are then sent to a cloud server. Specifically, users take a photo of their breakfast and tap the "upload" button in the app, uploading the data to the server. The server receives this data and prepares it for analysis.

[1505] Step 2: Data analysis

[1506] The server analyzes the received data. For video analysis, OpenCV is used to divide the video into frames, and facial color and facial expressions are evaluated using a facial recognition model with TensorFlow. For food photo analysis, TensorFlow and PyTorch are used to identify ingredients in the photo and evaluate their nutritional value. The input data are video frames and food photos, and the output is facial features, a list of ingredients, and the corresponding nutritional information.

[1507] Step 3: Emotion Recognition

[1508] The server recognizes the user's emotions from the selfie video. It uses a pre-trained emotion recognition model (e.g., CNN trained on FER-2013) to classify emotions based on facial expressions. The input is facial features extracted from the video, and the output is the type of emotion (e.g., happiness, fatigue, anxiety, etc.). The server uses this result to understand the user's mental state.

[1509] Step 4: Feedback generation

[1510] The server generates customized feedback based on the results of video and photo analysis and emotion recognition. First, it makes dietary suggestions based on the nutritional assessment results, and then adds mental support advice that takes into account the user's emotional state. For example, it might suggest, "You're lacking in vitamin D, so we recommend including fish in your diet," or, "You seem tired today, so we recommend some light stretching." The input is nutritional and emotional data, and the output is a personalized feedback message.

[1511] Step 5: Data Sync and Notifications

[1512] The feedback generated by the server is sent to the user's device. At the same time, it links with the maternal and child health handbook database to update the user's health information. In addition, reminder notifications are sent to the device when checkup dates approach. The input is the generated feedback message and the maternal and child health handbook database, and the output is the updated health information and notification message.

[1513] Step 6: Show feedback to the user

[1514] The user checks the feedback and suggestions on the device. The app displays breakfast advice, medical checkup reminders, and emotional support advice. The input is the feedback message, and the output is the displayed advice and notifications. The user can use this information to manage their health and mental health.

[1515] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1516] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1517] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1518] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1519] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1520] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1521] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1522] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1523] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1524] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1525] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1526] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1527] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1528] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1529] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1530] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1531] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1532] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1533] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1534] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1535] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1536] The following is further disclosed regarding the above embodiment.

[1537] (Claim 1)

[1538] A means for a user to input an image;

[1539] artificial intelligence means for analyzing user-entered images;

[1540] a feedback generating means for providing customized feedback to the user based on the analysis results;

[1541] A management means for user schedule management and notification;

[1542] A system including:

[1543] (Claim 2)

[1544] The system according to claim 1, further comprising a suggestion means for suggesting an appropriate meal menu and exercise plan based on the analysis results.

[1545] (Claim 3)

[1546] 10. The system of claim 1, further comprising an acquisition means for acquiring the user's health information from an external database.

[1547] "Example 1"

[1548] (Claim 1)

[1549] a means for users to input images and videos;

[1550] artificial intelligence means for analyzing user-entered images and videos;

[1551] a feedback generating means for providing customized feedback to the user based on the analysis results;

[1552] A management means for user schedule management and notification;

[1553] a means for users to view the feedback and suggestions they receive;

[1554] A system including:

[1555] (Claim 2)

[1556] The system according to claim 1, further comprising a suggestion means for suggesting an appropriate meal menu and exercise plan based on the analysis results.

[1557] (Claim 3)

[1558] 10. The system of claim 1, further comprising an acquisition means for acquiring the user's health information from an external database.

[1559] "Application Example 1"

[1560] (Claim 1)

[1561] A means for a user to input an image;

[1562] artificial intelligence means for analyzing user-entered images;

[1563] a feedback generating means for providing customized feedback to the user based on the analysis results;

[1564] A management means for user schedule management and notification;

[1565] a means for providing specific dietary suggestions;

[1566] ordering means for ordering the suggested meal;

[1567] A system including:

[1568] (Claim 2)

[1569] The system according to claim 1, further comprising a suggestion means for suggesting an appropriate meal menu and exercise plan based on the analysis results.

[1570] (Claim 3)

[1571] 10. The system of claim 1, further comprising an acquisition means for acquiring the user's health information from an external database.

[1572] "Example 2: Combining Emotion Engines"

[1573] (Claim 1)

[1574] a means for users to input images and videos;

[1575] artificial intelligence means for analyzing user-entered images and videos;

[1576] a feedback generating means for providing customized feedback to the user based on the analysis result and the user's emotion recognition result;

[1577] A management means for user schedule management and notification;

[1578] A system including:

[1579] (Claim 2)

[1580] 10. The system according to claim 1, further comprising a suggestion means for suggesting an appropriate meal menu and exercise plan based on the analysis result and the emotion recognition result.

[1581] (Claim 3)

[1582] 10. The system of claim 1, further comprising an acquisition means for acquiring the user's health information from an external database.

[1583] "Application example 2 when combining emotion engines"

[1584] (Claim 1)

[1585] a means for a user to input an image;

[1586] artificial intelligence means for analyzing user-entered images and videos;

[1587] a feedback generating means for providing the user with customized feedback and meal suggestions based on the analysis results;

[1588] an emotion engine means for recognizing an emotional state of a user;

[1589] A management means for user schedule management and notification;

[1590] A means for providing emotional and health-appropriate advice using generative AI models; and

[1591] A system including:

[1592] (Claim 2)

[1593] The system according to claim 1, further comprising a suggestion means for suggesting an appropriate meal menu and exercise plan based on the analysis results.

[1594] (Claim 3)

[1595] 10. The system of claim 1, further comprising an acquisition means for acquiring the user's health information from an external database. [Explanation of symbols]

[1596] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to input an image; artificial intelligence means for analyzing user-entered images; a feedback generating means for providing customized feedback to the user based on the analysis results; A management means for user schedule management and notification; A system including:

2. The system according to claim 1 , further comprising a suggestion unit that suggests an appropriate meal menu and exercise plan based on the analysis results.

3. The system of claim 1 , further comprising an acquisition means for acquiring the user's health information from an external database.

Images