System

Abstract

A system is provided.SOLUTION: A system, comprising: image recognition means for analyzing a menu image to extract a drink list; health condition collection means for collecting health condition data of a user; analysis means for analyzing a past drinking record and the health condition data to determine an appropriate alcohol intake; advice means for providing an appropriate advice when the user selects a next drink; and monitoring means for monitoring the health condition of the user in real time and issuing a warning when an abnormality is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people today face health risks associated with drinking alcohol. In particular, hangovers and other health problems caused by drinking too much, as well as nuisances to others, are problems. This is caused by people drinking excessively without accurately understanding their own health condition. This is a major problem, especially for young people and those who often drink without being conscious of their health condition. To solve this issue, a system is needed that monitors the user's health condition in real time and provides advice on appropriate amounts and types of alcohol. [Means for solving the problem]

[0005] The present invention provides a system for reducing health risks when users enjoy drinking alcohol. This system includes an image recognition unit that analyzes menu images and extracts a drink list, a health status collection unit that collects the user's health status data, an analysis unit that analyzes past drinking records and health status data to determine an appropriate amount of alcohol intake, an advice unit that provides appropriate advice when the user selects their next drink, and a monitoring unit that monitors the user's health status in real time and issues a warning if an abnormality is detected. This allows users to understand their own health status and select appropriate drinking habits, thereby avoiding the health risks associated with excessive drinking.

[0006] "Menu image" refers to a photo or scanned image of a restaurant menu, which is used to list the drinks available.

[0007] The "drink list" refers to a list of drink types and names extracted from the menu image, and indicates the drinks that the user can select.

[0008] "Image recognition means" refers to a program or hardware for analyzing menu images and extracting a drink list, and refers to technology that has that function.

[0009] "Health status data" refers to vital physical data including the user's heart rate, body temperature, number of steps taken, sleep status, etc., which is used to evaluate the user's health status.

[0010] "Health status collection means" refers to devices or programs used to collect users' health status data, such as smartwatches and smartphones.

[0011] "Past drinking records" refers to data such as the amount of alcohol a user has consumed in the past, the type of alcohol consumed, the time spent drinking, and their physical condition the following day, and serves as the basic data for analysis.

[0012] "Analysis means" refers to a program or hardware that analyzes collected health data and past drinking records to determine the appropriate amount and speed of alcohol intake.

[0013] "Advice means" refers to a program or interface that provides advice to the user on the next drink they should choose and appropriate drinking behavior based on the results of the analysis means.

[0014] "Monitoring means" refers to a program or hardware that monitors the user's health status in real time and issues an alert if an abnormality is detected. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that helps users enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health data, and analyzes past drinking records and health data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, monitors the user's health in real time, and issues a warning if it detects any abnormalities.

[0037] Program processing

[0038] Menu image analysis

[0039] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[0040] Examples:

[0041] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[0042] Health status collection

[0043] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones and sends this to a server.

[0044] Examples:

[0045] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user and uses it to evaluate their health.

[0046] Analysis of drinking data

[0047] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0048] Examples:

[0049] Based on data on how the user felt the next day when they previously drank three beers between 8:00 p.m. and 10:00 p.m., the system will advise them to avoid excessive drinking in the same situation.

[0050] Drink selection advice

[0051] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0052] Examples:

[0053] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking another beer. We recommend a soda or non-alcoholic beverage instead." It will also simultaneously suggest drinking a glass of water.

[0054] Health monitoring and alerts

[0055] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0056] Examples:

[0057] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." In particular, by quickly detecting a sudden rise in heart rate or abnormal changes in body temperature while drinking and notifying the user immediately, serious health risks can be prevented.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[0061] Step 2:

[0062] The terminal transmits the captured menu image to the server.

[0063] Step 3:

[0064] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[0065] Step 4:

[0066] The server sends the recognized drink list to the terminal.

[0067] Step 5:

[0068] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[0069] Step 6:

[0070] The terminal transmits the collected health condition data to the server.

[0071] Step 7:

[0072] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data.

[0073] Step 8:

[0074] The server uses the analysis data to calculate the recommended alcohol allowance and consumption speed, and provides appropriate advice to the user.

[0075] Step 9:

[0076] When the user selects their next drink, the device sends the selected drink to the server.

[0077] Step 10:

[0078] The server determines whether the selected drink is appropriate based on the current health status and past analysis results.

[0079] Step 11:

[0080] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[0081] Step 12:

[0082] The device displays the received advice to the user.

[0083] Step 13:

[0084] The server continuously monitors the health status data collected in real time.

[0085] Step 14:

[0086] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[0087] Step 15:

[0088] The device will display the received alert to the user.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] In recent years, it has been pointed out that many people consume excessive amounts of alcohol in drinking establishments and fail to properly manage their own health. Excessive drinking poses a high risk of harm to health, and sudden changes in heart rate and body temperature in particular can pose serious health risks. Furthermore, users often find it difficult to judge the appropriate amount and type of alcohol they are drinking, which can result in excessive alcohol intake and dehydration. To solve these issues, a system is needed that allows users to enjoy alcohol safely and appropriately.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes image recognition means for analyzing menu images and extracting a list of foods and beverages, biometric information collection means for collecting user health status data, and data analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, enabling users to accurately select foods and beverages at restaurants, determine an appropriate amount of alcohol intake based on their own health status, and enjoy drinking safely.

[0094] A "menu image" is a digital image of a medium that visually displays a list of food and drink items offered at a restaurant or other establishment.

[0095] The "food and drink list" is a list of specific drink and food names extracted from the menu image.

[0096] "Image recognition means" refers to algorithms or techniques used to extract specific information from digital images.

[0097] "User" refers to an individual who uses this system to manage their drinking and health.

[0098] "Health data" refers to information about the user's physical condition, such as heart rate, body temperature, sleep status, and number of steps taken.

[0099] "Biometric information collection means" refers to devices and sensors used to collect a user's health status data.

[0100] "Drinking records" are data on the amount of alcohol a user has consumed in the past, the time they drank, and their physical condition afterwards.

[0101] "Data analysis means" refers to systems or software that analyze collected health data and drinking records to determine the appropriate amount of alcohol intake for the user.

[0102] "Selection advice means" refers to a function for determining whether the next food or drink the user selects is appropriate and providing advice.

[0103] "Condition monitoring means" refers to technologies and systems that continuously monitor the user's health condition in real time and issue warnings if an abnormality is detected.

[0104] This invention is a system that allows users to enjoy alcohol safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and analyzes past drinking records and health status data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0105] The main components of the system are:

[0106] 1. Analysis of Menu Images

[0107] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server analyzes the received menu image using the Google Cloud Vision API. As a result of the analysis, a list of the foods and drinks listed on the menu is extracted. The server then sends the recognized list of foods and drinks to the device.

[0108] Examples:

[0109] When a user enters a restaurant, they take a photo of the menu with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image using the Google Cloud Vision API. The resulting list of beer, wine, cocktails, etc. is generated and sent to the device.

[0110] 2. Collecting health information

[0111] The device collects data such as heart rate, sleep status, and body temperature from connected smartwatches and smartphones, and sends this health status data to a server.

[0112] Examples:

[0113] The server periodically collects heart rate, step count, and sleep data from the smartwatch (e.g., Apple Watch) worn by the user and uses this data to evaluate health status.

[0114] 3. Analysis of drinking data

[0115] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0116] Examples:

[0117] The server generates advice for the same situation based on the user's physical condition data from when they drank three beers at a specific time in the past.

[0118] 4. Drink selection advice

[0119] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0120] Examples:

[0121] The next time the user selects a beer, the device will notify them, "You might want to avoid that next beer. We recommend a soda or non-alcoholic beverage instead," and also suggest that they "drink a glass of water."

[0122] 5. Health monitoring and alerts

[0123] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0124] Examples:

[0125] If the user's heart rate suddenly rises, the server detects the abnormality and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." The device then displays this warning to the user as a pop-up notification.

[0126] Example prompts for generative AI models

[0127] "It analyzes the menu photo to generate a list of the food and drink options available. It then processes the user's current health data and drinking history to provide them with appropriate drinking advice."

[0128] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0129] Step 1:

[0130] A user takes a photo of a restaurant menu and uploads it through the app. The device sends the photo to the server. The server then analyzes the received menu image through an image recognition engine and extracts a drink list. The input is the menu image, and the output is the extracted drink list.

[0131] Specific behavior:

[0132] The user launches the smartphone app and takes a photo of the menu with the camera.

[0133] The smartphone app uploads menu images to the server.

[0134] The server performs image analysis using the Google Cloud Vision API.

[0135] The server extracts a list of food and beverages from the analysis results and sends it to the user's terminal.

[0136] Step 2:

[0137] The device acquires the user's health status data (heart rate, sleep status, body temperature, etc.) from a connected smartwatch or smartphone. It sends this data to a server. The input is the health status data from the smartwatch or smartphone, and the output is the health data sent to the server.

[0138] Specific behavior:

[0139] The smartwatch automatically records the user's heart rate, steps, and sleep data.

[0140] The smartphone periodically collects this data and sends it to the server.

[0141] The server stores the received data in a database and uses it for subsequent analysis.

[0142] Step 3:

[0143] The server integrates and analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data. This is used to calculate the recommended alcohol allowance and consumption pace. The input is the past drinking record and current health status data, and the output is the analyzed recommended alcohol allowance and consumption pace.

[0144] Specific behavior:

[0145] The server queries a historical drinking database to obtain the user's historical drinking record.

[0146] The server runs the newly received health data and these drinking records through an analysis algorithm.

[0147] Based on the analysis results, the server calculates the recommended alcohol allowance and drinking pace.

[0148] The server sends the calculation results to the user's device and uses them for future drinking advice.

[0149] Step 4:

[0150] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and generates and sends advice to the device. The input is the user's selected drink, and the output is an advice message from the server.

[0151] Specific behavior:

[0152] Users select their next drink using a smartphone app.

[0153] The terminal transmits the selected drink information to the server.

[0154] The server re-analyzes the health status and past drinking records and generates appropriate advice.

[0155] The server sends the generated advice message to the terminal, and the terminal notifies the user of the advice.

[0156] Step 5:

[0157] The server continuously monitors the health status data collected in real time. If an abnormality is detected, a warning message is generated and sent to the user's device. The input is the health status data acquired in real time, and the output is a warning message when an abnormality is detected.

[0158] Specific behavior:

[0159] The server continuously monitors the health data.

[0160] If the server detects an abnormality, it generates a warning message.

[0161] A warning message is sent to the user terminal, and the terminal displays the warning to the user.

[0162] (Application example 1)

[0163] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0164] Conventional drinking support systems have difficulty managing a user's health status comprehensively and in real time, making it difficult to provide appropriate drinking advice. Furthermore, they lack the functionality to suggest the next drink based on past drinking records and current health status, making it impossible to prevent drinking-related illness and health risks. The purpose of this invention is to solve these problems and provide a system that allows users to enjoy drinking safely and appropriately.

[0165] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0166] In this invention, the server includes image recognition means for analyzing menu images and extracting a beverage list, health status collection means for collecting user health status data, analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, display means for displaying the beverage list extracted from the menu images, determination means for determining whether the beverage selected by the user is appropriate based on the user's health status and displaying the result, advice means for providing appropriate advice when the user selects their next beverage, and monitoring means for monitoring the user's health status in real time and issuing an alert if an abnormality is detected. This makes it possible to manage the user's health status comprehensively and in real time and provide appropriate drinking advice.

[0167] A "menu image" is an image, in paper or digital format, that lists and details the drinks served at a restaurant.

[0168] "Image recognition means" refers to software or hardware for analyzing menu images and extracting a drink list.

[0169] "Health Data" refers to information that indicates the user's health status, such as the user's heart rate, body temperature, number of steps, and sleep data.

[0170] "Health status collection means" refers to a means for collecting a user's health status data using sensors in a smartwatch or smartphone.

[0171] "Drinking history" is data that indicates the amount of alcohol a user has consumed in the past, the frequency of drinking, and related health conditions.

[0172] "Analysis means" refers to software or hardware functions for determining appropriate alcohol intake based on health status data and drinking records.

[0173] "Advice Measure" refers to a software or hardware function that provides appropriate advice to the user on the next beverage they should select.

[0174] The "monitoring means" is a system that monitors the user's health condition in real time and issues an alert if an abnormality is detected.

[0175] "Display means" refers to the display or software functionality for displaying to the user the beverage list extracted from the menu image.

[0176] The "determination means" is a function for determining whether the drink selected by the user is appropriate based on the current health condition and displaying the result.

[0177] "Warning" refers to a message that alerts the user when an abnormality is detected in the user's health condition.

[0178] This invention is a system that allows users to enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and determines the appropriate amount of alcohol intake by analyzing past drinking records and health status data. Furthermore, it has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0179] Hardware

[0180] To realize this system, the following hardware is required:

[0181] Smartphone: A device that allows users to take pictures of the menu and upload them to the app.

[0182] Smartwatch: A device that collects health data such as a user's heart rate, body temperature, and steps taken.

[0183] Server: A computational system that performs data analysis and provides appropriate advice.

[0184] software

[0185] To realize this system, the following software is required:

[0186] Image recognition engine (such as Google Cloud Vision API): Recognizes characters from menu images and extracts the drink list.

[0187] Mobile application (iOS / Android): An application that allows users to take pictures of menu items, collect health data, and receive advice.

[0188] Health data collection API (Apple HealthKit, Google Fit, etc.): Collects user health data and sends it to a server.

[0189] Real-time data analysis server (AWS Lambda, AWS SageMaker, etc.): A server system that analyzes health status data and drinking records to determine appropriate alcohol intake.

[0190] Specific examples

[0191] 1. Analysis of Menu Images

[0192] Users take a photo of a restaurant menu with their smartphone and upload it through the app. The server then uses the Google Cloud Vision API to extract the drink list from the menu. The server then sends the recognized drink list to the smartphone and displays it to the user.

[0193] 2. Collecting health status data

[0194] The smartwatch collects real-time data such as heart rate, body temperature, and number of steps taken by the wearer, and this data is periodically sent to a server via a mobile app.

[0195] 3. Analysis of drinking data

[0196] The server analyzes the user's past drinking history (amount of alcohol consumed, time of drinking, and how they felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user.

[0197] 4. Drink selection advice

[0198] When a user selects their next drink, the smartphone sends that information to the server, which determines whether the selected drink is appropriate based on their current health status and sends advice to the smartphone, which then displays the advice to the user.

[0199] 5. Health monitoring and alerts

[0200] The server monitors health status data in real time, and if it detects any abnormalities, it immediately generates a warning message and sends it to the user's smartphone, allowing the user to receive the warning and take appropriate action.

[0201] Prompt Sentence Examples

[0202] Parse the drink list from a menu photo. Design an application that can collect users' health data and provide advice on their next drink selection based on their past drinking history. Propose a software solution to monitor health status in real time and support safe drinking.

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] A user enters a restaurant, takes a photo of the menu page with their smartphone, and uploads the image to the server through the application. The server analyzes the received menu image using the Google Cloud Vision API. The input is the menu image, and the output is a list of drinks extracted through image analysis.

[0206] Step 2:

[0207] The server uses an image analysis engine (Google Cloud Vision API) to perform character recognition on menu images and generate a drink list. Specifically, character recognition technology is used to extract drink names and categories listed on the menu and convert them into a text list. The input is image-recognized text data, and the output is the extracted drink list.

[0208] Step 3:

[0209] The server sends the extracted beverage list to the smartphone, and the smartphone displays the beverage list to the user. The input is a text-format beverage list, and the output is a beverage list displayed on the user's smartphone. This step allows the user to check the beverage list.

[0210] Step 4:

[0211] When a user wears a smartwatch, the smartphone periodically collects health status data such as heart rate, body temperature, and number of steps from the smartwatch. The collected data is temporarily stored on the smartphone and then sent to a server. The input is the health status data obtained from the smartwatch, and the output is the health status data sent to the server.

[0212] Step 5:

[0213] The server receives the health status data sent from the smartwatch and stores it in a database along with past drinking records. The server analyzes this data and calculates the appropriate amount of alcohol intake based on the user's health status and past drinking patterns. The input is the health status data and past drinking records, and the output is the analysis result, the recommended alcohol intake amount.

[0214] Step 6:

[0215] When the user selects their next drink, the smartphone sends the selection to the server. The input is the user's selected drink, and the output is the selection information sent to the server.

[0216] Step 7:

[0217] The server determines whether the selected beverage is appropriate based on the user's current health status and sends the result to the smartphone. Specifically, it calculates whether the user's selected beverage is recommended based on past drinking data and current health status data. The input is the user's health status data and the selected beverage, and the output is appropriate advice.

[0218] Step 8:

[0219] The terminal receives advice from the server and displays it to the user. The input is the advice information sent from the server, and the output is the advice message that the user checks on their smartphone.

[0220] Step 9:

[0221] The server monitors the user's health status data in real time and generates a warning message if an abnormality is detected. This warning message is sent to the user's smartphone, so the user is immediately alerted. The input is real-time health status data, and the output is a warning message when an abnormality is detected.

[0222] Step 10:

[0223] Users receive a warning message on their smartphone and take appropriate action. This allows users to prevent health risks caused by excessive drinking. The input is the warning message, and the output is the user's response.

[0224] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0225] This invention is a system that allows users to enjoy drinking safely and appropriately, and includes an emotion engine that recognizes the user's emotions. This allows the system to take into account the user's emotions and stress level when drinking and provide more appropriate advice. The system analyzes menu images to extract a drink list, collects the user's health status and emotion data, and analyzes this to determine the appropriate amount of alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the ability to monitor the user's health status and emotion in real time and issue a warning if an abnormality is detected.

[0226] Program processing

[0227] Menu image analysis

[0228] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[0229] Examples:

[0230] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[0231] Health and Emotion Collection

[0232] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from the connected smartwatch or smartphone and sends it to a server. It also collects the user's emotional data (through facial recognition and voice analysis, etc.).

[0233] Examples:

[0234] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user, and also uses the smartphone camera to collect emotional data from the user's facial expressions.

[0235] Analysis of drinking and emotion data

[0236] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) along with newly collected health and emotional data. From the analyzed data, it calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[0237] Examples:

[0238] If a user previously experienced a headache after drinking three beers between 8:00 and 10:00 PM, the system will use that data to advise them to avoid excessive drinking in the same situation. Furthermore, if the user's facial expression indicates fatigue or stress, the system will take that information into account and recommend limiting alcohol intake.

[0239] Drink selection advice

[0240] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and emotional data, as well as past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0241] Examples:

[0242] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking that next beer. We recommend a soda or non-alcoholic beverage instead." The device will also simultaneously suggest drinking a glass of water. If the user appears tired or stressed, the device will also display personalized advice, such as, "Take it easy today and drink plenty of fluids to help you relax."

[0243] Health and emotional monitoring and alerts

[0244] The server continuously monitors the health and emotional data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0245] Examples:

[0246] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." If the user's facial expression shows signs of stress, the server will warn, "Your stress level is high. Please try to relax." In particular, by quickly detecting sudden increases in heart rate, abnormal changes in body temperature, and abnormalities in emotional data and notifying the user immediately, serious health risks can be prevented.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[0250] Step 2:

[0251] The terminal transmits the captured menu image to the server.

[0252] Step 3:

[0253] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[0254] Step 4:

[0255] The server sends the recognized drink list to the terminal.

[0256] Step 5:

[0257] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[0258] Step 6:

[0259] The device collects user emotional data by recognizing the user's facial expressions using the smartphone camera and then uses an emotion engine to analyze their stress level and emotional state.

[0260] Step 7:

[0261] The device transmits the collected health status data and emotion data to a server.

[0262] Step 8:

[0263] The server analyzes past drinking records and newly collected health and emotional data.

[0264] Step 9:

[0265] The server uses the analyzed data to calculate the recommended alcohol tolerance and consumption speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[0266] Step 10:

[0267] When the user selects their next drink, the device sends the selected drink to the server.

[0268] Step 11:

[0269] The server determines whether the selected drink is appropriate based on the current health and emotional data, as well as past analysis results.

[0270] Step 12:

[0271] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[0272] Step 13:

[0273] The device will then display the advice it receives to the user, such as a message like, "You might want to avoid your next beer. We recommend a soda or non-alcoholic beverage instead."

[0274] Step 14:

[0275] The server continuously monitors the health and emotional data collected in real time.

[0276] Step 15:

[0277] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[0278] Step 16:

[0279] The device will then display the received warning to the user, such as "Your health is deteriorating. Stop drinking alcohol and take a break by drinking water" or "Your stress level is high. Please try to relax."

[0280] Example 2

[0281] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0282] Conventional drinking management systems only consider the user's health status and are unable to provide advice that takes into account the user's emotions and stress level. Therefore, there is a need for systems that provide appropriate advice based on the emotional fluctuations and stress levels that accompany drinking, enabling users to drink safely and with consideration for their health. Furthermore, real-time monitoring of health status and emotions is insufficient, making it difficult to detect abnormalities early and issue warnings. Therefore, a system that comprehensively monitors and analyzes the user's emotions and health status and provides appropriate advice is needed.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0284] In this invention, the server includes image recognition means for analyzing menu images and extracting a drink list, biometric information collection means for collecting a user's biometric information, emotional information collection means for collecting the user's emotional data, analysis means for analyzing past drinking records, the biometric information, and the emotional data to determine an appropriate amount of alcohol intake, advice means for providing appropriate advice when the user selects their next drink, and monitoring means for monitoring the user's biometric information and emotional data in real time and issuing a warning if an abnormality is detected. This makes it possible to comprehensively manage and analyze the user's health condition and emotions, provide more appropriate drinking advice, and detect abnormalities early to reduce the user's health risks.

[0285] A "menu image" is an image that lists the food and drink items offered at a restaurant or other establishment, and is taken by the user and entered into the system.

[0286] The "drink list" is a list of drink names and information extracted from the menu image.

[0287] "Image recognition means" refers to a system or device that analyzes input image data and extracts specific information.

[0288] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and sleep state.

[0289] A "biometric information collection means" is a device or system for measuring and collecting a user's biometric information.

[0290] "Emotional data" is data that indicates the user's emotional state, analyzed from the user's facial expressions, tone of voice, etc.

[0291] The "emotion information collection means" is a system or device for collecting user emotion data.

[0292] An "analysis means" is a system or device for analyzing collected data and deriving results.

[0293] The "advice means" is a system or device that instructs the user on appropriate actions based on the results obtained by the analysis means.

[0294] "Monitoring means" refers to a system or device for monitoring a user's biometric information and emotional data in real time and detecting abnormalities.

[0295] An "alert" is a notification or message that notifies the user of an abnormal situation.

[0296] This invention is a system that allows users to enjoy drinking safely and appropriately. Specifically, it includes an emotion engine that recognizes the user's emotions and provides more appropriate advice by taking into account the user's emotions and stress level when drinking. This system uses the following hardware and software to collect, analyze data, and provide advice.

[0297] Hardware and software used

[0298] Hardware:

[0299] Smartphone (taking photos and running applications)

[0300] Smartwatch (biometric information measurement)

[0301] Server (data analysis and storage)

[0302] software:

[0303] Image recognition engine (e.g. Google Cloud Vision API)

[0304] Biometric information collection application (smartwatch integration)

[0305] Emotion recognition engines (e.g., Python OpenCV and speech recognition libraries)

[0306] Data analysis engines (e.g., Python's scikit-learn library)

[0307] Specific processing

[0308] Menu image analysis

[0309] When a user visits a restaurant, they take a photo of the menu with their smartphone and upload it to the server via the app. The device then sends the image to the server, which then uses an image recognition engine to analyze the image and extract the list of drinks listed on the menu. The server then sends the recognized list to the device and displays it to the user.

[0310] Examples:

[0311] For example, a user can take a picture of a restaurant menu with their smartphone and send it to the server through the app. The server then uses the Google Cloud Vision API to analyze the image, extract the drink list, and send it to the device to display to the user.

[0312] Health and Emotion Collection

[0313] The device collects the user's biometric information (heart rate, body temperature, sleep data, etc.) from the smartwatch and sends it to the server. The device also collects emotional data through facial expression and voice analysis and sends it to the server.

[0314] Examples:

[0315] The app periodically transmits heart rate and body temperature data from the smartwatch worn by the user to a server, and when the user takes a selfie with their smartphone, the device uses a facial expression recognition engine to collect emotional data and transmit it to the server.

[0316] Analysis of drinking and emotion data

[0317] The server accumulates past drinking records, current biometric information, and emotional data, and stores them in a database. Using an analytical engine, it calculates the appropriate amount of alcohol intake and the appropriate speed of consumption. It also evaluates stress levels and prepares to provide appropriate advice to users.

[0318] Examples:

[0319] If data is stored showing that a user previously experienced a headache after drinking three beers, the system will analyze that data along with current biometric and emotional data to generate advice on how much to limit next time.

[0320] Drink selection advice

[0321] When the user selects their next drink, the device sends the selection information to the server, which determines whether the drink is appropriate based on the user's current biometric and emotional data, generates and sends advice to the device, and displays the advice to the user.

[0322] Examples:

[0323] When the user chooses to drink wine next, the server analyzes biometric information and past data to generate advice such as, "You should refrain from drinking wine next time. We recommend drinking water," and notifies the device.

[0324] Health and emotional monitoring and alerts

[0325] The server monitors biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device, which then immediately displays the warning to the user and prompts them to take appropriate measures.

[0326] Examples:

[0327] If the heart rate suddenly rises, the server generates a warning saying, "Your heart rate is high. Stop drinking, drink some water and take a break," and sends it to the device to notify the user.

[0328] Prompt Sentence Examples

[0329] "In a drinking management system, generate prompts to advise users on the appropriate amount of beer they should drink."

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Step 1: Take and upload your menu images

[0332] The user takes a photo of a restaurant menu. Using the camera app on their smartphone, they take a photo that shows the entire menu. They then upload the menu image to the server through a dedicated application. The input is the menu image taken by the user, and the output is the image file sent to the server. Specifically, the user takes a photo of the menu and taps the "upload" button on the app.

[0333] Step 2: Receiving and analyzing menu images

[0334] The device sends the captured menu image to the server. The server runs the received image through an image recognition engine to extract the list of drinks listed on the menu. At this time, the Google Cloud Vision API is used to identify the text portion of the image and convert it into a list of drinks. The input is the received menu image, and the output is the identified list of drinks. Specifically, the server calls the API to extract the drink menu from the image.

[0335] Step 3: Send and display the drink list

[0336] The server sends the recognized drink list to the device. The device displays the received list to the user. The input is the recognized drink list, and the output is the list displayed on the user's device. Specifically, the device app displays the drink list and allows the user to select the next drink.

[0337] Step 4: Collect health and emotions

[0338] The device collects the user's biometric information (heart rate, body temperature, sleep state, etc.) from a connected smartwatch or smartphone and sends this to a server. It also analyzes the user's facial expressions and voice to collect emotional data, which it then sends to the server. The input is the biometric information and emotional data from the smartwatch or smartphone, and the output is this data sent to the server. Specifically, it acquires data from the smartwatch and captures images for facial recognition.

[0339] Step 5: Analyze drinking and emotion data

[0340] The server accumulates past drinking records, current biometric information, and emotional data and stores them in a database. An analysis engine is used to analyze this data and calculate the appropriate amount of alcohol intake and the appropriate speed of intake. It also evaluates stress levels. The input is the past drinking records and current biometric information and emotional data, and the output is the calculated appropriate amount of alcohol intake and the appropriate speed of intake. Specifically, the server runs a data analysis algorithm to obtain the results.

[0341] Step 6: Drink Selection Advice

[0342] When the user selects their next drink, the device sends the selection information to the server. The server determines whether the drink is appropriate based on the user's current biometric and emotional data, generates advice, and sends it to the device. The input is the selected drink information and the user's current biometric and emotional data, and the output is advice sent from the server to the device. Specifically, the device may display a message saying, "You should refrain from having the next drink."

[0343] Step 7: Health and Emotion Monitoring and Alerts

[0344] The server monitors the biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device. The device immediately displays the warning to the user. The input is the biometric and emotional data collected in real time, and the output is a warning message when an abnormality is detected. Specifically, if a sudden increase in heart rate or an abnormality is detected, the server immediately sends a warning to the device to notify the user, such as "Your heart rate is high. Stop drinking, drink water, and take a break."

[0345] (Application example 2)

[0346] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] There is a need to improve work efficiency and reduce health risks by monitoring the health and emotions of factory workers in real time and providing advice on appropriate breaks and hydration. However, there is currently a lack of systems that can properly evaluate workers' health and emotions and provide effective advice based on that assessment.

[0348] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means that analyzes menu images and extracts a drink list, a health status collection means that collects the user's health status data, an emotion collection means that collects the user's emotion data, an analysis means that analyzes past drinking records and the health status data and emotion data to determine an appropriate amount of alcohol intake, an advice means that provides appropriate advice when the user selects their next drink, and a monitoring means that monitors the user's health status and emotions in real time and issues an alert if an abnormality is detected. This makes it possible to effectively monitor the health status and emotions of factory workers, improving work efficiency and reducing health risks.

[0349] A "menu image" is an image that visually shows a list of drinks and food served at a restaurant, cafe, etc.

[0350] A "drink list" is a list of information such as the names, types, and prices of drinks extracted from the menu image.

[0351] "Image recognition means" refers to technology or devices for analyzing menu images and extracting a drink list.

[0352] "Health Data" refers to data that indicates the user's physical health, such as heart rate, body temperature, number of steps taken, and sleep status.

[0353] "Health status collection means" refers to a means for collecting a user's health status data using a device such as a smartwatch or smartphone.

[0354] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[0355] "Emotion collection means" refers to a means of collecting user emotional data using a camera or microphone.

[0356] "Analysis means" refers to technology or devices that analyze collected health data, emotional data, past drinking records, etc. to determine appropriate alcohol intake amounts and appropriate rest periods.

[0357] The "advice means" refers to a means for providing the user with advice on appropriate alcohol intake, rest periods, and hydration based on the results determined by the analysis means.

[0358] "Monitoring means" refers to technologies and devices that monitor the user's health and emotions in real time and issue warnings if any abnormalities are detected.

[0359] This invention is a system for managing the health of factory workers, which monitors the user's health condition and emotions in real time and provides advice on appropriate breaks and hydration. Specific embodiments of this system are described below.

[0360] 1. Overall system configuration

[0361] The system consists of the following main components:

[0362] Device (smartphone)

[0363] server

[0364] Smartwatch

[0365] The device is operated by the user and works in conjunction with the smartwatch to collect health status data, and also uses the device's camera to capture the user's facial expressions and collect emotional data.

[0366] The server analyzes the collected health and emotional data to generate appropriate advice and warnings. Additionally, the server analyzes past data to make more accurate decisions.

[0367] A smartwatch is a device that collects real-time health data such as a user's heart rate, body temperature, and number of steps taken.

[0368] 2. Program Processing

[0369] First, the device connects to the smartwatch and periodically collects health status data (heart rate, body temperature, number of steps, etc.), which is then sent to a server via Bluetooth or Wi-Fi. At the same time, the device's camera is used to collect the user's facial expression data, which is also sent to the server.

[0370] Upon receiving this data, the server analyzes the data using the health status collection means and emotion collection means to evaluate the user's current health status and emotion, and determines the user's stress level and fatigue level based on the analysis results.

[0371] The server then uses analytical tools to generate advice on appropriate rest times and hydration based on past drinking records, historical health and emotional data, and sends this advice to the device, which then notifies the user.

[0372] Furthermore, the server uses monitoring means to issue a warning if it detects an abnormality in the data collected in real time. For example, if the user's heart rate suddenly rises or if their facial expression indicates high stress, a warning message will be sent to the device to notify the user.

[0373] 3. Hardware and Software Used

[0374] Smartwatches (e.g., Apple Watch, Fitbit): Collect health data such as heart rate, body temperature, and steps taken.

[0375] Smartphones (e.g., iPhone, Android): Use the camera to collect facial expression data and send the data to a server.

[0376] Server: Analyzes the collected data and generates advice and warnings.

[0377] EmotionRecognizer: A library that analyzes facial expression data to recognize emotions (e.g., OpenCV, dlib).

[0378] HealthMonitor: A module that collects health data.

[0379] AlertSystem: A system that notifies workers of alerts (e.g., push notifications).

[0380] 4. Adding concrete examples

[0381] Health and Emotion Collection

[0382] While the user works in the factory, the smartwatch continuously collects data such as heart rate and body temperature, while the smartphone periodically captures the user's facial expressions and analyzes their emotional data. This data is then sent to a server, which then generates advice on appropriate breaks and hydration.

[0383] Prompt Sentence Examples

[0384] Write a sample program that uses health and emotional data collected from a smartwatch or smartphone to provide advice to factory workers on how to take appropriate breaks and stay hydrated.

[0385] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0386] Step 1:

[0387] The device works in conjunction with the smartwatch to collect the user's health data. Specifically, the smartwatch collects data such as heart rate, body temperature, and number of steps via Bluetooth, and sends this data to the device.

[0388] Input: Heart rate, body temperature, and step count data from a smartwatch

[0389] Output: Health status data stored on the device

[0390] Step 2:

[0391] The device uses a camera to capture the user's facial expressions and collect emotional data, which is then analyzed using image recognition technology to quantify the user's emotional state.

[0392] Input: Facial expression image taken by the device camera

[0393] Output: Analyzed emotion data (e.g., stress level, happiness)

[0394] Step 3:

[0395] The device periodically transmits the collected health and emotion data to a server, where the data is uploaded via the Internet.

[0396] Input: Health status data and emotion data stored on the device

[0397] Output: Health status data and emotion data uploaded to the server

[0398] Step 4:

[0399] The server analyzes the received health and emotion data using an analytical means, evaluating fluctuations in health and emotion to determine the user's current situation.

[0400] Input: Health status data and emotion data received from the device

[0401] Output: Analysis results (e.g. current health status, stress level, fatigue level)

[0402] Step 5:

[0403] The server analyzes past drinking records, health status data, and emotional data to dynamically generate advice on appropriate rest times and hydration.

[0404] Input: Analysis results, past drinking records, health status data, emotional data

[0405] Output: Generated advice (e.g., time to take a break, recommended hydration)

[0406] Step 6:

[0407] The server sends the generated advice to the device, which then notifies the user of the received advice and prompts them to take the necessary action.

[0408] Input: Advice from the server

[0409] Output: Advice notice displayed on terminal

[0410] Step 7:

[0411] The server continuously monitors the data collected in real time using the monitoring means, and if an abnormality is detected, such as a sudden increase in heart rate or high stress level, the server generates a warning message and sends it to the device.

[0412] Input: Real-time collected health status data, emotional data

[0413] Output: Warning message when an abnormality is detected

[0414] Step 8:

[0415] The device will notify the user of the received warning message and prompt them to take appropriate measures.

[0416] Input: Warning message from the server

[0417] Output: Warning notice displayed on terminal

[0418] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0419] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0420] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0421] [Second embodiment]

[0422] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0423] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0424] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0425] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0426] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0427] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0428] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0429] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0430] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0431] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0432] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0433] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0434] This invention is a system that helps users enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health data, and analyzes past drinking records and health data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, monitors the user's health in real time, and issues a warning if it detects any abnormalities.

[0435] Program processing

[0436] Menu image analysis

[0437] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[0438] Examples:

[0439] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[0440] Health status collection

[0441] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones and sends this to a server.

[0442] Examples:

[0443] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user and uses it to evaluate their health.

[0444] Analysis of drinking data

[0445] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0446] Examples:

[0447] Based on data on how the user felt the next day when they previously drank three beers between 8:00 p.m. and 10:00 p.m., the system will advise them to avoid excessive drinking in the same situation.

[0448] Drink selection advice

[0449] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0450] Examples:

[0451] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking another beer. We recommend a soda or non-alcoholic beverage instead." It will also simultaneously suggest drinking a glass of water.

[0452] Health monitoring and alerts

[0453] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0454] Examples:

[0455] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." In particular, by quickly detecting a sudden rise in heart rate or abnormal changes in body temperature while drinking and notifying the user immediately, serious health risks can be prevented.

[0456] The processing flow will be explained below.

[0457] Step 1:

[0458] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[0459] Step 2:

[0460] The terminal transmits the captured menu image to the server.

[0461] Step 3:

[0462] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[0463] Step 4:

[0464] The server sends the recognized drink list to the terminal.

[0465] Step 5:

[0466] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[0467] Step 6:

[0468] The terminal transmits the collected health condition data to the server.

[0469] Step 7:

[0470] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data.

[0471] Step 8:

[0472] The server uses the analysis data to calculate the recommended alcohol allowance and consumption speed, and provides appropriate advice to the user.

[0473] Step 9:

[0474] When the user selects their next drink, the device sends the selected drink to the server.

[0475] Step 10:

[0476] The server determines whether the selected drink is appropriate based on the current health status and past analysis results.

[0477] Step 11:

[0478] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[0479] Step 12:

[0480] The device displays the received advice to the user.

[0481] Step 13:

[0482] The server continuously monitors the health status data collected in real time.

[0483] Step 14:

[0484] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[0485] Step 15:

[0486] The device will display the received alert to the user.

[0487] Example 1

[0488] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] In recent years, it has been pointed out that many people consume excessive amounts of alcohol in drinking establishments and fail to properly manage their own health. Excessive drinking poses a high risk of harm to health, and sudden changes in heart rate and body temperature in particular can pose serious health risks. Furthermore, users often find it difficult to judge the appropriate amount and type of alcohol they are drinking, which can result in excessive alcohol intake and dehydration. To solve these issues, a system is needed that allows users to enjoy alcohol safely and appropriately.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes image recognition means for analyzing menu images and extracting a list of foods and beverages, biometric information collection means for collecting user health status data, and data analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, enabling users to accurately select foods and beverages at restaurants, determine an appropriate amount of alcohol intake based on their own health status, and enjoy drinking safely.

[0492] A "menu image" is a digital image of a medium that visually displays a list of food and drink items offered at a restaurant or other establishment.

[0493] The "food and drink list" is a list of specific drink and food names extracted from the menu image.

[0494] "Image recognition means" refers to algorithms or techniques used to extract specific information from digital images.

[0495] "User" refers to an individual who uses this system to manage their drinking and health.

[0496] "Health data" refers to information about the user's physical condition, such as heart rate, body temperature, sleep status, and number of steps taken.

[0497] "Biometric information collection means" refers to devices and sensors used to collect a user's health status data.

[0498] "Drinking records" are data on the amount of alcohol a user has consumed in the past, the time they drank, and their physical condition afterwards.

[0499] "Data analysis means" refers to systems or software that analyze collected health data and drinking records to determine the appropriate amount of alcohol intake for the user.

[0500] "Selection advice means" refers to a function for determining whether the next food or drink the user selects is appropriate and providing advice.

[0501] "Condition monitoring means" refers to technologies and systems that continuously monitor the user's health condition in real time and issue warnings if an abnormality is detected.

[0502] This invention is a system that allows users to enjoy alcohol safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and analyzes past drinking records and health status data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0503] The main components of the system are:

[0504] 1. Analysis of Menu Images

[0505] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server analyzes the received menu image using the Google Cloud Vision API. As a result of the analysis, a list of the foods and drinks listed on the menu is extracted. The server then sends the recognized list of foods and drinks to the device.

[0506] Examples:

[0507] When a user enters a restaurant, they take a photo of the menu with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image using the Google Cloud Vision API. The resulting list of beer, wine, cocktails, etc. is generated and sent to the device.

[0508] 2. Collecting health information

[0509] The device collects data such as heart rate, sleep status, and body temperature from connected smartwatches and smartphones, and sends this health status data to a server.

[0510] Examples:

[0511] The server periodically collects heart rate, step count, and sleep data from the smartwatch (e.g., Apple Watch) worn by the user and uses this data to evaluate health status.

[0512] 3. Analysis of drinking data

[0513] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0514] Examples:

[0515] The server generates advice for the same situation based on the user's physical condition data from when they drank three beers at a specific time in the past.

[0516] 4. Drink selection advice

[0517] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0518] Examples:

[0519] The next time the user selects a beer, the device will notify them, "You might want to avoid that next beer. We recommend a soda or non-alcoholic beverage instead," and also suggest that they "drink a glass of water."

[0520] 5. Health monitoring and alerts

[0521] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0522] Examples:

[0523] If the user's heart rate suddenly rises, the server detects the abnormality and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." The device then displays this warning to the user as a pop-up notification.

[0524] Example prompts for generative AI models

[0525] "It analyzes the menu photo to generate a list of the food and drink options available. It then processes the user's current health data and drinking history to provide them with appropriate drinking advice."

[0526] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0527] Step 1:

[0528] A user takes a photo of a restaurant menu and uploads it through the app. The device sends the photo to the server. The server then analyzes the received menu image through an image recognition engine and extracts a drink list. The input is the menu image, and the output is the extracted drink list.

[0529] Specific behavior:

[0530] The user launches the smartphone app and takes a photo of the menu with the camera.

[0531] The smartphone app uploads menu images to the server.

[0532] The server performs image analysis using the Google Cloud Vision API.

[0533] The server extracts a list of food and beverages from the analysis results and sends it to the user's terminal.

[0534] Step 2:

[0535] The device acquires the user's health status data (heart rate, sleep status, body temperature, etc.) from a connected smartwatch or smartphone. It sends this data to a server. The input is the health status data from the smartwatch or smartphone, and the output is the health data sent to the server.

[0536] Specific behavior:

[0537] The smartwatch automatically records the user's heart rate, steps, and sleep data.

[0538] The smartphone periodically collects this data and sends it to the server.

[0539] The server stores the received data in a database and uses it for subsequent analysis.

[0540] Step 3:

[0541] The server integrates and analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data. This is used to calculate the recommended alcohol allowance and consumption pace. The input is the past drinking record and current health status data, and the output is the analyzed recommended alcohol allowance and consumption pace.

[0542] Specific behavior:

[0543] The server queries a historical drinking database to obtain the user's historical drinking record.

[0544] The server runs the newly received health data and these drinking records through an analysis algorithm.

[0545] Based on the analysis results, the server calculates the recommended alcohol allowance and drinking pace.

[0546] The server sends the calculation results to the user's device and uses them for future drinking advice.

[0547] Step 4:

[0548] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and generates and sends advice to the device. The input is the user's selected drink, and the output is an advice message from the server.

[0549] Specific behavior:

[0550] Users select their next drink using a smartphone app.

[0551] The terminal transmits the selected drink information to the server.

[0552] The server re-analyzes the health status and past drinking records and generates appropriate advice.

[0553] The server sends the generated advice message to the terminal, and the terminal notifies the user of the advice.

[0554] Step 5:

[0555] The server continuously monitors the health status data collected in real time. If an abnormality is detected, a warning message is generated and sent to the user's device. The input is the health status data acquired in real time, and the output is a warning message when an abnormality is detected.

[0556] Specific behavior:

[0557] The server continuously monitors the health data.

[0558] If the server detects an abnormality, it generates a warning message.

[0559] A warning message is sent to the user terminal, and the terminal displays the warning to the user.

[0560] (Application example 1)

[0561] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0562] Conventional drinking support systems have difficulty managing a user's health status comprehensively and in real time, making it difficult to provide appropriate drinking advice. Furthermore, they lack the functionality to suggest the next drink based on past drinking records and current health status, making it impossible to prevent drinking-related illness and health risks. The purpose of this invention is to solve these problems and provide a system that allows users to enjoy drinking safely and appropriately.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0564] In this invention, the server includes image recognition means for analyzing menu images and extracting a beverage list, health status collection means for collecting user health status data, analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, display means for displaying the beverage list extracted from the menu images, determination means for determining whether the beverage selected by the user is appropriate based on the user's health status and displaying the result, advice means for providing appropriate advice when the user selects their next beverage, and monitoring means for monitoring the user's health status in real time and issuing an alert if an abnormality is detected. This makes it possible to manage the user's health status comprehensively and in real time and provide appropriate drinking advice.

[0565] A "menu image" is an image, in paper or digital format, that lists and details the drinks served at a restaurant.

[0566] "Image recognition means" refers to software or hardware for analyzing menu images and extracting a drink list.

[0567] "Health Data" refers to information that indicates the user's health status, such as the user's heart rate, body temperature, number of steps, and sleep data.

[0568] "Health status collection means" refers to a means for collecting a user's health status data using sensors in a smartwatch or smartphone.

[0569] "Drinking history" is data that indicates the amount of alcohol a user has consumed in the past, the frequency of drinking, and related health conditions.

[0570] "Analysis means" refers to software or hardware functions for determining appropriate alcohol intake based on health status data and drinking records.

[0571] "Advice Measure" refers to a software or hardware function that provides appropriate advice to the user on the next beverage they should select.

[0572] The "monitoring means" is a system that monitors the user's health condition in real time and issues an alert if an abnormality is detected.

[0573] "Display means" refers to the display or software functionality for displaying to the user the beverage list extracted from the menu image.

[0574] The "determination means" is a function for determining whether the drink selected by the user is appropriate based on the current health condition and displaying the result.

[0575] "Warning" refers to a message that alerts the user when an abnormality is detected in the user's health condition.

[0576] This invention is a system that allows users to enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and determines the appropriate amount of alcohol intake by analyzing past drinking records and health status data. Furthermore, it has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0577] Hardware

[0578] To realize this system, the following hardware is required:

[0579] Smartphone: A device that allows users to take pictures of the menu and upload them to the app.

[0580] Smartwatch: A device that collects health data such as a user's heart rate, body temperature, and steps taken.

[0581] Server: A computational system that performs data analysis and provides appropriate advice.

[0582] software

[0583] To realize this system, the following software is required:

[0584] Image recognition engine (such as Google Cloud Vision API): Recognizes characters from menu images and extracts the drink list.

[0585] Mobile application (iOS / Android): An application that allows users to take pictures of menu items, collect health data, and receive advice.

[0586] Health data collection API (Apple HealthKit, Google Fit, etc.): Collects user health data and sends it to a server.

[0587] Real-time data analysis server (AWS Lambda, AWS SageMaker, etc.): A server system that analyzes health status data and drinking records to determine appropriate alcohol intake.

[0588] Specific examples

[0589] 1. Analysis of Menu Images

[0590] Users take a photo of a restaurant menu with their smartphone and upload it through the app. The server then uses the Google Cloud Vision API to extract the drink list from the menu. The server then sends the recognized drink list to the smartphone and displays it to the user.

[0591] 2. Collecting health status data

[0592] The smartwatch collects real-time data such as heart rate, body temperature, and number of steps taken by the wearer, and this data is periodically sent to a server via a mobile app.

[0593] 3. Analysis of drinking data

[0594] The server analyzes the user's past drinking history (amount of alcohol consumed, time of drinking, and how they felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user.

[0595] 4. Drink selection advice

[0596] When a user selects their next drink, the smartphone sends that information to the server, which determines whether the selected drink is appropriate based on their current health status and sends advice to the smartphone, which then displays the advice to the user.

[0597] 5. Health monitoring and alerts

[0598] The server monitors health status data in real time, and if it detects any abnormalities, it immediately generates a warning message and sends it to the user's smartphone, allowing the user to receive the warning and take appropriate action.

[0599] Prompt Sentence Examples

[0600] Parse the drink list from a menu photo. Design an application that can collect users' health data and provide advice on their next drink selection based on their past drinking history. Propose a software solution to monitor health status in real time and support safe drinking.

[0601] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0602] Step 1:

[0603] A user enters a restaurant, takes a photo of the menu page with their smartphone, and uploads the image to the server through the application. The server analyzes the received menu image using the Google Cloud Vision API. The input is the menu image, and the output is a list of drinks extracted through image analysis.

[0604] Step 2:

[0605] The server uses an image analysis engine (Google Cloud Vision API) to perform character recognition on menu images and generate a drink list. Specifically, character recognition technology is used to extract drink names and categories listed on the menu and convert them into a text list. The input is image-recognized text data, and the output is the extracted drink list.

[0606] Step 3:

[0607] The server sends the extracted beverage list to the smartphone, and the smartphone displays the beverage list to the user. The input is a text-format beverage list, and the output is a beverage list displayed on the user's smartphone. This step allows the user to check the beverage list.

[0608] Step 4:

[0609] When a user wears a smartwatch, the smartphone periodically collects health status data such as heart rate, body temperature, and number of steps from the smartwatch. The collected data is temporarily stored on the smartphone and then sent to a server. The input is the health status data obtained from the smartwatch, and the output is the health status data sent to the server.

[0610] Step 5:

[0611] The server receives the health status data sent from the smartwatch and stores it in a database along with past drinking records. The server analyzes this data and calculates the appropriate amount of alcohol intake based on the user's health status and past drinking patterns. The input is the health status data and past drinking records, and the output is the analysis result, the recommended alcohol intake amount.

[0612] Step 6:

[0613] When the user selects their next drink, the smartphone sends the selection to the server. The input is the user's selected drink, and the output is the selection information sent to the server.

[0614] Step 7:

[0615] The server determines whether the selected beverage is appropriate based on the user's current health status and sends the result to the smartphone. Specifically, it calculates whether the user's selected beverage is recommended based on past drinking data and current health status data. The input is the user's health status data and the selected beverage, and the output is appropriate advice.

[0616] Step 8:

[0617] The terminal receives advice from the server and displays it to the user. The input is the advice information sent from the server, and the output is the advice message that the user checks on their smartphone.

[0618] Step 9:

[0619] The server monitors the user's health status data in real time and generates a warning message if an abnormality is detected. This warning message is sent to the user's smartphone, so the user is immediately alerted. The input is real-time health status data, and the output is a warning message when an abnormality is detected.

[0620] Step 10:

[0621] Users receive a warning message on their smartphone and take appropriate action. This allows users to prevent health risks caused by excessive drinking. The input is the warning message, and the output is the user's response.

[0622] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0623] This invention is a system that allows users to enjoy drinking safely and appropriately, and includes an emotion engine that recognizes the user's emotions. This allows the system to take into account the user's emotions and stress level when drinking and provide more appropriate advice. The system analyzes menu images to extract a drink list, collects the user's health status and emotion data, and analyzes this to determine the appropriate amount of alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the ability to monitor the user's health status and emotion in real time and issue a warning if an abnormality is detected.

[0624] Program processing

[0625] Menu image analysis

[0626] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[0627] Examples:

[0628] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[0629] Health and Emotion Collection

[0630] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from the connected smartwatch or smartphone and sends it to a server. It also collects the user's emotional data (through facial recognition and voice analysis, etc.).

[0631] Examples:

[0632] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user, and also uses the smartphone camera to collect emotional data from the user's facial expressions.

[0633] Analysis of drinking and emotion data

[0634] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) along with newly collected health and emotional data. From the analyzed data, it calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[0635] Examples:

[0636] If a user previously experienced a headache after drinking three beers between 8:00 and 10:00 PM, the system will use that data to advise them to avoid excessive drinking in the same situation. Furthermore, if the user's facial expression indicates fatigue or stress, the system will take that information into account and recommend limiting alcohol intake.

[0637] Drink selection advice

[0638] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and emotional data, as well as past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0639] Examples:

[0640] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking that next beer. We recommend a soda or non-alcoholic beverage instead." The device will also simultaneously suggest drinking a glass of water. If the user appears tired or stressed, the device will also display personalized advice, such as, "Take it easy today and drink plenty of fluids to help you relax."

[0641] Health and emotional monitoring and alerts

[0642] The server continuously monitors the health and emotional data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0643] Examples:

[0644] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." If the user's facial expression shows signs of stress, the server will warn, "Your stress level is high. Please try to relax." In particular, by quickly detecting sudden increases in heart rate, abnormal changes in body temperature, and abnormalities in emotional data and notifying the user immediately, serious health risks can be prevented.

[0645] The processing flow will be explained below.

[0646] Step 1:

[0647] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[0648] Step 2:

[0649] The terminal transmits the captured menu image to the server.

[0650] Step 3:

[0651] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[0652] Step 4:

[0653] The server sends the recognized drink list to the terminal.

[0654] Step 5:

[0655] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[0656] Step 6:

[0657] The device collects user emotional data by recognizing the user's facial expressions using the smartphone camera and then uses an emotion engine to analyze their stress level and emotional state.

[0658] Step 7:

[0659] The device transmits the collected health status data and emotion data to a server.

[0660] Step 8:

[0661] The server analyzes past drinking records and newly collected health and emotional data.

[0662] Step 9:

[0663] The server uses the analyzed data to calculate the recommended alcohol tolerance and consumption speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[0664] Step 10:

[0665] When the user selects their next drink, the device sends the selected drink to the server.

[0666] Step 11:

[0667] The server determines whether the selected drink is appropriate based on the current health and emotional data, as well as past analysis results.

[0668] Step 12:

[0669] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[0670] Step 13:

[0671] The device will then display the advice it receives to the user, such as a message like, "You might want to avoid your next beer. We recommend a soda or non-alcoholic beverage instead."

[0672] Step 14:

[0673] The server continuously monitors the health and emotional data collected in real time.

[0674] Step 15:

[0675] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[0676] Step 16:

[0677] The device will then display the received warning to the user, such as "Your health is deteriorating. Stop drinking alcohol and take a break by drinking water" or "Your stress level is high. Please try to relax."

[0678] Example 2

[0679] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Conventional drinking management systems only consider the user's health status and are unable to provide advice that takes into account the user's emotions and stress level. Therefore, there is a need for systems that provide appropriate advice based on the emotional fluctuations and stress levels that accompany drinking, enabling users to drink safely and with consideration for their health. Furthermore, real-time monitoring of health status and emotions is insufficient, making it difficult to detect abnormalities early and issue warnings. Therefore, a system that comprehensively monitors and analyzes the user's emotions and health status and provides appropriate advice is needed.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0682] In this invention, the server includes image recognition means for analyzing menu images and extracting a drink list, biometric information collection means for collecting a user's biometric information, emotional information collection means for collecting the user's emotional data, analysis means for analyzing past drinking records, the biometric information, and the emotional data to determine an appropriate amount of alcohol intake, advice means for providing appropriate advice when the user selects their next drink, and monitoring means for monitoring the user's biometric information and emotional data in real time and issuing a warning if an abnormality is detected. This makes it possible to comprehensively manage and analyze the user's health condition and emotions, provide more appropriate drinking advice, and detect abnormalities early to reduce the user's health risks.

[0683] A "menu image" is an image that lists the food and drink items offered at a restaurant or other establishment, and is taken by the user and entered into the system.

[0684] The "drink list" is a list of drink names and information extracted from the menu image.

[0685] "Image recognition means" refers to a system or device that analyzes input image data and extracts specific information.

[0686] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and sleep state.

[0687] A "biometric information collection means" is a device or system for measuring and collecting a user's biometric information.

[0688] "Emotional data" is data that indicates the user's emotional state, analyzed from the user's facial expressions, tone of voice, etc.

[0689] The "emotion information collection means" is a system or device for collecting user emotion data.

[0690] An "analysis means" is a system or device for analyzing collected data and deriving results.

[0691] The "advice means" is a system or device that instructs the user on appropriate actions based on the results obtained by the analysis means.

[0692] "Monitoring means" refers to a system or device for monitoring a user's biometric information and emotional data in real time and detecting abnormalities.

[0693] An "alert" is a notification or message that notifies the user of an abnormal situation.

[0694] This invention is a system that allows users to enjoy drinking safely and appropriately. Specifically, it includes an emotion engine that recognizes the user's emotions and provides more appropriate advice by taking into account the user's emotions and stress level when drinking. This system uses the following hardware and software to collect, analyze data, and provide advice.

[0695] Hardware and software used

[0696] Hardware:

[0697] Smartphone (taking photos and running applications)

[0698] Smartwatch (biometric information measurement)

[0699] Server (data analysis and storage)

[0700] software:

[0701] Image recognition engine (e.g. Google Cloud Vision API)

[0702] Biometric information collection application (smartwatch integration)

[0703] Emotion recognition engines (e.g., Python OpenCV and speech recognition libraries)

[0704] Data analysis engines (e.g., Python's scikit-learn library)

[0705] Specific processing

[0706] Menu image analysis

[0707] When a user visits a restaurant, they take a photo of the menu with their smartphone and upload it to the server via the app. The device then sends the image to the server, which then uses an image recognition engine to analyze the image and extract the list of drinks listed on the menu. The server then sends the recognized list to the device and displays it to the user.

[0708] Examples:

[0709] For example, a user can take a picture of a restaurant menu with their smartphone and send it to the server through the app. The server then uses the Google Cloud Vision API to analyze the image, extract the drink list, and send it to the device to display to the user.

[0710] Health and Emotion Collection

[0711] The device collects the user's biometric information (heart rate, body temperature, sleep data, etc.) from the smartwatch and sends it to the server. The device also collects emotional data through facial expression and voice analysis and sends it to the server.

[0712] Examples:

[0713] The app periodically transmits heart rate and body temperature data from the smartwatch worn by the user to a server, and when the user takes a selfie with their smartphone, the device uses a facial expression recognition engine to collect emotional data and transmit it to the server.

[0714] Analysis of drinking and emotion data

[0715] The server accumulates past drinking records, current biometric information, and emotional data, and stores them in a database. Using an analytical engine, it calculates the appropriate amount of alcohol intake and the appropriate speed of consumption. It also evaluates stress levels and prepares to provide appropriate advice to users.

[0716] Examples:

[0717] If data is stored showing that a user previously experienced a headache after drinking three beers, the system will analyze that data along with current biometric and emotional data to generate advice on how much to limit next time.

[0718] Drink selection advice

[0719] When the user selects their next drink, the device sends the selection information to the server, which determines whether the drink is appropriate based on the user's current biometric and emotional data, generates and sends advice to the device, and displays the advice to the user.

[0720] Examples:

[0721] When the user chooses to drink wine next, the server analyzes biometric information and past data to generate advice such as, "You should refrain from drinking wine next time. We recommend drinking water," and notifies the device.

[0722] Health and emotional monitoring and alerts

[0723] The server monitors biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device, which then immediately displays the warning to the user and prompts them to take appropriate measures.

[0724] Examples:

[0725] If the heart rate suddenly rises, the server generates a warning saying, "Your heart rate is high. Stop drinking, drink some water and take a break," and sends it to the device to notify the user.

[0726] Prompt Sentence Examples

[0727] "In a drinking management system, generate prompts to advise users on the appropriate amount of beer they should drink."

[0728] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0729] Step 1: Take and upload your menu images

[0730] The user takes a photo of a restaurant menu. Using the camera app on their smartphone, they take a photo that shows the entire menu. They then upload the menu image to the server through a dedicated application. The input is the menu image taken by the user, and the output is the image file sent to the server. Specifically, the user takes a photo of the menu and taps the "upload" button on the app.

[0731] Step 2: Receiving and analyzing menu images

[0732] The device sends the captured menu image to the server. The server runs the received image through an image recognition engine to extract the list of drinks listed on the menu. At this time, the Google Cloud Vision API is used to identify the text portion of the image and convert it into a list of drinks. The input is the received menu image, and the output is the identified list of drinks. Specifically, the server calls the API to extract the drink menu from the image.

[0733] Step 3: Send and display the drink list

[0734] The server sends the recognized drink list to the device. The device displays the received list to the user. The input is the recognized drink list, and the output is the list displayed on the user's device. Specifically, the device app displays the drink list and allows the user to select the next drink.

[0735] Step 4: Collect health and emotions

[0736] The device collects the user's biometric information (heart rate, body temperature, sleep state, etc.) from a connected smartwatch or smartphone and sends this to a server. It also analyzes the user's facial expressions and voice to collect emotional data, which it then sends to the server. The input is the biometric information and emotional data from the smartwatch or smartphone, and the output is this data sent to the server. Specifically, it acquires data from the smartwatch and captures images for facial recognition.

[0737] Step 5: Analyze drinking and emotion data

[0738] The server accumulates past drinking records, current biometric information, and emotional data and stores them in a database. An analysis engine is used to analyze this data and calculate the appropriate amount of alcohol intake and the appropriate speed of intake. It also evaluates stress levels. The input is the past drinking records and current biometric information and emotional data, and the output is the calculated appropriate amount of alcohol intake and the appropriate speed of intake. Specifically, the server runs a data analysis algorithm to obtain the results.

[0739] Step 6: Drink Selection Advice

[0740] When the user selects their next drink, the device sends the selection information to the server. The server determines whether the drink is appropriate based on the user's current biometric and emotional data, generates advice, and sends it to the device. The input is the selected drink information and the user's current biometric and emotional data, and the output is advice sent from the server to the device. Specifically, the device may display a message saying, "You should refrain from having the next drink."

[0741] Step 7: Health and Emotion Monitoring and Alerts

[0742] The server monitors the biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device. The device immediately displays the warning to the user. The input is the biometric and emotional data collected in real time, and the output is a warning message when an abnormality is detected. Specifically, if a sudden increase in heart rate or an abnormality is detected, the server immediately sends a warning to the device to notify the user, such as "Your heart rate is high. Stop drinking, drink water, and take a break."

[0743] (Application example 2)

[0744] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] There is a need to improve work efficiency and reduce health risks by monitoring the health and emotions of factory workers in real time and providing advice on appropriate breaks and hydration. However, there is currently a lack of systems that can properly evaluate workers' health and emotions and provide effective advice based on that assessment.

[0746] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means that analyzes menu images and extracts a drink list, a health status collection means that collects the user's health status data, an emotion collection means that collects the user's emotion data, an analysis means that analyzes past drinking records and the health status data and emotion data to determine an appropriate amount of alcohol intake, an advice means that provides appropriate advice when the user selects their next drink, and a monitoring means that monitors the user's health status and emotions in real time and issues an alert if an abnormality is detected. This makes it possible to effectively monitor the health status and emotions of factory workers, improving work efficiency and reducing health risks.

[0747] A "menu image" is an image that visually shows a list of drinks and food served at a restaurant, cafe, etc.

[0748] A "drink list" is a list of information such as the names, types, and prices of drinks extracted from the menu image.

[0749] "Image recognition means" refers to technology or devices for analyzing menu images and extracting a drink list.

[0750] "Health Data" refers to data that indicates the user's physical health, such as heart rate, body temperature, number of steps taken, and sleep status.

[0751] "Health status collection means" refers to a means for collecting a user's health status data using a device such as a smartwatch or smartphone.

[0752] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[0753] "Emotion collection means" refers to a means of collecting user emotional data using a camera or microphone.

[0754] "Analysis means" refers to technology or devices that analyze collected health data, emotional data, past drinking records, etc. to determine appropriate alcohol intake amounts and appropriate rest periods.

[0755] The "advice means" refers to a means for providing the user with advice on appropriate alcohol intake, rest periods, and hydration based on the results determined by the analysis means.

[0756] "Monitoring means" refers to technologies and devices that monitor the user's health and emotions in real time and issue warnings if any abnormalities are detected.

[0757] This invention is a system for managing the health of factory workers, which monitors the user's health condition and emotions in real time and provides advice on appropriate breaks and hydration. Specific embodiments of this system are described below.

[0758] 1. Overall system configuration

[0759] The system consists of the following main components:

[0760] Device (smartphone)

[0761] server

[0762] Smartwatch

[0763] The device is operated by the user and works in conjunction with the smartwatch to collect health status data, and also uses the device's camera to capture the user's facial expressions and collect emotional data.

[0764] The server analyzes the collected health and emotional data to generate appropriate advice and warnings. Additionally, the server analyzes past data to make more accurate decisions.

[0765] A smartwatch is a device that collects real-time health data such as a user's heart rate, body temperature, and number of steps taken.

[0766] 2. Program Processing

[0767] First, the device connects to the smartwatch and periodically collects health status data (heart rate, body temperature, number of steps, etc.), which is then sent to a server via Bluetooth or Wi-Fi. At the same time, the device's camera is used to collect the user's facial expression data, which is also sent to the server.

[0768] Upon receiving this data, the server analyzes the data using the health status collection means and emotion collection means to evaluate the user's current health status and emotion, and determines the user's stress level and fatigue level based on the analysis results.

[0769] The server then uses analytical tools to generate advice on appropriate rest times and hydration based on past drinking records, historical health and emotional data, and sends this advice to the device, which then notifies the user.

[0770] Furthermore, the server uses monitoring means to issue a warning if it detects an abnormality in the data collected in real time. For example, if the user's heart rate suddenly rises or if their facial expression indicates high stress, a warning message will be sent to the device to notify the user.

[0771] 3. Hardware and Software Used

[0772] Smartwatches (e.g., Apple Watch, Fitbit): Collect health data such as heart rate, body temperature, and steps taken.

[0773] Smartphones (e.g., iPhone, Android): Use the camera to collect facial expression data and send the data to a server.

[0774] Server: Analyzes the collected data and generates advice and warnings.

[0775] EmotionRecognizer: A library that analyzes facial expression data to recognize emotions (e.g., OpenCV, dlib).

[0776] HealthMonitor: A module that collects health data.

[0777] AlertSystem: A system that notifies workers of alerts (e.g., push notifications).

[0778] 4. Adding concrete examples

[0779] Health and Emotion Collection

[0780] While the user works in the factory, the smartwatch continuously collects data such as heart rate and body temperature, while the smartphone periodically captures the user's facial expressions and analyzes their emotional data. This data is then sent to a server, which then generates advice on appropriate breaks and hydration.

[0781] Prompt Sentence Examples

[0782] Write a sample program that uses health and emotional data collected from a smartwatch or smartphone to provide advice to factory workers on how to take appropriate breaks and stay hydrated.

[0783] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0784] Step 1:

[0785] The device works in conjunction with the smartwatch to collect the user's health data. Specifically, the smartwatch collects data such as heart rate, body temperature, and number of steps via Bluetooth, and sends this data to the device.

[0786] Input: Heart rate, body temperature, and step count data from a smartwatch

[0787] Output: Health status data stored on the device

[0788] Step 2:

[0789] The device uses a camera to capture the user's facial expressions and collect emotional data, which is then analyzed using image recognition technology to quantify the user's emotional state.

[0790] Input: Facial expression image taken by the device camera

[0791] Output: Analyzed emotion data (e.g., stress level, happiness)

[0792] Step 3:

[0793] The device periodically transmits the collected health and emotion data to a server, where the data is uploaded via the Internet.

[0794] Input: Health status data and emotion data stored on the device

[0795] Output: Health status data and emotion data uploaded to the server

[0796] Step 4:

[0797] The server analyzes the received health and emotion data using an analytical means, evaluating fluctuations in health and emotion to determine the user's current situation.

[0798] Input: Health status data and emotion data received from the device

[0799] Output: Analysis results (e.g. current health status, stress level, fatigue level)

[0800] Step 5:

[0801] The server analyzes past drinking records, health status data, and emotional data to dynamically generate advice on appropriate rest times and hydration.

[0802] Input: Analysis results, past drinking records, health status data, emotional data

[0803] Output: Generated advice (e.g., time to take a break, recommended hydration)

[0804] Step 6:

[0805] The server sends the generated advice to the device, which then notifies the user of the received advice and prompts them to take the necessary action.

[0806] Input: Advice from the server

[0807] Output: Advice notice displayed on terminal

[0808] Step 7:

[0809] The server continuously monitors the data collected in real time using the monitoring means, and if an abnormality is detected, such as a sudden increase in heart rate or high stress level, the server generates a warning message and sends it to the device.

[0810] Input: Real-time collected health status data, emotional data

[0811] Output: Warning message when an abnormality is detected

[0812] Step 8:

[0813] The device will notify the user of the received warning message and prompt them to take appropriate measures.

[0814] Input: Warning message from the server

[0815] Output: Warning notice displayed on terminal

[0816] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0817] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0819] [Third embodiment]

[0820] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0821] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0822] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0823] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0824] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0825] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0826] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0827] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0828] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0829] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0830] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0831] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0832] This invention is a system that helps users enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health data, and analyzes past drinking records and health data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, monitors the user's health in real time, and issues a warning if it detects any abnormalities.

[0833] Program processing

[0834] Menu image analysis

[0835] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[0836] Examples:

[0837] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[0838] Health status collection

[0839] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones and sends this to a server.

[0840] Examples:

[0841] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user and uses it to evaluate their health.

[0842] Analysis of drinking data

[0843] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0844] Examples:

[0845] Based on data on how the user felt the next day when they previously drank three beers between 8:00 p.m. and 10:00 p.m., the system will advise them to avoid excessive drinking in the same situation.

[0846] Drink selection advice

[0847] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0848] Examples:

[0849] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking another beer. We recommend a soda or non-alcoholic beverage instead." It will also simultaneously suggest drinking a glass of water.

[0850] Health monitoring and alerts

[0851] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0852] Examples:

[0853] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." In particular, by quickly detecting a sudden rise in heart rate or abnormal changes in body temperature while drinking and notifying the user immediately, serious health risks can be prevented.

[0854] The processing flow will be explained below.

[0855] Step 1:

[0856] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[0857] Step 2:

[0858] The terminal transmits the captured menu image to the server.

[0859] Step 3:

[0860] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[0861] Step 4:

[0862] The server sends the recognized drink list to the terminal.

[0863] Step 5:

[0864] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[0865] Step 6:

[0866] The terminal transmits the collected health condition data to the server.

[0867] Step 7:

[0868] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data.

[0869] Step 8:

[0870] The server uses the analysis data to calculate the recommended alcohol allowance and consumption speed, and provides appropriate advice to the user.

[0871] Step 9:

[0872] When the user selects their next drink, the device sends the selected drink to the server.

[0873] Step 10:

[0874] The server determines whether the selected drink is appropriate based on the current health status and past analysis results.

[0875] Step 11:

[0876] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[0877] Step 12:

[0878] The device displays the received advice to the user.

[0879] Step 13:

[0880] The server continuously monitors the health status data collected in real time.

[0881] Step 14:

[0882] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[0883] Step 15:

[0884] The device will display the received alert to the user.

[0885] Example 1

[0886] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0887] In recent years, it has been pointed out that many people consume excessive amounts of alcohol in drinking establishments and fail to properly manage their own health. Excessive drinking poses a high risk of harm to health, and sudden changes in heart rate and body temperature in particular can pose serious health risks. Furthermore, users often find it difficult to judge the appropriate amount and type of alcohol they are drinking, which can result in excessive alcohol intake and dehydration. To solve these issues, a system is needed that allows users to enjoy alcohol safely and appropriately.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0889] In this invention, the server includes image recognition means for analyzing menu images and extracting a list of foods and beverages, biometric information collection means for collecting user health status data, and data analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, enabling users to accurately select foods and beverages at restaurants, determine an appropriate amount of alcohol intake based on their own health status, and enjoy drinking safely.

[0890] A "menu image" is a digital image of a medium that visually displays a list of food and drink items offered at a restaurant or other establishment.

[0891] The "food and drink list" is a list of specific drink and food names extracted from the menu image.

[0892] "Image recognition means" refers to algorithms or techniques used to extract specific information from digital images.

[0893] "User" refers to an individual who uses this system to manage their drinking and health.

[0894] "Health data" refers to information about the user's physical condition, such as heart rate, body temperature, sleep status, and number of steps taken.

[0895] "Biometric information collection means" refers to devices and sensors used to collect a user's health status data.

[0896] "Drinking records" are data on the amount of alcohol a user has consumed in the past, the time they drank, and their physical condition afterwards.

[0897] "Data analysis means" refers to systems or software that analyze collected health data and drinking records to determine the appropriate amount of alcohol intake for the user.

[0898] "Selection advice means" refers to a function for determining whether the next food or drink the user selects is appropriate and providing advice.

[0899] "Condition monitoring means" refers to technologies and systems that continuously monitor the user's health condition in real time and issue warnings if an abnormality is detected.

[0900] This invention is a system that allows users to enjoy alcohol safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and analyzes past drinking records and health status data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0901] The main components of the system are:

[0902] 1. Analysis of Menu Images

[0903] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server analyzes the received menu image using the Google Cloud Vision API. As a result of the analysis, a list of the foods and drinks listed on the menu is extracted. The server then sends the recognized list of foods and drinks to the device.

[0904] Examples:

[0905] When a user enters a restaurant, they take a photo of the menu with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image using the Google Cloud Vision API. The resulting list of beer, wine, cocktails, etc. is generated and sent to the device.

[0906] 2. Collecting health information

[0907] The device collects data such as heart rate, sleep status, and body temperature from connected smartwatches and smartphones, and sends this health status data to a server.

[0908] Examples:

[0909] The server periodically collects heart rate, step count, and sleep data from the smartwatch (e.g., Apple Watch) worn by the user and uses this data to evaluate health status.

[0910] 3. Analysis of drinking data

[0911] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[0912] Examples:

[0913] The server generates advice for the same situation based on the user's physical condition data from when they drank three beers at a specific time in the past.

[0914] 4. Drink selection advice

[0915] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[0916] Examples:

[0917] The next time the user selects a beer, the device will notify them, "You might want to avoid that next beer. We recommend a soda or non-alcoholic beverage instead," and also suggest that they "drink a glass of water."

[0918] 5. Health monitoring and alerts

[0919] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[0920] Examples:

[0921] If the user's heart rate suddenly rises, the server detects the abnormality and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." The device then displays this warning to the user as a pop-up notification.

[0922] Example prompts for generative AI models

[0923] "It analyzes the menu photo to generate a list of the food and drink options available. It then processes the user's current health data and drinking history to provide them with appropriate drinking advice."

[0924] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0925] Step 1:

[0926] A user takes a photo of a restaurant menu and uploads it through the app. The device sends the photo to the server. The server then analyzes the received menu image through an image recognition engine and extracts a drink list. The input is the menu image, and the output is the extracted drink list.

[0927] Specific behavior:

[0928] The user launches the smartphone app and takes a photo of the menu with the camera.

[0929] The smartphone app uploads menu images to the server.

[0930] The server performs image analysis using the Google Cloud Vision API.

[0931] The server extracts a list of food and beverages from the analysis results and sends it to the user's terminal.

[0932] Step 2:

[0933] The device acquires the user's health status data (heart rate, sleep status, body temperature, etc.) from a connected smartwatch or smartphone. It sends this data to a server. The input is the health status data from the smartwatch or smartphone, and the output is the health data sent to the server.

[0934] Specific behavior:

[0935] The smartwatch automatically records the user's heart rate, steps, and sleep data.

[0936] The smartphone periodically collects this data and sends it to the server.

[0937] The server stores the received data in a database and uses it for subsequent analysis.

[0938] Step 3:

[0939] The server integrates and analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data. This is used to calculate the recommended alcohol allowance and consumption pace. The input is the past drinking record and current health status data, and the output is the analyzed recommended alcohol allowance and consumption pace.

[0940] Specific behavior:

[0941] The server queries a historical drinking database to obtain the user's historical drinking record.

[0942] The server runs the newly received health data and these drinking records through an analysis algorithm.

[0943] Based on the analysis results, the server calculates the recommended alcohol allowance and drinking pace.

[0944] The server sends the calculation results to the user's device and uses them for future drinking advice.

[0945] Step 4:

[0946] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and generates and sends advice to the device. The input is the user's selected drink, and the output is an advice message from the server.

[0947] Specific behavior:

[0948] Users select their next drink using a smartphone app.

[0949] The terminal transmits the selected drink information to the server.

[0950] The server re-analyzes the health status and past drinking records and generates appropriate advice.

[0951] The server sends the generated advice message to the terminal, and the terminal notifies the user of the advice.

[0952] Step 5:

[0953] The server continuously monitors the health status data collected in real time. If an abnormality is detected, a warning message is generated and sent to the user's device. The input is the health status data acquired in real time, and the output is a warning message when an abnormality is detected.

[0954] Specific behavior:

[0955] The server continuously monitors the health data.

[0956] If the server detects an abnormality, it generates a warning message.

[0957] A warning message is sent to the user terminal, and the terminal displays the warning to the user.

[0958] (Application example 1)

[0959] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0960] Conventional drinking support systems have difficulty managing a user's health status comprehensively and in real time, making it difficult to provide appropriate drinking advice. Furthermore, they lack the functionality to suggest the next drink based on past drinking records and current health status, making it impossible to prevent drinking-related illness and health risks. The purpose of this invention is to solve these problems and provide a system that allows users to enjoy drinking safely and appropriately.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0962] In this invention, the server includes image recognition means for analyzing menu images and extracting a beverage list, health status collection means for collecting user health status data, analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, display means for displaying the beverage list extracted from the menu images, determination means for determining whether the beverage selected by the user is appropriate based on the user's health status and displaying the result, advice means for providing appropriate advice when the user selects their next beverage, and monitoring means for monitoring the user's health status in real time and issuing an alert if an abnormality is detected. This makes it possible to manage the user's health status comprehensively and in real time and provide appropriate drinking advice.

[0963] A "menu image" is an image, in paper or digital format, that lists and details the drinks served at a restaurant.

[0964] "Image recognition means" refers to software or hardware for analyzing menu images and extracting a drink list.

[0965] "Health Data" refers to information that indicates the user's health status, such as the user's heart rate, body temperature, number of steps, and sleep data.

[0966] "Health status collection means" refers to a means for collecting a user's health status data using sensors in a smartwatch or smartphone.

[0967] "Drinking history" is data that indicates the amount of alcohol a user has consumed in the past, the frequency of drinking, and related health conditions.

[0968] "Analysis means" refers to software or hardware functions for determining appropriate alcohol intake based on health status data and drinking records.

[0969] "Advice Measure" refers to a software or hardware function that provides appropriate advice to the user on the next beverage they should select.

[0970] The "monitoring means" is a system that monitors the user's health condition in real time and issues an alert if an abnormality is detected.

[0971] "Display means" refers to the display or software functionality for displaying to the user the beverage list extracted from the menu image.

[0972] The "determination means" is a function for determining whether the drink selected by the user is appropriate based on the current health condition and displaying the result.

[0973] "Warning" refers to a message that alerts the user when an abnormality is detected in the user's health condition.

[0974] This invention is a system that allows users to enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and determines the appropriate amount of alcohol intake by analyzing past drinking records and health status data. Furthermore, it has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[0975] Hardware

[0976] To realize this system, the following hardware is required:

[0977] Smartphone: A device that allows users to take pictures of the menu and upload them to the app.

[0978] Smartwatch: A device that collects health data such as a user's heart rate, body temperature, and steps taken.

[0979] Server: A computational system that performs data analysis and provides appropriate advice.

[0980] software

[0981] To realize this system, the following software is required:

[0982] Image recognition engine (such as Google Cloud Vision API): Recognizes characters from menu images and extracts the drink list.

[0983] Mobile application (iOS / Android): An application that allows users to take pictures of menu items, collect health data, and receive advice.

[0984] Health data collection API (Apple HealthKit, Google Fit, etc.): Collects user health data and sends it to a server.

[0985] Real-time data analysis server (AWS Lambda, AWS SageMaker, etc.): A server system that analyzes health status data and drinking records to determine appropriate alcohol intake.

[0986] Specific examples

[0987] 1. Analysis of Menu Images

[0988] Users take a photo of a restaurant menu with their smartphone and upload it through the app. The server then uses the Google Cloud Vision API to extract the drink list from the menu. The server then sends the recognized drink list to the smartphone and displays it to the user.

[0989] 2. Collecting health status data

[0990] The smartwatch collects real-time data such as heart rate, body temperature, and number of steps taken by the wearer, and this data is periodically sent to a server via a mobile app.

[0991] 3. Analysis of drinking data

[0992] The server analyzes the user's past drinking history (amount of alcohol consumed, time of drinking, and how they felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user.

[0993] 4. Drink selection advice

[0994] When a user selects their next drink, the smartphone sends that information to the server, which determines whether the selected drink is appropriate based on their current health status and sends advice to the smartphone, which then displays the advice to the user.

[0995] 5. Health monitoring and alerts

[0996] The server monitors health status data in real time, and if it detects any abnormalities, it immediately generates a warning message and sends it to the user's smartphone, allowing the user to receive the warning and take appropriate action.

[0997] Prompt Sentence Examples

[0998] Parse the drink list from a menu photo. Design an application that can collect users' health data and provide advice on their next drink selection based on their past drinking history. Propose a software solution to monitor health status in real time and support safe drinking.

[0999] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1000] Step 1:

[1001] A user enters a restaurant, takes a photo of the menu page with their smartphone, and uploads the image to the server through the application. The server analyzes the received menu image using the Google Cloud Vision API. The input is the menu image, and the output is a list of drinks extracted through image analysis.

[1002] Step 2:

[1003] The server uses an image analysis engine (Google Cloud Vision API) to perform character recognition on menu images and generate a drink list. Specifically, character recognition technology is used to extract drink names and categories listed on the menu and convert them into a text list. The input is image-recognized text data, and the output is the extracted drink list.

[1004] Step 3:

[1005] The server sends the extracted beverage list to the smartphone, and the smartphone displays the beverage list to the user. The input is a text-format beverage list, and the output is a beverage list displayed on the user's smartphone. This step allows the user to check the beverage list.

[1006] Step 4:

[1007] When a user wears a smartwatch, the smartphone periodically collects health status data such as heart rate, body temperature, and number of steps from the smartwatch. The collected data is temporarily stored on the smartphone and then sent to a server. The input is the health status data obtained from the smartwatch, and the output is the health status data sent to the server.

[1008] Step 5:

[1009] The server receives the health status data sent from the smartwatch and stores it in a database along with past drinking records. The server analyzes this data and calculates the appropriate amount of alcohol intake based on the user's health status and past drinking patterns. The input is the health status data and past drinking records, and the output is the analysis result, the recommended alcohol intake amount.

[1010] Step 6:

[1011] When the user selects their next drink, the smartphone sends the selection to the server. The input is the user's selected drink, and the output is the selection information sent to the server.

[1012] Step 7:

[1013] The server determines whether the selected beverage is appropriate based on the user's current health status and sends the result to the smartphone. Specifically, it calculates whether the user's selected beverage is recommended based on past drinking data and current health status data. The input is the user's health status data and the selected beverage, and the output is appropriate advice.

[1014] Step 8:

[1015] The terminal receives advice from the server and displays it to the user. The input is the advice information sent from the server, and the output is the advice message that the user checks on their smartphone.

[1016] Step 9:

[1017] The server monitors the user's health status data in real time and generates a warning message if an abnormality is detected. This warning message is sent to the user's smartphone, so the user is immediately alerted. The input is real-time health status data, and the output is a warning message when an abnormality is detected.

[1018] Step 10:

[1019] Users receive a warning message on their smartphone and take appropriate action. This allows users to prevent health risks caused by excessive drinking. The input is the warning message, and the output is the user's response.

[1020] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1021] This invention is a system that allows users to enjoy drinking safely and appropriately, and includes an emotion engine that recognizes the user's emotions. This allows the system to take into account the user's emotions and stress level when drinking and provide more appropriate advice. The system analyzes menu images to extract a drink list, collects the user's health status and emotion data, and analyzes this to determine the appropriate amount of alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the ability to monitor the user's health status and emotion in real time and issue a warning if an abnormality is detected.

[1022] Program processing

[1023] Menu image analysis

[1024] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[1025] Examples:

[1026] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[1027] Health and Emotion Collection

[1028] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from the connected smartwatch or smartphone and sends it to a server. It also collects the user's emotional data (through facial recognition and voice analysis, etc.).

[1029] Examples:

[1030] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user, and also uses the smartphone camera to collect emotional data from the user's facial expressions.

[1031] Analysis of drinking and emotion data

[1032] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) along with newly collected health and emotional data. From the analyzed data, it calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[1033] Examples:

[1034] If a user previously experienced a headache after drinking three beers between 8:00 and 10:00 PM, the system will use that data to advise them to avoid excessive drinking in the same situation. Furthermore, if the user's facial expression indicates fatigue or stress, the system will take that information into account and recommend limiting alcohol intake.

[1035] Drink selection advice

[1036] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and emotional data, as well as past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[1037] Examples:

[1038] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking that next beer. We recommend a soda or non-alcoholic beverage instead." The device will also simultaneously suggest drinking a glass of water. If the user appears tired or stressed, the device will also display personalized advice, such as, "Take it easy today and drink plenty of fluids to help you relax."

[1039] Health and emotional monitoring and alerts

[1040] The server continuously monitors the health and emotional data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[1041] Examples:

[1042] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." If the user's facial expression shows signs of stress, the server will warn, "Your stress level is high. Please try to relax." In particular, by quickly detecting sudden increases in heart rate, abnormal changes in body temperature, and abnormalities in emotional data and notifying the user immediately, serious health risks can be prevented.

[1043] The processing flow will be explained below.

[1044] Step 1:

[1045] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[1046] Step 2:

[1047] The terminal transmits the captured menu image to the server.

[1048] Step 3:

[1049] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[1050] Step 4:

[1051] The server sends the recognized drink list to the terminal.

[1052] Step 5:

[1053] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[1054] Step 6:

[1055] The device collects user emotional data by recognizing the user's facial expressions using the smartphone camera and then uses an emotion engine to analyze their stress level and emotional state.

[1056] Step 7:

[1057] The device transmits the collected health status data and emotion data to a server.

[1058] Step 8:

[1059] The server analyzes past drinking records and newly collected health and emotional data.

[1060] Step 9:

[1061] The server uses the analyzed data to calculate the recommended alcohol tolerance and consumption speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[1062] Step 10:

[1063] When the user selects their next drink, the device sends the selected drink to the server.

[1064] Step 11:

[1065] The server determines whether the selected drink is appropriate based on the current health and emotional data, as well as past analysis results.

[1066] Step 12:

[1067] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[1068] Step 13:

[1069] The device will then display the advice it receives to the user, such as a message like, "You might want to avoid your next beer. We recommend a soda or non-alcoholic beverage instead."

[1070] Step 14:

[1071] The server continuously monitors the health and emotional data collected in real time.

[1072] Step 15:

[1073] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[1074] Step 16:

[1075] The device will then display the received warning to the user, such as "Your health is deteriorating. Stop drinking alcohol and take a break by drinking water" or "Your stress level is high. Please try to relax."

[1076] Example 2

[1077] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] Conventional drinking management systems only consider the user's health status and are unable to provide advice that takes into account the user's emotions and stress level. Therefore, there is a need for systems that provide appropriate advice based on the emotional fluctuations and stress levels that accompany drinking, enabling users to drink safely and with consideration for their health. Furthermore, real-time monitoring of health status and emotions is insufficient, making it difficult to detect abnormalities early and issue warnings. Therefore, a system that comprehensively monitors and analyzes the user's emotions and health status and provides appropriate advice is needed.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1080] In this invention, the server includes image recognition means for analyzing menu images and extracting a drink list, biometric information collection means for collecting a user's biometric information, emotional information collection means for collecting the user's emotional data, analysis means for analyzing past drinking records, the biometric information, and the emotional data to determine an appropriate amount of alcohol intake, advice means for providing appropriate advice when the user selects their next drink, and monitoring means for monitoring the user's biometric information and emotional data in real time and issuing a warning if an abnormality is detected. This makes it possible to comprehensively manage and analyze the user's health condition and emotions, provide more appropriate drinking advice, and detect abnormalities early to reduce the user's health risks.

[1081] A "menu image" is an image that lists the food and drink items offered at a restaurant or other establishment, and is taken by the user and entered into the system.

[1082] The "drink list" is a list of drink names and information extracted from the menu image.

[1083] "Image recognition means" refers to a system or device that analyzes input image data and extracts specific information.

[1084] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and sleep state.

[1085] A "biometric information collection means" is a device or system for measuring and collecting a user's biometric information.

[1086] "Emotional data" is data that indicates the user's emotional state, analyzed from the user's facial expressions, tone of voice, etc.

[1087] The "emotion information collection means" is a system or device for collecting user emotion data.

[1088] An "analysis means" is a system or device for analyzing collected data and deriving results.

[1089] The "advice means" is a system or device that instructs the user on appropriate actions based on the results obtained by the analysis means.

[1090] "Monitoring means" refers to a system or device for monitoring a user's biometric information and emotional data in real time and detecting abnormalities.

[1091] An "alert" is a notification or message that notifies the user of an abnormal situation.

[1092] This invention is a system that allows users to enjoy drinking safely and appropriately. Specifically, it includes an emotion engine that recognizes the user's emotions and provides more appropriate advice by taking into account the user's emotions and stress level when drinking. This system uses the following hardware and software to collect, analyze data, and provide advice.

[1093] Hardware and software used

[1094] Hardware:

[1095] Smartphone (taking photos and running applications)

[1096] Smartwatch (biometric information measurement)

[1097] Server (data analysis and storage)

[1098] software:

[1099] Image recognition engine (e.g. Google Cloud Vision API)

[1100] Biometric information collection application (smartwatch integration)

[1101] Emotion recognition engines (e.g., Python OpenCV and speech recognition libraries)

[1102] Data analysis engines (e.g., Python's scikit-learn library)

[1103] Specific processing

[1104] Menu image analysis

[1105] When a user visits a restaurant, they take a photo of the menu with their smartphone and upload it to the server via the app. The device then sends the image to the server, which then uses an image recognition engine to analyze the image and extract the list of drinks listed on the menu. The server then sends the recognized list to the device and displays it to the user.

[1106] Examples:

[1107] For example, a user can take a picture of a restaurant menu with their smartphone and send it to the server through the app. The server then uses the Google Cloud Vision API to analyze the image, extract the drink list, and send it to the device to display to the user.

[1108] Health and Emotion Collection

[1109] The device collects the user's biometric information (heart rate, body temperature, sleep data, etc.) from the smartwatch and sends it to the server. The device also collects emotional data through facial expression and voice analysis and sends it to the server.

[1110] Examples:

[1111] The app periodically transmits heart rate and body temperature data from the smartwatch worn by the user to a server, and when the user takes a selfie with their smartphone, the device uses a facial expression recognition engine to collect emotional data and transmit it to the server.

[1112] Analysis of drinking and emotion data

[1113] The server accumulates past drinking records, current biometric information, and emotional data, and stores them in a database. Using an analytical engine, it calculates the appropriate amount of alcohol intake and the appropriate speed of consumption. It also evaluates stress levels and prepares to provide appropriate advice to users.

[1114] Examples:

[1115] If data is stored showing that a user previously experienced a headache after drinking three beers, the system will analyze that data along with current biometric and emotional data to generate advice on how much to limit next time.

[1116] Drink selection advice

[1117] When the user selects their next drink, the device sends the selection information to the server, which determines whether the drink is appropriate based on the user's current biometric and emotional data, generates and sends advice to the device, and displays the advice to the user.

[1118] Examples:

[1119] When the user chooses to drink wine next, the server analyzes biometric information and past data to generate advice such as, "You should refrain from drinking wine next time. We recommend drinking water," and notifies the device.

[1120] Health and emotional monitoring and alerts

[1121] The server monitors biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device, which then immediately displays the warning to the user and prompts them to take appropriate measures.

[1122] Examples:

[1123] If the heart rate suddenly rises, the server generates a warning saying, "Your heart rate is high. Stop drinking, drink some water and take a break," and sends it to the device to notify the user.

[1124] Prompt Sentence Examples

[1125] "In a drinking management system, generate prompts to advise users on the appropriate amount of beer they should drink."

[1126] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1127] Step 1: Take and upload your menu images

[1128] The user takes a photo of a restaurant menu. Using the camera app on their smartphone, they take a photo that shows the entire menu. They then upload the menu image to the server through a dedicated application. The input is the menu image taken by the user, and the output is the image file sent to the server. Specifically, the user takes a photo of the menu and taps the "upload" button on the app.

[1129] Step 2: Receiving and analyzing menu images

[1130] The device sends the captured menu image to the server. The server runs the received image through an image recognition engine to extract the list of drinks listed on the menu. At this time, the Google Cloud Vision API is used to identify the text portion of the image and convert it into a list of drinks. The input is the received menu image, and the output is the identified list of drinks. Specifically, the server calls the API to extract the drink menu from the image.

[1131] Step 3: Send and display the drink list

[1132] The server sends the recognized drink list to the device. The device displays the received list to the user. The input is the recognized drink list, and the output is the list displayed on the user's device. Specifically, the device app displays the drink list and allows the user to select the next drink.

[1133] Step 4: Collect health and emotions

[1134] The device collects the user's biometric information (heart rate, body temperature, sleep state, etc.) from a connected smartwatch or smartphone and sends this to a server. It also analyzes the user's facial expressions and voice to collect emotional data, which it then sends to the server. The input is the biometric information and emotional data from the smartwatch or smartphone, and the output is this data sent to the server. Specifically, it acquires data from the smartwatch and captures images for facial recognition.

[1135] Step 5: Analyze drinking and emotion data

[1136] The server accumulates past drinking records, current biometric information, and emotional data and stores them in a database. An analysis engine is used to analyze this data and calculate the appropriate amount of alcohol intake and the appropriate speed of intake. It also evaluates stress levels. The input is the past drinking records and current biometric information and emotional data, and the output is the calculated appropriate amount of alcohol intake and the appropriate speed of intake. Specifically, the server runs a data analysis algorithm to obtain the results.

[1137] Step 6: Drink Selection Advice

[1138] When the user selects their next drink, the device sends the selection information to the server. The server determines whether the drink is appropriate based on the user's current biometric and emotional data, generates advice, and sends it to the device. The input is the selected drink information and the user's current biometric and emotional data, and the output is advice sent from the server to the device. Specifically, the device may display a message saying, "You should refrain from having the next drink."

[1139] Step 7: Health and Emotion Monitoring and Alerts

[1140] The server monitors the biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device. The device immediately displays the warning to the user. The input is the biometric and emotional data collected in real time, and the output is a warning message when an abnormality is detected. Specifically, if a sudden increase in heart rate or an abnormality is detected, the server immediately sends a warning to the device to notify the user, such as "Your heart rate is high. Stop drinking, drink water, and take a break."

[1141] (Application example 2)

[1142] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1143] There is a need to improve work efficiency and reduce health risks by monitoring the health and emotions of factory workers in real time and providing advice on appropriate breaks and hydration. However, there is currently a lack of systems that can properly evaluate workers' health and emotions and provide effective advice based on that assessment.

[1144] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means that analyzes menu images and extracts a drink list, a health status collection means that collects the user's health status data, an emotion collection means that collects the user's emotion data, an analysis means that analyzes past drinking records and the health status data and emotion data to determine an appropriate amount of alcohol intake, an advice means that provides appropriate advice when the user selects their next drink, and a monitoring means that monitors the user's health status and emotions in real time and issues an alert if an abnormality is detected. This makes it possible to effectively monitor the health status and emotions of factory workers, improving work efficiency and reducing health risks.

[1145] A "menu image" is an image that visually shows a list of drinks and food served at a restaurant, cafe, etc.

[1146] A "drink list" is a list of information such as the names, types, and prices of drinks extracted from the menu image.

[1147] "Image recognition means" refers to technology or devices for analyzing menu images and extracting a drink list.

[1148] "Health Data" refers to data that indicates the user's physical health, such as heart rate, body temperature, number of steps taken, and sleep status.

[1149] "Health status collection means" refers to a means for collecting a user's health status data using a device such as a smartwatch or smartphone.

[1150] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[1151] "Emotion collection means" refers to a means of collecting user emotional data using a camera or microphone.

[1152] "Analysis means" refers to technology or devices that analyze collected health data, emotional data, past drinking records, etc. to determine appropriate alcohol intake amounts and appropriate rest periods.

[1153] The "advice means" refers to a means for providing the user with advice on appropriate alcohol intake, rest periods, and hydration based on the results determined by the analysis means.

[1154] "Monitoring means" refers to technologies and devices that monitor the user's health and emotions in real time and issue warnings if any abnormalities are detected.

[1155] This invention is a system for managing the health of factory workers, which monitors the user's health condition and emotions in real time and provides advice on appropriate breaks and hydration. Specific embodiments of this system are described below.

[1156] 1. Overall system configuration

[1157] The system consists of the following main components:

[1158] Device (smartphone)

[1159] server

[1160] Smartwatch

[1161] The device is operated by the user and works in conjunction with the smartwatch to collect health status data, and also uses the device's camera to capture the user's facial expressions and collect emotional data.

[1162] The server analyzes the collected health and emotional data to generate appropriate advice and warnings. Additionally, the server analyzes past data to make more accurate decisions.

[1163] A smartwatch is a device that collects real-time health data such as a user's heart rate, body temperature, and number of steps taken.

[1164] 2. Program Processing

[1165] First, the device connects to the smartwatch and periodically collects health status data (heart rate, body temperature, number of steps, etc.), which is then sent to a server via Bluetooth or Wi-Fi. At the same time, the device's camera is used to collect the user's facial expression data, which is also sent to the server.

[1166] Upon receiving this data, the server analyzes the data using the health status collection means and emotion collection means to evaluate the user's current health status and emotion, and determines the user's stress level and fatigue level based on the analysis results.

[1167] The server then uses analytical tools to generate advice on appropriate rest times and hydration based on past drinking records, historical health and emotional data, and sends this advice to the device, which then notifies the user.

[1168] Furthermore, the server uses monitoring means to issue a warning if it detects an abnormality in the data collected in real time. For example, if the user's heart rate suddenly rises or if their facial expression indicates high stress, a warning message will be sent to the device to notify the user.

[1169] 3. Hardware and Software Used

[1170] Smartwatches (e.g., Apple Watch, Fitbit): Collect health data such as heart rate, body temperature, and steps taken.

[1171] Smartphones (e.g., iPhone, Android): Use the camera to collect facial expression data and send the data to a server.

[1172] Server: Analyzes the collected data and generates advice and warnings.

[1173] EmotionRecognizer: A library that analyzes facial expression data to recognize emotions (e.g., OpenCV, dlib).

[1174] HealthMonitor: A module that collects health data.

[1175] AlertSystem: A system that notifies workers of alerts (e.g., push notifications).

[1176] 4. Adding concrete examples

[1177] Health and Emotion Collection

[1178] While the user works in the factory, the smartwatch continuously collects data such as heart rate and body temperature, while the smartphone periodically captures the user's facial expressions and analyzes their emotional data. This data is then sent to a server, which then generates advice on appropriate breaks and hydration.

[1179] Prompt Sentence Examples

[1180] Write a sample program that uses health and emotional data collected from a smartwatch or smartphone to provide advice to factory workers on how to take appropriate breaks and stay hydrated.

[1181] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1182] Step 1:

[1183] The device works in conjunction with the smartwatch to collect the user's health data. Specifically, the smartwatch collects data such as heart rate, body temperature, and number of steps via Bluetooth, and sends this data to the device.

[1184] Input: Heart rate, body temperature, and step count data from a smartwatch

[1185] Output: Health status data stored on the device

[1186] Step 2:

[1187] The device uses a camera to capture the user's facial expressions and collect emotional data, which is then analyzed using image recognition technology to quantify the user's emotional state.

[1188] Input: Facial expression image taken by the device camera

[1189] Output: Analyzed emotion data (e.g., stress level, happiness)

[1190] Step 3:

[1191] The device periodically transmits the collected health and emotion data to a server, where the data is uploaded via the Internet.

[1192] Input: Health status data and emotion data stored on the device

[1193] Output: Health status data and emotion data uploaded to the server

[1194] Step 4:

[1195] The server analyzes the received health and emotion data using an analytical means, evaluating fluctuations in health and emotion to determine the user's current situation.

[1196] Input: Health status data and emotion data received from the device

[1197] Output: Analysis results (e.g. current health status, stress level, fatigue level)

[1198] Step 5:

[1199] The server analyzes past drinking records, health status data, and emotional data to dynamically generate advice on appropriate rest times and hydration.

[1200] Input: Analysis results, past drinking records, health status data, emotional data

[1201] Output: Generated advice (e.g., time to take a break, recommended hydration)

[1202] Step 6:

[1203] The server sends the generated advice to the device, which then notifies the user of the received advice and prompts them to take the necessary action.

[1204] Input: Advice from the server

[1205] Output: Advice notice displayed on terminal

[1206] Step 7:

[1207] The server continuously monitors the data collected in real time using the monitoring means, and if an abnormality is detected, such as a sudden increase in heart rate or high stress level, the server generates a warning message and sends it to the device.

[1208] Input: Real-time collected health status data, emotional data

[1209] Output: Warning message when an abnormality is detected

[1210] Step 8:

[1211] The device will notify the user of the received warning message and prompt them to take appropriate measures.

[1212] Input: Warning message from the server

[1213] Output: Warning notice displayed on terminal

[1214] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1215] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1216] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1217] [Fourth embodiment]

[1218] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1219] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1220] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1221] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1222] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1223] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1224] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1225] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1226] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1227] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1228] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1229] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1230] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1231] This invention is a system that helps users enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health data, and analyzes past drinking records and health data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, monitors the user's health in real time, and issues a warning if it detects any abnormalities.

[1232] Program processing

[1233] Menu image analysis

[1234] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[1235] Examples:

[1236] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[1237] Health status collection

[1238] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones and sends this to a server.

[1239] Examples:

[1240] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user and uses it to evaluate their health.

[1241] Analysis of drinking data

[1242] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[1243] Examples:

[1244] Based on data on how the user felt the next day when they previously drank three beers between 8:00 p.m. and 10:00 p.m., the system will advise them to avoid excessive drinking in the same situation.

[1245] Drink selection advice

[1246] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[1247] Examples:

[1248] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking another beer. We recommend a soda or non-alcoholic beverage instead." It will also simultaneously suggest drinking a glass of water.

[1249] Health monitoring and alerts

[1250] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[1251] Examples:

[1252] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." In particular, by quickly detecting a sudden rise in heart rate or abnormal changes in body temperature while drinking and notifying the user immediately, serious health risks can be prevented.

[1253] The processing flow will be explained below.

[1254] Step 1:

[1255] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[1256] Step 2:

[1257] The terminal transmits the captured menu image to the server.

[1258] Step 3:

[1259] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[1260] Step 4:

[1261] The server sends the recognized drink list to the terminal.

[1262] Step 5:

[1263] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[1264] Step 6:

[1265] The terminal transmits the collected health condition data to the server.

[1266] Step 7:

[1267] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data.

[1268] Step 8:

[1269] The server uses the analysis data to calculate the recommended alcohol allowance and consumption speed, and provides appropriate advice to the user.

[1270] Step 9:

[1271] When the user selects their next drink, the device sends the selected drink to the server.

[1272] Step 10:

[1273] The server determines whether the selected drink is appropriate based on the current health status and past analysis results.

[1274] Step 11:

[1275] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[1276] Step 12:

[1277] The device displays the received advice to the user.

[1278] Step 13:

[1279] The server continuously monitors the health status data collected in real time.

[1280] Step 14:

[1281] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[1282] Step 15:

[1283] The device will display the received alert to the user.

[1284] Example 1

[1285] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] In recent years, it has been pointed out that many people consume excessive amounts of alcohol in drinking establishments and fail to properly manage their own health. Excessive drinking poses a high risk of harm to health, and sudden changes in heart rate and body temperature in particular can pose serious health risks. Furthermore, users often find it difficult to judge the appropriate amount and type of alcohol they are drinking, which can result in excessive alcohol intake and dehydration. To solve these issues, a system is needed that allows users to enjoy alcohol safely and appropriately.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1288] In this invention, the server includes image recognition means for analyzing menu images and extracting a list of foods and beverages, biometric information collection means for collecting user health status data, and data analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, enabling users to accurately select foods and beverages at restaurants, determine an appropriate amount of alcohol intake based on their own health status, and enjoy drinking safely.

[1289] A "menu image" is a digital image of a medium that visually displays a list of food and drink items offered at a restaurant or other establishment.

[1290] The "food and drink list" is a list of specific drink and food names extracted from the menu image.

[1291] "Image recognition means" refers to algorithms or techniques used to extract specific information from digital images.

[1292] "User" refers to an individual who uses this system to manage their drinking and health.

[1293] "Health data" refers to information about the user's physical condition, such as heart rate, body temperature, sleep status, and number of steps taken.

[1294] "Biometric information collection means" refers to devices and sensors used to collect a user's health status data.

[1295] "Drinking records" are data on the amount of alcohol a user has consumed in the past, the time they drank, and their physical condition afterwards.

[1296] "Data analysis means" refers to systems or software that analyze collected health data and drinking records to determine the appropriate amount of alcohol intake for the user.

[1297] "Selection advice means" refers to a function for determining whether the next food or drink the user selects is appropriate and providing advice.

[1298] "Condition monitoring means" refers to technologies and systems that continuously monitor the user's health condition in real time and issue warnings if an abnormality is detected.

[1299] This invention is a system that allows users to enjoy alcohol safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and analyzes past drinking records and health status data to determine appropriate alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[1300] The main components of the system are:

[1301] 1. Analysis of Menu Images

[1302] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server analyzes the received menu image using the Google Cloud Vision API. As a result of the analysis, a list of the foods and drinks listed on the menu is extracted. The server then sends the recognized list of foods and drinks to the device.

[1303] Examples:

[1304] When a user enters a restaurant, they take a photo of the menu with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image using the Google Cloud Vision API. The resulting list of beer, wine, cocktails, etc. is generated and sent to the device.

[1305] 2. Collecting health information

[1306] The device collects data such as heart rate, sleep status, and body temperature from connected smartwatches and smartphones, and sends this health status data to a server.

[1307] Examples:

[1308] The server periodically collects heart rate, step count, and sleep data from the smartwatch (e.g., Apple Watch) worn by the user and uses this data to evaluate health status.

[1309] 3. Analysis of drinking data

[1310] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, providing appropriate advice to the user.

[1311] Examples:

[1312] The server generates advice for the same situation based on the user's physical condition data from when they drank three beers at a specific time in the past.

[1313] 4. Drink selection advice

[1314] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[1315] Examples:

[1316] The next time the user selects a beer, the device will notify them, "You might want to avoid that next beer. We recommend a soda or non-alcoholic beverage instead," and also suggest that they "drink a glass of water."

[1317] 5. Health monitoring and alerts

[1318] The server continuously monitors the health status data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[1319] Examples:

[1320] If the user's heart rate suddenly rises, the server detects the abnormality and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." The device then displays this warning to the user as a pop-up notification.

[1321] Example prompts for generative AI models

[1322] "It analyzes the menu photo to generate a list of the food and drink options available. It then processes the user's current health data and drinking history to provide them with appropriate drinking advice."

[1323] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1324] Step 1:

[1325] A user takes a photo of a restaurant menu and uploads it through the app. The device sends the photo to the server. The server then analyzes the received menu image through an image recognition engine and extracts a drink list. The input is the menu image, and the output is the extracted drink list.

[1326] Specific behavior:

[1327] The user launches the smartphone app and takes a photo of the menu with the camera.

[1328] The smartphone app uploads menu images to the server.

[1329] The server performs image analysis using the Google Cloud Vision API.

[1330] The server extracts a list of food and beverages from the analysis results and sends it to the user's terminal.

[1331] Step 2:

[1332] The device acquires the user's health status data (heart rate, sleep status, body temperature, etc.) from a connected smartwatch or smartphone. It sends this data to a server. The input is the health status data from the smartwatch or smartphone, and the output is the health data sent to the server.

[1333] Specific behavior:

[1334] The smartwatch automatically records the user's heart rate, steps, and sleep data.

[1335] The smartphone periodically collects this data and sends it to the server.

[1336] The server stores the received data in a database and uses it for subsequent analysis.

[1337] Step 3:

[1338] The server integrates and analyzes past drinking records (amount of alcohol consumed, time of drinking, and physical condition the next day) and newly collected health status data. This is used to calculate the recommended alcohol allowance and consumption pace. The input is the past drinking record and current health status data, and the output is the analyzed recommended alcohol allowance and consumption pace.

[1339] Specific behavior:

[1340] The server queries a historical drinking database to obtain the user's historical drinking record.

[1341] The server runs the newly received health data and these drinking records through an analysis algorithm.

[1342] Based on the analysis results, the server calculates the recommended alcohol allowance and drinking pace.

[1343] The server sends the calculation results to the user's device and uses them for future drinking advice.

[1344] Step 4:

[1345] When the user selects their next drink, the device sends the selected drink to the server. The server determines whether the selected drink is appropriate based on the user's current health status and past analysis results, and generates and sends advice to the device. The input is the user's selected drink, and the output is an advice message from the server.

[1346] Specific behavior:

[1347] Users select their next drink using a smartphone app.

[1348] The terminal transmits the selected drink information to the server.

[1349] The server re-analyzes the health status and past drinking records and generates appropriate advice.

[1350] The server sends the generated advice message to the terminal, and the terminal notifies the user of the advice.

[1351] Step 5:

[1352] The server continuously monitors the health status data collected in real time. If an abnormality is detected, a warning message is generated and sent to the user's device. The input is the health status data acquired in real time, and the output is a warning message when an abnormality is detected.

[1353] Specific behavior:

[1354] The server continuously monitors the health data.

[1355] If the server detects an abnormality, it generates a warning message.

[1356] A warning message is sent to the user terminal, and the terminal displays the warning to the user.

[1357] (Application example 1)

[1358] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1359] Conventional drinking support systems have difficulty managing a user's health status comprehensively and in real time, making it difficult to provide appropriate drinking advice. Furthermore, they lack the functionality to suggest the next drink based on past drinking records and current health status, making it impossible to prevent drinking-related illness and health risks. The purpose of this invention is to solve these problems and provide a system that allows users to enjoy drinking safely and appropriately.

[1360] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1361] In this invention, the server includes image recognition means for analyzing menu images and extracting a beverage list, health status collection means for collecting user health status data, analysis means for analyzing past drinking records and health status data to determine an appropriate amount of alcohol intake, display means for displaying the beverage list extracted from the menu images, determination means for determining whether the beverage selected by the user is appropriate based on the user's health status and displaying the result, advice means for providing appropriate advice when the user selects their next beverage, and monitoring means for monitoring the user's health status in real time and issuing an alert if an abnormality is detected. This makes it possible to manage the user's health status comprehensively and in real time and provide appropriate drinking advice.

[1362] A "menu image" is an image, in paper or digital format, that lists and details the drinks served at a restaurant.

[1363] "Image recognition means" refers to software or hardware for analyzing menu images and extracting a drink list.

[1364] "Health Data" refers to information that indicates the user's health status, such as the user's heart rate, body temperature, number of steps, and sleep data.

[1365] "Health status collection means" refers to a means for collecting a user's health status data using sensors in a smartwatch or smartphone.

[1366] "Drinking history" is data that indicates the amount of alcohol a user has consumed in the past, the frequency of drinking, and related health conditions.

[1367] "Analysis means" refers to software or hardware functions for determining appropriate alcohol intake based on health status data and drinking records.

[1368] "Advice Measure" refers to a software or hardware function that provides appropriate advice to the user on the next beverage they should select.

[1369] The "monitoring means" is a system that monitors the user's health condition in real time and issues an alert if an abnormality is detected.

[1370] "Display means" refers to the display or software functionality for displaying to the user the beverage list extracted from the menu image.

[1371] The "determination means" is a function for determining whether the drink selected by the user is appropriate based on the current health condition and displaying the result.

[1372] "Warning" refers to a message that alerts the user when an abnormality is detected in the user's health condition.

[1373] This invention is a system that allows users to enjoy drinking safely and appropriately. The system analyzes menu images to extract a drink list, collects the user's health status data, and determines the appropriate amount of alcohol intake by analyzing past drinking records and health status data. Furthermore, it has the function of monitoring the user's health status in real time and issuing a warning if an abnormality is detected.

[1374] Hardware

[1375] To realize this system, the following hardware is required:

[1376] Smartphone: A device that allows users to take pictures of the menu and upload them to the app.

[1377] Smartwatch: A device that collects health data such as a user's heart rate, body temperature, and steps taken.

[1378] Server: A computational system that performs data analysis and provides appropriate advice.

[1379] software

[1380] To realize this system, the following software is required:

[1381] Image recognition engine (such as Google Cloud Vision API): Recognizes characters from menu images and extracts the drink list.

[1382] Mobile application (iOS / Android): An application that allows users to take pictures of menu items, collect health data, and receive advice.

[1383] Health data collection API (Apple HealthKit, Google Fit, etc.): Collects user health data and sends it to a server.

[1384] Real-time data analysis server (AWS Lambda, AWS SageMaker, etc.): A server system that analyzes health status data and drinking records to determine appropriate alcohol intake.

[1385] Specific examples

[1386] 1. Analysis of Menu Images

[1387] Users take a photo of a restaurant menu with their smartphone and upload it through the app. The server then uses the Google Cloud Vision API to extract the drink list from the menu. The server then sends the recognized drink list to the smartphone and displays it to the user.

[1388] 2. Collecting health status data

[1389] The smartwatch collects real-time data such as heart rate, body temperature, and number of steps taken by the wearer, and this data is periodically sent to a server via a mobile app.

[1390] 3. Analysis of drinking data

[1391] The server analyzes the user's past drinking history (amount of alcohol consumed, time of drinking, and how they felt the next day) and newly collected health status data, and from the analyzed data calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user.

[1392] 4. Drink selection advice

[1393] When a user selects their next drink, the smartphone sends that information to the server, which determines whether the selected drink is appropriate based on their current health status and sends advice to the smartphone, which then displays the advice to the user.

[1394] 5. Health monitoring and alerts

[1395] The server monitors health status data in real time, and if it detects any abnormalities, it immediately generates a warning message and sends it to the user's smartphone, allowing the user to receive the warning and take appropriate action.

[1396] Prompt Sentence Examples

[1397] Parse the drink list from a menu photo. Design an application that can collect users' health data and provide advice on their next drink selection based on their past drinking history. Propose a software solution to monitor health status in real time and support safe drinking.

[1398] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1399] Step 1:

[1400] A user enters a restaurant, takes a photo of the menu page with their smartphone, and uploads the image to the server through the application. The server analyzes the received menu image using the Google Cloud Vision API. The input is the menu image, and the output is a list of drinks extracted through image analysis.

[1401] Step 2:

[1402] The server uses an image analysis engine (Google Cloud Vision API) to perform character recognition on menu images and generate a drink list. Specifically, character recognition technology is used to extract drink names and categories listed on the menu and convert them into a text list. The input is image-recognized text data, and the output is the extracted drink list.

[1403] Step 3:

[1404] The server sends the extracted beverage list to the smartphone, and the smartphone displays the beverage list to the user. The input is a text-format beverage list, and the output is a beverage list displayed on the user's smartphone. This step allows the user to check the beverage list.

[1405] Step 4:

[1406] When a user wears a smartwatch, the smartphone periodically collects health status data such as heart rate, body temperature, and number of steps from the smartwatch. The collected data is temporarily stored on the smartphone and then sent to a server. The input is the health status data obtained from the smartwatch, and the output is the health status data sent to the server.

[1407] Step 5:

[1408] The server receives the health status data sent from the smartwatch and stores it in a database along with past drinking records. The server analyzes this data and calculates the appropriate amount of alcohol intake based on the user's health status and past drinking patterns. The input is the health status data and past drinking records, and the output is the analysis result, the recommended alcohol intake amount.

[1409] Step 6:

[1410] When the user selects their next drink, the smartphone sends the selection to the server. The input is the user's selected drink, and the output is the selection information sent to the server.

[1411] Step 7:

[1412] The server determines whether the selected beverage is appropriate based on the user's current health status and sends the result to the smartphone. Specifically, it calculates whether the user's selected beverage is recommended based on past drinking data and current health status data. The input is the user's health status data and the selected beverage, and the output is appropriate advice.

[1413] Step 8:

[1414] The terminal receives advice from the server and displays it to the user. The input is the advice information sent from the server, and the output is the advice message that the user checks on their smartphone.

[1415] Step 9:

[1416] The server monitors the user's health status data in real time and generates a warning message if an abnormality is detected. This warning message is sent to the user's smartphone, so the user is immediately alerted. The input is real-time health status data, and the output is a warning message when an abnormality is detected.

[1417] Step 10:

[1418] Users receive a warning message on their smartphone and take appropriate action. This allows users to prevent health risks caused by excessive drinking. The input is the warning message, and the output is the user's response.

[1419] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1420] This invention is a system that allows users to enjoy drinking safely and appropriately, and includes an emotion engine that recognizes the user's emotions. This allows the system to take into account the user's emotions and stress level when drinking and provide more appropriate advice. The system analyzes menu images to extract a drink list, collects the user's health status and emotion data, and analyzes this to determine the appropriate amount of alcohol intake. It also provides appropriate advice when the user selects their next drink, and has the ability to monitor the user's health status and emotion in real time and issue a warning if an abnormality is detected.

[1421] Program processing

[1422] Menu image analysis

[1423] Users take a photo of a restaurant menu and upload it through the app. The device sends the photo to the server. The server then runs the image recognition engine through the menu and extracts the drink list from the menu. The server then sends the recognized drink list to the user's device.

[1424] Examples:

[1425] When a user enters a restaurant, they take a photo of the menu page with their smartphone. The smartphone app automatically uploads the menu image to the server, which then analyzes the image to generate a list of beer, wine, cocktails, and other items, which is then displayed on the device.

[1426] Health and Emotion Collection

[1427] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from the connected smartwatch or smartphone and sends it to a server. It also collects the user's emotional data (through facial recognition and voice analysis, etc.).

[1428] Examples:

[1429] The server periodically collects heart rate, step count, and sleep data from the smartwatch worn by the user, and also uses the smartphone camera to collect emotional data from the user's facial expressions.

[1430] Analysis of drinking and emotion data

[1431] The server analyzes past drinking records (amount of alcohol consumed, time of drinking, and how well you felt the next day) along with newly collected health and emotional data. From the analyzed data, it calculates the recommended alcohol allowance and drinking speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[1432] Examples:

[1433] If a user previously experienced a headache after drinking three beers between 8:00 and 10:00 PM, the system will use that data to advise them to avoid excessive drinking in the same situation. Furthermore, if the user's facial expression indicates fatigue or stress, the system will take that information into account and recommend limiting alcohol intake.

[1434] Drink selection advice

[1435] When the user selects their next drink, the device sends the selection to the server. The server determines whether the selected drink is appropriate based on the user's current health status and emotional data, as well as past analysis results, and sends a OK / NG rating for the drink and advice on necessary fluid intake to the device. The device then displays the received advice to the user.

[1436] Examples:

[1437] The next time a user is about to drink a beer, the device will display a message saying, "It's best to refrain from drinking that next beer. We recommend a soda or non-alcoholic beverage instead." The device will also simultaneously suggest drinking a glass of water. If the user appears tired or stressed, the device will also display personalized advice, such as, "Take it easy today and drink plenty of fluids to help you relax."

[1438] Health and emotional monitoring and alerts

[1439] The server continuously monitors the health and emotional data collected in real time. If an abnormality is detected in the collected data, it generates a warning message and sends it to the user's device. The device then displays the received warning to the user.

[1440] Examples:

[1441] If the user's heart rate suddenly rises, the server analyzes the information and sends a warning message to the device saying, "Your health is deteriorating. Stop drinking and take a break by drinking water, etc." If the user's facial expression shows signs of stress, the server will warn, "Your stress level is high. Please try to relax." In particular, by quickly detecting sudden increases in heart rate, abnormal changes in body temperature, and abnormalities in emotional data and notifying the user immediately, serious health risks can be prevented.

[1442] The processing flow will be explained below.

[1443] Step 1:

[1444] Users take a photo of the menu at a restaurant with their smartphone and upload it through the app.

[1445] Step 2:

[1446] The terminal transmits the captured menu image to the server.

[1447] Step 3:

[1448] The server runs the received menu image through an image recognition engine and extracts the drink list listed on the menu.

[1449] Step 4:

[1450] The server sends the recognized drink list to the terminal.

[1451] Step 5:

[1452] The device collects the user's health data (heart rate, sleep status, body temperature, etc.) from connected smartwatches and smartphones.

[1453] Step 6:

[1454] The device collects user emotional data by recognizing the user's facial expressions using the smartphone camera and then uses an emotion engine to analyze their stress level and emotional state.

[1455] Step 7:

[1456] The device transmits the collected health status data and emotion data to a server.

[1457] Step 8:

[1458] The server analyzes past drinking records and newly collected health and emotional data.

[1459] Step 9:

[1460] The server uses the analyzed data to calculate the recommended alcohol tolerance and consumption speed, and provides appropriate advice to the user. It also assesses the user's stress level and adjusts the recommendations accordingly.

[1461] Step 10:

[1462] When the user selects their next drink, the device sends the selected drink to the server.

[1463] Step 11:

[1464] The server determines whether the selected drink is appropriate based on the current health and emotional data, as well as past analysis results.

[1465] Step 12:

[1466] The server sends a judgment of whether the drink is OK or not and advice on necessary fluid intake to the terminal.

[1467] Step 13:

[1468] The device will then display the advice it receives to the user, such as a message like, "You might want to avoid your next beer. We recommend a soda or non-alcoholic beverage instead."

[1469] Step 14:

[1470] The server continuously monitors the health and emotional data collected in real time.

[1471] Step 15:

[1472] If the server detects an abnormality in the collected data, it generates a warning message and sends it to the user's terminal.

[1473] Step 16:

[1474] The device will then display the received warning to the user, such as "Your health is deteriorating. Stop drinking alcohol and take a break by drinking water" or "Your stress level is high. Please try to relax."

[1475] Example 2

[1476] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1477] Conventional drinking management systems only consider the user's health status and are unable to provide advice that takes into account the user's emotions and stress level. Therefore, there is a need for systems that provide appropriate advice based on the emotional fluctuations and stress levels that accompany drinking, enabling users to drink safely and with consideration for their health. Furthermore, real-time monitoring of health status and emotions is insufficient, making it difficult to detect abnormalities early and issue warnings. Therefore, a system that comprehensively monitors and analyzes the user's emotions and health status and provides appropriate advice is needed.

[1478] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1479] In this invention, the server includes image recognition means for analyzing menu images and extracting a drink list, biometric information collection means for collecting a user's biometric information, emotional information collection means for collecting the user's emotional data, analysis means for analyzing past drinking records, the biometric information, and the emotional data to determine an appropriate amount of alcohol intake, advice means for providing appropriate advice when the user selects their next drink, and monitoring means for monitoring the user's biometric information and emotional data in real time and issuing a warning if an abnormality is detected. This makes it possible to comprehensively manage and analyze the user's health condition and emotions, provide more appropriate drinking advice, and detect abnormalities early to reduce the user's health risks.

[1480] A "menu image" is an image that lists the food and drink items offered at a restaurant or other establishment, and is taken by the user and entered into the system.

[1481] The "drink list" is a list of drink names and information extracted from the menu image.

[1482] "Image recognition means" refers to a system or device that analyzes input image data and extracts specific information.

[1483] "Biometric information" refers to data that indicates the user's physical condition, such as heart rate, body temperature, and sleep state.

[1484] A "biometric information collection means" is a device or system for measuring and collecting a user's biometric information.

[1485] "Emotional data" is data that indicates the user's emotional state, analyzed from the user's facial expressions, tone of voice, etc.

[1486] The "emotion information collection means" is a system or device for collecting user emotion data.

[1487] An "analysis means" is a system or device for analyzing collected data and deriving results.

[1488] The "advice means" is a system or device that instructs the user on appropriate actions based on the results obtained by the analysis means.

[1489] "Monitoring means" refers to a system or device for monitoring a user's biometric information and emotional data in real time and detecting abnormalities.

[1490] An "alert" is a notification or message that notifies the user of an abnormal situation.

[1491] This invention is a system that allows users to enjoy drinking safely and appropriately. Specifically, it includes an emotion engine that recognizes the user's emotions and provides more appropriate advice by taking into account the user's emotions and stress level when drinking. This system uses the following hardware and software to collect, analyze data, and provide advice.

[1492] Hardware and software used

[1493] Hardware:

[1494] Smartphone (taking photos and running applications)

[1495] Smartwatch (biometric information measurement)

[1496] Server (data analysis and storage)

[1497] software:

[1498] Image recognition engine (e.g. Google Cloud Vision API)

[1499] Biometric information collection application (smartwatch integration)

[1500] Emotion recognition engines (e.g., Python OpenCV and speech recognition libraries)

[1501] Data analysis engines (e.g., Python's scikit-learn library)

[1502] Specific processing

[1503] Menu image analysis

[1504] When a user visits a restaurant, they take a photo of the menu with their smartphone and upload it to the server via the app. The device then sends the image to the server, which then uses an image recognition engine to analyze the image and extract the list of drinks listed on the menu. The server then sends the recognized list to the device and displays it to the user.

[1505] Examples:

[1506] For example, a user can take a picture of a restaurant menu with their smartphone and send it to the server through the app. The server then uses the Google Cloud Vision API to analyze the image, extract the drink list, and send it to the device to display to the user.

[1507] Health and Emotion Collection

[1508] The device collects the user's biometric information (heart rate, body temperature, sleep data, etc.) from the smartwatch and sends it to the server. The device also collects emotional data through facial expression and voice analysis and sends it to the server.

[1509] Examples:

[1510] The app periodically transmits heart rate and body temperature data from the smartwatch worn by the user to a server, and when the user takes a selfie with their smartphone, the device uses a facial expression recognition engine to collect emotional data and transmit it to the server.

[1511] Analysis of drinking and emotion data

[1512] The server accumulates past drinking records, current biometric information, and emotional data, and stores them in a database. Using an analytical engine, it calculates the appropriate amount of alcohol intake and the appropriate speed of consumption. It also evaluates stress levels and prepares to provide appropriate advice to users.

[1513] Examples:

[1514] If data is stored showing that a user previously experienced a headache after drinking three beers, the system will analyze that data along with current biometric and emotional data to generate advice on how much to limit next time.

[1515] Drink selection advice

[1516] When the user selects their next drink, the device sends the selection information to the server, which determines whether the drink is appropriate based on the user's current biometric and emotional data, generates and sends advice to the device, and displays the advice to the user.

[1517] Examples:

[1518] When the user chooses to drink wine next, the server analyzes biometric information and past data to generate advice such as, "You should refrain from drinking wine next time. We recommend drinking water," and notifies the device.

[1519] Health and emotional monitoring and alerts

[1520] The server monitors biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device, which then immediately displays the warning to the user and prompts them to take appropriate measures.

[1521] Examples:

[1522] If the heart rate suddenly rises, the server generates a warning saying, "Your heart rate is high. Stop drinking, drink some water and take a break," and sends it to the device to notify the user.

[1523] Prompt Sentence Examples

[1524] "In a drinking management system, generate prompts to advise users on the appropriate amount of beer they should drink."

[1525] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1526] Step 1: Take and upload your menu images

[1527] The user takes a photo of a restaurant menu. Using the camera app on their smartphone, they take a photo that shows the entire menu. They then upload the menu image to the server through a dedicated application. The input is the menu image taken by the user, and the output is the image file sent to the server. Specifically, the user takes a photo of the menu and taps the "upload" button on the app.

[1528] Step 2: Receiving and analyzing menu images

[1529] The device sends the captured menu image to the server. The server runs the received image through an image recognition engine to extract the list of drinks listed on the menu. At this time, the Google Cloud Vision API is used to identify the text portion of the image and convert it into a list of drinks. The input is the received menu image, and the output is the identified list of drinks. Specifically, the server calls the API to extract the drink menu from the image.

[1530] Step 3: Send and display the drink list

[1531] The server sends the recognized drink list to the device. The device displays the received list to the user. The input is the recognized drink list, and the output is the list displayed on the user's device. Specifically, the device app displays the drink list and allows the user to select the next drink.

[1532] Step 4: Collect health and emotions

[1533] The device collects the user's biometric information (heart rate, body temperature, sleep state, etc.) from a connected smartwatch or smartphone and sends this to a server. It also analyzes the user's facial expressions and voice to collect emotional data, which it then sends to the server. The input is the biometric information and emotional data from the smartwatch or smartphone, and the output is this data sent to the server. Specifically, it acquires data from the smartwatch and captures images for facial recognition.

[1534] Step 5: Analyze drinking and emotion data

[1535] The server accumulates past drinking records, current biometric information, and emotional data and stores them in a database. An analysis engine is used to analyze this data and calculate the appropriate amount of alcohol intake and the appropriate speed of intake. It also evaluates stress levels. The input is the past drinking records and current biometric information and emotional data, and the output is the calculated appropriate amount of alcohol intake and the appropriate speed of intake. Specifically, the server runs a data analysis algorithm to obtain the results.

[1536] Step 6: Drink Selection Advice

[1537] When the user selects their next drink, the device sends the selection information to the server. The server determines whether the drink is appropriate based on the user's current biometric and emotional data, generates advice, and sends it to the device. The input is the selected drink information and the user's current biometric and emotional data, and the output is advice sent from the server to the device. Specifically, the device may display a message saying, "You should refrain from having the next drink."

[1538] Step 7: Health and Emotion Monitoring and Alerts

[1539] The server monitors the biometric and emotional data collected in real time, and if it detects an abnormality, it generates a warning and sends it to the device. The device immediately displays the warning to the user. The input is the biometric and emotional data collected in real time, and the output is a warning message when an abnormality is detected. Specifically, if a sudden increase in heart rate or an abnormality is detected, the server immediately sends a warning to the device to notify the user, such as "Your heart rate is high. Stop drinking, drink water, and take a break."

[1540] (Application example 2)

[1541] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1542] There is a need to improve work efficiency and reduce health risks by monitoring the health and emotions of factory workers in real time and providing advice on appropriate breaks and hydration. However, there is currently a lack of systems that can properly evaluate workers' health and emotions and provide effective advice based on that assessment.

[1543] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an image recognition means that analyzes menu images and extracts a drink list, a health status collection means that collects the user's health status data, an emotion collection means that collects the user's emotion data, an analysis means that analyzes past drinking records and the health status data and emotion data to determine an appropriate amount of alcohol intake, an advice means that provides appropriate advice when the user selects their next drink, and a monitoring means that monitors the user's health status and emotions in real time and issues an alert if an abnormality is detected. This makes it possible to effectively monitor the health status and emotions of factory workers, improving work efficiency and reducing health risks.

[1544] A "menu image" is an image that visually shows a list of drinks and food served at a restaurant, cafe, etc.

[1545] A "drink list" is a list of information such as the names, types, and prices of drinks extracted from the menu image.

[1546] "Image recognition means" refers to technology or devices for analyzing menu images and extracting a drink list.

[1547] "Health Data" refers to data that indicates the user's physical health, such as heart rate, body temperature, number of steps taken, and sleep status.

[1548] "Health status collection means" refers to a means for collecting a user's health status data using a device such as a smartwatch or smartphone.

[1549] "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[1550] "Emotion collection means" refers to a means of collecting user emotional data using a camera or microphone.

[1551] "Analysis means" refers to technology or devices that analyze collected health data, emotional data, past drinking records, etc. to determine appropriate alcohol intake amounts and appropriate rest periods.

[1552] The "advice means" refers to a means for providing the user with advice on appropriate alcohol intake, rest periods, and hydration based on the results determined by the analysis means.

[1553] "Monitoring means" refers to technologies and devices that monitor the user's health and emotions in real time and issue warnings if any abnormalities are detected.

[1554] This invention is a system for managing the health of factory workers, which monitors the user's health condition and emotions in real time and provides advice on appropriate breaks and hydration. Specific embodiments of this system are described below.

[1555] 1. Overall system configuration

[1556] The system consists of the following main components:

[1557] Device (smartphone)

[1558] server

[1559] Smartwatch

[1560] The device is operated by the user and works in conjunction with the smartwatch to collect health status data, and also uses the device's camera to capture the user's facial expressions and collect emotional data.

[1561] The server analyzes the collected health and emotional data to generate appropriate advice and warnings. Additionally, the server analyzes past data to make more accurate decisions.

[1562] A smartwatch is a device that collects real-time health data such as a user's heart rate, body temperature, and number of steps taken.

[1563] 2. Program Processing

[1564] First, the device connects to the smartwatch and periodically collects health status data (heart rate, body temperature, number of steps, etc.), which is then sent to a server via Bluetooth or Wi-Fi. At the same time, the device's camera is used to collect the user's facial expression data, which is also sent to the server.

[1565] Upon receiving this data, the server analyzes the data using the health status collection means and emotion collection means to evaluate the user's current health status and emotion, and determines the user's stress level and fatigue level based on the analysis results.

[1566] The server then uses analytical tools to generate advice on appropriate rest times and hydration based on past drinking records, historical health and emotional data, and sends this advice to the device, which then notifies the user.

[1567] Furthermore, the server uses monitoring means to issue a warning if it detects an abnormality in the data collected in real time. For example, if the user's heart rate suddenly rises or if their facial expression indicates high stress, a warning message will be sent to the device to notify the user.

[1568] 3. Hardware and Software Used

[1569] Smartwatches (e.g., Apple Watch, Fitbit): Collect health data such as heart rate, body temperature, and steps taken.

[1570] Smartphones (e.g., iPhone, Android): Use the camera to collect facial expression data and send the data to a server.

[1571] Server: Analyzes the collected data and generates advice and warnings.

[1572] EmotionRecognizer: A library that analyzes facial expression data to recognize emotions (e.g., OpenCV, dlib).

[1573] HealthMonitor: A module that collects health data.

[1574] AlertSystem: A system that notifies workers of alerts (e.g., push notifications).

[1575] 4. Adding concrete examples

[1576] Health and Emotion Collection

[1577] While the user works in the factory, the smartwatch continuously collects data such as heart rate and body temperature, while the smartphone periodically captures the user's facial expressions and analyzes their emotional data. This data is then sent to a server, which then generates advice on appropriate breaks and hydration.

[1578] Prompt Sentence Examples

[1579] Write a sample program that uses health and emotional data collected from a smartwatch or smartphone to provide advice to factory workers on how to take appropriate breaks and stay hydrated.

[1580] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1581] Step 1:

[1582] The device works in conjunction with the smartwatch to collect the user's health data. Specifically, the smartwatch collects data such as heart rate, body temperature, and number of steps via Bluetooth, and sends this data to the device.

[1583] Input: Heart rate, body temperature, and step count data from a smartwatch

[1584] Output: Health status data stored on the device

[1585] Step 2:

[1586] The device uses a camera to capture the user's facial expressions and collect emotional data, which is then analyzed using image recognition technology to quantify the user's emotional state.

[1587] Input: Facial expression image taken by the device camera

[1588] Output: Analyzed emotion data (e.g., stress level, happiness)

[1589] Step 3:

[1590] The device periodically transmits the collected health and emotion data to a server, where the data is uploaded via the Internet.

[1591] Input: Health status data and emotion data stored on the device

[1592] Output: Health status data and emotion data uploaded to the server

[1593] Step 4:

[1594] The server analyzes the received health and emotion data using an analytical means, evaluating fluctuations in health and emotion to determine the user's current situation.

[1595] Input: Health status data and emotion data received from the device

[1596] Output: Analysis results (e.g. current health status, stress level, fatigue level)

[1597] Step 5:

[1598] The server analyzes past drinking records, health status data, and emotional data to dynamically generate advice on appropriate rest times and hydration.

[1599] Input: Analysis results, past drinking records, health status data, emotional data

[1600] Output: Generated advice (e.g., time to take a break, recommended hydration)

[1601] Step 6:

[1602] The server sends the generated advice to the device, which then notifies the user of the received advice and prompts them to take the necessary action.

[1603] Input: Advice from the server

[1604] Output: Advice notice displayed on terminal

[1605] Step 7:

[1606] The server continuously monitors the data collected in real time using the monitoring means, and if an abnormality is detected, such as a sudden increase in heart rate or high stress level, the server generates a warning message and sends it to the device.

[1607] Input: Real-time collected health status data, emotional data

[1608] Output: Warning message when an abnormality is detected

[1609] Step 8:

[1610] The device will notify the user of the received warning message and prompt them to take appropriate measures.

[1611] Input: Warning message from the server

[1612] Output: Warning notice displayed on terminal

[1613] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1614] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1615] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1616] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1617] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1618] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1619] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1620] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1621] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1622] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1623] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1624] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1625] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1626] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1627] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1628] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1629] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1630] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1631] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1632] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1633] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1634] The following is further disclosed regarding the above embodiment.

[1635] (Claim 1)

[1636] Image recognition means for analyzing menu images and extracting a drink list;

[1637] a health status collection means for collecting health status data of a user;

[1638] An analytical method for analyzing past drinking records and health status data to determine appropriate alcohol intake amounts;

[1639] an advice means for providing appropriate advice to the user when selecting their next drink;

[1640] A monitoring method that monitors the user's health status in real time and issues an alert if an abnormality is detected;

[1641] A system including:

[1642] (Claim 2)

[1643] 10. The system of claim 1, wherein the system provides advice on appropriate beverages and necessary fluid intake when the user selects their next drink.

[1644] (Claim 3)

[1645] The system of claim 1, characterized in that it analyzes the user's health status data and past drinking records to determine the amount and speed of drinking.

[1646] "Example 1"

[1647] (Claim 1)

[1648] image recognition means for analyzing menu images and extracting a list of food and drink items;

[1649] a biometric information collection means for collecting health status data of a user;

[1650] A data analysis means for analyzing past drinking records and health status data to determine appropriate alcohol intake amounts;

[1651] A selection advice means for providing appropriate advice to a user when selecting a next food or beverage;

[1652] A status monitoring means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected;

[1653] A system including:

[1654] (Claim 2)

[1655] 10. The system of claim 1, wherein the system provides advice on appropriate drinks and necessary fluid intake when the user selects their next food or drink.

[1656] (Claim 3)

[1657] The system of claim 1, characterized in that it analyzes the user's health status data and past drinking records to determine the amount and speed of drinking.

[1658] "Application Example 1"

[1659] (Claim 1)

[1660] image recognition means for analyzing menu images and extracting a beverage list;

[1661] a health status collection means for collecting health status data of a user;

[1662] An analytical method for analyzing past drinking records and health status data to determine appropriate alcohol intake amounts;

[1663] an advisory means for providing appropriate advice to the user when selecting their next beverage;

[1664] A monitoring method that monitors the user's health status in real time and issues an alert if an abnormality is detected.

[1665] a display means for displaying a beverage list extracted from the menu image;

[1666] a determining means for determining whether the beverage selected by the user is appropriate based on the user's health condition and displaying the result;

[1667] A system including:

[1668] (Claim 2)

[1669] 10. The system of claim 1, wherein the system provides advice on appropriate beverages and necessary fluid intake when the user selects their next beverage.

[1670] (Claim 3)

[1671] The system of claim 1, characterized in that it analyzes the user's health status data and past drinking records to determine the amount and speed of drinking.

[1672] (Claim 4)

[1673] 10. The system of claim 1, wherein the system provides real-time health status monitoring and alerting functions and sends an alert to the user when an abnormality is detected.

[1674] "Example 2: Combining Emotion Engines"

[1675] (Claim 1)

[1676] image recognition means for analyzing menu images and extracting a beverage list;

[1677] a biometric information collection means for collecting biometric information of a user;

[1678] Emotion information collection means for collecting user emotion data;

[1679] an analysis means for analyzing past drinking records, biological information, and emotional data to determine an appropriate amount of alcohol intake;

[1680] an advice means for providing appropriate advice to the user when selecting their next beverage;

[1681] A monitoring means that monitors the user's biometric information and emotional data in real time and issues a warning if an abnormality is detected;

[1682] A system including:

[1683] (Claim 2)

[1684] 10. The system of claim 1, wherein the system provides advice on appropriate beverages and necessary fluid intake when the user selects their next beverage.

[1685] (Claim 3)

[1686] The system according to claim 1, characterized in that it analyzes the user's biometric information, emotional data and past drinking records to determine the amount and rate of alcohol consumption.

[1687] "Application example 2 when combining emotion engines"

[1688] (Claim 1)

[1689] Image recognition means for analyzing menu images and extracting a drink list;

[1690] a health status collection means for collecting health status data of a user;

[1691] An emotion collection means for collecting emotion data of a user;

[1692] an analysis means for analyzing past drinking records, health condition data, and emotional data to determine an appropriate amount of alcohol intake;

[1693] an advice means for providing appropriate advice to the user when selecting their next drink;

[1694] A monitoring method that monitors the user's health and emotions in real time and issues a warning if an abnormality is detected.

[1695] A system including:

[1696] (Claim 2)

[1697] 10. The system of claim 1, wherein the system provides advice on appropriate beverages and necessary fluid intake when the user selects their next drink.

[1698] (Claim 3)

[1699] The system of claim 1, characterized in that it analyzes the user's health condition data, emotional data, and past drinking records to determine the amount and speed of drinking. [Explanation of symbols]

[1700] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. Image recognition means for analyzing menu images and extracting a drink list; a health status collection means for collecting health status data of a user; An analytical method for analyzing past drinking records and health status data to determine appropriate alcohol intake amounts; an advice means for providing appropriate advice to the user when selecting their next drink; A monitoring method that monitors the user's health status in real time and issues an alert if an abnormality is detected; A system including:

2. 10. The system of claim 1, wherein the system provides advice on appropriate beverages and necessary fluid intake when the user selects their next drink.

3. The system of claim 1, wherein the system analyzes the user's health status data and past drinking records to determine the amount and speed of drinking.

Images