System

Abstract

A system is provided.SOLUTION: A system comprising: means for recognizing an image of a menu of a restaurant photographed by a user and extracting a type and details of a drink; means for acquiring health data of the user and evaluating a current health condition; means for collating a past drinking history and physical condition data with the current health condition and determining an appropriate alcohol intake; and means for monitoring the health condition of the user in real time and issuing a warning when an abnormality is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people face the risks and health consequences of drinking too much, especially when they continue drinking without knowing their own health condition or limits. Conventional methods make it difficult to make on-the-spot judgments and provide appropriate advice, especially when health conditions change from moment to moment. This can lead to inconvenience to others, hangovers, and even serious health problems. To address this situation, there is a need for a system that can provide appropriate advice in real time while drinking and minimize health risks. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. It provides a means for performing image recognition on a restaurant menu photographed by the user and extracting the types and details of drinks. It also provides a means for acquiring the user's health data and assessing their current health condition. It also provides a means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake. It also includes a means for providing advice on selecting the next drink based on the assessment results. It also includes a means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected. This realizes a system that enables users to maintain an appropriate drinking pace and avoid health risks.

[0006] "User" refers to any individual or organization that uses this system.

[0007] "Restaurant" refers to an establishment or place that serves food and beverages.

[0008] "Menu" refers to a list of food and beverages offered by a restaurant.

[0009] "Image recognition" refers to the technology of extracting and analyzing information from captured images.

[0010] "Drinks" refers to beverages, including alcoholic and non-alcoholic beverages.

[0011] "Health data" refers to data that indicates the user's health condition, such as heart rate, sleep time, and amount of exercise.

[0012] "Current health status" refers to the user's current health status as assessed based on the acquired health data.

[0013] "Past drinking history" refers to the details of the user's past drinking and data related thereto.

[0014] "Physical condition data" refers to data relating to the user's physical condition, such as physical condition after drinking alcohol, the severity of a hangover, etc.

[0015] "Alcohol intake" refers to the total amount of alcohol consumed by a user within a certain period of time.

[0016] "Determination result" refers to the conclusion calculated by the system regarding the amount of alcohol a user may consume in the future.

[0017] "Advice" refers to advice on the user's drinking pace and which drinks to choose.

[0018] "Real-time monitoring" refers to technology that continuously monitors a user's health status.

[0019] "Warning" refers to a warning issued when a person's health condition reaches a dangerous level. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition and drinking habits will be described in detail.

[0042] 1. Image recognition for menus

[0043] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[0044] 2. Health Data Collection

[0045] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0046] 3. Determining alcohol intake

[0047] The server extracts past drinking history and physical condition data from a database. Based on this, it compares it with the user's current health condition and determines how much alcohol the user needs to consume and at what speed. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[0048] 4. Advice for choosing your next drink

[0049] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[0050] 5. Real-time health monitoring and alerts

[0051] The device continuously monitors the user's health condition in real time. Data is collected and analyzed periodically. If the user's health condition suddenly deteriorates, the device immediately issues a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0052] The system of the present invention provides support for users to enjoy drinking in a healthy manner and reduces the health risks associated with excessive drinking. As a specific example, if a user attempts to select a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and maintain their health.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[0056] Step 2:

[0057] The device uses image recognition technology to analyze the menu image, and the extracted information, such as the name, type, and price of the drink, is stored in the device's memory.

[0058] Step 3:

[0059] The device sends the extracted information to the server, which then registers the information in a database and generates a drink list.

[0060] Step 4:

[0061] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. The data is collected periodically and stored on the device.

[0062] Step 5:

[0063] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[0064] Step 6:

[0065] The server extracts past drinking history and physical condition data from a database, and based on this, compares it with the current health condition to determine the appropriate amount of alcohol intake.

[0066] Step 7:

[0067] The server sends the result of the judgment to the terminal, which then stores the result.

[0068] Step 8:

[0069] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[0070] Step 9:

[0071] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[0072] Step 10:

[0073] The device monitors the user's health status in real time, with data collected periodically and analyzed to detect abnormalities.

[0074] Step 11:

[0075] If your health condition suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level, the device will display a message saying, "Your current heart rate has reached a dangerous level. Stop drinking alcohol and drink water."

[0076] The above are the specific processing steps of this system.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] In today's world, managing alcohol consumption is important for maintaining good health for many people. However, it is difficult for users to determine the appropriate amount of alcohol intake based on their own health condition and drinking history. Furthermore, there is a lack of methods to monitor health status in real time while drinking and respond immediately if an abnormality occurs. To address these issues, a system is needed that provides appropriate advice based on health status and drinking habits, and supports users in enjoying healthy drinking.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment results, and means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected. This allows the user to monitor their health condition in real time while maintaining an appropriate drinking pace according to their own health condition, and receive appropriate advice and warnings as needed.

[0082] A "user" is an individual who uses the system to manage their drinking habits and health.

[0083] "Eating and drinking establishments" refers to places where users eat and drink, including restaurants and bars.

[0084] A "menu" is a document or booklet listing the drinks and food served at a restaurant.

[0085] "Image recognition" refers to the technique of extracting information from images using computer vision techniques.

[0086] "Beverages" refers to alcoholic and non-alcoholic beverages served at restaurants.

[0087] "Health data" refers to data that indicates the user's health condition, such as heart rate, sleep time, and amount of exercise.

[0088] "Health status" refers to the user's current physical condition and includes parameters such as heart rate, blood pressure, and activity level.

[0089] "Drinking history" is a record of alcoholic beverages consumed by a user in the past.

[0090] "Physical condition data" is a general term for data related to the user's physical condition and health.

[0091] "Alcohol intake" refers to the amount of alcohol consumed by a user within a certain period of time.

[0092] The "judgment result" is an assessment of the appropriate alcohol intake tolerance calculated by the system based on the user's health data, drinking history, and current health condition.

[0093] "Advice" refers to recommendations or recommendations about drinking that the system provides to the user.

[0094] "Real-time monitoring" refers to a technology that continuously monitors a user's health status and updates the data instantly.

[0095] A "warning" is a message issued by the system to alert the user when the user's health condition reaches an abnormal state.

[0096] "Server" refers to a computer system for processing and storing data.

[0097] A "database" is a data structure or system used to organize and manage information.

[0098] "Decision-making" is the process by which a system draws a conclusion based on multiple data and conditions.

[0099] The present invention relates to a system that provides appropriate advice based on a user's health condition and drinking habits. This system provides support for users to maintain their health while enjoying alcohol.

[0100] First, the user takes a photo of a restaurant menu with their smartphone. The device analyzes the menu image using Google Cloud Vision API to extract the type of drink and details. The extracted information is sent to the server and registered in a database. This is how menu information is managed.

[0101] Next, the device acquires health data (heart rate, sleep time, exercise volume, etc.) from the user's smartwatch or smartphone. This health data is collected through software such as Apple Health or Google Fit. The collected data is sent to a server and stored in a database. This allows the user's health status to be monitored appropriately.

[0102] The server extracts past drinking history and health data from a database and compares it with the user's current health status. This determines how much alcohol the user can safely consume. For example, it makes a specific determination such as whether two beers or one cocktail is safe. The result of the determination is sent from the server to the device.

[0103] When a user inputs or selects the next drink they want to order, the terminal displays a "OK" or "NG" result based on the judgment result from the server. Even if the drink is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the terminal will display "NG," but suggest "One beer is OK."

[0104] Furthermore, the device continuously monitors the user's health condition in real time, collecting and analyzing data periodically. If the user's health condition suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0105] For example, if a user tries to choose a cocktail at a restaurant, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain an appropriate drinking pace and maintain their health.

[0106] An example of a prompt to be input to a generative AI model is, "Please describe a system that provides appropriate advice based on the user's health status and drinking habits. Specifically, please provide details on image recognition of menus, collection of health data, determination of alcohol intake, advice on next drink selection, and real-time monitoring and alerting of health status."

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1:

[0109] A user takes a photo of a restaurant menu with their smartphone, and the image becomes the input data.

[0110] Step 2:

[0111] The device analyzes the menu image captured using the Google Cloud Vision API. Using the menu image as input, an image recognition algorithm extracts the drink type and details. The output data is the drink type and its detailed information.

[0112] Step 3:

[0113] The device sends the extracted drink information to the server. The input data is the type of drink and detailed information, which is converted into a format that is sent to the server as output data. This data is also registered in the database as menu information.

[0114] Step 4:

[0115] The device acquires health data (heart rate, sleep time, exercise amount, etc.) from the user's smartwatch or smartphone. The input data is various sensor data from the health device, and the output data is organized as health data.

[0116] Step 5:

[0117] The health data collected by the device is sent to a server. The input data is organized health data, and the output data is data converted into a format suitable for sending to the server. This saves the health data in the server's database.

[0118] Step 6:

[0119] The server extracts past drinking history and health data from the database, with input data being the user's drinking history and health data stored in the database and output data being the extracted history data.

[0120] Step 7:

[0121] The server compares the current health status with past data. The input data is the past drinking history and health data, and the current health data. The data matching algorithm determines the appropriate alcohol intake. The output data is the determined acceptable alcohol intake limit.

[0122] Step 8:

[0123] The server sends the result of the determination to the terminal. The input data is the determined alcohol intake tolerance, and the output data is data converted into a format that can be sent to the terminal.

[0124] Step 9:

[0125] The user inputs or selects the next drink to order into the terminal, and the input data is information about the drink selected or input by the user.

[0126] Step 10:

[0127] The device displays a verdict of "OK" or "NG" based on the judgment result received from the server. The input data is the information about the drink selected by the user and the judgment result from the server, and the output data is a verdict of "OK" or "NG." It also suggests alternative drinks.

[0128] Step 11:

[0129] The device monitors the user's health condition in real time. The input data is real-time data from various sensor devices, and the output data is an assessment of the user's current health condition.

[0130] Step 12:

[0131] If the device detects an abnormality, it will issue a warning to the user. The input data is health data collected in real time, and if an abnormality is detected through analysis, a warning message is displayed as output data.

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Currently, health management when drinking alcohol at restaurants is left up to the user, and inappropriate alcohol consumption can pose health risks. To solve this problem, a system is needed that can grasp the user's health status and drinking habits in real time and provide appropriate advice. In addition, a system is needed that can provide specific advice when the user selects their next drink and issue warnings according to changes in their health status.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting drink types and details, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment result, means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected, and means for displaying the advice and warning in real time using smart glasses. This enables users to achieve appropriate alcohol intake while reducing health risks at restaurants.

[0137] "User" refers to an individual or corporation that uses this system.

[0138] "Restaurant" means a commercial establishment that serves beverages and meals.

[0139] A "menu" is a list of the types of food and drink served by a restaurant.

[0140] "Image recognition" refers to the technology of extracting specific information from captured images.

[0141] "Types of drinks" refers to the various drinks listed on a restaurant menu.

[0142] "Health data" refers to information related to a user's health, such as their heart rate, sleep time, and amount of exercise.

[0143] "Health status" refers to the current physical condition and degree of health of the user, as assessed based on the user's health data.

[0144] "Drinking history" refers to a record of what types and amounts of alcoholic beverages a user has consumed in the past.

[0145] "Alcohol intake" refers to the total amount of alcohol consumed by a user within a certain period of time.

[0146] "Real-time monitoring" refers to the act of continuously monitoring a user's health status.

[0147] "Warning" refers to a warning that is given when the user's health condition reaches a dangerous state.

[0148] "Smart glasses" refers to a type of wearable device that can present information visually.

[0149] "Server" refers to the device that serves as the core of this system and analyzes, stores, and distributes data.

[0150] As an embodiment of the present invention, a system for enabling users to consume alcohol appropriately while reducing health risks at restaurants will be described in detail.

[0151] Hardware and Software Overview

[0152] The server is the core of this system and is a device that analyzes, stores, and distributes data. This system uses smart glasses, smartphones, and wearable devices (e.g., smartwatches). Smart glasses are a type of wearable device that can present information visually, and Google Glass can be used. Image recognition APIs such as Google Cloud Vision API are used for image recognition. Health data is collected using health data analysis platforms such as Apple HealthKit and Google Fit. Firebase can also be used as a real-time communication server.

[0153] Program processing explanation

[0154] The server receives restaurant menu images taken by the user using smart glasses or a smartphone and analyzes them using an image recognition API. This extracts drink types and details from the menu items. The server then obtains the user's health data from the smartwatch or smartphone and evaluates their current health status. This evaluation uses data such as heart rate, sleep time, and exercise volume.

[0155] The server compares the user's current health condition with their past drinking history and physical condition data, and determines the appropriate amount of alcohol intake based on this. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. Based on this judgment, it provides advice on the user's next drink. The smart glasses display shows a "OK" or "NG" result, and also suggests alternatives.

[0156] The smart glasses and smartphone also continuously monitor the user's health status in real time. If the user's heart rate reaches a dangerous level, a warning message will be displayed immediately. The specific warning message will be displayed, such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0157] Examples and prompts

[0158] As a concrete example, consider the case where a user uses smart glasses to select a cocktail from a restaurant menu. Because the user's heart rate is already high, the system judges the selection as "NG" and the smart glasses display shows "One beer is OK." In this case, the user can order a beer instead of a cocktail to reduce the health risk.

[0159] Example prompt sentence:

[0160] When a user browses a menu and chooses a cocktail at a restaurant, they provide the following information:

[0161] 1. If a cocktail is "NG," please briefly explain why it is "NG."

[0162] 2. Suggest a specific drink name as an alternative.

[0163] 3. Based on the current user health data (high heart rate), advise what action to take.

[0164] example:

[0165] Cocktails: No - High heart rate, try a lower alcohol content drink. Alternative: 1 beer

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] A user takes a photo of a restaurant menu using smart glasses or a smartphone.

[0169] Input: Restaurant menu image

[0170] Output: Menu image taken

[0171] The terminal acquires the menu image photographed by the user and temporarily stores the image.

[0172] Step 2:

[0173] The terminal transmits the captured menu image to the server.

[0174] Input: Menu image taken

[0175] Output: Image data sent to the server

[0176] The device uploads the image data to a specified server and sends a request to an API for image recognition.

[0177] Step 3:

[0178] The server uses an image recognition API to analyze the menu images and extract drink types and details.

[0179] Input: Image data sent to the server

[0180] Output: List of drink types and details

[0181] The server uses the Google Cloud Vision API and other tools to analyze the text in the images and categorize the menu items by drink type.

[0182] Step 4:

[0183] The server obtains the user's health data from the health data analysis platform.

[0184] Input: User identification information

[0185] Output: User's health data (heart rate, sleep time, exercise amount, etc.)

[0186] The server retrieves health data through APIs from platforms such as HealthKit and Google Fit and uses it to assess the user's current health status.

[0187] Step 5:

[0188] The server compares the acquired health data with past drinking history and physical condition data to determine the appropriate amount of alcohol intake.

[0189] Input: User's health data, past drinking history, and physical condition data

[0190] Output: Appropriate alcohol intake and judgment result

[0191] The server compares past drinking history and current health data from a database to calculate alcohol intake and determine drinking pace.

[0192] Step 6:

[0193] The server generates advice for the next drink selection based on the judgment result and transmits it to the terminal.

[0194] Input: Judgment result

[0195] Output: Advice ("OK" or "NG")

[0196] The server uses the generative AI model to generate advice appropriate for the user and outputs a judgment result such as "OK" or "NG" when the next drink is selected, or provides specific advice based on the prompt text.

[0197] Step 7:

[0198] The terminal receives the determination result from the server and displays it on the display of the smart glasses.

[0199] Input: Advice content

[0200] Output: Advice displayed on the smart glasses display

[0201] The terminal displays the received advice in real time on the display of the smart glasses so that the user can check it.

[0202] Step 8:

[0203] When the user orders the next drink, the terminal will prompt the user to enter or select the name of the drink and display the decision.

[0204] Input: User-selected drink name

[0205] Output: Drink judgment result ("OK" or "NG") and alternatives

[0206] The device sends the name of the drink selected by the user to the server and displays the result on the display. If the drink is judged to be "NG," an alternative option will be presented.

[0207] Step 9:

[0208] The server continuously monitors the user's health status in real time and immediately issues an alert if it detects any abnormalities.

[0209] Input: User's health data (real-time)

[0210] Output: Warning message

[0211] The server continuously monitors the user's health data, and if, for example, the heart rate reaches a dangerous level, it sends a warning message to the device, such as, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[0212] Through these steps, users can enjoy alcohol in a healthy manner at restaurants. This system provides real-time advice and warnings through the collaboration of smart glasses and a server, reducing health risks for users.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition, drinking status, and emotional state will be described in detail.

[0215] 1. Image recognition for menus

[0216] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[0217] 2. Health Data Collection

[0218] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0219] 3. Collecting Emotional Data

[0220] The device collects emotional data from the user's facial expressions and voice. This data is collected using a camera and microphone, and the emotion engine analyzes it. The analysis results are used to evaluate the user's emotional state (e.g., joy, sadness, anger, etc.), and the results are sent to the server and stored in a database.

[0221] 4. Determining alcohol intake

[0222] The server extracts past drinking history, physical condition data, and emotional data from a database. Based on this, it compares the user's current health and emotional state and determines the appropriate amount of alcohol intake. For example, it makes a specific judgment such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[0223] 5. Advice for choosing your next drink

[0224] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[0225] 6. Real-time health and emotional status monitoring and alerts

[0226] The device continuously monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[0227] The system of the present invention supports users in enjoying alcohol in a healthy and appropriate emotional state, reducing the health risks and emotional downfalls associated with excessive drinking. A specific example is a system in which, if a user attempts to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and emotional state, thereby maintaining their health and social relationships.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[0231] Step 2:

[0232] The device uses image recognition technology to analyze the menu image, extracting detailed information such as the drink name, type, and price, and storing this information in memory.

[0233] Step 3:

[0234] The extracted drink information is sent to the server, which then registers the received information in a database and generates a drink list.

[0235] Step 4:

[0236] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. This data is collected at regular intervals and stored on the device.

[0237] Step 5:

[0238] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[0239] Step 6:

[0240] The device uses a camera and microphone to collect the user's emotional data (facial expressions and voice), and the emotion engine analyzes this data to evaluate the user's current emotional state.

[0241] Step 7:

[0242] The evaluation results of the emotional state are sent to a server and stored in a database.

[0243] Step 8:

[0244] The server extracts past drinking history, physical condition data, and emotional data from a database, and compares it with the current health and emotional state to determine the appropriate amount of alcohol intake.

[0245] Step 9:

[0246] The server sends the result of the judgment to the terminal, which then stores the result.

[0247] Step 10:

[0248] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[0249] Step 11:

[0250] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[0251] Step 12:

[0252] The device monitors the user's health and emotional state in real time, with data periodically captured and analyzed to detect anomalies.

[0253] Step 13:

[0254] If your health or emotional state suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level or you become extremely irritable, a warning message will appear saying, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[0255] The above are the specific processing steps of this system.

[0256] Example 2

[0257] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0258] Currently, many people consume alcohol at restaurants without fully considering their health and emotional state, which can result in health risks and emotional problems. Furthermore, there are few systems that provide real-time information to help users determine the appropriate amount of alcohol they should consume. This makes it difficult to properly manage users' health and the effects of drinking.

[0259] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0260] In this invention, the server includes means for performing image recognition on restaurant menus photographed by the user and extracting drink types and details, means for acquiring the user's health data and evaluating their current health condition, and means for analyzing the user's facial expressions and voice to collect emotional data. This makes it possible to monitor the user's health and emotional state in real time, determine the appropriate amount of alcohol intake, and provide advice on selecting the next drink.

[0261] "User" refers to a person who uses the system.

[0262] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or smartwatch.

[0263] A "menu image" refers to an image of a menu taken by a user at a restaurant.

[0264] "Image recognition technology" refers to technology for extracting specific information from captured images.

[0265] "Drink type" refers to the category of drinks listed on the menu, for example, beer, wine, cocktails, etc.

[0266] "Health data" refers to physiological data such as the user's heart rate, sleep time, and amount of exercise.

[0267] "Emotion data" refers to the emotional state of the user analyzed from their facial expressions and voice, such as joy, sadness, anger, etc.

[0268] "Drinking history" refers to a record of alcohol consumed by a user in the past.

[0269] "Alcohol intake" refers to the amount of alcohol that is appropriate for a user to drink.

[0270] "Advice" refers to appropriate instructions or suggestions regarding drinking that the system provides to the user.

[0271] "Monitoring" refers to the continuous observation of a user's health and emotional state.

[0272] "Warning" refers to a warning message sent by the system when an abnormality is detected in the user's condition.

[0273] A "database" refers to a structured collection of information for storing and managing data within a system.

[0274] "Analysis" refers to the process of evaluating the user's condition based on collected data.

[0275] "Real-time" refers to near-simultaneous processing and responses.

[0276] The present invention is a system that provides appropriate advice based on a user's health condition, drinking habits, and emotional state. This system operates through the following specific steps.

[0277] Hardware and Software Configuration

[0278] Device: A smartphone, tablet, smartwatch, etc. used by a user. Devices are equipped with cameras and microphones, which are used to collect data.

[0279] Server: A computer system that analyzes and manages data. The server is connected to a database that stores users' health and emotional data.

[0280] Database: A data storage space located within the server. It stores user health data, emotional data, drinking history, etc.

[0281] Software and Technology

[0282] Image recognition technology: Google Cloud Vision API and OpenCV are used as image recognition technologies to analyze menu images captured by the device's camera.

[0283] Health data collection platform: Uses Apple HealthKit and Google Fit to collect user health data (heart rate, sleep time, exercise amount, etc.).

[0284] Sentiment analysis engine: Uses Amazon Rekognition and Microsoft Azure Face API to analyze facial and voice data collected from the camera and microphone to generate emotion data.

[0285] AI model: Using TensorFlow and PyTorch, it analyzes a user's health data, emotional data, and past drinking history to determine the appropriate amount of alcohol intake.

[0286] Specific operation of the system

[0287] 1. Menu Image Recognition:

[0288] A user takes a photo of a restaurant menu with their smartphone. The device then analyzes the menu image using the Google Cloud Vision API to extract drink types and details. For example, if the menu includes "beer," "wine," and "sangria," the device recognizes these and sends the information to the server.

[0289] Example prompt sentence:

[0290] "Perform image recognition on the menu and extract the types of drinks."

[0291] 2. Health Data Collection:

[0292] The device collects health data such as heart rate, sleep time, and exercise volume through Apple HealthKit or Google Fit, and this data is periodically sent to a server and used as the basis for determining appropriate alcohol intake.

[0293] Example prompt sentence:

[0294] "Collect health data and assess the user's current health status."

[0295] 3. Collecting Emotional Data:

[0296] The device uses a camera and microphone to capture the user's facial expressions and voice, and then analyzes the emotion data using Amazon Rekognition or Microsoft Azure Face API. For example, if the user is smiling, it will be evaluated as "happy" and the data will be sent to the server.

[0297] Example prompt sentence:

[0298] "Analyze emotions from the user's facial expressions and voice and evaluate the results."

[0299] 4. Determining Alcohol Intake:

[0300] The server uses an AI model based on past drinking history, health data, and emotional data to determine the appropriate amount of alcohol intake—for example, "two beers or one cocktail"—and sends this information to the device.

[0301] Example prompt sentence:

[0302] "Based on past data, determine the appropriate amount of alcohol you should consume today."

[0303] 5. Advice for choosing your next drink:

[0304] When the user inputs or selects the next drink they want to order into the terminal, the terminal displays a verdict of "OK" or "NG" based on the judgment result from the server. For example, if the user selects a cocktail, the terminal will display "NG" and suggest that "one beer is OK."

[0305] Example prompt sentence:

[0306] "Show me advice for the drink I'm about to order."

[0307] 6. Real-time health and emotional status monitoring and alerts:

[0308] The device uses Apple HealthKit and Google Fit to collect the user's health and emotional data in real time. If an abnormality is detected, for example, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0309] Example prompt sentence:

[0310] "Monitor your health and emotional state in real time and alert you if anything unusual is detected."

[0311] These steps help users enjoy alcohol in a healthy and appropriate emotional state, reducing health risks and emotional problems while also helping users maintain social relationships.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] Image recognition for menus

[0315] A user takes a photo of a restaurant menu with their smartphone. The input is the menu image taken by the user. The device analyzes the menu image using the Google Cloud Vision API. As a result of the analysis, the type of drink and its details are extracted. This information is output and sent to the server. Specifically, the device analyzes the image and recognizes items such as "beer," "wine," and "sangria."

[0316] Step 2:

[0317] Health data collection

[0318] The device acquires the user's health data using Apple HealthKit or Google Fit. Input includes the user's heart rate, sleep time, and exercise amount. The acquired data is pre-processed on the device and sent to the server as output. Specifically, the device collects data every 30 minutes and sends information such as a heart rate of 70 bpm, 7 hours of sleep, and 30 minutes of exercise per day to the server.

[0319] Step 3:

[0320] Collecting Emotional Data

[0321] The device uses the smartphone's camera and microphone to collect the user's facial expressions and voice. The inputs include the user's facial expression data and voice data. Using Amazon Rekognition or Microsoft Azure Face API, the device analyzes the emotional data, and the output emotional state (e.g., happiness, sadness, anger) is sent to the server. Specifically, if the user is smiling, it evaluates the user as "happy," and this data is sent to the server.

[0322] Step 4:

[0323] Determining alcohol intake

[0324] The server extracts past drinking history, health data, and emotional data from a database. This past data is included as input. The server analyzes the data using a generative AI model using TensorFlow and PyTorch, and the output determines the appropriate amount of alcohol intake. Specifically, the server determines that "two beers or one cocktail is appropriate" and sends this information to the device.

[0325] Step 5:

[0326] Advice for choosing your next drink

[0327] The user inputs or selects the next drink they wish to order into the terminal. The input is the drink information selected by the user. Based on the result of the judgment received from the server, the terminal displays a judgment of "OK" or "NG" as output. In concrete terms, if the user selects a cocktail, the terminal displays "NG" and suggests that "one beer is OK."

[0328] Step 6:

[0329] Real-time health and emotional status monitoring and alerts

[0330] The device continues to collect health and emotional data in real time. The input includes the user's current health and emotional data. If the device detects an abnormality based on the analyzed data, it displays a warning message as output. Specifically, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0331] (Application example 2)

[0332] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0333] Currently, drinking in restaurants and bars presents a challenge in that it is difficult for patrons to determine the appropriate amount of alcohol they should consume while keeping track of their health and emotional state. In particular, because excessive drinking increases health risks and fluctuations in emotional state can have a negative impact on social relationships, there is a lack of mechanisms to support appropriate drinking. This has led to a demand for systems that allow patrons to enjoy a safe and comfortable drinking experience.

[0334] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0335] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for acquiring the user's emotional data and assessing their current emotional state, means for comparing their past drinking history and physical condition data with their current health and emotional state to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the results of this assessment, and means for monitoring the user's health and emotional state in real time and issuing a warning if an abnormality is detected. This makes it possible to provide a safe and comfortable drinking experience while taking into consideration the user's health and emotional state comprehensively.

[0336] "Image recognition" is a technology that analyzes captured images and recognizes specific objects and characters.

[0337] "Health data" refers to collected and recorded information about an individual's health status, such as heart rate, sleep time, and amount of exercise.

[0338] "Emotion data" is data that indicates an emotional state such as joy, sadness, or anger, which is analyzed from the user's facial expressions and voice.

[0339] "Drinking history" refers to a record of the types and amounts of alcoholic beverages consumed in the past, as well as the dates and times of consumption.

[0340] "Physical condition data" is data that includes various information about the user's past and present health conditions.

[0341] "Alcohol intake" is a measure of the total amount of alcohol consumed within a specified period of time.

[0342] The "determination result" is judgment information about the safety of the drink provided by the server based on the user's health data, drinking history, and emotional data.

[0343] The "means for providing advice" is a function in which the system determines whether the next drink the user selects is appropriate or inappropriate, and makes suggestions, including alternatives, based on the results.

[0344] "Real-time monitoring" is a function that constantly monitors and instantly analyzes the user's health and emotional state.

[0345] The "means for issuing warnings" is a function that immediately sends a warning message when an abnormality occurs in the user's health or emotional state.

[0346] An embodiment of the present invention will now be described. This system provides appropriate advice based on the user's health condition, drinking status, and emotional state, and supports a safe and comfortable drinking experience.

[0347] First, the user takes a photo of a restaurant menu using a smartphone or smart glasses. The device then analyzes the menu image using image recognition technology to extract the drink type and details. This process uses image processing libraries such as OpenCV. The extracted information is then sent to a server and registered in a database.

[0348] Next, the device acquires the user's health data. This data includes information such as heart rate, sleep time, and exercise volume acquired from the smartwatch or smartphone. This data is collected periodically and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0349] In addition, the device collects emotional data from the user's facial expressions and voice. This step uses a camera and microphone, and the emotion engine analyzes the data. The resulting emotional state (happiness, sadness, anger, etc.) is sent to the server and stored in a database.

[0350] The server compares the collected past drinking history, physical condition data, and emotional data with the user's current health and emotional state to determine the appropriate amount of alcohol intake. Specifically, it makes a specific judgment, such as whether the user can safely consume up to two beers or one cocktail. This judgment result is sent from the server to the device.

[0351] When a user selects or inputs the next drink to order, the device will display "OK" or "NG" based on the judgment from the server, indicating whether the drink is appropriate or inappropriate. For example, if a user tries to order a cocktail, the device will display "NG," but an alternative suggestion such as "One beer is OK" will be displayed.

[0352] The device also monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[0353] For example, if a user tries to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain a proper drinking pace and maintain their health and social relationships.

[0354] Here are some examples of prompts for generative AI models:

[0355] "Please provide a concrete example of a system that determines the appropriate amount of alcohol intake based on the user's current health status (heart rate, sleep time, amount of exercise) and emotional state (happiness, sadness, anger), and compares it with the user's current health and emotional state. Also, please display drink suggestions."

[0356] The above is an embodiment of the present invention. This system allows users to enjoy a safe and comfortable drinking experience while maintaining an appropriate alcohol intake amount according to their health and emotional state.

[0357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0358] Step 1:

[0359] The user takes a photo of a restaurant menu using a smartphone or smart glasses.

[0360] Specific behavior:

[0361] A user takes a picture of a restaurant menu with a camera to obtain a menu image.

[0362] Input: Menu Image

[0363] Output: Photographed menu image data

[0364] Step 2:

[0365] The device uses image recognition technology to analyze the menu image captured and extract the drink type and details.

[0366] Specific behavior:

[0367] The device uses an image processing library such as OpenCV to analyze the image and recognize the menu items.

[0368] Input: Photographed menu image data

[0369] Output: Recognized drink type and details

[0370] Step 3:

[0371] The extracted information is sent to a server and registered in a database.

[0372] Specific behavior:

[0373] The drink information extracted from the device is sent to the server, which stores the information in a database.

[0374] Input: Recognized drink information

[0375] Output: Drink information stored in the database

[0376] Step 4:

[0377] The device acquires the user's health data.

[0378] Specific behavior:

[0379] The device collects health data such as heart rate, sleep time, and exercise volume from smartwatches and smartphones.

[0380] Input: Health data obtained from a smartwatch or smartphone

[0381] Output: Retrieved health data

[0382] Step 5:

[0383] The acquired health data is sent to a server and registered in a database.

[0384] Specific behavior:

[0385] The device sends the acquired health data to a server, which stores the data in a database.

[0386] Input: Acquired health data

[0387] Output: Health data stored in a database

[0388] Step 6:

[0389] The device collects emotional data from the user's facial expressions and voice.

[0390] Specific behavior:

[0391] The device uses a camera and microphone to capture the user's facial expressions and voice, which are then analyzed using an emotion engine.

[0392] Input: User's facial expressions and voice data

[0393] Output: Parsed emotion data

[0394] Step 7:

[0395] The analyzed emotional data is sent to a server and registered in a database.

[0396] Specific behavior:

[0397] The device sends the analyzed emotion data to the server, which stores the data in a database.

[0398] Input: Parsed emotion data

[0399] Output: Emotion data stored in a database

[0400] Step 8:

[0401] The server compares past drinking history, physical condition data, current health and emotional state, and determines the appropriate amount of alcohol intake.

[0402] Specific behavior:

[0403] The server retrieves past drinking history, physical condition data, current health and emotional state from the database and analyzes them using an algorithm.

[0404] Input: Drinking history, physical condition data, health status, emotional state stored in the database

[0405] Output: Determined appropriate alcohol intake

[0406] Step 9:

[0407] When the user enters or selects the next drink they want to order, the terminal will display an "OK" or "NG" result and alternatives.

[0408] Specific behavior:

[0409] The terminal receives the judgment result from the server and displays the status of "OK" or "NG" as well as alternatives for the next drink the user orders.

[0410] Input: Verification result from the server

[0411] Output: Display advice to the user when choosing a drink

[0412] Step 10:

[0413] The device monitors the user's health and emotional state in real time and issues an alert if it detects any abnormalities.

[0414] Specific behavior:

[0415] The device periodically collects and analyzes the user's health and emotional data, and displays a warning message in real time if any abnormalities are detected.

[0416] Input: User health and emotional data

[0417] Output: Warning message when an abnormality is detected

[0418] These are the specific processing steps of this system. Each step allows the user to enjoy an optimal drinking experience that is tailored to their own health and emotional state.

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0422] [Second embodiment]

[0423] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0424] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0430] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0435] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition and drinking habits will be described in detail.

[0436] 1. Image recognition for menus

[0437] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[0438] 2. Health Data Collection

[0439] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0440] 3. Determining alcohol intake

[0441] The server extracts past drinking history and physical condition data from a database. Based on this, it compares it with the user's current health condition and determines how much alcohol the user needs to consume and at what speed. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[0442] 4. Advice for choosing your next drink

[0443] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[0444] 5. Real-time health monitoring and alerts

[0445] The device continuously monitors the user's health condition in real time. Data is collected and analyzed periodically. If the user's health condition suddenly deteriorates, the device immediately issues a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0446] The system of the present invention provides support for users to enjoy drinking in a healthy manner and reduces the health risks associated with excessive drinking. As a specific example, if a user attempts to select a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and maintain their health.

[0447] The processing flow will be explained below.

[0448] Step 1:

[0449] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[0450] Step 2:

[0451] The device uses image recognition technology to analyze the menu image, and the extracted information, such as the name, type, and price of the drink, is stored in the device's memory.

[0452] Step 3:

[0453] The device sends the extracted information to the server, which then registers the information in a database and generates a drink list.

[0454] Step 4:

[0455] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. The data is collected periodically and stored on the device.

[0456] Step 5:

[0457] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[0458] Step 6:

[0459] The server extracts past drinking history and physical condition data from a database, and based on this, compares it with the current health condition to determine the appropriate amount of alcohol intake.

[0460] Step 7:

[0461] The server sends the result of the judgment to the terminal, which then stores the result.

[0462] Step 8:

[0463] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[0464] Step 9:

[0465] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[0466] Step 10:

[0467] The device monitors the user's health status in real time, with data collected periodically and analyzed to detect abnormalities.

[0468] Step 11:

[0469] If your health condition suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level, the device will display a message saying, "Your current heart rate has reached a dangerous level. Stop drinking alcohol and drink water."

[0470] The above are the specific processing steps of this system.

[0471] Example 1

[0472] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0473] In today's world, managing alcohol consumption is important for maintaining good health for many people. However, it is difficult for users to determine the appropriate amount of alcohol intake based on their own health condition and drinking history. Furthermore, there is a lack of methods to monitor health status in real time while drinking and respond immediately if an abnormality occurs. To address these issues, a system is needed that provides appropriate advice based on health status and drinking habits, and supports users in enjoying healthy drinking.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0475] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment results, and means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected. This allows the user to monitor their health condition in real time while maintaining an appropriate drinking pace according to their own health condition, and receive appropriate advice and warnings as needed.

[0476] A "user" is an individual who uses the system to manage their drinking habits and health.

[0477] "Eating and drinking establishments" refers to places where users eat and drink, including restaurants and bars.

[0478] A "menu" is a document or booklet listing the drinks and food served at a restaurant.

[0479] "Image recognition" refers to the technique of extracting information from images using computer vision techniques.

[0480] "Beverages" refers to alcoholic and non-alcoholic beverages served at restaurants.

[0481] "Health data" refers to data that indicates the user's health condition, such as heart rate, sleep time, and amount of exercise.

[0482] "Health status" refers to the user's current physical condition and includes parameters such as heart rate, blood pressure, and activity level.

[0483] "Drinking history" is a record of alcoholic beverages consumed by a user in the past.

[0484] "Physical condition data" is a general term for data related to the user's physical condition and health.

[0485] "Alcohol intake" refers to the amount of alcohol consumed by a user within a certain period of time.

[0486] The "judgment result" is an assessment of the appropriate alcohol intake tolerance calculated by the system based on the user's health data, drinking history, and current health condition.

[0487] "Advice" refers to recommendations or recommendations about drinking that the system provides to the user.

[0488] "Real-time monitoring" refers to a technology that continuously monitors a user's health status and updates the data instantly.

[0489] A "warning" is a message issued by the system to alert the user when the user's health condition reaches an abnormal state.

[0490] "Server" refers to a computer system for processing and storing data.

[0491] A "database" is a data structure or system used to organize and manage information.

[0492] "Decision-making" is the process by which a system draws a conclusion based on multiple data and conditions.

[0493] The present invention relates to a system that provides appropriate advice based on a user's health condition and drinking habits. This system provides support for users to maintain their health while enjoying alcohol.

[0494] First, the user takes a photo of a restaurant menu with their smartphone. The device analyzes the menu image using Google Cloud Vision API to extract the type of drink and details. The extracted information is sent to the server and registered in a database. This is how menu information is managed.

[0495] Next, the device acquires health data (heart rate, sleep time, exercise volume, etc.) from the user's smartwatch or smartphone. This health data is collected through software such as Apple Health or Google Fit. The collected data is sent to a server and stored in a database. This allows the user's health status to be monitored appropriately.

[0496] The server extracts past drinking history and health data from a database and compares it with the user's current health status. This determines how much alcohol the user can safely consume. For example, it makes a specific determination such as whether two beers or one cocktail is safe. The result of the determination is sent from the server to the device.

[0497] When a user inputs or selects the next drink they want to order, the terminal displays a "OK" or "NG" result based on the judgment result from the server. Even if the drink is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the terminal will display "NG," but suggest "One beer is OK."

[0498] Furthermore, the device continuously monitors the user's health condition in real time, collecting and analyzing data periodically. If the user's health condition suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0499] For example, if a user tries to choose a cocktail at a restaurant, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain an appropriate drinking pace and maintain their health.

[0500] An example of a prompt to be input to a generative AI model is, "Please describe a system that provides appropriate advice based on the user's health status and drinking habits. Specifically, please provide details on image recognition of menus, collection of health data, determination of alcohol intake, advice on next drink selection, and real-time monitoring and alerting of health status."

[0501] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0502] Step 1:

[0503] A user takes a photo of a restaurant menu with their smartphone, and the image becomes the input data.

[0504] Step 2:

[0505] The device analyzes the menu image captured using the Google Cloud Vision API. Using the menu image as input, an image recognition algorithm extracts the drink type and details. The output data is the drink type and its detailed information.

[0506] Step 3:

[0507] The device sends the extracted drink information to the server. The input data is the type of drink and detailed information, which is converted into a format that is sent to the server as output data. This data is also registered in the database as menu information.

[0508] Step 4:

[0509] The device acquires health data (heart rate, sleep time, exercise amount, etc.) from the user's smartwatch or smartphone. The input data is various sensor data from the health device, and the output data is organized as health data.

[0510] Step 5:

[0511] The health data collected by the device is sent to a server. The input data is organized health data, and the output data is data converted into a format suitable for sending to the server. This saves the health data in the server's database.

[0512] Step 6:

[0513] The server extracts past drinking history and health data from the database, with input data being the user's drinking history and health data stored in the database and output data being the extracted history data.

[0514] Step 7:

[0515] The server compares the current health status with past data. The input data is the past drinking history and health data, and the current health data. The data matching algorithm determines the appropriate alcohol intake. The output data is the determined acceptable alcohol intake limit.

[0516] Step 8:

[0517] The server sends the result of the determination to the terminal. The input data is the determined alcohol intake tolerance, and the output data is data converted into a format that can be sent to the terminal.

[0518] Step 9:

[0519] The user inputs or selects the next drink to order into the terminal, and the input data is information about the drink selected or input by the user.

[0520] Step 10:

[0521] The device displays a verdict of "OK" or "NG" based on the judgment result received from the server. The input data is the information about the drink selected by the user and the judgment result from the server, and the output data is a verdict of "OK" or "NG." It also suggests alternative drinks.

[0522] Step 11:

[0523] The device monitors the user's health condition in real time. The input data is real-time data from various sensor devices, and the output data is an assessment of the user's current health condition.

[0524] Step 12:

[0525] If the device detects an abnormality, it will issue a warning to the user. The input data is health data collected in real time, and if an abnormality is detected through analysis, a warning message is displayed as output data.

[0526] (Application example 1)

[0527] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] Currently, health management when drinking alcohol at restaurants is left up to the user, and inappropriate alcohol consumption can pose health risks. To solve this problem, a system is needed that can grasp the user's health status and drinking habits in real time and provide appropriate advice. In addition, a system is needed that can provide specific advice when the user selects their next drink and issue warnings according to changes in their health status.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0530] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting drink types and details, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment result, means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected, and means for displaying the advice and warning in real time using smart glasses. This enables users to achieve appropriate alcohol intake while reducing health risks at restaurants.

[0531] "User" refers to an individual or corporation that uses this system.

[0532] "Restaurant" means a commercial establishment that serves beverages and meals.

[0533] A "menu" is a list of the types of food and drink served by a restaurant.

[0534] "Image recognition" refers to the technology of extracting specific information from captured images.

[0535] "Types of drinks" refers to the various drinks listed on a restaurant menu.

[0536] "Health data" refers to information related to a user's health, such as their heart rate, sleep time, and amount of exercise.

[0537] "Health status" refers to the current physical condition and degree of health of the user, as assessed based on the user's health data.

[0538] "Drinking history" refers to a record of what types and amounts of alcoholic beverages a user has consumed in the past.

[0539] "Alcohol intake" refers to the total amount of alcohol consumed by a user within a certain period of time.

[0540] "Real-time monitoring" refers to the act of continuously monitoring a user's health status.

[0541] "Warning" refers to a warning that is given when the user's health condition reaches a dangerous state.

[0542] "Smart glasses" refers to a type of wearable device that can present information visually.

[0543] "Server" refers to the device that serves as the core of this system and analyzes, stores, and distributes data.

[0544] As an embodiment of the present invention, a system for enabling users to consume alcohol appropriately while reducing health risks at restaurants will be described in detail.

[0545] Hardware and Software Overview

[0546] The server is the core of this system and is a device that analyzes, stores, and distributes data. This system uses smart glasses, smartphones, and wearable devices (e.g., smartwatches). Smart glasses are a type of wearable device that can present information visually, and Google Glass can be used. Image recognition APIs such as Google Cloud Vision API are used for image recognition. Health data is collected using health data analysis platforms such as Apple HealthKit and Google Fit. Firebase can also be used as a real-time communication server.

[0547] Program processing explanation

[0548] The server receives restaurant menu images taken by the user using smart glasses or a smartphone and analyzes them using an image recognition API. This extracts drink types and details from the menu items. The server then obtains the user's health data from the smartwatch or smartphone and evaluates their current health status. This evaluation uses data such as heart rate, sleep time, and exercise volume.

[0549] The server compares the user's current health condition with their past drinking history and physical condition data, and determines the appropriate amount of alcohol intake based on this. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. Based on this judgment, it provides advice on the user's next drink. The smart glasses display shows a "OK" or "NG" result, and also suggests alternatives.

[0550] The smart glasses and smartphone also continuously monitor the user's health status in real time. If the user's heart rate reaches a dangerous level, a warning message will be displayed immediately. The specific warning message will be displayed, such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0551] Examples and prompts

[0552] As a concrete example, consider the case where a user uses smart glasses to select a cocktail from a restaurant menu. Because the user's heart rate is already high, the system judges the selection as "NG" and the smart glasses display shows "One beer is OK." In this case, the user can order a beer instead of a cocktail to reduce the health risk.

[0553] Example prompt sentence:

[0554] When a user browses a menu and chooses a cocktail at a restaurant, they provide the following information:

[0555] 1. If a cocktail is "NG," please briefly explain why it is "NG."

[0556] 2. Suggest a specific drink name as an alternative.

[0557] 3. Based on the current user health data (high heart rate), advise what action to take.

[0558] example:

[0559] Cocktails: No - High heart rate, try a lower alcohol content drink. Alternative: 1 beer

[0560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0561] Step 1:

[0562] A user takes a photo of a restaurant menu using smart glasses or a smartphone.

[0563] Input: Restaurant menu image

[0564] Output: Menu image taken

[0565] The terminal acquires the menu image photographed by the user and temporarily stores the image.

[0566] Step 2:

[0567] The terminal transmits the captured menu image to the server.

[0568] Input: Menu image taken

[0569] Output: Image data sent to the server

[0570] The device uploads the image data to a specified server and sends a request to an API for image recognition.

[0571] Step 3:

[0572] The server uses an image recognition API to analyze the menu images and extract drink types and details.

[0573] Input: Image data sent to the server

[0574] Output: List of drink types and details

[0575] The server uses the Google Cloud Vision API and other tools to analyze the text in the images and categorize the menu items by drink type.

[0576] Step 4:

[0577] The server obtains the user's health data from the health data analysis platform.

[0578] Input: User identification information

[0579] Output: User's health data (heart rate, sleep time, exercise amount, etc.)

[0580] The server retrieves health data through APIs from platforms such as HealthKit and Google Fit and uses it to assess the user's current health status.

[0581] Step 5:

[0582] The server compares the acquired health data with past drinking history and physical condition data to determine the appropriate amount of alcohol intake.

[0583] Input: User's health data, past drinking history, and physical condition data

[0584] Output: Appropriate alcohol intake and judgment result

[0585] The server compares past drinking history and current health data from a database to calculate alcohol intake and determine drinking pace.

[0586] Step 6:

[0587] The server generates advice for the next drink selection based on the judgment result and transmits it to the terminal.

[0588] Input: Judgment result

[0589] Output: Advice ("OK" or "NG")

[0590] The server uses the generative AI model to generate advice appropriate for the user and outputs a judgment result such as "OK" or "NG" when the next drink is selected, or provides specific advice based on the prompt text.

[0591] Step 7:

[0592] The terminal receives the determination result from the server and displays it on the display of the smart glasses.

[0593] Input: Advice content

[0594] Output: Advice displayed on the smart glasses display

[0595] The terminal displays the received advice in real time on the display of the smart glasses so that the user can check it.

[0596] Step 8:

[0597] When the user orders the next drink, the terminal will prompt the user to enter or select the name of the drink and display the decision.

[0598] Input: User-selected drink name

[0599] Output: Drink judgment result ("OK" or "NG") and alternatives

[0600] The device sends the name of the drink selected by the user to the server and displays the result on the display. If the drink is judged to be "NG," an alternative option will be presented.

[0601] Step 9:

[0602] The server continuously monitors the user's health status in real time and immediately issues an alert if it detects any abnormalities.

[0603] Input: User's health data (real-time)

[0604] Output: Warning message

[0605] The server continuously monitors the user's health data, and if, for example, the heart rate reaches a dangerous level, it sends a warning message to the device, such as, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[0606] Through these steps, users can enjoy alcohol in a healthy manner at restaurants. This system provides real-time advice and warnings through the collaboration of smart glasses and a server, reducing health risks for users.

[0607] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0608] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition, drinking status, and emotional state will be described in detail.

[0609] 1. Image recognition for menus

[0610] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[0611] 2. Health Data Collection

[0612] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0613] 3. Collecting Emotional Data

[0614] The device collects emotional data from the user's facial expressions and voice. This data is collected using a camera and microphone, and the emotion engine analyzes it. The analysis results are used to evaluate the user's emotional state (e.g., joy, sadness, anger, etc.), and the results are sent to the server and stored in a database.

[0615] 4. Determining alcohol intake

[0616] The server extracts past drinking history, physical condition data, and emotional data from a database. Based on this, it compares the user's current health and emotional state and determines the appropriate amount of alcohol intake. For example, it makes a specific judgment such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[0617] 5. Advice for choosing your next drink

[0618] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[0619] 6. Real-time health and emotional status monitoring and alerts

[0620] The device continuously monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[0621] The system of the present invention supports users in enjoying alcohol in a healthy and appropriate emotional state, reducing the health risks and emotional downfalls associated with excessive drinking. A specific example is a system in which, if a user attempts to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and emotional state, thereby maintaining their health and social relationships.

[0622] The processing flow will be explained below.

[0623] Step 1:

[0624] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[0625] Step 2:

[0626] The device uses image recognition technology to analyze the menu image, extracting detailed information such as the drink name, type, and price, and storing this information in memory.

[0627] Step 3:

[0628] The extracted drink information is sent to the server, which then registers the received information in a database and generates a drink list.

[0629] Step 4:

[0630] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. This data is collected at regular intervals and stored on the device.

[0631] Step 5:

[0632] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[0633] Step 6:

[0634] The device uses a camera and microphone to collect the user's emotional data (facial expressions and voice), and the emotion engine analyzes this data to evaluate the user's current emotional state.

[0635] Step 7:

[0636] The evaluation results of the emotional state are sent to a server and stored in a database.

[0637] Step 8:

[0638] The server extracts past drinking history, physical condition data, and emotional data from a database, and compares it with the current health and emotional state to determine the appropriate amount of alcohol intake.

[0639] Step 9:

[0640] The server sends the result of the judgment to the terminal, which then stores the result.

[0641] Step 10:

[0642] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[0643] Step 11:

[0644] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[0645] Step 12:

[0646] The device monitors the user's health and emotional state in real time, with data periodically captured and analyzed to detect anomalies.

[0647] Step 13:

[0648] If your health or emotional state suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level or you become extremely irritable, a warning message will appear saying, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[0649] The above are the specific processing steps of this system.

[0650] Example 2

[0651] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] Currently, many people consume alcohol at restaurants without fully considering their health and emotional state, which can result in health risks and emotional problems. Furthermore, there are few systems that provide real-time information to help users determine the appropriate amount of alcohol they should consume. This makes it difficult to properly manage users' health and the effects of drinking.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0654] In this invention, the server includes means for performing image recognition on restaurant menus photographed by the user and extracting drink types and details, means for acquiring the user's health data and evaluating their current health condition, and means for analyzing the user's facial expressions and voice to collect emotional data. This makes it possible to monitor the user's health and emotional state in real time, determine the appropriate amount of alcohol intake, and provide advice on selecting the next drink.

[0655] "User" refers to a person who uses the system.

[0656] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or smartwatch.

[0657] A "menu image" refers to an image of a menu taken by a user at a restaurant.

[0658] "Image recognition technology" refers to technology for extracting specific information from captured images.

[0659] "Drink type" refers to the category of drinks listed on the menu, for example, beer, wine, cocktails, etc.

[0660] "Health data" refers to physiological data such as the user's heart rate, sleep time, and amount of exercise.

[0661] "Emotion data" refers to the emotional state of the user analyzed from their facial expressions and voice, such as joy, sadness, anger, etc.

[0662] "Drinking history" refers to a record of alcohol consumed by a user in the past.

[0663] "Alcohol intake" refers to the amount of alcohol that is appropriate for a user to drink.

[0664] "Advice" refers to appropriate instructions or suggestions regarding drinking that the system provides to the user.

[0665] "Monitoring" refers to the continuous observation of a user's health and emotional state.

[0666] "Warning" refers to a warning message sent by the system when an abnormality is detected in the user's condition.

[0667] A "database" refers to a structured collection of information for storing and managing data within a system.

[0668] "Analysis" refers to the process of evaluating the user's condition based on collected data.

[0669] "Real-time" refers to near-simultaneous processing and responses.

[0670] The present invention is a system that provides appropriate advice based on a user's health condition, drinking habits, and emotional state. This system operates through the following specific steps.

[0671] Hardware and Software Configuration

[0672] Device: A smartphone, tablet, smartwatch, etc. used by a user. Devices are equipped with cameras and microphones, which are used to collect data.

[0673] Server: A computer system that analyzes and manages data. The server is connected to a database that stores users' health and emotional data.

[0674] Database: A data storage space located within the server. It stores user health data, emotional data, drinking history, etc.

[0675] Software and Technology

[0676] Image recognition technology: Google Cloud Vision API and OpenCV are used as image recognition technologies to analyze menu images captured by the device's camera.

[0677] Health data collection platform: Uses Apple HealthKit and Google Fit to collect user health data (heart rate, sleep time, exercise amount, etc.).

[0678] Sentiment analysis engine: Uses Amazon Rekognition and Microsoft Azure Face API to analyze facial and voice data collected from the camera and microphone to generate emotion data.

[0679] AI model: Using TensorFlow and PyTorch, it analyzes a user's health data, emotional data, and past drinking history to determine the appropriate amount of alcohol intake.

[0680] Specific operation of the system

[0681] 1. Menu Image Recognition:

[0682] A user takes a photo of a restaurant menu with their smartphone. The device then analyzes the menu image using the Google Cloud Vision API to extract drink types and details. For example, if the menu includes "beer," "wine," and "sangria," the device recognizes these and sends the information to the server.

[0683] Example prompt sentence:

[0684] "Perform image recognition on the menu and extract the types of drinks."

[0685] 2. Health Data Collection:

[0686] The device collects health data such as heart rate, sleep time, and exercise volume through Apple HealthKit or Google Fit, and this data is periodically sent to a server and used as the basis for determining appropriate alcohol intake.

[0687] Example prompt sentence:

[0688] "Collect health data and assess the user's current health status."

[0689] 3. Collecting Emotional Data:

[0690] The device uses a camera and microphone to capture the user's facial expressions and voice, and then analyzes the emotion data using Amazon Rekognition or Microsoft Azure Face API. For example, if the user is smiling, it will be evaluated as "happy" and the data will be sent to the server.

[0691] Example prompt sentence:

[0692] "Analyze emotions from the user's facial expressions and voice and evaluate the results."

[0693] 4. Determining Alcohol Intake:

[0694] The server uses an AI model based on past drinking history, health data, and emotional data to determine the appropriate amount of alcohol intake—for example, "two beers or one cocktail"—and sends this information to the device.

[0695] Example prompt sentence:

[0696] "Based on past data, determine the appropriate amount of alcohol you should consume today."

[0697] 5. Advice for choosing your next drink:

[0698] When the user inputs or selects the next drink they want to order into the terminal, the terminal displays a verdict of "OK" or "NG" based on the judgment result from the server. For example, if the user selects a cocktail, the terminal will display "NG" and suggest that "one beer is OK."

[0699] Example prompt sentence:

[0700] "Show me advice for the drink I'm about to order."

[0701] 6. Real-time health and emotional status monitoring and alerts:

[0702] The device uses Apple HealthKit and Google Fit to collect the user's health and emotional data in real time. If an abnormality is detected, for example, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0703] Example prompt sentence:

[0704] "Monitor your health and emotional state in real time and alert you if anything unusual is detected."

[0705] These steps help users enjoy alcohol in a healthy and appropriate emotional state, reducing health risks and emotional problems while also helping users maintain social relationships.

[0706] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] Image recognition for menus

[0709] A user takes a photo of a restaurant menu with their smartphone. The input is the menu image taken by the user. The device analyzes the menu image using the Google Cloud Vision API. As a result of the analysis, the type of drink and its details are extracted. This information is output and sent to the server. Specifically, the device analyzes the image and recognizes items such as "beer," "wine," and "sangria."

[0710] Step 2:

[0711] Health data collection

[0712] The device acquires the user's health data using Apple HealthKit or Google Fit. Input includes the user's heart rate, sleep time, and exercise amount. The acquired data is pre-processed on the device and sent to the server as output. Specifically, the device collects data every 30 minutes and sends information such as a heart rate of 70 bpm, 7 hours of sleep, and 30 minutes of exercise per day to the server.

[0713] Step 3:

[0714] Collecting Emotional Data

[0715] The device uses the smartphone's camera and microphone to collect the user's facial expressions and voice. The inputs include the user's facial expression data and voice data. Using Amazon Rekognition or Microsoft Azure Face API, the device analyzes the emotional data, and the output emotional state (e.g., happiness, sadness, anger) is sent to the server. Specifically, if the user is smiling, it evaluates the user as "happy," and this data is sent to the server.

[0716] Step 4:

[0717] Determining alcohol intake

[0718] The server extracts past drinking history, health data, and emotional data from a database. This past data is included as input. The server analyzes the data using a generative AI model using TensorFlow and PyTorch, and the output determines the appropriate amount of alcohol intake. Specifically, the server determines that "two beers or one cocktail is appropriate" and sends this information to the device.

[0719] Step 5:

[0720] Advice for choosing your next drink

[0721] The user inputs or selects the next drink they wish to order into the terminal. The input is the drink information selected by the user. Based on the result of the judgment received from the server, the terminal displays a judgment of "OK" or "NG" as output. In concrete terms, if the user selects a cocktail, the terminal displays "NG" and suggests that "one beer is OK."

[0722] Step 6:

[0723] Real-time health and emotional status monitoring and alerts

[0724] The device continues to collect health and emotional data in real time. The input includes the user's current health and emotional data. If the device detects an abnormality based on the analyzed data, it displays a warning message as output. Specifically, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0725] (Application example 2)

[0726] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] Currently, drinking in restaurants and bars presents a challenge in that it is difficult for patrons to determine the appropriate amount of alcohol they should consume while keeping track of their health and emotional state. In particular, because excessive drinking increases health risks and fluctuations in emotional state can have a negative impact on social relationships, there is a lack of mechanisms to support appropriate drinking. This has led to a demand for systems that allow patrons to enjoy a safe and comfortable drinking experience.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0729] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for acquiring the user's emotional data and assessing their current emotional state, means for comparing their past drinking history and physical condition data with their current health and emotional state to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the results of this assessment, and means for monitoring the user's health and emotional state in real time and issuing a warning if an abnormality is detected. This makes it possible to provide a safe and comfortable drinking experience while taking into consideration the user's health and emotional state comprehensively.

[0730] "Image recognition" is a technology that analyzes captured images and recognizes specific objects and characters.

[0731] "Health data" refers to collected and recorded information about an individual's health status, such as heart rate, sleep time, and amount of exercise.

[0732] "Emotion data" is data that indicates an emotional state such as joy, sadness, or anger, which is analyzed from the user's facial expressions and voice.

[0733] "Drinking history" refers to a record of the types and amounts of alcoholic beverages consumed in the past, as well as the dates and times of consumption.

[0734] "Physical condition data" is data that includes various information about the user's past and present health conditions.

[0735] "Alcohol intake" is a measure of the total amount of alcohol consumed within a specified period of time.

[0736] The "determination result" is judgment information about the safety of the drink provided by the server based on the user's health data, drinking history, and emotional data.

[0737] The "means for providing advice" is a function in which the system determines whether the next drink the user selects is appropriate or inappropriate, and makes suggestions, including alternatives, based on the results.

[0738] "Real-time monitoring" is a function that constantly monitors and instantly analyzes the user's health and emotional state.

[0739] The "means for issuing warnings" is a function that immediately sends a warning message when an abnormality occurs in the user's health or emotional state.

[0740] An embodiment of the present invention will now be described. This system provides appropriate advice based on the user's health condition, drinking status, and emotional state, and supports a safe and comfortable drinking experience.

[0741] First, the user takes a photo of a restaurant menu using a smartphone or smart glasses. The device then analyzes the menu image using image recognition technology to extract the drink type and details. This process uses image processing libraries such as OpenCV. The extracted information is then sent to a server and registered in a database.

[0742] Next, the device acquires the user's health data. This data includes information such as heart rate, sleep time, and exercise volume acquired from the smartwatch or smartphone. This data is collected periodically and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0743] In addition, the device collects emotional data from the user's facial expressions and voice. This step uses a camera and microphone, and the emotion engine analyzes the data. The resulting emotional state (happiness, sadness, anger, etc.) is sent to the server and stored in a database.

[0744] The server compares the collected past drinking history, physical condition data, and emotional data with the user's current health and emotional state to determine the appropriate amount of alcohol intake. Specifically, it makes a specific judgment, such as whether the user can safely consume up to two beers or one cocktail. This judgment result is sent from the server to the device.

[0745] When a user selects or inputs the next drink to order, the device will display "OK" or "NG" based on the judgment from the server, indicating whether the drink is appropriate or inappropriate. For example, if a user tries to order a cocktail, the device will display "NG," but an alternative suggestion such as "One beer is OK" will be displayed.

[0746] The device also monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[0747] For example, if a user tries to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain a proper drinking pace and maintain their health and social relationships.

[0748] Here are some examples of prompts for generative AI models:

[0749] "Please provide a concrete example of a system that determines the appropriate amount of alcohol intake based on the user's current health status (heart rate, sleep time, amount of exercise) and emotional state (happiness, sadness, anger), and compares it with the user's current health and emotional state. Also, please display drink suggestions."

[0750] The above is an embodiment of the present invention. This system allows users to enjoy a safe and comfortable drinking experience while maintaining an appropriate alcohol intake amount according to their health and emotional state.

[0751] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0752] Step 1:

[0753] The user takes a photo of a restaurant menu using a smartphone or smart glasses.

[0754] Specific behavior:

[0755] A user takes a picture of a restaurant menu with a camera to obtain a menu image.

[0756] Input: Menu Image

[0757] Output: Photographed menu image data

[0758] Step 2:

[0759] The device uses image recognition technology to analyze the menu image captured and extract the drink type and details.

[0760] Specific behavior:

[0761] The device uses an image processing library such as OpenCV to analyze the image and recognize the menu items.

[0762] Input: Photographed menu image data

[0763] Output: Recognized drink type and details

[0764] Step 3:

[0765] The extracted information is sent to a server and registered in a database.

[0766] Specific behavior:

[0767] The drink information extracted from the device is sent to the server, which stores the information in a database.

[0768] Input: Recognized drink information

[0769] Output: Drink information stored in the database

[0770] Step 4:

[0771] The device acquires the user's health data.

[0772] Specific behavior:

[0773] The device collects health data such as heart rate, sleep time, and exercise volume from smartwatches and smartphones.

[0774] Input: Health data obtained from a smartwatch or smartphone

[0775] Output: Retrieved health data

[0776] Step 5:

[0777] The acquired health data is sent to a server and registered in a database.

[0778] Specific behavior:

[0779] The device sends the acquired health data to a server, which stores the data in a database.

[0780] Input: Acquired health data

[0781] Output: Health data stored in a database

[0782] Step 6:

[0783] The device collects emotional data from the user's facial expressions and voice.

[0784] Specific behavior:

[0785] The device uses a camera and microphone to capture the user's facial expressions and voice, which are then analyzed using an emotion engine.

[0786] Input: User's facial expressions and voice data

[0787] Output: Parsed emotion data

[0788] Step 7:

[0789] The analyzed emotional data is sent to a server and registered in a database.

[0790] Specific behavior:

[0791] The device sends the analyzed emotion data to the server, which stores the data in a database.

[0792] Input: Parsed emotion data

[0793] Output: Emotion data stored in a database

[0794] Step 8:

[0795] The server compares past drinking history, physical condition data, current health and emotional state, and determines the appropriate amount of alcohol intake.

[0796] Specific behavior:

[0797] The server retrieves past drinking history, physical condition data, current health and emotional state from the database and analyzes them using an algorithm.

[0798] Input: Drinking history, physical condition data, health status, emotional state stored in the database

[0799] Output: Determined appropriate alcohol intake

[0800] Step 9:

[0801] When the user enters or selects the next drink they want to order, the terminal will display an "OK" or "NG" result and alternatives.

[0802] Specific behavior:

[0803] The terminal receives the judgment result from the server and displays the status of "OK" or "NG" as well as alternatives for the next drink the user orders.

[0804] Input: Verification result from the server

[0805] Output: Display advice to the user when choosing a drink

[0806] Step 10:

[0807] The device monitors the user's health and emotional state in real time and issues an alert if it detects any abnormalities.

[0808] Specific behavior:

[0809] The device periodically collects and analyzes the user's health and emotional data, and displays a warning message in real time if any abnormalities are detected.

[0810] Input: User health and emotional data

[0811] Output: Warning message when an abnormality is detected

[0812] These are the specific processing steps of this system. Each step allows the user to enjoy an optimal drinking experience that is tailored to their own health and emotional state.

[0813] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0814] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0815] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0816] [Third embodiment]

[0817] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0818] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0819] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0820] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0821] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0822] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0823] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0824] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0825] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0826] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0827] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0828] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0829] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition and drinking habits will be described in detail.

[0830] 1. Image recognition for menus

[0831] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[0832] 2. Health Data Collection

[0833] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[0834] 3. Determining alcohol intake

[0835] The server extracts past drinking history and physical condition data from a database. Based on this, it compares it with the user's current health condition and determines how much alcohol the user needs to consume and at what speed. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[0836] 4. Advice for choosing your next drink

[0837] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[0838] 5. Real-time health monitoring and alerts

[0839] The device continuously monitors the user's health condition in real time. Data is collected and analyzed periodically. If the user's health condition suddenly deteriorates, the device immediately issues a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0840] The system of the present invention provides support for users to enjoy drinking in a healthy manner and reduces the health risks associated with excessive drinking. As a specific example, if a user attempts to select a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and maintain their health.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[0844] Step 2:

[0845] The device uses image recognition technology to analyze the menu image, and the extracted information, such as the name, type, and price of the drink, is stored in the device's memory.

[0846] Step 3:

[0847] The device sends the extracted information to the server, which then registers the information in a database and generates a drink list.

[0848] Step 4:

[0849] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. The data is collected periodically and stored on the device.

[0850] Step 5:

[0851] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[0852] Step 6:

[0853] The server extracts past drinking history and physical condition data from a database, and based on this, compares it with the current health condition to determine the appropriate amount of alcohol intake.

[0854] Step 7:

[0855] The server sends the result of the judgment to the terminal, which then stores the result.

[0856] Step 8:

[0857] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[0858] Step 9:

[0859] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[0860] Step 10:

[0861] The device monitors the user's health status in real time, with data collected periodically and analyzed to detect abnormalities.

[0862] Step 11:

[0863] If your health condition suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level, the device will display a message saying, "Your current heart rate has reached a dangerous level. Stop drinking alcohol and drink water."

[0864] The above are the specific processing steps of this system.

[0865] Example 1

[0866] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0867] In today's world, managing alcohol consumption is important for maintaining good health for many people. However, it is difficult for users to determine the appropriate amount of alcohol intake based on their own health condition and drinking history. Furthermore, there is a lack of methods to monitor health status in real time while drinking and respond immediately if an abnormality occurs. To address these issues, a system is needed that provides appropriate advice based on health status and drinking habits, and supports users in enjoying healthy drinking.

[0868] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0869] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment results, and means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected. This allows the user to monitor their health condition in real time while maintaining an appropriate drinking pace according to their own health condition, and receive appropriate advice and warnings as needed.

[0870] A "user" is an individual who uses the system to manage their drinking habits and health.

[0871] "Eating and drinking establishments" refers to places where users eat and drink, including restaurants and bars.

[0872] A "menu" is a document or booklet listing the drinks and food served at a restaurant.

[0873] "Image recognition" refers to the technique of extracting information from images using computer vision techniques.

[0874] "Beverages" refers to alcoholic and non-alcoholic beverages served at restaurants.

[0875] "Health data" refers to data that indicates the user's health condition, such as heart rate, sleep time, and amount of exercise.

[0876] "Health status" refers to the user's current physical condition and includes parameters such as heart rate, blood pressure, and activity level.

[0877] "Drinking history" is a record of alcoholic beverages consumed by a user in the past.

[0878] "Physical condition data" is a general term for data related to the user's physical condition and health.

[0879] "Alcohol intake" refers to the amount of alcohol consumed by a user within a certain period of time.

[0880] The "judgment result" is an assessment of the appropriate alcohol intake tolerance calculated by the system based on the user's health data, drinking history, and current health condition.

[0881] "Advice" refers to recommendations or recommendations about drinking that the system provides to the user.

[0882] "Real-time monitoring" refers to a technology that continuously monitors a user's health status and updates the data instantly.

[0883] A "warning" is a message issued by the system to alert the user when the user's health condition reaches an abnormal state.

[0884] "Server" refers to a computer system for processing and storing data.

[0885] A "database" is a data structure or system used to organize and manage information.

[0886] "Decision-making" is the process by which a system draws a conclusion based on multiple data and conditions.

[0887] The present invention relates to a system that provides appropriate advice based on a user's health condition and drinking habits. This system provides support for users to maintain their health while enjoying alcohol.

[0888] First, the user takes a photo of a restaurant menu with their smartphone. The device analyzes the menu image using Google Cloud Vision API to extract the type of drink and details. The extracted information is sent to the server and registered in a database. This is how menu information is managed.

[0889] Next, the device acquires health data (heart rate, sleep time, exercise volume, etc.) from the user's smartwatch or smartphone. This health data is collected through software such as Apple Health or Google Fit. The collected data is sent to a server and stored in a database. This allows the user's health status to be monitored appropriately.

[0890] The server extracts past drinking history and health data from a database and compares it with the user's current health status. This determines how much alcohol the user can safely consume. For example, it makes a specific determination such as whether two beers or one cocktail is safe. The result of the determination is sent from the server to the device.

[0891] When a user inputs or selects the next drink they want to order, the terminal displays a "OK" or "NG" result based on the judgment result from the server. Even if the drink is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the terminal will display "NG," but suggest "One beer is OK."

[0892] Furthermore, the device continuously monitors the user's health condition in real time, collecting and analyzing data periodically. If the user's health condition suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0893] For example, if a user tries to choose a cocktail at a restaurant, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain an appropriate drinking pace and maintain their health.

[0894] An example of a prompt to be input to a generative AI model is, "Please describe a system that provides appropriate advice based on the user's health status and drinking habits. Specifically, please provide details on image recognition of menus, collection of health data, determination of alcohol intake, advice on next drink selection, and real-time monitoring and alerting of health status."

[0895] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0896] Step 1:

[0897] A user takes a photo of a restaurant menu with their smartphone, and the image becomes the input data.

[0898] Step 2:

[0899] The device analyzes the menu image captured using the Google Cloud Vision API. Using the menu image as input, an image recognition algorithm extracts the drink type and details. The output data is the drink type and its detailed information.

[0900] Step 3:

[0901] The device sends the extracted drink information to the server. The input data is the type of drink and detailed information, which is converted into a format that is sent to the server as output data. This data is also registered in the database as menu information.

[0902] Step 4:

[0903] The device acquires health data (heart rate, sleep time, exercise amount, etc.) from the user's smartwatch or smartphone. The input data is various sensor data from the health device, and the output data is organized as health data.

[0904] Step 5:

[0905] The health data collected by the device is sent to a server. The input data is organized health data, and the output data is data converted into a format suitable for sending to the server. This saves the health data in the server's database.

[0906] Step 6:

[0907] The server extracts past drinking history and health data from the database, with input data being the user's drinking history and health data stored in the database and output data being the extracted history data.

[0908] Step 7:

[0909] The server compares the current health status with past data. The input data is the past drinking history and health data, and the current health data. The data matching algorithm determines the appropriate alcohol intake. The output data is the determined acceptable alcohol intake limit.

[0910] Step 8:

[0911] The server sends the result of the determination to the terminal. The input data is the determined alcohol intake tolerance, and the output data is data converted into a format that can be sent to the terminal.

[0912] Step 9:

[0913] The user inputs or selects the next drink to order into the terminal, and the input data is information about the drink selected or input by the user.

[0914] Step 10:

[0915] The device displays a verdict of "OK" or "NG" based on the judgment result received from the server. The input data is the information about the drink selected by the user and the judgment result from the server, and the output data is a verdict of "OK" or "NG." It also suggests alternative drinks.

[0916] Step 11:

[0917] The device monitors the user's health condition in real time. The input data is real-time data from various sensor devices, and the output data is an assessment of the user's current health condition.

[0918] Step 12:

[0919] If the device detects an abnormality, it will issue a warning to the user. The input data is health data collected in real time, and if an abnormality is detected through analysis, a warning message is displayed as output data.

[0920] (Application example 1)

[0921] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0922] Currently, health management when drinking alcohol at restaurants is left up to the user, and inappropriate alcohol consumption can pose health risks. To solve this problem, a system is needed that can grasp the user's health status and drinking habits in real time and provide appropriate advice. In addition, a system is needed that can provide specific advice when the user selects their next drink and issue warnings according to changes in their health status.

[0923] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0924] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting drink types and details, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment result, means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected, and means for displaying the advice and warning in real time using smart glasses. This enables users to achieve appropriate alcohol intake while reducing health risks at restaurants.

[0925] "User" refers to an individual or corporation that uses this system.

[0926] "Restaurant" means a commercial establishment that serves beverages and meals.

[0927] A "menu" is a list of the types of food and drink served by a restaurant.

[0928] "Image recognition" refers to the technology of extracting specific information from captured images.

[0929] "Types of drinks" refers to the various drinks listed on a restaurant menu.

[0930] "Health data" refers to information related to a user's health, such as their heart rate, sleep time, and amount of exercise.

[0931] "Health status" refers to the current physical condition and degree of health of the user, as assessed based on the user's health data.

[0932] "Drinking history" refers to a record of what types and amounts of alcoholic beverages a user has consumed in the past.

[0933] "Alcohol intake" refers to the total amount of alcohol consumed by a user within a certain period of time.

[0934] "Real-time monitoring" refers to the act of continuously monitoring a user's health status.

[0935] "Warning" refers to a warning that is given when the user's health condition reaches a dangerous state.

[0936] "Smart glasses" refers to a type of wearable device that can present information visually.

[0937] "Server" refers to the device that serves as the core of this system and analyzes, stores, and distributes data.

[0938] As an embodiment of the present invention, a system for enabling users to consume alcohol appropriately while reducing health risks at restaurants will be described in detail.

[0939] Hardware and Software Overview

[0940] The server is the core of this system and is a device that analyzes, stores, and distributes data. This system uses smart glasses, smartphones, and wearable devices (e.g., smartwatches). Smart glasses are a type of wearable device that can present information visually, and Google Glass can be used. Image recognition APIs such as Google Cloud Vision API are used for image recognition. Health data is collected using health data analysis platforms such as Apple HealthKit and Google Fit. Firebase can also be used as a real-time communication server.

[0941] Program processing explanation

[0942] The server receives restaurant menu images taken by the user using smart glasses or a smartphone and analyzes them using an image recognition API. This extracts drink types and details from the menu items. The server then obtains the user's health data from the smartwatch or smartphone and evaluates their current health status. This evaluation uses data such as heart rate, sleep time, and exercise volume.

[0943] The server compares the user's current health condition with their past drinking history and physical condition data, and determines the appropriate amount of alcohol intake based on this. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. Based on this judgment, it provides advice on the user's next drink. The smart glasses display shows a "OK" or "NG" result, and also suggests alternatives.

[0944] The smart glasses and smartphone also continuously monitor the user's health status in real time. If the user's heart rate reaches a dangerous level, a warning message will be displayed immediately. The specific warning message will be displayed, such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[0945] Examples and prompts

[0946] As a concrete example, consider the case where a user uses smart glasses to select a cocktail from a restaurant menu. Because the user's heart rate is already high, the system judges the selection as "NG" and the smart glasses display shows "One beer is OK." In this case, the user can order a beer instead of a cocktail to reduce the health risk.

[0947] Example prompt sentence:

[0948] When a user browses a menu and chooses a cocktail at a restaurant, they provide the following information:

[0949] 1. If a cocktail is "NG," please briefly explain why it is "NG."

[0950] 2. Suggest a specific drink name as an alternative.

[0951] 3. Based on the current user health data (high heart rate), advise what action to take.

[0952] example:

[0953] Cocktails: No - High heart rate, try a lower alcohol content drink. Alternative: 1 beer

[0954] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0955] Step 1:

[0956] A user takes a photo of a restaurant menu using smart glasses or a smartphone.

[0957] Input: Restaurant menu image

[0958] Output: Menu image taken

[0959] The terminal acquires the menu image photographed by the user and temporarily stores the image.

[0960] Step 2:

[0961] The terminal transmits the captured menu image to the server.

[0962] Input: Menu image taken

[0963] Output: Image data sent to the server

[0964] The device uploads the image data to a specified server and sends a request to an API for image recognition.

[0965] Step 3:

[0966] The server uses an image recognition API to analyze the menu images and extract drink types and details.

[0967] Input: Image data sent to the server

[0968] Output: List of drink types and details

[0969] The server uses the Google Cloud Vision API and other tools to analyze the text in the images and categorize the menu items by drink type.

[0970] Step 4:

[0971] The server obtains the user's health data from the health data analysis platform.

[0972] Input: User identification information

[0973] Output: User's health data (heart rate, sleep time, exercise amount, etc.)

[0974] The server retrieves health data through APIs from platforms such as HealthKit and Google Fit and uses it to assess the user's current health status.

[0975] Step 5:

[0976] The server compares the acquired health data with past drinking history and physical condition data to determine the appropriate amount of alcohol intake.

[0977] Input: User's health data, past drinking history, and physical condition data

[0978] Output: Appropriate alcohol intake and judgment result

[0979] The server compares past drinking history and current health data from a database to calculate alcohol intake and determine drinking pace.

[0980] Step 6:

[0981] The server generates advice for the next drink selection based on the judgment result and transmits it to the terminal.

[0982] Input: Judgment result

[0983] Output: Advice ("OK" or "NG")

[0984] The server uses the generative AI model to generate advice appropriate for the user and outputs a judgment result such as "OK" or "NG" when the next drink is selected, or provides specific advice based on the prompt text.

[0985] Step 7:

[0986] The terminal receives the determination result from the server and displays it on the display of the smart glasses.

[0987] Input: Advice content

[0988] Output: Advice displayed on the smart glasses display

[0989] The terminal displays the received advice in real time on the display of the smart glasses so that the user can check it.

[0990] Step 8:

[0991] When the user orders the next drink, the terminal will prompt the user to enter or select the name of the drink and display the decision.

[0992] Input: User-selected drink name

[0993] Output: Drink judgment result ("OK" or "NG") and alternatives

[0994] The device sends the name of the drink selected by the user to the server and displays the result on the display. If the drink is judged to be "NG," an alternative option will be presented.

[0995] Step 9:

[0996] The server continuously monitors the user's health status in real time and immediately issues an alert if it detects any abnormalities.

[0997] Input: User's health data (real-time)

[0998] Output: Warning message

[0999] The server continuously monitors the user's health data, and if, for example, the heart rate reaches a dangerous level, it sends a warning message to the device, such as, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[1000] Through these steps, users can enjoy alcohol in a healthy manner at restaurants. This system provides real-time advice and warnings through the collaboration of smart glasses and a server, reducing health risks for users.

[1001] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1002] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition, drinking status, and emotional state will be described in detail.

[1003] 1. Image recognition for menus

[1004] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[1005] 2. Health Data Collection

[1006] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[1007] 3. Collecting Emotional Data

[1008] The device collects emotional data from the user's facial expressions and voice. This data is collected using a camera and microphone, and the emotion engine analyzes it. The analysis results are used to evaluate the user's emotional state (e.g., joy, sadness, anger, etc.), and the results are sent to the server and stored in a database.

[1009] 4. Determining alcohol intake

[1010] The server extracts past drinking history, physical condition data, and emotional data from a database. Based on this, it compares the user's current health and emotional state and determines the appropriate amount of alcohol intake. For example, it makes a specific judgment such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[1011] 5. Advice for choosing your next drink

[1012] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[1013] 6. Real-time health and emotional status monitoring and alerts

[1014] The device continuously monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[1015] The system of the present invention supports users in enjoying alcohol in a healthy and appropriate emotional state, reducing the health risks and emotional downfalls associated with excessive drinking. A specific example is a system in which, if a user attempts to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and emotional state, thereby maintaining their health and social relationships.

[1016] The processing flow will be explained below.

[1017] Step 1:

[1018] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[1019] Step 2:

[1020] The device uses image recognition technology to analyze the menu image, extracting detailed information such as the drink name, type, and price, and storing this information in memory.

[1021] Step 3:

[1022] The extracted drink information is sent to the server, which then registers the received information in a database and generates a drink list.

[1023] Step 4:

[1024] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. This data is collected at regular intervals and stored on the device.

[1025] Step 5:

[1026] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[1027] Step 6:

[1028] The device uses a camera and microphone to collect the user's emotional data (facial expressions and voice), and the emotion engine analyzes this data to evaluate the user's current emotional state.

[1029] Step 7:

[1030] The evaluation results of the emotional state are sent to a server and stored in a database.

[1031] Step 8:

[1032] The server extracts past drinking history, physical condition data, and emotional data from a database, and compares it with the current health and emotional state to determine the appropriate amount of alcohol intake.

[1033] Step 9:

[1034] The server sends the result of the judgment to the terminal, which then stores the result.

[1035] Step 10:

[1036] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[1037] Step 11:

[1038] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[1039] Step 12:

[1040] The device monitors the user's health and emotional state in real time, with data periodically captured and analyzed to detect anomalies.

[1041] Step 13:

[1042] If your health or emotional state suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level or you become extremely irritable, a warning message will appear saying, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[1043] The above are the specific processing steps of this system.

[1044] Example 2

[1045] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1046] Currently, many people consume alcohol at restaurants without fully considering their health and emotional state, which can result in health risks and emotional problems. Furthermore, there are few systems that provide real-time information to help users determine the appropriate amount of alcohol they should consume. This makes it difficult to properly manage users' health and the effects of drinking.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1048] In this invention, the server includes means for performing image recognition on restaurant menus photographed by the user and extracting drink types and details, means for acquiring the user's health data and evaluating their current health condition, and means for analyzing the user's facial expressions and voice to collect emotional data. This makes it possible to monitor the user's health and emotional state in real time, determine the appropriate amount of alcohol intake, and provide advice on selecting the next drink.

[1049] "User" refers to a person who uses the system.

[1050] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or smartwatch.

[1051] A "menu image" refers to an image of a menu taken by a user at a restaurant.

[1052] "Image recognition technology" refers to technology for extracting specific information from captured images.

[1053] "Drink type" refers to the category of drinks listed on the menu, for example, beer, wine, cocktails, etc.

[1054] "Health data" refers to physiological data such as the user's heart rate, sleep time, and amount of exercise.

[1055] "Emotion data" refers to the emotional state of the user analyzed from their facial expressions and voice, such as joy, sadness, anger, etc.

[1056] "Drinking history" refers to a record of alcohol consumed by a user in the past.

[1057] "Alcohol intake" refers to the amount of alcohol that is appropriate for a user to drink.

[1058] "Advice" refers to appropriate instructions or suggestions regarding drinking that the system provides to the user.

[1059] "Monitoring" refers to the continuous observation of a user's health and emotional state.

[1060] "Warning" refers to a warning message sent by the system when an abnormality is detected in the user's condition.

[1061] A "database" refers to a structured collection of information for storing and managing data within a system.

[1062] "Analysis" refers to the process of evaluating the user's condition based on collected data.

[1063] "Real-time" refers to near-simultaneous processing and responses.

[1064] The present invention is a system that provides appropriate advice based on a user's health condition, drinking habits, and emotional state. This system operates through the following specific steps.

[1065] Hardware and Software Configuration

[1066] Device: A smartphone, tablet, smartwatch, etc. used by a user. Devices are equipped with cameras and microphones, which are used to collect data.

[1067] Server: A computer system that analyzes and manages data. The server is connected to a database that stores users' health and emotional data.

[1068] Database: A data storage space located within the server. It stores user health data, emotional data, drinking history, etc.

[1069] Software and Technology

[1070] Image recognition technology: Google Cloud Vision API and OpenCV are used as image recognition technologies to analyze menu images captured by the device's camera.

[1071] Health data collection platform: Uses Apple HealthKit and Google Fit to collect user health data (heart rate, sleep time, exercise amount, etc.).

[1072] Sentiment analysis engine: Uses Amazon Rekognition and Microsoft Azure Face API to analyze facial and voice data collected from the camera and microphone to generate emotion data.

[1073] AI model: Using TensorFlow and PyTorch, it analyzes a user's health data, emotional data, and past drinking history to determine the appropriate amount of alcohol intake.

[1074] Specific operation of the system

[1075] 1. Menu Image Recognition:

[1076] A user takes a photo of a restaurant menu with their smartphone. The device then analyzes the menu image using the Google Cloud Vision API to extract drink types and details. For example, if the menu includes "beer," "wine," and "sangria," the device recognizes these and sends the information to the server.

[1077] Example prompt sentence:

[1078] "Perform image recognition on the menu and extract the types of drinks."

[1079] 2. Health Data Collection:

[1080] The device collects health data such as heart rate, sleep time, and exercise volume through Apple HealthKit or Google Fit, and this data is periodically sent to a server and used as the basis for determining appropriate alcohol intake.

[1081] Example prompt sentence:

[1082] "Collect health data and assess the user's current health status."

[1083] 3. Collecting Emotional Data:

[1084] The device uses a camera and microphone to capture the user's facial expressions and voice, and then analyzes the emotion data using Amazon Rekognition or Microsoft Azure Face API. For example, if the user is smiling, it will be evaluated as "happy" and the data will be sent to the server.

[1085] Example prompt sentence:

[1086] "Analyze emotions from the user's facial expressions and voice and evaluate the results."

[1087] 4. Determining Alcohol Intake:

[1088] The server uses an AI model based on past drinking history, health data, and emotional data to determine the appropriate amount of alcohol intake—for example, "two beers or one cocktail"—and sends this information to the device.

[1089] Example prompt sentence:

[1090] "Based on past data, determine the appropriate amount of alcohol you should consume today."

[1091] 5. Advice for choosing your next drink:

[1092] When the user inputs or selects the next drink they want to order into the terminal, the terminal displays a verdict of "OK" or "NG" based on the judgment result from the server. For example, if the user selects a cocktail, the terminal will display "NG" and suggest that "one beer is OK."

[1093] Example prompt sentence:

[1094] "Show me advice for the drink I'm about to order."

[1095] 6. Real-time health and emotional status monitoring and alerts:

[1096] The device uses Apple HealthKit and Google Fit to collect the user's health and emotional data in real time. If an abnormality is detected, for example, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1097] Example prompt sentence:

[1098] "Monitor your health and emotional state in real time and alert you if anything unusual is detected."

[1099] These steps help users enjoy alcohol in a healthy and appropriate emotional state, reducing health risks and emotional problems while also helping users maintain social relationships.

[1100] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1101] Step 1:

[1102] Image recognition for menus

[1103] A user takes a photo of a restaurant menu with their smartphone. The input is the menu image taken by the user. The device analyzes the menu image using the Google Cloud Vision API. As a result of the analysis, the type of drink and its details are extracted. This information is output and sent to the server. Specifically, the device analyzes the image and recognizes items such as "beer," "wine," and "sangria."

[1104] Step 2:

[1105] Health data collection

[1106] The device acquires the user's health data using Apple HealthKit or Google Fit. Input includes the user's heart rate, sleep time, and exercise amount. The acquired data is pre-processed on the device and sent to the server as output. Specifically, the device collects data every 30 minutes and sends information such as a heart rate of 70 bpm, 7 hours of sleep, and 30 minutes of exercise per day to the server.

[1107] Step 3:

[1108] Collecting Emotional Data

[1109] The device uses the smartphone's camera and microphone to collect the user's facial expressions and voice. The inputs include the user's facial expression data and voice data. Using Amazon Rekognition or Microsoft Azure Face API, the device analyzes the emotional data, and the output emotional state (e.g., happiness, sadness, anger) is sent to the server. Specifically, if the user is smiling, it evaluates the user as "happy," and this data is sent to the server.

[1110] Step 4:

[1111] Determining alcohol intake

[1112] The server extracts past drinking history, health data, and emotional data from a database. This past data is included as input. The server analyzes the data using a generative AI model using TensorFlow and PyTorch, and the output determines the appropriate amount of alcohol intake. Specifically, the server determines that "two beers or one cocktail is appropriate" and sends this information to the device.

[1113] Step 5:

[1114] Advice for choosing your next drink

[1115] The user inputs or selects the next drink they wish to order into the terminal. The input is the drink information selected by the user. Based on the result of the judgment received from the server, the terminal displays a judgment of "OK" or "NG" as output. In concrete terms, if the user selects a cocktail, the terminal displays "NG" and suggests that "one beer is OK."

[1116] Step 6:

[1117] Real-time health and emotional status monitoring and alerts

[1118] The device continues to collect health and emotional data in real time. The input includes the user's current health and emotional data. If the device detects an abnormality based on the analyzed data, it displays a warning message as output. Specifically, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1119] (Application example 2)

[1120] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] Currently, drinking in restaurants and bars presents a challenge in that it is difficult for patrons to determine the appropriate amount of alcohol they should consume while keeping track of their health and emotional state. In particular, because excessive drinking increases health risks and fluctuations in emotional state can have a negative impact on social relationships, there is a lack of mechanisms to support appropriate drinking. This has led to a demand for systems that allow patrons to enjoy a safe and comfortable drinking experience.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1123] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for acquiring the user's emotional data and assessing their current emotional state, means for comparing their past drinking history and physical condition data with their current health and emotional state to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the results of this assessment, and means for monitoring the user's health and emotional state in real time and issuing a warning if an abnormality is detected. This makes it possible to provide a safe and comfortable drinking experience while taking into consideration the user's health and emotional state comprehensively.

[1124] "Image recognition" is a technology that analyzes captured images and recognizes specific objects and characters.

[1125] "Health data" refers to collected and recorded information about an individual's health status, such as heart rate, sleep time, and amount of exercise.

[1126] "Emotion data" is data that indicates an emotional state such as joy, sadness, or anger, which is analyzed from the user's facial expressions and voice.

[1127] "Drinking history" refers to a record of the types and amounts of alcoholic beverages consumed in the past, as well as the dates and times of consumption.

[1128] "Physical condition data" is data that includes various information about the user's past and present health conditions.

[1129] "Alcohol intake" is a measure of the total amount of alcohol consumed within a specified period of time.

[1130] The "determination result" is judgment information about the safety of the drink provided by the server based on the user's health data, drinking history, and emotional data.

[1131] The "means for providing advice" is a function in which the system determines whether the next drink the user selects is appropriate or inappropriate, and makes suggestions, including alternatives, based on the results.

[1132] "Real-time monitoring" is a function that constantly monitors and instantly analyzes the user's health and emotional state.

[1133] The "means for issuing warnings" is a function that immediately sends a warning message when an abnormality occurs in the user's health or emotional state.

[1134] An embodiment of the present invention will now be described. This system provides appropriate advice based on the user's health condition, drinking status, and emotional state, and supports a safe and comfortable drinking experience.

[1135] First, the user takes a photo of a restaurant menu using a smartphone or smart glasses. The device then analyzes the menu image using image recognition technology to extract the drink type and details. This process uses image processing libraries such as OpenCV. The extracted information is then sent to a server and registered in a database.

[1136] Next, the device acquires the user's health data. This data includes information such as heart rate, sleep time, and exercise volume acquired from the smartwatch or smartphone. This data is collected periodically and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[1137] In addition, the device collects emotional data from the user's facial expressions and voice. This step uses a camera and microphone, and the emotion engine analyzes the data. The resulting emotional state (happiness, sadness, anger, etc.) is sent to the server and stored in a database.

[1138] The server compares the collected past drinking history, physical condition data, and emotional data with the user's current health and emotional state to determine the appropriate amount of alcohol intake. Specifically, it makes a specific judgment, such as whether the user can safely consume up to two beers or one cocktail. This judgment result is sent from the server to the device.

[1139] When a user selects or inputs the next drink to order, the device will display "OK" or "NG" based on the judgment from the server, indicating whether the drink is appropriate or inappropriate. For example, if a user tries to order a cocktail, the device will display "NG," but an alternative suggestion such as "One beer is OK" will be displayed.

[1140] The device also monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[1141] For example, if a user tries to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain a proper drinking pace and maintain their health and social relationships.

[1142] Here are some examples of prompts for generative AI models:

[1143] "Please provide a concrete example of a system that determines the appropriate amount of alcohol intake based on the user's current health status (heart rate, sleep time, amount of exercise) and emotional state (happiness, sadness, anger), and compares it with the user's current health and emotional state. Also, please display drink suggestions."

[1144] The above is an embodiment of the present invention. This system allows users to enjoy a safe and comfortable drinking experience while maintaining an appropriate alcohol intake amount according to their health and emotional state.

[1145] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1146] Step 1:

[1147] The user takes a photo of a restaurant menu using a smartphone or smart glasses.

[1148] Specific behavior:

[1149] A user takes a picture of a restaurant menu with a camera to obtain a menu image.

[1150] Input: Menu Image

[1151] Output: Photographed menu image data

[1152] Step 2:

[1153] The device uses image recognition technology to analyze the menu image captured and extract the drink type and details.

[1154] Specific behavior:

[1155] The device uses an image processing library such as OpenCV to analyze the image and recognize the menu items.

[1156] Input: Photographed menu image data

[1157] Output: Recognized drink type and details

[1158] Step 3:

[1159] The extracted information is sent to a server and registered in a database.

[1160] Specific behavior:

[1161] The drink information extracted from the device is sent to the server, which stores the information in a database.

[1162] Input: Recognized drink information

[1163] Output: Drink information stored in the database

[1164] Step 4:

[1165] The device acquires the user's health data.

[1166] Specific behavior:

[1167] The device collects health data such as heart rate, sleep time, and exercise volume from smartwatches and smartphones.

[1168] Input: Health data obtained from a smartwatch or smartphone

[1169] Output: Retrieved health data

[1170] Step 5:

[1171] The acquired health data is sent to a server and registered in a database.

[1172] Specific behavior:

[1173] The device sends the acquired health data to a server, which stores the data in a database.

[1174] Input: Acquired health data

[1175] Output: Health data stored in a database

[1176] Step 6:

[1177] The device collects emotional data from the user's facial expressions and voice.

[1178] Specific behavior:

[1179] The device uses a camera and microphone to capture the user's facial expressions and voice, which are then analyzed using an emotion engine.

[1180] Input: User's facial expressions and voice data

[1181] Output: Parsed emotion data

[1182] Step 7:

[1183] The analyzed emotional data is sent to a server and registered in a database.

[1184] Specific behavior:

[1185] The device sends the analyzed emotion data to the server, which stores the data in a database.

[1186] Input: Parsed emotion data

[1187] Output: Emotion data stored in a database

[1188] Step 8:

[1189] The server compares past drinking history, physical condition data, current health and emotional state, and determines the appropriate amount of alcohol intake.

[1190] Specific behavior:

[1191] The server retrieves past drinking history, physical condition data, current health and emotional state from the database and analyzes them using an algorithm.

[1192] Input: Drinking history, physical condition data, health status, emotional state stored in the database

[1193] Output: Determined appropriate alcohol intake

[1194] Step 9:

[1195] When the user enters or selects the next drink they want to order, the terminal will display an "OK" or "NG" result and alternatives.

[1196] Specific behavior:

[1197] The terminal receives the judgment result from the server and displays the status of "OK" or "NG" as well as alternatives for the next drink the user orders.

[1198] Input: Verification result from the server

[1199] Output: Display advice to the user when choosing a drink

[1200] Step 10:

[1201] The device monitors the user's health and emotional state in real time and issues an alert if it detects any abnormalities.

[1202] Specific behavior:

[1203] The device periodically collects and analyzes the user's health and emotional data, and displays a warning message in real time if any abnormalities are detected.

[1204] Input: User health and emotional data

[1205] Output: Warning message when an abnormality is detected

[1206] These are the specific processing steps of this system. Each step allows the user to enjoy an optimal drinking experience that is tailored to their own health and emotional state.

[1207] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1208] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1209] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1210] [Fourth embodiment]

[1211] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1212] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1213] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1214] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1215] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1216] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1217] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1218] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1219] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1220] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1221] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1222] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1223] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1224] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition and drinking habits will be described in detail.

[1225] 1. Image recognition for menus

[1226] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[1227] 2. Health Data Collection

[1228] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[1229] 3. Determining alcohol intake

[1230] The server extracts past drinking history and physical condition data from a database. Based on this, it compares it with the user's current health condition and determines how much alcohol the user needs to consume and at what speed. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[1231] 4. Advice for choosing your next drink

[1232] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[1233] 5. Real-time health monitoring and alerts

[1234] The device continuously monitors the user's health condition in real time. Data is collected and analyzed periodically. If the user's health condition suddenly deteriorates, the device immediately issues a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1235] The system of the present invention provides support for users to enjoy drinking in a healthy manner and reduces the health risks associated with excessive drinking. As a specific example, if a user attempts to select a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and maintain their health.

[1236] The processing flow will be explained below.

[1237] Step 1:

[1238] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[1239] Step 2:

[1240] The device uses image recognition technology to analyze the menu image, and the extracted information, such as the name, type, and price of the drink, is stored in the device's memory.

[1241] Step 3:

[1242] The device sends the extracted information to the server, which then registers the information in a database and generates a drink list.

[1243] Step 4:

[1244] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. The data is collected periodically and stored on the device.

[1245] Step 5:

[1246] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[1247] Step 6:

[1248] The server extracts past drinking history and physical condition data from a database, and based on this, compares it with the current health condition to determine the appropriate amount of alcohol intake.

[1249] Step 7:

[1250] The server sends the result of the judgment to the terminal, which then stores the result.

[1251] Step 8:

[1252] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[1253] Step 9:

[1254] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[1255] Step 10:

[1256] The device monitors the user's health status in real time, with data collected periodically and analyzed to detect abnormalities.

[1257] Step 11:

[1258] If your health condition suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level, the device will display a message saying, "Your current heart rate has reached a dangerous level. Stop drinking alcohol and drink water."

[1259] The above are the specific processing steps of this system.

[1260] Example 1

[1261] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1262] In today's world, managing alcohol consumption is important for maintaining good health for many people. However, it is difficult for users to determine the appropriate amount of alcohol intake based on their own health condition and drinking history. Furthermore, there is a lack of methods to monitor health status in real time while drinking and respond immediately if an abnormality occurs. To address these issues, a system is needed that provides appropriate advice based on health status and drinking habits, and supports users in enjoying healthy drinking.

[1263] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1264] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment results, and means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected. This allows the user to monitor their health condition in real time while maintaining an appropriate drinking pace according to their own health condition, and receive appropriate advice and warnings as needed.

[1265] A "user" is an individual who uses the system to manage their drinking habits and health.

[1266] "Eating and drinking establishments" refers to places where users eat and drink, including restaurants and bars.

[1267] A "menu" is a document or booklet listing the drinks and food served at a restaurant.

[1268] "Image recognition" refers to the technique of extracting information from images using computer vision techniques.

[1269] "Beverages" refers to alcoholic and non-alcoholic beverages served at restaurants.

[1270] "Health data" refers to data that indicates the user's health condition, such as heart rate, sleep time, and amount of exercise.

[1271] "Health status" refers to the user's current physical condition and includes parameters such as heart rate, blood pressure, and activity level.

[1272] "Drinking history" is a record of alcoholic beverages consumed by a user in the past.

[1273] "Physical condition data" is a general term for data related to the user's physical condition and health.

[1274] "Alcohol intake" refers to the amount of alcohol consumed by a user within a certain period of time.

[1275] The "judgment result" is an assessment of the appropriate alcohol intake tolerance calculated by the system based on the user's health data, drinking history, and current health condition.

[1276] "Advice" refers to recommendations or recommendations about drinking that the system provides to the user.

[1277] "Real-time monitoring" refers to a technology that continuously monitors a user's health status and updates the data instantly.

[1278] A "warning" is a message issued by the system to alert the user when the user's health condition reaches an abnormal state.

[1279] "Server" refers to a computer system for processing and storing data.

[1280] A "database" is a data structure or system used to organize and manage information.

[1281] "Decision-making" is the process by which a system draws a conclusion based on multiple data and conditions.

[1282] The present invention relates to a system that provides appropriate advice based on a user's health condition and drinking habits. This system provides support for users to maintain their health while enjoying alcohol.

[1283] First, the user takes a photo of a restaurant menu with their smartphone. The device analyzes the menu image using the Google Cloud Vision API to extract the type of drink and its details. The extracted information is sent to a server and registered in a database. This is how menu information is managed.

[1284] Next, the device acquires health data (heart rate, sleep time, exercise volume, etc.) from the user's smartwatch or smartphone. This health data is collected through software such as Apple Health or Google Fit. The collected data is sent to a server and stored in a database. This allows the user's health status to be monitored appropriately.

[1285] The server extracts past drinking history and health data from a database and compares it with the user's current health status. This determines how much alcohol the user can safely consume. For example, it makes a specific determination such as whether two beers or one cocktail is safe. The result of the determination is sent from the server to the device.

[1286] When a user inputs or selects the next drink they want to order, the terminal displays a "OK" or "NG" result based on the judgment result from the server. Even if the drink is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the terminal will display "NG," but suggest "One beer is OK."

[1287] Furthermore, the device continuously monitors the user's health condition in real time, collecting and analyzing data periodically. If the user's health condition suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level, the device will display a warning message such as, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1288] For example, if a user tries to choose a cocktail at a restaurant, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain an appropriate drinking pace and maintain their health.

[1289] An example of a prompt to be input to a generative AI model is, "Please describe a system that provides appropriate advice based on the user's health status and drinking habits. Specifically, please provide details on image recognition of menus, collection of health data, determination of alcohol intake, advice on next drink selection, and real-time monitoring and alerting of health status."

[1290] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1291] Step 1:

[1292] A user takes a photo of a restaurant menu with their smartphone, and the image becomes the input data.

[1293] Step 2:

[1294] The device analyzes the menu image captured using the Google Cloud Vision API. Using the menu image as input, an image recognition algorithm extracts the drink type and details. The output data is the drink type and its detailed information.

[1295] Step 3:

[1296] The device sends the extracted drink information to the server. The input data is the type of drink and detailed information, which is converted into a format that is sent to the server as output data. This data is also registered in the database as menu information.

[1297] Step 4:

[1298] The device acquires health data (heart rate, sleep time, exercise amount, etc.) from the user's smartwatch or smartphone. The input data is various sensor data from the health device, and the output data is organized as health data.

[1299] Step 5:

[1300] The health data collected by the device is sent to a server. The input data is organized health data, and the output data is data converted into a format suitable for sending to the server. This saves the health data in the server's database.

[1301] Step 6:

[1302] The server extracts past drinking history and health data from the database, with input data being the user's drinking history and health data stored in the database and output data being the extracted history data.

[1303] Step 7:

[1304] The server compares the current health status with past data. The input data is the past drinking history and health data, and the current health data. The data matching algorithm determines the appropriate alcohol intake. The output data is the determined acceptable alcohol intake limit.

[1305] Step 8:

[1306] The server sends the result of the determination to the terminal. The input data is the determined alcohol intake tolerance, and the output data is data converted into a format that can be sent to the terminal.

[1307] Step 9:

[1308] The user inputs or selects the next drink to order into the terminal, and the input data is information about the drink selected or input by the user.

[1309] Step 10:

[1310] The device displays a verdict of "OK" or "NG" based on the judgment result received from the server. The input data is the information about the drink selected by the user and the judgment result from the server, and the output data is a verdict of "OK" or "NG." It also suggests alternative drinks.

[1311] Step 11:

[1312] The device monitors the user's health condition in real time. The input data is real-time data from various sensor devices, and the output data is an assessment of the user's current health condition.

[1313] Step 12:

[1314] If the device detects an abnormality, it will issue a warning to the user. The input data is health data collected in real time, and if an abnormality is detected through analysis, a warning message is displayed as output data.

[1315] (Application example 1)

[1316] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1317] Currently, health management when drinking alcohol at restaurants is left up to the user, and inappropriate alcohol consumption can pose health risks. To solve this problem, a system is needed that can grasp the user's health status and drinking habits in real time and provide appropriate advice. In addition, a system is needed that can provide specific advice when the user selects their next drink and issue warnings according to changes in their health status.

[1318] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1319] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting drink types and details, means for acquiring the user's health data and assessing their current health condition, means for comparing their current health condition with their past drinking history and physical condition data to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the assessment result, means for monitoring the user's health condition in real time and issuing a warning if an abnormality is detected, and means for displaying the advice and warning in real time using smart glasses. This enables users to achieve appropriate alcohol intake while reducing health risks at restaurants.

[1320] "User" refers to an individual or corporation that uses this system.

[1321] "Restaurant" means a commercial establishment that serves beverages and meals.

[1322] A "menu" is a list of the types of food and drink served by a restaurant.

[1323] "Image recognition" refers to the technology of extracting specific information from captured images.

[1324] "Types of drinks" refers to the various drinks listed on a restaurant menu.

[1325] "Health data" refers to information related to a user's health, such as their heart rate, sleep time, and amount of exercise.

[1326] "Health status" refers to the current physical condition and degree of health of the user, as assessed based on the user's health data.

[1327] "Drinking history" refers to a record of what types and amounts of alcoholic beverages a user has consumed in the past.

[1328] "Alcohol intake" refers to the total amount of alcohol consumed by a user within a certain period of time.

[1329] "Real-time monitoring" refers to the act of continuously monitoring a user's health status.

[1330] "Warning" refers to a warning that is given when the user's health condition reaches a dangerous state.

[1331] "Smart glasses" refers to a type of wearable device that can present information visually.

[1332] "Server" refers to the device that serves as the core of this system and analyzes, stores, and distributes data.

[1333] As an embodiment of the present invention, a system for enabling users to consume alcohol appropriately while reducing health risks at restaurants will be described in detail.

[1334] Hardware and Software Overview

[1335] The server is the core of this system and is a device that analyzes, stores, and distributes data. This system uses smart glasses, smartphones, and wearable devices (e.g., smartwatches). Smart glasses are a type of wearable device that can present information visually, and Google Glass can be used. Image recognition APIs such as Google Cloud Vision API are used for image recognition. Health data is collected using health data analysis platforms such as Apple HealthKit and Google Fit. Firebase can also be used as a real-time communication server.

[1336] Program processing explanation

[1337] The server receives restaurant menu images taken by the user using smart glasses or a smartphone and analyzes them using an image recognition API. This extracts drink types and details from the menu items. The server then obtains the user's health data from the smartwatch or smartphone and evaluates their current health status. This evaluation uses data such as heart rate, sleep time, and exercise volume.

[1338] The server compares the user's current health condition with their past drinking history and physical condition data, and determines the appropriate amount of alcohol intake based on this. For example, it makes a specific judgment, such as whether two beers or one cocktail is safe. Based on this judgment, it provides advice on the user's next drink. The smart glasses display shows a "OK" or "NG" result, and also suggests alternatives.

[1339] The smart glasses and smartphone also continuously monitor the user's health status in real time. If the user's heart rate reaches a dangerous level, a warning message will be displayed immediately. The specific warning message will be displayed, such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1340] Examples and prompts

[1341] As a concrete example, consider the case where a user uses smart glasses to select a cocktail from a restaurant menu. Because the user's heart rate is already high, the system judges the selection as "NG" and the smart glasses display shows "One beer is OK." In this case, the user can order a beer instead of a cocktail to reduce the health risk.

[1342] Example prompt sentence:

[1343] When a user browses a menu and chooses a cocktail at a restaurant, they provide the following information:

[1344] 1. If a cocktail is "NG," please briefly explain why it is "NG."

[1345] 2. Suggest a specific drink name as an alternative.

[1346] 3. Based on the current user health data (high heart rate), advise what action to take.

[1347] example:

[1348] Cocktails: No - High heart rate, try a lower alcohol content drink. Alternative: 1 beer

[1349] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1350] Step 1:

[1351] A user takes a photo of a restaurant menu using smart glasses or a smartphone.

[1352] Input: Restaurant menu image

[1353] Output: Menu image taken

[1354] The terminal acquires the menu image photographed by the user and temporarily stores the image.

[1355] Step 2:

[1356] The terminal transmits the captured menu image to the server.

[1357] Input: Menu image taken

[1358] Output: Image data sent to the server

[1359] The device uploads the image data to a specified server and sends a request to an API for image recognition.

[1360] Step 3:

[1361] The server uses an image recognition API to analyze the menu images and extract drink types and details.

[1362] Input: Image data sent to the server

[1363] Output: List of drink types and details

[1364] The server uses the Google Cloud Vision API and other tools to analyze the text in the images and categorize the menu items by drink type.

[1365] Step 4:

[1366] The server obtains the user's health data from the health data analysis platform.

[1367] Input: User's identity

[1368] Output: User's health data (heart rate, sleep time, exercise amount, etc.)

[1369] The server retrieves health data from platforms such as HealthKit and Google Fit through APIs and uses it to assess the user's current health status.

[1370] Step 5:

[1371] The server compares the acquired health data with past drinking history and physical condition data to determine the appropriate amount of alcohol intake.

[1372] Input: User's health data, past drinking history, and physical condition data

[1373] Output: Appropriate alcohol intake and judgment result

[1374] The server compares past drinking history and current health data from a database to calculate alcohol intake and determine drinking pace.

[1375] Step 6:

[1376] The server generates advice for the next drink selection based on the judgment result and transmits it to the terminal.

[1377] Input: Judgment result

[1378] Output: Advice ("OK" or "NG")

[1379] The server uses the generative AI model to generate advice appropriate for the user and outputs a judgment result such as "OK" or "NG" when the next drink is selected, or provides specific advice based on the prompt text.

[1380] Step 7:

[1381] The terminal receives the determination result from the server and displays it on the display of the smart glasses.

[1382] Input: Advice content

[1383] Output: Advice displayed on the smart glasses display

[1384] The terminal displays the received advice in real time on the display of the smart glasses so that the user can check it.

[1385] Step 8:

[1386] When the user orders the next drink, the terminal will prompt the user to enter or select the name of the drink and display the decision.

[1387] Input: User-selected drink name

[1388] Output: Drink judgment result ("OK" or "NG") and alternatives

[1389] The device sends the name of the drink selected by the user to the server and displays the result on the display. If the drink is judged to be "NG," an alternative option will be presented.

[1390] Step 9:

[1391] The server continuously monitors the user's health status in real time and immediately issues an alert if it detects any abnormalities.

[1392] Input: User's health data (real-time)

[1393] Output: Warning message

[1394] The server continuously monitors the user's health data, and if, for example, the heart rate reaches a dangerous level, it sends a warning message to the device, such as, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[1395] Through these steps, users can enjoy alcohol in a healthy manner at restaurants. This system provides real-time advice and warnings through the collaboration of smart glasses and a server, reducing the risk to users' health.

[1396] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1397] As an embodiment of the present invention, a system for providing appropriate advice based on a user's health condition, drinking status, and emotional state will be described in detail.

[1398] 1. Image recognition for menus

[1399] First, the user takes a photo of a restaurant menu with their smartphone. The device then uses image recognition technology to analyze the menu image and extract the drink type and details. For example, categories such as beer, cocktails, and sake are recognized, and this information is sent to the server and registered in a database.

[1400] 2. Health Data Collection

[1401] The device collects the user's health data from the watch or smartphone. Specifically, data such as heart rate, sleep time, and exercise volume are collected. This data is collected at regular intervals and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[1402] 3. Collecting Emotional Data

[1403] The device collects emotional data from the user's facial expressions and voice. This data is collected using a camera and microphone, and the emotion engine analyzes it. The analysis results are used to evaluate the user's emotional state (e.g., joy, sadness, anger, etc.), and the results are sent to the server and stored in a database.

[1404] 4. Determining alcohol intake

[1405] The server extracts past drinking history, physical condition data, and emotional data from a database. Based on this, it compares the user's current health and emotional state and determines the appropriate amount of alcohol intake. For example, it makes a specific judgment such as whether two beers or one cocktail is safe. The result of the judgment is sent from the server to the device.

[1406] 5. Advice for choosing your next drink

[1407] When a user inputs or selects the next drink they want to order, the device displays a "OK" or "NG" result based on the server's judgment. Even if the result is "NG," an alternative "OK" drink will be suggested. For example, if a user tries to order a cocktail, the device will display "NG," but suggest "One beer is OK."

[1408] 6. Real-time health and emotional status monitoring and alerts

[1409] The device continuously monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[1410] The system of the present invention supports users in enjoying alcohol in a healthy and appropriate emotional state, reducing the health risks and emotional downfalls associated with excessive drinking. A specific example is a system in which, if a user attempts to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." This allows users to maintain an appropriate drinking pace and emotional state, thereby maintaining their health and social relationships.

[1411] The processing flow will be explained below.

[1412] Step 1:

[1413] A user takes a photo of a restaurant menu with their smartphone, and the image is imported into the device.

[1414] Step 2:

[1415] The device uses image recognition technology to analyze the menu image, extracting detailed information such as the drink name, type, and price, and storing this information in memory.

[1416] Step 3:

[1417] The extracted drink information is sent to the server, which then registers the received information in a database and generates a drink list.

[1418] Step 4:

[1419] The device collects the user's health data (heart rate, sleep time, exercise amount, etc.) from the watch or smartphone. This data is collected at regular intervals and stored on the device.

[1420] Step 5:

[1421] The device analyzes the acquired health data and evaluates the user's current health condition. The evaluation results are sent to the server and stored in a database.

[1422] Step 6:

[1423] The device uses a camera and microphone to collect the user's emotional data (facial expressions and voice), and the emotion engine analyzes this data to evaluate the user's current emotional state.

[1424] Step 7:

[1425] The evaluation results of the emotional state are sent to a server and stored in a database.

[1426] Step 8:

[1427] The server extracts past drinking history, physical condition data, and emotional data from a database, and compares it with the current health and emotional state to determine the appropriate amount of alcohol intake.

[1428] Step 9:

[1429] The server sends the result of the judgment to the terminal, which then stores the result.

[1430] Step 10:

[1431] The user inputs or selects the next drink to order into the terminal, and the input drink information is sent to the terminal.

[1432] Step 11:

[1433] The device will display a "OK" or "NG" verdict based on the results of the server's assessment. If the result is "NG," it will also suggest an "OK" alternative drink.

[1434] Step 12:

[1435] The device monitors the user's health and emotional state in real time, with data periodically captured and analyzed to detect anomalies.

[1436] Step 13:

[1437] If your health or emotional state suddenly deteriorates, the device will issue a warning. For example, if your heart rate reaches a dangerous level or you become extremely irritable, a warning message will appear saying, "Your current heart rate has reached a dangerous level. Stop drinking and drink water."

[1438] The above are the specific processing steps of this system.

[1439] Example 2

[1440] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1441] Currently, many people consume alcohol at restaurants without fully considering their health and emotional state, which can result in health risks and emotional problems. Furthermore, there are few systems that provide real-time information to help users determine the appropriate amount of alcohol they should consume. This makes it difficult to properly manage users' health and the effects of drinking.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1443] In this invention, the server includes means for performing image recognition on restaurant menus photographed by the user and extracting drink types and details, means for acquiring the user's health data and evaluating their current health condition, and means for analyzing the user's facial expressions and voice to collect emotional data. This makes it possible to monitor the user's health and emotional state in real time, determine the appropriate amount of alcohol intake, and provide advice on selecting the next drink.

[1444] "User" refers to a person who uses the system.

[1445] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or smartwatch.

[1446] A "menu image" refers to an image of a menu taken by a user at a restaurant.

[1447] "Image recognition technology" refers to technology for extracting specific information from captured images.

[1448] "Drink type" refers to the category of drinks listed on the menu, for example, beer, wine, cocktails, etc.

[1449] "Health data" refers to physiological data such as the user's heart rate, sleep time, and amount of exercise.

[1450] "Emotion data" refers to the emotional state of the user analyzed from their facial expressions and voice, such as joy, sadness, anger, etc.

[1451] "Drinking history" refers to a record of alcohol consumed by a user in the past.

[1452] "Alcohol intake" refers to the amount of alcohol that is appropriate for a user to drink.

[1453] "Advice" refers to appropriate instructions or suggestions regarding drinking that the system provides to the user.

[1454] "Monitoring" refers to the continuous observation of a user's health and emotional state.

[1455] "Warning" refers to a warning message sent by the system when an abnormality is detected in the user's condition.

[1456] A "database" refers to a structured collection of information for storing and managing data within a system.

[1457] "Analysis" refers to the process of evaluating the user's condition based on collected data.

[1458] "Real-time" refers to near-simultaneous processing and responses.

[1459] The present invention is a system that provides appropriate advice based on a user's health condition, drinking habits, and emotional state. This system operates through the following specific steps.

[1460] Hardware and Software Configuration

[1461] Device: A smartphone, tablet, smartwatch, etc. used by a user. Devices are equipped with cameras and microphones, which are used to collect data.

[1462] Server: A computer system that analyzes and manages data. The server is connected to a database that stores users' health and emotional data.

[1463] Database: A data storage space located within the server. It stores user health data, emotional data, drinking history, etc.

[1464] Software and Technology

[1465] Image recognition technology: Google Cloud Vision API and OpenCV are used as image recognition technologies to analyze menu images captured by the device's camera.

[1466] Health data collection platform: Uses Apple HealthKit and Google Fit to collect user health data (heart rate, sleep time, exercise amount, etc.).

[1467] Sentiment analysis engine: Uses Amazon Rekognition and Microsoft Azure Face API to analyze facial and voice data collected from the camera and microphone to generate emotion data.

[1468] AI model: Using TensorFlow and PyTorch, it analyzes a user's health data, emotional data, and past drinking history to determine the appropriate amount of alcohol intake.

[1469] Specific operation of the system

[1470] 1. Menu Image Recognition:

[1471] A user takes a photo of a restaurant menu with their smartphone. The device then analyzes the menu image using the Google Cloud Vision API to extract drink types and details. For example, if the menu includes "beer," "wine," and "sangria," the device recognizes these and sends the information to the server.

[1472] Example prompt sentence:

[1473] "Perform image recognition on the menu and extract the types of drinks."

[1474] 2. Health Data Collection:

[1475] The device collects health data such as heart rate, sleep time, and exercise volume through Apple HealthKit or Google Fit, and this data is periodically sent to a server and used as the basis for determining appropriate alcohol intake.

[1476] Example prompt sentence:

[1477] "Collect health data and assess the user's current health status."

[1478] 3. Collecting Emotional Data:

[1479] The device uses a camera and microphone to capture the user's facial expressions and voice, and then analyzes the emotion data using Amazon Rekognition or Microsoft Azure Face API. For example, if the user is smiling, it will be evaluated as "happy" and the data will be sent to the server.

[1480] Example prompt sentence:

[1481] "Analyze emotions from the user's facial expressions and voice and evaluate the results."

[1482] 4. Determining Alcohol Intake:

[1483] The server uses an AI model based on past drinking history, health data, and emotional data to determine the appropriate amount of alcohol intake—for example, "two beers or one cocktail"—and sends this information to the device.

[1484] Example prompt sentence:

[1485] "Based on past data, determine the appropriate amount of alcohol you should consume today."

[1486] 5. Advice for choosing your next drink:

[1487] When the user inputs or selects the next drink they want to order into the terminal, the terminal displays a verdict of "OK" or "NG" based on the judgment result from the server. For example, if the user selects a cocktail, the terminal will display "NG" and suggest that "one beer is OK."

[1488] Example prompt sentence:

[1489] "Show me advice for the drink I'm about to order."

[1490] 6. Real-time health and emotional status monitoring and alerts:

[1491] The device uses Apple HealthKit and Google Fit to collect the user's health and emotional data in real time. If an abnormality is detected, for example, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1492] Example prompt sentence:

[1493] "Monitor your health and emotional state in real time and alert you if anything unusual is detected."

[1494] These steps help users enjoy alcohol in a healthy and appropriate emotional state, reducing health risks and emotional problems while also helping users maintain social relationships.

[1495] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1496] Step 1:

[1497] Image recognition for menus

[1498] A user takes a photo of a restaurant menu with their smartphone. The input is the menu image taken by the user. The device analyzes the menu image using the Google Cloud Vision API. As a result of the analysis, the type of drink and its details are extracted. This information is output and sent to the server. Specifically, the device analyzes the image and recognizes items such as "beer," "wine," and "sangria."

[1499] Step 2:

[1500] Health data collection

[1501] The device acquires the user's health data using Apple HealthKit or Google Fit. Input includes the user's heart rate, sleep time, and exercise amount. The acquired data is pre-processed on the device and sent to the server as output. Specifically, the device collects data every 30 minutes and sends information such as a heart rate of 70 bpm, 7 hours of sleep, and 30 minutes of exercise per day to the server.

[1502] Step 3:

[1503] Collecting Emotional Data

[1504] The device uses the smartphone's camera and microphone to collect the user's facial expressions and voice. The inputs include the user's facial expression data and voice data. Using Amazon Rekognition or Microsoft Azure Face API, the device analyzes the emotional data, and the output emotional state (e.g., happiness, sadness, anger) is sent to the server. Specifically, if the user is smiling, it evaluates the user as "happy," and this data is sent to the server.

[1505] Step 4:

[1506] Determining alcohol intake

[1507] The server extracts past drinking history, health data, and emotional data from a database. This past data is included as input. The server analyzes the data using a generative AI model using TensorFlow and PyTorch, and the output determines the appropriate amount of alcohol intake. Specifically, the server determines that "two beers or one cocktail is appropriate" and sends this information to the device.

[1508] Step 5:

[1509] Advice for choosing your next drink

[1510] The user inputs or selects the next drink they wish to order into the terminal. The input is the drink information selected by the user. Based on the result of the judgment received from the server, the terminal displays a judgment of "OK" or "NG" as output. In concrete terms, if the user selects a cocktail, the terminal displays "NG" and suggests that "one beer is OK."

[1511] Step 6:

[1512] Real-time health and emotional status monitoring and alerts

[1513] The device continues to collect health and emotional data in real time. The input includes the user's current health and emotional data. If the device detects an abnormality based on the analyzed data, it displays a warning message as output. Specifically, if the heart rate reaches a dangerous level, the device will display a warning saying, "Your current heart rate has reached a dangerous level. Please stop drinking and drink water."

[1514] (Application example 2)

[1515] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] Currently, drinking in restaurants and bars presents a challenge in that it is difficult for patrons to determine the appropriate amount of alcohol they should consume while keeping track of their health and emotional state. In particular, because excessive drinking increases health risks and fluctuations in emotional state can have a negative impact on social relationships, there is a lack of mechanisms to support appropriate drinking. This has led to a demand for systems that allow patrons to enjoy a safe and comfortable drinking experience.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1518] In this invention, the server includes means for performing image recognition on a restaurant menu photographed by the user and extracting the type and details of the drink, means for acquiring the user's health data and assessing their current health condition, means for acquiring the user's emotional data and assessing their current emotional state, means for comparing their past drinking history and physical condition data with their current health and emotional state to determine an appropriate amount of alcohol intake, means for providing advice on selecting the next drink based on the results of this assessment, and means for monitoring the user's health and emotional state in real time and issuing a warning if an abnormality is detected. This makes it possible to provide a safe and comfortable drinking experience while taking into consideration the user's health and emotional state comprehensively.

[1519] "Image recognition" is a technology that analyzes captured images and recognizes specific objects and characters.

[1520] "Health data" refers to collected and recorded information about an individual's health status, such as heart rate, sleep time, and amount of exercise.

[1521] "Emotion data" is data that indicates an emotional state such as joy, sadness, or anger, which is analyzed from the user's facial expressions and voice.

[1522] "Drinking history" refers to a record of the types and amounts of alcoholic beverages consumed in the past, as well as the dates and times of consumption.

[1523] "Physical condition data" is data that includes various information about the user's past and present health conditions.

[1524] "Alcohol intake" is a measure of the total amount of alcohol consumed within a specified period of time.

[1525] The "determination result" is judgment information about the safety of the drink provided by the server based on the user's health data, drinking history, and emotional data.

[1526] The "means for providing advice" is a function in which the system determines whether the next drink the user selects is appropriate or inappropriate, and makes suggestions, including alternatives, based on the results.

[1527] "Real-time monitoring" is a function that constantly monitors and instantly analyzes the user's health and emotional state.

[1528] The "means for issuing warnings" is a function that immediately sends a warning message when an abnormality occurs in the user's health or emotional state.

[1529] An embodiment of the present invention will now be described. This system provides appropriate advice based on the user's health condition, drinking status, and emotional state, and supports a safe and comfortable drinking experience.

[1530] First, the user takes a photo of a restaurant menu using a smartphone or smart glasses. The device then analyzes the menu image using image recognition technology to extract the drink type and details. This process uses image processing libraries such as OpenCV. The extracted information is then sent to a server and registered in a database.

[1531] Next, the device acquires the user's health data. This data includes information such as heart rate, sleep time, and exercise volume acquired from the smartwatch or smartphone. This data is collected periodically and used to evaluate the user's current health status. The evaluation results are sent to a server and stored in a database.

[1532] In addition, the device collects emotional data from the user's facial expressions and voice. This step uses a camera and microphone, and the emotion engine analyzes the data. The resulting emotional state (happiness, sadness, anger, etc.) is sent to the server and stored in a database.

[1533] The server compares the collected past drinking history, physical condition data, and emotional data with the user's current health and emotional state to determine the appropriate amount of alcohol intake. Specifically, it makes a specific judgment, such as whether the user can safely consume up to two beers or one cocktail. This judgment result is sent from the server to the device.

[1534] When a user selects or inputs the next drink to order, the device will display "OK" or "NG" based on the judgment from the server, indicating whether the drink is appropriate or inappropriate. For example, if a user tries to order a cocktail, the device will display "NG," but an alternative suggestion such as "One beer is OK" will be displayed.

[1535] The device also monitors the user's health and emotional state in real time. Data is collected and analyzed periodically. If the user's health or emotional state suddenly deteriorates, the device will immediately issue a warning. For example, if the user's heart rate reaches a dangerous level or if the user becomes extremely irritable, a warning message such as "Your current heart rate has reached a dangerous level. Please stop drinking and drink water" will be displayed.

[1536] For example, if a user tries to choose a cocktail, the device will judge it as "NG" and provide advice such as "One beer is OK." In this way, users can maintain a proper drinking pace and maintain their health and social relationships.

[1537] Here are some examples of prompts for generative AI models:

[1538] "Please provide a concrete example of a system that determines the appropriate amount of alcohol intake based on the user's current health status (heart rate, sleep time, amount of exercise) and emotional state (happiness, sadness, anger), and compares it with the user's current health and emotional state. Also, please display drink suggestions."

[1539] The above is an embodiment of the present invention. This system allows users to enjoy a safe and comfortable drinking experience while maintaining an appropriate alcohol intake amount according to their health and emotional state.

[1540] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1541] Step 1:

[1542] The user takes a photo of a restaurant menu using a smartphone or smart glasses.

[1543] Specific behavior:

[1544] A user takes a picture of a restaurant menu with a camera to obtain a menu image.

[1545] Input: Menu Image

[1546] Output: Photographed menu image data

[1547] Step 2:

[1548] The device uses image recognition technology to analyze the menu image captured and extract the drink type and details.

[1549] Specific behavior:

[1550] The device uses an image processing library such as OpenCV to analyze the image and recognize the menu items.

[1551] Input: Photographed menu image data

[1552] Output: Recognized drink type and details

[1553] Step 3:

[1554] The extracted information is sent to a server and registered in a database.

[1555] Specific behavior:

[1556] The drink information extracted from the device is sent to the server, which stores the information in a database.

[1557] Input: Recognized drink information

[1558] Output: Drink information stored in the database

[1559] Step 4:

[1560] The device acquires the user's health data.

[1561] Specific behavior:

[1562] The device collects health data such as heart rate, sleep time, and exercise volume from smartwatches and smartphones.

[1563] Input: Health data obtained from a smartwatch or smartphone

[1564] Output: Retrieved health data

[1565] Step 5:

[1566] The acquired health data is sent to a server and registered in a database.

[1567] Specific behavior:

[1568] The device sends the acquired health data to a server, which stores the data in a database.

[1569] Input: Acquired health data

[1570] Output: Health data stored in a database

[1571] Step 6:

[1572] The device collects emotional data from the user's facial expressions and voice.

[1573] Specific behavior:

[1574] The device uses a camera and microphone to capture the user's facial expressions and voice, which are then analyzed using an emotion engine.

[1575] Input: User's facial expressions and voice data

[1576] Output: Parsed emotion data

[1577] Step 7:

[1578] The analyzed emotional data is sent to a server and registered in a database.

[1579] Specific behavior:

[1580] The device sends the analyzed emotion data to the server, which stores the data in a database.

[1581] Input: Parsed emotion data

[1582] Output: Emotion data stored in a database

[1583] Step 8:

[1584] The server compares past drinking history, physical condition data, current health and emotional state, and determines the appropriate amount of alcohol intake.

[1585] Specific behavior:

[1586] The server retrieves past drinking history, physical condition data, current health and emotional state from the database and analyzes them using an algorithm.

[1587] Input: Drinking history, physical condition data, health status, emotional state stored in the database

[1588] Output: Determined appropriate alcohol intake

[1589] Step 9:

[1590] When the user enters or selects the next drink they want to order, the terminal will display an "OK" or "NG" result and alternatives.

[1591] Specific behavior:

[1592] The terminal receives the judgment result from the server and displays the status of "OK" or "NG" as well as alternatives for the next drink the user orders.

[1593] Input: Verification result from the server

[1594] Output: Display advice to the user when choosing a drink

[1595] Step 10:

[1596] The device monitors the user's health and emotional state in real time and issues an alert if it detects any abnormalities.

[1597] Specific behavior:

[1598] The device periodically collects and analyzes the user's health and emotional data, and displays a warning message in real time if any abnormalities are detected.

[1599] Input: User health and emotional data

[1600] Output: Warning message when an abnormality is detected

[1601] These are the specific processing steps of this system. Each step allows the user to enjoy an optimal drinking experience that is tailored to their own health and emotional state.

[1602] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1603] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1604] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1605] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1606] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1607] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1608] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1609] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1610] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1611] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1612] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1613] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1614] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1615] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1616] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1617] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1618] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1619] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1620] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1621] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1622] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1623] The following is further disclosed regarding the above embodiment.

[1624] (Claim 1)

[1625] A means for performing image recognition on a restaurant menu photographed by a user and extracting drink types and details;

[1626] a means for acquiring health data of a user and assessing a current health status;

[1627] A means for comparing past drinking history and physical condition data with current health status to determine appropriate alcohol intake;

[1628] A means for providing advice on the next drink selection based on the judgment result;

[1629] A means for monitoring the user's health status in real time and issuing a warning if an abnormality is detected;

[1630] A system including:

[1631] (Claim 2)

[1632] 2. The system according to claim 1, wherein a drink list is generated from a menu image taken by a user and registered in a database.

[1633] (Claim 3)

[1634] The system according to claim 1, wherein the user inputs or selects the next drink to order, and displays a judgment of "OK" or "NG" based on the judgment result.

[1635] "Example 1"

[1636] (Claim 1)

[1637] A means for performing image recognition on a restaurant menu photographed by a user and extracting the type and details of drinks;

[1638] a means for acquiring health data of a user and assessing a current health status;

[1639] A means for comparing past drinking history and physical condition data with current health status to determine appropriate alcohol intake;

[1640] A means for providing advice on the next beverage selection based on the judgment result;

[1641] A means for monitoring the user's health status in real time and issuing a warning if an abnormality is detected;

[1642] A system including:

[1643] (Claim 2)

[1644] 2. The system according to claim 1, wherein a beverage list is generated from a menu image taken by a user and registered in a database.

[1645] (Claim 3)

[1646] The system according to claim 1, wherein the user inputs or selects the next drink to order, and displays a judgment of "OK" or "NG" based on the judgment result.

[1647] "Application Example 1"

[1648] (Claim 1)

[1649] A means for performing image recognition on a restaurant menu photographed by a user and extracting drink types and details;

[1650] a means for acquiring health data of a user and assessing a current health status;

[1651] A means for comparing past drinking history and physical condition data with current health status to determine appropriate alcohol intake;

[1652] A means for providing advice on the next drink selection based on the judgment result;

[1653] A means for monitoring the user's health status in real time and issuing a warning if an abnormality is detected;

[1654] means for displaying said advice and warnings in real time using smart glasses;

[1655] A system including:

[1656] (Claim 2)

[1657] 2. The system according to claim 1, wherein a drink list is generated from a menu image taken by a user and registered in a database.

[1658] (Claim 3)

[1659] The system of claim 1, wherein the user inputs or selects the next drink to order, and the smart glasses display a judgment of "OK" or "NG" based on the judgment result.

[1660] "Example 2: Combining Emotion Engines"

[1661] (Claim 1)

[1662] A means for performing image recognition on a restaurant menu photographed by a user and extracting drink types and details;

[1663] a means for acquiring health data of a user and assessing a current health status;

[1664] A means for analyzing a user's facial expressions and voice to collect emotional data;

[1665] A means for comparing past drinking history, physical condition data, and emotional data with current health status to determine appropriate alcohol intake;

[1666] A means for providing advice on the next drink selection based on the judgment result;

[1667] A means for monitoring the user's health and emotional state in real time and issuing an alert when an abnormality is detected;

[1668] A system including:

[1669] (Claim 2)

[1670] 2. The system according to claim 1, wherein a drink list is generated from a menu image taken by a user and registered in a database.

[1671] (Claim 3)

[1672] The system according to claim 1, wherein the user inputs or selects the next drink to order, and based on the judgment result, displays a verdict of "OK" or "NG" and suggests an alternative drink.

[1673] "Application example 2 when combining emotion engines"

[1674] (Claim 1)

[1675] A means for performing image recognition on a restaurant menu photographed by a user and extracting drink types and details;

[1676] a means for acquiring health data of a user and assessing a current health status;

[1677] means for obtaining emotional data of a user and assessing the user's current emotional state;

[1678] A means for comparing past drinking history and physical condition data with current health and emotional state to determine an appropriate amount of alcohol intake;

[1679] A means for providing advice on the next drink selection based on the judgment result;

[1680] A means for monitoring the user's health and emotional state in real time and issuing an alert when an abnormality is detected;

[1681] A system including:

[1682] (Claim 2)

[1683] 2. The system according to claim 1, wherein a drink list is generated from a menu image taken by a user and registered in a database.

[1684] (Claim 3)

[1685] The system according to claim 1, wherein the user inputs or selects the next drink to order, and based on the judgment result, displays a judgment of "OK" or "NG" and alternatives. [Explanation of symbols]

[1686] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for performing image recognition on a restaurant menu photographed by a user and extracting drink types and details; a means for acquiring health data of a user and assessing a current health status; A means for comparing past drinking history and physical condition data with current health status to determine appropriate alcohol intake; A means for providing advice on the next drink selection based on the judgment result; A means for monitoring the user's health status in real time and issuing a warning if an abnormality is detected; A system including:

2. 2. The system according to claim 1, wherein a drink list is generated from a menu image taken by a user and registered in a database.

3. The system according to claim 1, wherein the user inputs or selects the next drink to order, and displays a judgment of "OK" or "NG" based on the judgment result.

Images