System

Abstract

A system is provided.SOLUTION: A system comprising: means for enabling natural interaction with a user using a generative artificial intelligence model for interactive interaction; means for recognizing gestures of the user and performing corresponding actions; and means for tracking the user using a remote camera and transmitting an interaction log and context information to a remote server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In an aging society, continuous communication and monitoring are important to reduce feelings of loneliness and the risk of dementia among the elderly. However, in reality, it is difficult for family members or caregivers to be nearby at all times, and conventional monitoring systems focus on one-way monitoring, which is insufficient to improve the quality of life of the elderly. Furthermore, while pets are known to have psychological healing effects, they are difficult to care for and place a significant burden on the elderly in particular. To solve these problems, a system that allows for more natural communication and monitoring that places less burden on the elderly is needed. [Means for solving the problem]

[0005] The present invention solves these problems by the following means. First, it provides a means for enabling natural conversation with the user by utilizing a generative artificial intelligence model that performs two-way dialogue. This allows elderly people to enjoy daily conversations and reduce feelings of loneliness. Furthermore, it provides a means for recognizing the user's gestures and performing corresponding actions, enabling natural interaction with the elderly. Furthermore, it provides a system that monitors the user using a remote camera and transmits interaction logs and situation information to a remote server. This remote monitoring function allows family members and caregivers to check the elderly's status even from a remote location and quickly respond to unexpected problems. Finally, it provides a means for detecting abnormalities in the elderly's behavior or physical condition based on the analysis results transmitted from the remote server and generating alerts, thereby supporting health management for the elderly. In this way, it provides a system that combines the soothing effects of a pet robot with advanced monitoring functions to improve the quality of life of the elderly.

[0006] "Two-way interaction" refers to communication in which the user and the system exchange information and respond in real time.

[0007] A "generative artificial intelligence model" is an algorithm or computer model that uses artificial intelligence technology to generate natural-looking linguistic responses and behavior, and is used for interacting with users and analyzing their behavior.

[0008] "Gesture recognition" refers to a technology that uses cameras and sensors to detect a user's hand or body movements, interpret them, and generate an appropriate response.

[0009] A "remote camera" is a camera used for observing and monitoring from a remote location, and has the ability to transmit video and images over a network.

[0010] "Monitoring" refers to the act of remotely monitoring the living conditions of elderly people, people with disabilities, and others, and providing necessary support and intervention.

[0011] An "interaction log" is data that records the history of contact and dialogue between the user and the robot, and is used to understand changes in the user's activities and physical condition.

[0012] "Analysis results" refers to the output of analysis conducted using artificial intelligence and machine learning technology based on data collected by the server.

[0013] An "alert" is a warning or notification issued when the system detects an abnormality or problem, and is used to prompt the user or caregiver to take prompt action.

[0014] "Movement" refers to the physical reaction or action that the robot takes in response to the user's gestures or instructions.

[0015] "Context" refers to the background information and context that are relevant in a dialogue or situation, and appropriate responses and actions are generated based on this. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a pet robot that monitors the elderly. This pet robot has three functions: two-way communication, gesture recognition, and remote camera monitoring. It can help relieve the elderly's sense of loneliness, prevent dementia, and provide remote monitoring. Details on how to implement this system are described below.

[0038] System Configuration

[0039] The system has three main components:

[0040] 1. Pet robot main body (terminal)

[0041] High-sensitivity microphone

[0042] speaker

[0043] camera

[0044] Contact Sensor

[0045] Processor equipped with generative artificial intelligence model

[0046] 2. Remote Server

[0047] Operation of generative artificial intelligence models

[0048] Database

[0049] Alert generation function

[0050] Dashboard (for families and caregivers)

[0051] 3. Users (elderly people and their families)

[0052] User interface (natural interaction with pet robots, gestures)

[0053] Program processing overview

[0054] The system programs work together to realize the functions of two-way dialogue, gesture recognition, and remote monitoring. Specifically, the following processes are performed:

[0055] Two-way dialogue

[0056] 1. The user speaks

[0057] The user talks to the robot, asking questions such as "How was your day?"

[0058] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[0059] 2. Sending and analyzing audio data

[0060] The device transmits the captured audio data to the server in real time.

[0061] The server analyzes the voice data, generates an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[0062] 3. Executing the response

[0063] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[0064] Example: When a user asks, "How are you today?", the robot responds, based on the analysis results from the server, with, "I'm feeling great today!" At this time, the robot also performs actions such as shaking its body.

[0065] Gesture Recognition

[0066] 1. Performing a gesture

[0067] The user makes a gesture such as waving or pointing.

[0068] The device's camera captures the user's gestures in real time.

[0069] 2. Sending and analyzing gesture data

[0070] The device transmits the captured gesture data to the server.

[0071] The server analyzes the gesture, understands the user's intention, generates an appropriate response, and sends the resulting data to the device.

[0072] 3. Response Execution

[0073] The device performs an appropriate action (for example, waving back) based on the received response data.

[0074] Example: When a user waves their hand, the robot recognizes the movement with a camera, and based on the analysis results of the server, the robot waves back.

[0075] Remote monitoring

[0076] 1. Touch detection

[0077] When the user touches the robot, the device's contact sensor detects the touch.

[0078] 2. Sending log data

[0079] The device collects interaction data and periodically sends the log to a server.

[0080] 3. Data storage and analysis

[0081] The server stores the received interaction logs in a database and analyzes them. If an abnormality is detected, an alert is generated.

[0082] 4. Display on the dashboard

[0083] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers, allowing them to check on the elderly person's condition remotely.

[0084] Example: When a user strokes the robot's back, the interaction data is sent to a server and can be viewed by family members on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[0085] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. This also allows family members and caregivers to remotely monitor the elderly with peace of mind.

[0086] The processing flow will be explained below.

[0087] Two-way interaction processing steps

[0088] Step 1:

[0089] The user talks to the robot, asking questions such as "How was your day?"

[0090] Step 2:

[0091] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[0092] Step 3:

[0093] The device transmits the acquired voice data to the server in real time.

[0094] Step 4:

[0095] The server analyzes the received voice data and uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue.

[0096] Step 5:

[0097] The server transmits the generated response data to the terminal.

[0098] Step 6:

[0099] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[0100] Examples:

[0101] When a user says, "How are you today?", the device captures the audio and sends it to the server. The server generates a reply, "I'm feeling great today!", which the device then relays to the user.

[0102] Gesture Recognizer Processing Steps

[0103] Step 1:

[0104] The user makes a gesture (e.g., a wave) to the robot.

[0105] Step 2:

[0106] The device's camera captures the user's gestures in real time.

[0107] Step 3:

[0108] The device transmits the captured gesture data to the server.

[0109] Step 4:

[0110] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[0111] Step 5:

[0112] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[0113] Step 6:

[0114] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[0115] Examples:

[0116] When a user waves to the robot, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[0117] Remote monitoring function processing steps

[0118] Step 1:

[0119] The user touches the robot (for example, stroking its back).

[0120] Step 2:

[0121] The device's contact sensor detects contact and collects the data.

[0122] Step 3:

[0123] The device periodically transmits the acquired contact data to the server.

[0124] Step 4:

[0125] The server stores the received interaction logs in a database.

[0126] Step 5:

[0127] The server analyzes the stored data and generates an alert if an anomaly is detected.

[0128] Step 6:

[0129] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[0130] Examples:

[0131] When a user strokes the robot's back, the device detects the touch with a contact sensor and sends the data to a server, which analyzes and stores the data and displays it on a dashboard that family members can view remotely.

[0132] The specific processing steps for each function have been explained above. The specific operations performed in each step enable this pet robot to alleviate the sense of loneliness of elderly people and provide daily monitoring.

[0133] Example 1

[0134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Current technology has not yet fully addressed the challenges of simultaneously alleviating loneliness among the elderly, preventing dementia, and remotely monitoring them. In particular, there is a lack of systems that allow elderly people to enjoy natural conversations and interact through gestures, while also allowing remote family members and caregivers to monitor the elderly's status in real time. Current technology has difficulty simultaneously meeting these multiple needs, and therefore has not yet succeeded in increasing the sense of security for the elderly and their families.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0137] In this invention, the server includes a two-way dialogue means for enabling natural dialogue with the user, a gesture recognition means for recognizing the user's gestures and performing corresponding actions, a remote monitoring means for monitoring the user's status using a remote camera and transmitting interaction logs and situation information to a remote server, a data transmission and response means for transmitting voice data and gesture data to the server and executing appropriate responses and response actions based on the analysis results, a voice and gesture analysis means equipped with a generative artificial intelligence model for analyzing the data received by the remote server, generating responses based on the analysis results, and transmitting the responses to the terminal, an alert generation means for analyzing the accumulated interaction logs and situation information and generating an alert if an abnormality is detected, and a dashboard means for displaying the analysis results and alert information. This makes it possible to simultaneously alleviate loneliness in the elderly, prevent dementia, and remotely monitor them.

[0138] The "two-way dialogue means" is a technology that uses voice recognition and voice synthesis to achieve two-way communication with the user in order to realize natural dialogue.

[0139] A "gesture recognition means" is a technology that uses a camera or other sensor to capture a user's physical movements, such as hand gestures or pointing, and analyzes those movements.

[0140] "Remote monitoring means" is a technology that uses remote cameras and other sensors to monitor the user's situation and transmits the data to a remote server.

[0141] "Data transmission and response means" refers to a technology that transmits voice data and gesture data collected from the terminal to a server in real time, receives analysis results from the server, and executes appropriate responses or response actions.

[0142] "Voice and gesture analysis means" refers to a technology that uses a generative artificial intelligence model installed on the server to analyze received voice and gesture data and generate a response based on the analysis results.

[0143] The "alert generation means" is a technology that analyzes accumulated interaction logs and situational information, and generates warning messages or notifications when an abnormality is detected.

[0144] "Dashboard means" is a technology that visually displays analysis results and alert information so that family members and caregivers can check them in real time.

[0145] This invention is a pet robot system for monitoring elderly people, which has three main functions: two-way interaction, gesture recognition, and remote camera monitoring. The system uses hardware such as a high-sensitivity microphone, speaker, camera, contact sensor, and processor equipped with a generative artificial intelligence model. Details for implementing this system are described below.

[0146] System Configuration

[0147] The system consists of three main components:

[0148] 1. Pet robot main body (terminal)

[0149] - High sensitivity microphone

[0150] - Speaker

[0151] - Camera

[0152] - Contact Sensor

[0153] - Processor equipped with generative artificial intelligence model

[0154] 2. Remote Server

[0155] - Operation of generative artificial intelligence models

[0156] - Database

[0157] - Alert generation function

[0158] - Dashboard (for families and caregivers)

[0159] 3. Users (elderly people and their families)

[0160] - User interface (natural interaction with pet robots, gestures)

[0161] Two-way dialogue

[0162] When a user speaks to the robot, the device's high-sensitivity microphone captures the voice. For example, if the user asks, "How was your day today?", the voice data is sent to the server in real time. The server analyzes the voice data, generates an appropriate response based on the context of the conversation, and sends this response data to the device. The device then plays back the received response data with natural voice and movements to respond to the user. For example, if the user asks, "How are you feeling today?" the robot will respond, "I'm feeling great today!" At this time, the robot will also perform movements such as shaking its body appropriately.

[0163] Gesture Recognition

[0164] When a user makes a gesture such as waving or pointing, the device's camera captures the user's gesture in real time. The device sends the captured gesture data to a server, which then analyzes the gesture. Based on the analysis results, the server understands the user's intention, generates an appropriate response action, and sends the resulting data to the device. The device then performs an appropriate action (for example, waving back) based on the received response data. As a specific example, when a user waves their hand, a robot recognizes the movement and waves back.

[0165] Remote monitoring

[0166] When a user touches the robot, the device's contact sensor detects the contact. The device collects the detected contact data and periodically sends the log to the server. The server stores the received contact log in a database and analyzes it. If an abnormality is detected, an alert is generated and family members or caregivers are notified. The server also displays the analysis results and contact log on a dashboard in real time, allowing the elderly person's condition to be checked remotely. For example, when a user strokes the robot's back, the contact data is sent to the server, and family members can check it on the remote dashboard. If an abnormality is detected, an alert is generated and a notification is sent.

[0167] Specific hardware and software names to be used

[0168] High-sensitivity microphone: General high-sensitivity microphone

[0169] Speaker: General speaker

[0170] Camera: General camera

[0171] Contact sensor: General contact sensor

[0172] Processor with generative artificial intelligence model: General processor

[0173] Remote Server: General Server

[0174] Generative Artificial Intelligence Model: General AI Model

[0175] Database: General database

[0176] Dashboard: General dashboard software

[0177] Specific examples of prompts to input to generative AI models

[0178] Speech analysis prompt: "What's the weather like tomorrow?"

[0179] Gesture analysis prompt: "Your hand wave has been recognized."

[0180] Interaction Log Prompt: "Back-pat sequence occurred"

[0181] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0182] Two-way dialogue

[0183] Step 1: User speaks

[0184] The user can ask the robot a question or start a conversation, for example, "How was your day today?"

[0185] Input: User's voice

[0186] Output: Audio data

[0187] Specific operation: When the user speaks to the pet robot, the voice is captured by a highly sensitive microphone.

[0188] Step 2: The device receives and transmits the audio data

[0189] The device collects the captured audio data.

[0190] Input: Captured audio data

[0191] Output: Audio data for transmission

[0192] What it does: It compresses audio data in real time and sends it securely to the server using TLS encryption.

[0193] Step 3: The server analyzes the voice data and generates a response

[0194] The server analyzes the received voice data, performs speech recognition, and converts the speech to text using a generative AI model.

[0195] Input: Received audio data

[0196] Output: Contextual response text

[0197] Specific operation: After converting the voice data into text, it performs context analysis and generates an appropriate response, such as "I'm feeling great today!"

[0198] Step 4: The device responds

[0199] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[0200] Input: Reply text

[0201] Output: Voice and action responses

[0202] Specific behavior: The response data is converted into voice and played back to the user. At the same time, the robot performs natural movements such as shaking its body.

[0203] Gesture Recognition

[0204] Step 1: The user makes a gesture

[0205] The user makes a gesture such as waving or pointing.

[0206] Input: User's physical movements

[0207] Output: Camera video data

[0208] Specific actions: When a user makes a gesture, the action is captured in real time by the camera.

[0209] Step 2: The device receives and sends gesture data

[0210] The device collects the captured gesture data.

[0211] Input: Captured video data

[0212] Output: Gesture data to send

[0213] How it works: Video data is compressed in real time and sent securely to the server using TLS encryption.

[0214] Step 3: The server analyzes the gesture data and generates a response

[0215] The server analyzes the received gesture data and understands the user's intention using a machine learning model.

[0216] Input: Received gesture data

[0217] Output: Response behavior data based on user intention

[0218] Specific Actions: Analyze gesture data and generate appropriate response actions, such as waving back.

[0219] Step 4: The device performs the response action

[0220] The terminal performs an appropriate operation based on the received response operation data.

[0221] Input: Response action data

[0222] Output: Actual device behavior

[0223] Specific actions: Based on the received action data, the robot will, for example, wave its hand back.

[0224] Remote monitoring

[0225] Step 1: Touch detection

[0226] When the user touches the robot, the device's contact sensor detects the touch.

[0227] Input: Physical touch

[0228] Output: Interaction data

[0229] Specific operation: When a user touches the robot, the contact sensor detects the touch and generates data.

[0230] Step 2: Device collects and sends log data

[0231] The device collects detected contact data and periodically sends the log data to a server.

[0232] Input: Interaction data

[0233] Output: Log data for sending

[0234] What it does: Collected log data is compressed in real time and sent securely to the server using TLS encryption.

[0235] Step 3: The server stores and analyzes the data

[0236] The server stores the received interaction logs in a database and analyzes them.

[0237] Input: Received log data

[0238] Output: Analysis results and anomaly detection results

[0239] Specific operation: The saved log data is analyzed periodically to check for any abnormalities. For example, if the frequency of contact decreases drastically, it will be recognized as an abnormality.

[0240] Step 4: View the server in the dashboard

[0241] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers.

[0242] Input: Analysis results and log data

[0243] Output: Visual information on a dashboard

[0244] Specific operation: Analysis results and log data are displayed on the dashboard, and if an abnormality is detected, an alert is displayed in real time. Visual information is also provided so that family members and caregivers can check the user's condition.

[0245] Specific examples of prompts to input to generative AI models

[0246] Speech analysis prompt: "What's the weather like tomorrow?"

[0247] Gesture analysis prompt: "Your hand wave has been recognized."

[0248] Interaction Log Prompt: "Back-pat sequence occurred"

[0249] (Application example 1)

[0250] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0251] Problems faced by elderly people when using food delivery services include the burden of completing the complicated ordering process on their own, the difficulty of responding quickly and safely to the delivery person's arrival, and the uncertainty of having family members or caregivers supervise the delivery remotely. There is a need to solve these problems and provide a more secure and comfortable food delivery experience.

[0252] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0253] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for supporting food delivery orders, and means for confirming delivery and generating alerts to notify the remote server. This reduces the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[0254] "Two-way dialogue" is a communication method that uses a generative artificial intelligence model that recognizes speech and generates responses to enable natural dialogue with the user.

[0255] A "generative artificial intelligence model" is an artificial intelligence algorithm that analyzes voice data and generates appropriate responses.

[0256] "Gesture recognition" is a technology that analyzes a user's actions captured by a camera, understands their meaning, and performs a corresponding action.

[0257] A "remote camera" is a camera device used to monitor the user's situation from a remote location.

[0258] An "interaction log" is information that records data when a user interacts with a robot and is used for later analysis.

[0259] A "remote server" is a server device in a remote location that has high-performance computing capabilities, analyzes data in real time, and stores necessary information.

[0260] "Food delivery" is a service that delivers food to a specific location.

[0261] "Ordering support" is a function that provides assistance through voice dialogue, etc., to make it easier for users to order food delivery.

[0262] "Delivery confirmation" is a way to confirm that the delivery person has arrived and to assist with safe pickup.

[0263] "Alert generation" is a function that generates a notification when an abnormality is detected.

[0264] This invention provides a pet robot system for supporting food delivery services for the elderly. The system includes functions such as two-way interaction, gesture recognition, and remote monitoring, allowing users to use food delivery services more safely and comfortably.

[0265] System Configuration

[0266] The system has three main components:

[0267] 1. Pet robot main body (terminal):

[0268] High-sensitivity microphone

[0269] speaker

[0270] camera

[0271] Contact Sensor

[0272] Processor equipped with generative artificial intelligence model

[0273] 2. Remote Server:

[0274] Operation of generative artificial intelligence models

[0275] Database

[0276] Alert generation function

[0277] Dashboard (for families and caregivers)

[0278] 3. Users (elderly people and their families):

[0279] User interface (natural interaction with pet robots, gestures)

[0280] Program processing overview

[0281] The system's programs work together to realize the functions of food delivery order support, delivery confirmation, and safety confirmation. Specifically, the following hardware and software are used to perform the processing.

[0282] Food delivery ordering assistance

[0283] 1. User: An elderly person speaks to their pet robot, saying, "I want to order dinner."

[0284] 2. Terminal: Captures audio using a sensitive microphone and transmits the data to a remote server.

[0285] 3. Remote server: A generative AI model analyzes the speech and generates an appropriate response, which is returned to the device.

[0286] 4. Terminal: Asks "Which restaurant would you like?" The user responds to proceed with the order.

[0287] Delivery Confirmation

[0288] 1. Terminal: When the delivery person arrives, the camera recognizes their arrival and sends the data to a remote server.

[0289] 2. Remote server: Notifies family members or caregivers based on the received data.

[0290] 3. Terminal: Respond appropriately to the delivery person by saying "Thank you for your hard work."

[0291] Safety confirmation

[0292] 1. User: When receiving meals, the robot will provide support to ensure that the elderly person receives the meal safely.

[0293] 2. Terminal: Capture the situation after receiving the item with a camera and send it to a remote server.

[0294] 3. Remote server: Based on the analysis results, if an abnormality is detected, an alert is generated to notify family members or caregivers.

[0295] Examples of concrete examples and prompts

[0296] As a specific scenario for operating this system, consider the case where elderly people use food delivery services.

[0297] Examples:

[0298] When an elderly person says to the robot, "I'd like to order sushi," the robot responds, "Which sushi restaurant would you like?" If the elderly person answers, "Sushida," the robot responds, "I'll open the Sushida menu," and proceeds with the order.

[0299] Example prompt sentence:

[0300] User: I'd like to order sushi.

[0301] Robot: Which sushi restaurant would you like?

[0302] User: Sushida.

[0303] Robot: Open the Sushida menu.

[0304] This will reduce the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[0305] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0306] Step 1:

[0307] The user speaks to the robot, saying, "I'd like to order dinner." The input data includes the user's voice data. The device uses a highly sensitive microphone to capture this voice data, and the voice data is acquired as input data.

[0308] Step 2:

[0309] The device transmits the captured voice data in real time to a remote server, where the voice data is processed for transmission. The remote server analyzes the voice data received as input. A generative AI model converts the voice to text, analyzes the context, and generates an appropriate response. In this process, the voice data is converted to text data and analyzed.

[0310] Step 3:

[0311] The remote server sends the generated response data to the device. The device then synthesizes the received response data into voice and responds to the user. Voice synthesis technology is used here, and text data is converted back into voice data. Specifically, the device asks the user, "Which restaurant would you like?"

[0312] Step 4:

[0313] The user responds with "Sushida." The user's voice data is again captured as input data, and the device captures and sends it to the remote server. As in the previous step, the server converts the voice data into text data and generates an analysis result.

[0314] Step 5:

[0315] The remote server generates an appropriate response, such as "Sushida's menu will be displayed," and sends it to the device. The device then synthesizes this response and responds to the user, for example, "Sushida's menu will be displayed."

[0316] Step 6:

[0317] When the delivery person arrives, the device's camera captures the video data, which is then sent as input data to a remote server, which analyzes the video data and confirms the delivery person's arrival.

[0318] Step 7:

[0319] The remote server then notifies the family or caregiver of the results of the check. The notification includes information about the delivery person's arrival and status. At the same time, the device responds appropriately to the delivery person by saying, "Thank you for your hard work."

[0320] Step 8:

[0321] When the user receives the meal, the device's camera and contact sensors monitor the situation. The video data and contact log at the time of receipt are sent as input data to a remote server. The remote server analyzes this data and checks for any abnormalities.

[0322] Step 9:

[0323] If the remote server detects any abnormalities based on the analysis results, it generates an alert and notifies the family or caregiver, allowing them to confirm safety. If everything is normal, it notifies them that the delivery has been completed successfully.

[0324] These steps will create a system that allows seniors to use food delivery services safely and easily.

[0325] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0326] This invention relates to a pet robot for elderly care that combines an emotion engine. This pet robot is equipped with two-way communication, gesture recognition, remote camera monitoring, and emotion recognition functions, helping to alleviate feelings of loneliness in elderly people, preventing dementia, and remotely monitoring them. Details for implementing this system are described below.

[0327] System Configuration

[0328] The system comprises four main components:

[0329] 1. Pet robot main body (terminal)

[0330] High-sensitivity microphone

[0331] speaker

[0332] camera

[0333] Contact Sensor

[0334] Processor equipped with generative artificial intelligence model

[0335] Emotion Engine

[0336] 2. Remote Server

[0337] Operation of generative artificial intelligence models

[0338] Database

[0339] Alert generation function

[0340] Dashboard (for families and caregivers)

[0341] 3. Users (elderly people and their families)

[0342] User interface (natural interaction with pet robots, gestures)

[0343] 4. Emotion Engine

[0344] Recognizing emotions from the user's voice and facial expressions

[0345] Program processing overview

[0346] The system programs work together to realize the functions of two-way dialogue, gesture recognition, remote monitoring, and emotion recognition. Specifically, the following processes are performed:

[0347] Two-way dialogue

[0348] 1. The user speaks

[0349] The user talks to the robot, asking questions such as "How was your day?"

[0350] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[0351] 2. Sending and analyzing audio data

[0352] The device transmits the acquired voice data to the server in real time.

[0353] The server analyzes the voice data and recognizes the user's emotions using an emotion engine.

[0354] 3. Response Generation and Execution

[0355] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[0356] The device responds to the user by expressing the received response data in voice and motion, thereby realizing a dialogue that responds to the user's emotions.

[0357] Examples:

[0358] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" If the emotion engine recognizes the emotion "lonely" from the tone of the user's voice, the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[0359] Gesture Recognition

[0360] 1. Performing a gesture

[0361] The user makes a gesture (e.g., a wave) to the robot.

[0362] The device's camera captures the user's gestures in real time.

[0363] 2. Sending and analyzing gesture data

[0364] The device transmits the captured gesture data to the server.

[0365] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[0366] 3. Response Execution

[0367] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[0368] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[0369] Examples:

[0370] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[0371] Remote monitoring

[0372] 1. Touch detection

[0373] The user touches the robot (for example, stroking its back).

[0374] The device's contact sensor detects contact and collects the data.

[0375] 2. Sending log data

[0376] The device periodically transmits the acquired contact data to the server.

[0377] 3. Data storage and analysis

[0378] The server stores the received interaction logs in a database.

[0379] 4. Displaying analysis results and generating alerts

[0380] The server analyzes the stored data and generates an alert if an anomaly is detected.

[0381] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[0382] Examples:

[0383] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, and displays it on a dashboard that family members can view remotely. If an abnormality is detected, an alert is generated and notified.

[0384] Emotion recognition function

[0385] 1. Recognizing the user's voice and facial expressions

[0386] The device captures the user's voice and facial expressions through a camera and microphone.

[0387] The emotion engine recognizes the user's emotions from the captured data.

[0388] 2. Transmission and analysis of emotional data

[0389] The device transmits the recognized emotion data to the server.

[0390] The server analyzes the emotional data to adjust dialogue and behavior.

[0391] 3. Generating and executing responses according to emotions

[0392] The server generates appropriate responses and actions based on the emotion data and sends that data to the terminal.

[0393] The terminal executes the received response data and expresses an appropriate emotion to the user.

[0394] Examples:

[0395] If a user says, "I'm feeling lonely after being at home all day today," the emotion engine recognizes the emotion of "lonely." The server generates data to respond in a cheerful tone, saying, "I see you're lonely. Let's play together!", and the device executes that response.

[0396] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. In addition, the addition of an emotion recognition function further improves emotional care for the elderly, allowing family members and caregivers to remotely monitor the elderly with peace of mind.

[0397] The processing flow will be explained below.

[0398] Two-way interaction processing steps

[0399] Step 1:

[0400] The user talks to the robot, for example, asking, "How was your day?"

[0401] Step 2:

[0402] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[0403] Step 3:

[0404] The device transmits the acquired voice data to the server in real time.

[0405] Step 4:

[0406] The server analyzes the received voice data using a generative artificial intelligence model to understand the content and context of the speech.

[0407] Step 5:

[0408] The server uses an emotion engine to analyze the user's tone of voice and phrasing to recognize the user's emotions.

[0409] Step 6:

[0410] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the conversation and the emotion recognition results, and sends the response data to the terminal.

[0411] Step 7:

[0412] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[0413] Examples:

[0414] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" At the same time, the emotion engine recognizes "lonely" from the tone of the user's voice, and the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[0415] Gesture Recognizer Processing Steps

[0416] Step 1:

[0417] The user makes a gesture (e.g., a wave) to the robot.

[0418] Step 2:

[0419] The device's camera captures the user's gestures in real time.

[0420] Step 3:

[0421] The device transmits the captured gesture data to the server.

[0422] Step 4:

[0423] The server analyzes the received gesture data using machine learning models to understand the meaning of the gesture.

[0424] Step 5:

[0425] The server generates an appropriate response action based on the user's gesture and transmits the response data to the terminal.

[0426] Step 6:

[0427] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[0428] Examples:

[0429] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[0430] Remote monitoring function processing steps

[0431] Step 1:

[0432] The user touches the robot (for example, stroking its back).

[0433] Step 2:

[0434] The device's contact sensor detects contact and collects the data.

[0435] Step 3:

[0436] The device periodically transmits the acquired contact data to the server.

[0437] Step 4:

[0438] The server stores the received interaction logs in a database.

[0439] Step 5:

[0440] The server analyzes the stored data to detect abnormalities in the elderly person's behavior and physical condition, and if an abnormality is detected, it generates an alert.

[0441] Step 6:

[0442] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[0443] Examples:

[0444] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, which family members can view on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[0445] Emotion Recognition Processing Steps

[0446] Step 1:

[0447] The user speaks to the robot or shows facial expressions.

[0448] The device's camera captures the user's facial expressions, and a highly sensitive microphone captures their voice.

[0449] Step 2:

[0450] The terminal transmits the user's voice data and facial expression data to the server in real time.

[0451] Step 3:

[0452] The server analyzes the received voice data and facial expression data and recognizes the user's emotions (e.g., joy, sadness, anger) using an emotion engine.

[0453] Step 4:

[0454] The server generates appropriate responses and actions based on the emotion recognition results and sends the data to the terminal.

[0455] Step 5:

[0456] The terminal executes the received response data and gives a natural response that matches the user's emotions.

[0457] Examples:

[0458] If the user says, "I've been lonely at home all day today," the emotion engine recognizes the emotion "lonely." The server generates a response, "I see you've been lonely. Let's play together!", and the robot conveys this to the user in a cheerful tone.

[0459] The specific processing steps for each function, including the emotion engine, have been explained above. The specific actions performed at each step enable this pet robot to alleviate the sense of loneliness felt by the elderly and provide daily monitoring.

[0460] Example 2

[0461] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0462] There is a need for systems that can prevent elderly people from feeling lonely and developing dementia, and that can also monitor them remotely. To alleviate the feelings of loneliness and anxiety that elderly people experience in their daily lives, natural dialogue that responds to emotions and interactive responses through gesture recognition are necessary. Remote monitoring functions are also required to make it easier for family members and caregivers to understand the condition of the elderly. Existing systems lack comprehensive solutions to meet these needs.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0464] In this invention, the server includes means for enabling natural dialogue with the user using an artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and situation information to the remote server, means for recognizing the user's voice and facial expressions and analyzing emotions, and means for generating appropriate responses based on the analyzed emotional data and providing them to the user. This makes it possible to reduce loneliness among the elderly, prevent dementia, and even monitor them remotely.

[0465] An "artificial intelligence model" is a program designed to naturally interact with users, using speech recognition and natural language processing technology to understand the meaning of what the user says and generate appropriate responses.

[0466] "Gesture recognition" is a technology that captures a user's actions (such as waving a hand) using a sensor such as a camera, and analyzes and determines the meaning of the action.

[0467] A "remote camera" is a device that captures video from a remote location and transmits it to a remote server in real time, and is used to monitor users.

[0468] An "interaction log" is a record of physical contact between the user and the robot (such as a back pat) and is sent to a remote server.

[0469] An "emotion engine" is a technology that analyzes a user's voice and facial expression data to recognize the user's emotions at that time (for example, joy, sadness, loneliness, etc.).

[0470] "Response generation based on analysis results" is a technology that generates appropriate responses and actions based on the results of analyzing the user's statements, gestures, and emotional data.

[0471] "Alert generation" is a technology that notifies family members or caregivers from a remote server when an abnormality is detected in the user's behavior or condition.

[0472] "Users" refer to the elderly people who use the system, as well as their families and caregivers, who interact with the robot through voice and gestures.

[0473] This invention is a system that combines an emotion engine with a pet robot for elderly care, alleviating the sense of loneliness of the elderly, preventing dementia, and remotely monitoring them. The components of this system include the pet robot itself, a remote server, users (elderly people and their families), and the emotion engine.

[0474] System Configuration

[0475] 1. Pet robot main body (terminal)

[0476] High-sensitivity microphone: Captures user voice in real time.

[0477] Speaker: Replies sent from the server are spoken to the user.

[0478] Camera: Captures user gestures and facial expressions and provides data for analysis.

[0479] Contact sensor: Acquires contact data when the user touches the robot.

[0480] Processor equipped with generative artificial intelligence model: Performs voice analysis, gesture recognition, and emotion analysis.

[0481] Emotion engine: Technology for recognizing and analyzing user emotions.

[0482] 2. Remote Server

[0483] Operation of generative artificial intelligence models: Analyze acquired data and generate appropriate responses and actions.

[0484] Database: Stores interaction data and emotion analysis results.

[0485] Alert generation function: If an abnormality is detected, an alert is generated to notify family members or caregivers.

[0486] Dashboard (for family and caregivers): Displays analysis results and interaction logs, allowing users to check their condition.

[0487] 3. Users (elderly people and their families)

[0488] User Interface: Interact with the pet robot through natural dialogue and gestures.

[0489] 4. Emotion Engine

[0490] Recognizing emotions from voice and facial expressions: Analyze the user's voice tone and facial expressions to generate emotional data.

[0491] System program processing overview

[0492] The system works to recognize the user's voice and gestures, generate appropriate responses and actions based on them, and realize two-way dialogue according to emotions.

[0493] For example, when a user asks, "How was your day?", the device's high-sensitivity microphone captures the voice and sends the voice data to the server. The server uses a generative artificial intelligence model to analyze the voice data, and an emotion engine recognizes the user's emotion. The server then generates an appropriate response and sends it to the device. The response, "I'm feeling great today!", is heard through the device's speaker.

[0494] When a user makes a waving gesture, the device's camera captures the gesture and sends it to the server. The server analyzes the gesture, generates the necessary response action, and sends it to the device. The device then performs the waving action.

[0495] Furthermore, when the user touches the robot, the device's contact sensor captures the data and sends it to the server, where it is stored in a database and can be viewed by family members or caregivers via a dashboard. If an abnormality is detected, an alert is generated and a remote notification is sent.

[0496] Examples of prompt statements

[0497] "Please explain in detail the process by which an elderly care robot recognizes emotions from the user's voice and facial expressions and returns an appropriate response."

[0498] In this way, each function of this system works together to reduce loneliness and provide care for the elderly. In addition, the addition of an emotion recognition function improves emotional care for the elderly and allows family members and caregivers to remotely monitor them with peace of mind.

[0499] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0500] Handling two-way interactions

[0501] Step 1: Capture audio

[0502] Input: The user speaks to the robot (e.g., "How was your day?").

[0503] The device uses a high-sensitivity microphone to capture the user's voice in real time.

[0504] Output: The audio data is captured on the device.

[0505] Step 2: Sending audio data

[0506] Input: Audio data captured on the device.

[0507] The device transmits the captured audio data to the server in real time.

[0508] Output: The audio data is sent to the server.

[0509] Step 3: Voice analysis and emotion recognition

[0510] Input: The audio data sent to the server.

[0511] The server uses a generative artificial intelligence model to analyze the voice data and understand what the user is saying.

[0512] The server uses an emotion engine to recognize the user's emotions (e.g., joy, sadness, loneliness) from the voice.

[0513] Output: The analyzed speech content and emotion data are obtained.

[0514] Step 4: Response generation and speech output

[0515] Input: Parsed speech and sentiment data.

[0516] The server generates an appropriate response based on the analysis results.

[0517] The server generates a response and sends it to the terminal.

[0518] The device provides a response to the user through the speaker (e.g., "I'm feeling great today!").

[0519] Output: A voice is output in response to the user.

[0520] Gesture Recognition Processing

[0521] Step 1: Gesture Capture

[0522] Input: The user makes a gesture towards the robot (e.g., waving).

[0523] The device uses a camera to capture the user's gestures in real time.

[0524] Output: Gesture data is captured on the device.

[0525] Step 2: Sending gesture data

[0526] Input: Gesture data captured on the device.

[0527] The device transmits the captured gesture data to the server.

[0528] Output: Gesture data is sent to the server.

[0529] Step 3: Gesture analysis

[0530] Input: The gesture data sent to the server.

[0531] The server analyzes the received gesture data and understands its meaning (e.g., greeting, request).

[0532] Output: The parsed meaning of the gesture is obtained.

[0533] Step 4: Generate and execute response actions

[0534] Input: Parsed gesture meaning.

[0535] Based on the analysis results, the server generates an appropriate response action and sends it to the device (e.g., waving back).

[0536] The terminal performs an action based on the received response action data.

[0537] Output: The action corresponding to the gesture is performed.

[0538] Remote monitoring processing

[0539] Step 1: Touch detection

[0540] Input: The user touches the robot (e.g., pats its back).

[0541] The device's contact sensor detects this contact and captures the data.

[0542] Output: Interaction data is captured on the device.

[0543] Step 2: Sending log data

[0544] Input: Interaction data captured on the device.

[0545] The device periodically transmits the acquired contact data to the server.

[0546] Output: Interaction data is sent to the server.

[0547] Step 3: Data storage and analysis

[0548] Input: Interaction data sent to the server.

[0549] The server stores the received interaction data in a database.

[0550] The server analyzes the stored data and generates alerts if anomalies are detected.

[0551] Output: Saved interaction data and analysis results are obtained.

[0552] Step 4: Viewing analysis results and generating alerts

[0553] Input: Saved interaction data and its analysis results.

[0554] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[0555] The server generates an alert and notifies you if an abnormality is detected.

[0556] Output: Data is displayed on the dashboard and alerts are sent.

[0557] Emotion recognition processing

[0558] Step 1: Capture your voice and facial expressions

[0559] Input: User's voice and facial expressions.

[0560] The device captures the user's voice and facial expressions in real time using a camera and microphone.

[0561] Output: Voice data and facial expression data are acquired on the terminal.

[0562] Step 2: Send and analyze emotion data

[0563] Input: Voice data and facial expression data acquired on the device.

[0564] The terminal transmits the acquired data to the server.

[0565] The server analyzes the received data using an emotion engine and recognizes the user's emotions.

[0566] Output: The analyzed emotion data is obtained.

[0567] Step 3: Generate and execute a response

[0568] Input: Parsed emotion data.

[0569] The server generates appropriate responses and actions based on the emotional data and sends them to the terminal.

[0570] The terminal executes the received response data and expresses an appropriate emotion to the user.

[0571] Output: A response according to the emotion is provided to the user.

[0572] The above is the specific flow of the program processing for this system. At each step, appropriate data processing and calculations are performed based on the input data, and the output is used in the processing of the next step. This enables interaction according to the user's condition and emotions, realizing a system that can alleviate the loneliness of elderly people while also allowing family members and caregivers to monitor them remotely.

[0573] (Application example 2)

[0574] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0575] The goal is to eliminate the difficulties that elderly people face when using food delivery services independently, ease their sense of loneliness, and enable them to order meals with peace of mind, check delivery status, and easily manage meal records. Another goal is to provide emotional care for elderly people by utilizing emotion recognition functions and realize appropriate remote monitoring.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0577] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for assisting the user in selecting a menu when ordering a meal, means for presenting the user with the progress of delivery, and means for automatically managing meal records and sending the record information to the remote server. This makes food delivery services easier and more secure for the elderly, alleviating their sense of loneliness and enabling remote monitoring.

[0578] "Two-way dialogue" is a mechanism in which users and systems communicate with each other in real time using voice and text, enabling natural conversation.

[0579] A "generative artificial intelligence model" is a type of artificial intelligence that learns from large amounts of data and has the ability to engage in natural conversations, recognize emotions, and generate responses.

[0580] "Gesture recognition" is a technology that uses cameras and sensors to detect a user's body movements and hand gestures and respond accordingly.

[0581] A "remote camera" is a camera installed in a physically separate location, and is a device that allows you to view images from a remote location.

[0582] "User monitoring" is a function in which the system monitors the user's behavior and situation and provides necessary support and alerts.

[0583] An "interaction log" is recorded data of contacts and interactions that occur between a user and a system.

[0584] "Menu selection assistance" is a function that shows the options presented when the user orders a meal and helps the user select an appropriate menu from among them.

[0585] "Delivery progress" is information that indicates the current status and progress of the delivery process of the ordered product.

[0586] "Dietary logging" is the process by which the system automatically saves and manages the food and drink intake of the user.

[0587] The system according to the present invention comprises the following major hardware and software components:

[0588] Hardware Configuration

[0589] 1. Device (smartphone)

[0590] High-sensitivity microphone: Captures the user's voice.

[0591] Speaker: Provides audio responses to the user.

[0592] Camera: Captures user gestures and facial expressions.

[0593] Display: Presents visual information and interfaces.

[0594] Processor: Responsible for processing information and running applications.

[0595] 2. Remote Server

[0596] High-performance processor: Responsible for running generative artificial intelligence models and emotion engines.

[0597] Database: Stores user data and log information.

[0598] Alert function: Detects and notifies abnormalities.

[0599] Software Configuration

[0600] 1. Generative AI models (e.g., OpenAI GPT-4)

[0601] It performs natural language analysis of the user's voice instructions and generates an appropriate response.

[0602] Determine an action based on the user's gesture.

[0603] 2. Emotion engine (e.g. Affectiva SDK)

[0604] It analyzes the user's facial expressions and tone of voice to recognize emotions.

[0605] 3. Admin Dashboard

[0606] An interface that allows family members and caregivers to check the user's condition remotely.

[0607] 4. Alert notification system (e.g. Twilio API)

[0608] A remote server sends a notification to family members / caregivers when an abnormality is detected.

[0609] Processing Overview

[0610] Food ordering support

[0611] When a user wants to order a meal, they launch an application on their device and give their voice command. The device's sensitive microphone captures the voice, and a generative artificial intelligence model analyzes the command. The device displays the appropriate menu and transmits the user's selection to the server.

[0612] Check delivery status

[0613] If the user wants to check the delivery status, the generative artificial intelligence model generates a voice response through the terminal, providing the user with the delivery progress status through voice and display.

[0614] Food log

[0615] After the user has finished eating, the device application automatically sends the record to the server and stores it in a database. This information is then periodically sent to family members or caregivers.

[0616] Emotion recognition function

[0617] The user's voice and facial expressions are captured by the device's camera and microphone, and analyzed by the emotion engine. Based on the analysis results, the generative AI model generates appropriate responses and actions, which are then provided to the user.

[0618] Specific examples

[0619] For example, if a user says, "What should I eat today?", the device's microphone captures the voice and sends it to the server. A generative artificial intelligence model installed on the server analyzes it and generates a response such as, "The recommended menu items are curry rice, pasta, and sushi. Which would you like?" If the user answers, "Maybe I'll have sushi," the system confirms this and completes the order.

[0620] Example prompts for generative AI models

[0621] User input:

[0622] "What should I eat today?"

[0623] Pet robot response generation:

[0624] "My recommended menu items are curry rice, pasta, and sushi. Which would you like?"

[0625] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0626] Step 1:

[0627] A user launches an application on a device to order a meal and issues a voice command such as, "What should I eat today?" The input is the user's voice command, and the device's high-sensitivity microphone captures the voice. The output is the captured voice data.

[0628] Step 2:

[0629] The device transmits captured audio data to a remote server in real time. The input is audio data, which is transmitted to the server via data communication. The output is the audio data received by the server.

[0630] Step 3:

[0631] The server receives the voice data and analyzes it using a generative artificial intelligence model (OpenAI GPT-4) to generate an appropriate response. The input is the voice data, and data analysis and natural language processing are performed. The output is the generated response text.

[0632] Step 4:

[0633] The server sends the generated response text to the terminal. The input is the generated response text, which is sent to the terminal using data communication. The output is the response text that arrives at the terminal.

[0634] Step 5:

[0635] The device converts the received response text into speech and provides a response to the user through a speaker. The input is the response text, converts the text to speech, and plays the response speech. The output is the audio response that the user hears.

[0636] Step 6:

[0637] The menu selection made by the user (e.g., "Shall I have sushi?") is captured again by the device's high-sensitivity microphone and sent to the remote server. The input is the user's secondary voice instruction, and the voice data is captured and sent. The output is the voice data received by the server.

[0638] Step 7:

[0639] The server then uses the artificial intelligence model to analyze the voice data, confirm the selected menu, and generate the order. The input is the voice data, and data analysis and order generation are performed. The output is the generated order information.

[0640] Step 8:

[0641] The server sends the generated order information to the online delivery service to complete the order. The input is the order information and data transmission. The output is the completed order.

[0642] Step 9:

[0643] As soon as delivery status progress information arrives at the server, it is sent to the terminal in real time. The input is delivery progress information, which is sent to the terminal using data communication. The output is the delivery progress information that has arrived at the terminal.

[0644] Step 10:

[0645] The terminal provides delivery progress information to the user through voice and screen display. The input is delivery progress information, which is played back as voice and displayed on the screen. The output is information on the delivery progress status provided to the user.

[0646] Step 11:

[0647] The device automatically records the user's eating behavior and sends it to a remote server. The input is the automatically collected eating behavior data, which is sent to the server via data communication. The output is the meal record data received by the server.

[0648] Step 12:

[0649] The server stores the food record data in a database and notifies family members and caregivers at regular intervals. The input is the food record data, and the server stores the data and sends notifications. The output is the stored data and sent notifications.

[0650] Step 13:

[0651] The server periodically analyzes the user's emotional state using an emotion engine and generates appropriate responses and suggestions as needed. The input is the user's voice and facial expression data, and emotion analysis is performed. The output is the generated responses and suggestions.

[0652] Step 14:

[0653] The generated response or suggestion is sent to the terminal and provided to the user. The input is the generated response or suggestion, which is sent to the terminal using data communication. The output is the response or suggestion provided to the user.

[0654] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0655] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0656] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0657] [Second embodiment]

[0658] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0659] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0660] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0661] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0662] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0663] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0664] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0665] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0666] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0669] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0670] This invention relates to a pet robot that monitors the elderly. This pet robot has three functions: two-way communication, gesture recognition, and remote camera monitoring. It can help relieve the elderly's sense of loneliness, prevent dementia, and provide remote monitoring. Details on how to implement this system are described below.

[0671] System Configuration

[0672] The system has three main components:

[0673] 1. Pet robot main body (terminal)

[0674] High-sensitivity microphone

[0675] speaker

[0676] camera

[0677] Contact Sensor

[0678] Processor equipped with generative artificial intelligence model

[0679] 2. Remote Server

[0680] Operation of generative artificial intelligence models

[0681] Database

[0682] Alert generation function

[0683] Dashboard (for families and caregivers)

[0684] 3. Users (elderly people and their families)

[0685] User interface (natural interaction with pet robots, gestures)

[0686] Program processing overview

[0687] The system programs work together to realize the functions of two-way dialogue, gesture recognition, and remote monitoring. Specifically, the following processes are performed:

[0688] Two-way dialogue

[0689] 1. The user speaks

[0690] The user talks to the robot, asking questions such as "How was your day?"

[0691] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[0692] 2. Sending and analyzing audio data

[0693] The device transmits the captured audio data to the server in real time.

[0694] The server analyzes the voice data, generates an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[0695] 3. Executing the response

[0696] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[0697] Example: When a user asks, "How are you today?", the robot responds, based on the analysis results from the server, with, "I'm feeling great today!" At this time, the robot also performs actions such as shaking its body.

[0698] Gesture Recognition

[0699] 1. Performing a gesture

[0700] The user makes a gesture such as waving or pointing.

[0701] The device's camera captures the user's gestures in real time.

[0702] 2. Sending and analyzing gesture data

[0703] The device transmits the captured gesture data to the server.

[0704] The server analyzes the gesture, understands the user's intention, generates an appropriate response, and sends the resulting data to the device.

[0705] 3. Response Execution

[0706] The device performs an appropriate action (for example, waving back) based on the received response data.

[0707] Example: When a user waves their hand, the robot recognizes the movement with a camera, and based on the analysis results of the server, the robot waves back.

[0708] Remote monitoring

[0709] 1. Touch detection

[0710] When the user touches the robot, the device's contact sensor detects the touch.

[0711] 2. Sending log data

[0712] The device collects interaction data and periodically sends the log to a server.

[0713] 3. Data storage and analysis

[0714] The server stores the received interaction logs in a database and analyzes them. If an abnormality is detected, an alert is generated.

[0715] 4. Display on the dashboard

[0716] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers, allowing them to check on the elderly person's condition remotely.

[0717] Example: When a user strokes the robot's back, the interaction data is sent to a server and can be viewed by family members on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[0718] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. This also allows family members and caregivers to remotely monitor the elderly with peace of mind.

[0719] The processing flow will be explained below.

[0720] Two-way interaction processing steps

[0721] Step 1:

[0722] The user talks to the robot, asking questions such as "How was your day?"

[0723] Step 2:

[0724] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[0725] Step 3:

[0726] The device transmits the acquired voice data to the server in real time.

[0727] Step 4:

[0728] The server analyzes the received voice data and uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue.

[0729] Step 5:

[0730] The server transmits the generated response data to the terminal.

[0731] Step 6:

[0732] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[0733] Examples:

[0734] When a user says, "How are you today?", the device captures the audio and sends it to the server. The server generates a reply, "I'm feeling great today!", which the device then relays to the user.

[0735] Gesture Recognizer Processing Steps

[0736] Step 1:

[0737] The user makes a gesture (e.g., a wave) to the robot.

[0738] Step 2:

[0739] The device's camera captures the user's gestures in real time.

[0740] Step 3:

[0741] The device transmits the captured gesture data to the server.

[0742] Step 4:

[0743] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[0744] Step 5:

[0745] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[0746] Step 6:

[0747] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[0748] Examples:

[0749] When a user waves to the robot, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[0750] Remote monitoring function processing steps

[0751] Step 1:

[0752] The user touches the robot (for example, stroking its back).

[0753] Step 2:

[0754] The device's contact sensor detects contact and collects the data.

[0755] Step 3:

[0756] The device periodically transmits the acquired contact data to the server.

[0757] Step 4:

[0758] The server stores the received interaction logs in a database.

[0759] Step 5:

[0760] The server analyzes the stored data and generates an alert if an anomaly is detected.

[0761] Step 6:

[0762] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[0763] Examples:

[0764] When a user strokes the robot's back, the device detects the touch with a contact sensor and sends the data to a server, which analyzes and stores the data and displays it on a dashboard that family members can view remotely.

[0765] The specific processing steps for each function have been explained above. The specific operations performed in each step enable this pet robot to alleviate the sense of loneliness of elderly people and provide daily monitoring.

[0766] Example 1

[0767] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Current technology has not yet fully addressed the challenges of simultaneously alleviating loneliness among the elderly, preventing dementia, and remotely monitoring them. In particular, there is a lack of systems that allow elderly people to enjoy natural conversations and interact through gestures, while also allowing remote family members and caregivers to monitor the elderly's status in real time. Current technology has difficulty simultaneously meeting these multiple needs, and therefore has not yet succeeded in increasing the sense of security for the elderly and their families.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0770] In this invention, the server includes a two-way dialogue means for enabling natural dialogue with the user, a gesture recognition means for recognizing the user's gestures and performing corresponding actions, a remote monitoring means for monitoring the user's status using a remote camera and transmitting interaction logs and situation information to a remote server, a data transmission and response means for transmitting voice data and gesture data to the server and executing appropriate responses and response actions based on the analysis results, a voice and gesture analysis means equipped with a generative artificial intelligence model for analyzing the data received by the remote server, generating responses based on the analysis results, and transmitting the responses to the terminal, an alert generation means for analyzing the accumulated interaction logs and situation information and generating an alert if an abnormality is detected, and a dashboard means for displaying the analysis results and alert information. This makes it possible to simultaneously alleviate loneliness in the elderly, prevent dementia, and remotely monitor them.

[0771] The "two-way dialogue means" is a technology that uses voice recognition and voice synthesis to achieve two-way communication with the user in order to realize natural dialogue.

[0772] A "gesture recognition means" is a technology that uses a camera or other sensor to capture a user's physical movements, such as hand gestures or pointing, and analyzes those movements.

[0773] "Remote monitoring means" is a technology that uses remote cameras and other sensors to monitor the user's situation and transmits the data to a remote server.

[0774] "Data transmission and response means" refers to a technology that transmits voice data and gesture data collected from the terminal to a server in real time, receives analysis results from the server, and executes appropriate responses or response actions.

[0775] "Voice and gesture analysis means" refers to a technology that uses a generative artificial intelligence model installed on the server to analyze received voice and gesture data and generate a response based on the analysis results.

[0776] The "alert generation means" is a technology that analyzes accumulated interaction logs and situational information, and generates warning messages or notifications when an abnormality is detected.

[0777] "Dashboard means" is a technology that visually displays analysis results and alert information so that family members and caregivers can check them in real time.

[0778] This invention is a pet robot system for monitoring elderly people, which has three main functions: two-way interaction, gesture recognition, and remote camera monitoring. The system uses hardware such as a high-sensitivity microphone, speaker, camera, contact sensor, and processor equipped with a generative artificial intelligence model. Details for implementing this system are described below.

[0779] System Configuration

[0780] The system consists of three main components:

[0781] 1. Pet robot main body (terminal)

[0782] - High sensitivity microphone

[0783] - Speaker

[0784] - Camera

[0785] - Contact Sensor

[0786] - Processor equipped with generative artificial intelligence model

[0787] 2. Remote Server

[0788] - Operation of generative artificial intelligence models

[0789] - Database

[0790] - Alert generation function

[0791] - Dashboard (for families and caregivers)

[0792] 3. Users (elderly people and their families)

[0793] - User interface (natural interaction with pet robots, gestures)

[0794] Two-way dialogue

[0795] When a user speaks to the robot, the device's high-sensitivity microphone captures the voice. For example, if the user asks, "How was your day today?", the voice data is sent to the server in real time. The server analyzes the voice data, generates an appropriate response based on the context of the conversation, and sends this response data to the device. The device then plays back the received response data with natural voice and movements to respond to the user. For example, if the user asks, "How are you feeling today?" the robot will respond, "I'm feeling great today!" At this time, the robot will also perform movements such as shaking its body appropriately.

[0796] Gesture Recognition

[0797] When a user makes a gesture such as waving or pointing, the device's camera captures the user's gesture in real time. The device sends the captured gesture data to a server, which then analyzes the gesture. Based on the analysis results, the server understands the user's intention, generates an appropriate response action, and sends the resulting data to the device. The device then performs an appropriate action (for example, waving back) based on the received response data. As a specific example, when a user waves their hand, a robot recognizes the movement and waves back.

[0798] Remote monitoring

[0799] When a user touches the robot, the device's contact sensor detects the contact. The device collects the detected contact data and periodically sends the log to the server. The server stores the received contact log in a database and analyzes it. If an abnormality is detected, an alert is generated and family members or caregivers are notified. The server also displays the analysis results and contact log on a dashboard in real time, allowing the elderly person's condition to be checked remotely. For example, when a user strokes the robot's back, the contact data is sent to the server, and family members can check it on the remote dashboard. If an abnormality is detected, an alert is generated and a notification is sent.

[0800] Specific hardware and software names to be used

[0801] High-sensitivity microphone: General high-sensitivity microphone

[0802] Speaker: General speaker

[0803] Camera: General camera

[0804] Contact sensor: General contact sensor

[0805] Processor with generative artificial intelligence model: General processor

[0806] Remote Server: General Server

[0807] Generative Artificial Intelligence Model: General AI Model

[0808] Database: General database

[0809] Dashboard: General dashboard software

[0810] Specific examples of prompts to input to generative AI models

[0811] Speech analysis prompt: "What's the weather like tomorrow?"

[0812] Gesture analysis prompt: "Your hand wave has been recognized."

[0813] Interaction Log Prompt: "Back-pat sequence occurred"

[0814] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0815] Two-way dialogue

[0816] Step 1: User speaks

[0817] The user can ask the robot a question or start a conversation, for example, "How was your day today?"

[0818] Input: User's voice

[0819] Output: Audio data

[0820] Specific operation: When the user speaks to the pet robot, the voice is captured by a highly sensitive microphone.

[0821] Step 2: The device receives and transmits the audio data

[0822] The device collects the captured audio data.

[0823] Input: Captured audio data

[0824] Output: Audio data for transmission

[0825] What it does: It compresses audio data in real time and sends it securely to the server using TLS encryption.

[0826] Step 3: The server analyzes the voice data and generates a response

[0827] The server analyzes the received voice data, performs speech recognition, and converts the speech to text using a generative AI model.

[0828] Input: Received audio data

[0829] Output: Contextual response text

[0830] Specific operation: After converting the voice data into text, it performs context analysis and generates an appropriate response, such as "I'm feeling great today!"

[0831] Step 4: The device responds

[0832] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[0833] Input: Reply text

[0834] Output: Voice and action responses

[0835] Specific behavior: The response data is converted into voice and played back to the user. At the same time, the robot performs natural movements such as shaking its body.

[0836] Gesture Recognition

[0837] Step 1: The user makes a gesture

[0838] The user makes a gesture such as waving or pointing.

[0839] Input: User's physical movements

[0840] Output: Camera video data

[0841] Specific actions: When a user makes a gesture, the action is captured in real time by the camera.

[0842] Step 2: The device receives and sends gesture data

[0843] The device collects the captured gesture data.

[0844] Input: Captured video data

[0845] Output: Gesture data to send

[0846] How it works: Video data is compressed in real time and sent securely to the server using TLS encryption.

[0847] Step 3: The server analyzes the gesture data and generates a response

[0848] The server analyzes the received gesture data and understands the user's intention using a machine learning model.

[0849] Input: Received gesture data

[0850] Output: Response behavior data based on user intention

[0851] Specific Actions: Analyze gesture data and generate appropriate response actions, such as waving back.

[0852] Step 4: The device performs the response action

[0853] The terminal performs an appropriate operation based on the received response operation data.

[0854] Input: Response action data

[0855] Output: Actual device behavior

[0856] Specific actions: Based on the received action data, the robot will, for example, wave its hand back.

[0857] Remote monitoring

[0858] Step 1: Touch detection

[0859] When the user touches the robot, the device's contact sensor detects the touch.

[0860] Input: Physical touch

[0861] Output: Interaction data

[0862] Specific operation: When a user touches the robot, the contact sensor detects the touch and generates data.

[0863] Step 2: Device collects and sends log data

[0864] The device collects detected contact data and periodically sends the log data to a server.

[0865] Input: Interaction data

[0866] Output: Log data for sending

[0867] What it does: Collected log data is compressed in real time and sent securely to the server using TLS encryption.

[0868] Step 3: The server stores and analyzes the data

[0869] The server stores the received interaction logs in a database and analyzes them.

[0870] Input: Received log data

[0871] Output: Analysis results and anomaly detection results

[0872] Specific operation: The saved log data is analyzed periodically to check for any abnormalities. For example, if the frequency of contact decreases drastically, it will be recognized as an abnormality.

[0873] Step 4: View the server in the dashboard

[0874] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers.

[0875] Input: Analysis results and log data

[0876] Output: Visual information on a dashboard

[0877] Specific operation: Analysis results and log data are displayed on the dashboard, and if an abnormality is detected, an alert is displayed in real time. Visual information is also provided so that family members and caregivers can check the user's condition.

[0878] Specific examples of prompts to input to generative AI models

[0879] Speech analysis prompt: "What's the weather like tomorrow?"

[0880] Gesture analysis prompt: "Your hand wave has been recognized."

[0881] Interaction Log Prompt: "Back-pat sequence occurred"

[0882] (Application example 1)

[0883] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Problems faced by elderly people when using food delivery services include the burden of completing the complicated ordering process on their own, the difficulty of responding quickly and safely to the delivery person's arrival, and the uncertainty of having family members or caregivers supervise the delivery remotely. There is a need to solve these problems and provide a more secure and comfortable food delivery experience.

[0885] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0886] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for supporting food delivery orders, and means for confirming delivery and generating alerts to notify the remote server. This reduces the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[0887] "Two-way dialogue" is a communication method that uses a generative artificial intelligence model that recognizes speech and generates responses to enable natural dialogue with the user.

[0888] A "generative artificial intelligence model" is an artificial intelligence algorithm that analyzes voice data and generates appropriate responses.

[0889] "Gesture recognition" is a technology that analyzes a user's actions captured by a camera, understands their meaning, and performs a corresponding action.

[0890] A "remote camera" is a camera device used to monitor the user's situation from a remote location.

[0891] An "interaction log" is information that records data when a user interacts with a robot and is used for later analysis.

[0892] A "remote server" is a server device in a remote location that has high-performance computing capabilities, analyzes data in real time, and stores necessary information.

[0893] "Food delivery" is a service that delivers food to a specific location.

[0894] "Ordering support" is a function that provides assistance through voice dialogue, etc., to make it easier for users to order food delivery.

[0895] "Delivery confirmation" is a way to confirm that the delivery person has arrived and to assist with safe pickup.

[0896] "Alert generation" is a function that generates a notification when an abnormality is detected.

[0897] This invention provides a pet robot system for supporting food delivery services for the elderly. The system includes functions such as two-way interaction, gesture recognition, and remote monitoring, allowing users to use food delivery services more safely and comfortably.

[0898] System Configuration

[0899] The system has three main components:

[0900] 1. Pet robot main body (terminal):

[0901] High-sensitivity microphone

[0902] speaker

[0903] camera

[0904] Contact Sensor

[0905] Processor equipped with generative artificial intelligence model

[0906] 2. Remote Server:

[0907] Operation of generative artificial intelligence models

[0908] Database

[0909] Alert generation function

[0910] Dashboard (for families and caregivers)

[0911] 3. Users (elderly people and their families):

[0912] User interface (natural interaction with pet robots, gestures)

[0913] Program processing overview

[0914] The system's programs work together to realize the functions of food delivery order support, delivery confirmation, and safety confirmation. Specifically, the following hardware and software are used to perform the processing.

[0915] Food delivery ordering assistance

[0916] 1. User: An elderly person speaks to their pet robot, saying, "I want to order dinner."

[0917] 2. Terminal: Captures audio using a sensitive microphone and transmits the data to a remote server.

[0918] 3. Remote server: A generative AI model analyzes the speech and generates an appropriate response, which is returned to the device.

[0919] 4. Terminal: Asks "Which restaurant would you like?" The user responds to proceed with the order.

[0920] Delivery Confirmation

[0921] 1. Terminal: When the delivery person arrives, the camera recognizes their arrival and sends the data to a remote server.

[0922] 2. Remote server: Notifies family members or caregivers based on the received data.

[0923] 3. Terminal: Respond appropriately to the delivery person by saying "Thank you for your hard work."

[0924] Safety confirmation

[0925] 1. User: When receiving meals, the robot will provide support to ensure that the elderly person receives the meal safely.

[0926] 2. Terminal: Capture the situation after receiving the item with a camera and send it to a remote server.

[0927] 3. Remote server: Based on the analysis results, if an abnormality is detected, an alert is generated to notify family members or caregivers.

[0928] Examples of concrete examples and prompts

[0929] As a specific scenario for operating this system, consider the case where elderly people use food delivery services.

[0930] Examples:

[0931] When an elderly person says to the robot, "I'd like to order sushi," the robot responds, "Which sushi restaurant would you like?" If the elderly person answers, "Sushida," the robot responds, "I'll open the Sushida menu," and proceeds with the order.

[0932] Example prompt sentence:

[0933] User: I'd like to order sushi.

[0934] Robot: Which sushi restaurant would you like?

[0935] User: Sushida.

[0936] Robot: Open the Sushida menu.

[0937] This will reduce the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[0938] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0939] Step 1:

[0940] The user speaks to the robot, saying, "I'd like to order dinner." The input data includes the user's voice data. The device uses a highly sensitive microphone to capture this voice data, and the voice data is acquired as input data.

[0941] Step 2:

[0942] The device transmits the captured voice data in real time to a remote server, where the voice data is processed for transmission. The remote server analyzes the voice data received as input. A generative AI model converts the voice to text, analyzes the context, and generates an appropriate response. In this process, the voice data is converted to text data and analyzed.

[0943] Step 3:

[0944] The remote server sends the generated response data to the device. The device then synthesizes the received response data into voice and responds to the user. Voice synthesis technology is used here, and text data is converted back into voice data. Specifically, the device asks the user, "Which restaurant would you like?"

[0945] Step 4:

[0946] The user responds with "Sushida." The user's voice data is again captured as input data, and the device captures and sends it to the remote server. As in the previous step, the server converts the voice data into text data and generates an analysis result.

[0947] Step 5:

[0948] The remote server generates an appropriate response, such as "Sushida's menu will be displayed," and sends it to the device. The device then synthesizes this response and responds to the user, for example, "Sushida's menu will be displayed."

[0949] Step 6:

[0950] When the delivery person arrives, the device's camera captures the video data, which is then sent as input data to a remote server, which analyzes the video data and confirms the delivery person's arrival.

[0951] Step 7:

[0952] The remote server then notifies the family or caregiver of the results of the check. The notification includes information about the delivery person's arrival and status. At the same time, the device responds appropriately to the delivery person by saying, "Thank you for your hard work."

[0953] Step 8:

[0954] When the user receives the meal, the device's camera and contact sensors monitor the situation. The video data and contact log at the time of receipt are sent as input data to a remote server. The remote server analyzes this data and checks for any abnormalities.

[0955] Step 9:

[0956] If the remote server detects any abnormalities based on the analysis results, it generates an alert and notifies the family or caregiver, allowing them to confirm safety. If everything is normal, it notifies them that the delivery has been completed successfully.

[0957] These steps will create a system that allows seniors to use food delivery services safely and easily.

[0958] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0959] This invention relates to a pet robot for elderly care that combines an emotion engine. This pet robot is equipped with two-way communication, gesture recognition, remote camera monitoring, and emotion recognition functions, helping to alleviate feelings of loneliness in elderly people, preventing dementia, and remotely monitoring them. Details for implementing this system are described below.

[0960] System Configuration

[0961] The system comprises four main components:

[0962] 1. Pet robot main body (terminal)

[0963] High-sensitivity microphone

[0964] speaker

[0965] camera

[0966] Contact Sensor

[0967] Processor equipped with generative artificial intelligence model

[0968] Emotion Engine

[0969] 2. Remote Server

[0970] Operation of generative artificial intelligence models

[0971] Database

[0972] Alert generation function

[0973] Dashboard (for families and caregivers)

[0974] 3. Users (elderly people and their families)

[0975] User interface (natural interaction with pet robots, gestures)

[0976] 4. Emotion Engine

[0977] Recognizing emotions from the user's voice and facial expressions

[0978] Program processing overview

[0979] The system programs work together to realize the functions of two-way dialogue, gesture recognition, remote monitoring, and emotion recognition. Specifically, the following processes are performed:

[0980] Two-way dialogue

[0981] 1. The user speaks

[0982] The user talks to the robot, asking questions such as "How was your day?"

[0983] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[0984] 2. Sending and analyzing audio data

[0985] The device transmits the acquired voice data to the server in real time.

[0986] The server analyzes the voice data and recognizes the user's emotions using an emotion engine.

[0987] 3. Response Generation and Execution

[0988] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[0989] The device responds to the user by expressing the received response data in voice and motion, thereby realizing a dialogue that responds to the user's emotions.

[0990] Examples:

[0991] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" If the emotion engine recognizes the emotion "lonely" from the tone of the user's voice, the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[0992] Gesture Recognition

[0993] 1. Performing a gesture

[0994] The user makes a gesture (e.g., a wave) to the robot.

[0995] The device's camera captures the user's gestures in real time.

[0996] 2. Sending and analyzing gesture data

[0997] The device transmits the captured gesture data to the server.

[0998] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[0999] 3. Response Execution

[1000] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[1001] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[1002] Examples:

[1003] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[1004] Remote monitoring

[1005] 1. Touch detection

[1006] The user touches the robot (for example, stroking its back).

[1007] The device's contact sensor detects contact and collects the data.

[1008] 2. Sending log data

[1009] The device periodically transmits the acquired contact data to the server.

[1010] 3. Data storage and analysis

[1011] The server stores the received interaction logs in a database.

[1012] 4. Displaying analysis results and generating alerts

[1013] The server analyzes the stored data and generates an alert if an anomaly is detected.

[1014] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1015] Examples:

[1016] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, and displays it on a dashboard that family members can view remotely. If an abnormality is detected, an alert is generated and notified.

[1017] Emotion recognition function

[1018] 1. Recognizing the user's voice and facial expressions

[1019] The device captures the user's voice and facial expressions through a camera and microphone.

[1020] The emotion engine recognizes the user's emotions from the captured data.

[1021] 2. Transmission and analysis of emotional data

[1022] The device transmits the recognized emotion data to the server.

[1023] The server analyzes the emotional data to adjust dialogue and behavior.

[1024] 3. Generating and executing responses according to emotions

[1025] The server generates appropriate responses and actions based on the emotion data and sends that data to the terminal.

[1026] The terminal executes the received response data and expresses an appropriate emotion to the user.

[1027] Examples:

[1028] If a user says, "I'm feeling lonely after being at home all day today," the emotion engine recognizes the emotion of "lonely." The server generates data to respond in a cheerful tone, saying, "I see you're lonely. Let's play together!", and the device executes that response.

[1029] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. In addition, the addition of an emotion recognition function further improves emotional care for the elderly, allowing family members and caregivers to remotely monitor the elderly with peace of mind.

[1030] The processing flow will be explained below.

[1031] Two-way interaction processing steps

[1032] Step 1:

[1033] The user talks to the robot, for example, asking, "How was your day?"

[1034] Step 2:

[1035] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[1036] Step 3:

[1037] The device transmits the acquired voice data to the server in real time.

[1038] Step 4:

[1039] The server analyzes the received voice data using a generative artificial intelligence model to understand the content and context of the speech.

[1040] Step 5:

[1041] The server uses an emotion engine to analyze the user's tone of voice and phrasing to recognize the user's emotions.

[1042] Step 6:

[1043] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the conversation and the emotion recognition results, and sends the response data to the terminal.

[1044] Step 7:

[1045] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[1046] Examples:

[1047] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" At the same time, the emotion engine recognizes "lonely" from the tone of the user's voice, and the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[1048] Gesture Recognizer Processing Steps

[1049] Step 1:

[1050] The user makes a gesture (e.g., a wave) to the robot.

[1051] Step 2:

[1052] The device's camera captures the user's gestures in real time.

[1053] Step 3:

[1054] The device transmits the captured gesture data to the server.

[1055] Step 4:

[1056] The server analyzes the received gesture data using machine learning models to understand the meaning of the gesture.

[1057] Step 5:

[1058] The server generates an appropriate response action based on the user's gesture and transmits the response data to the terminal.

[1059] Step 6:

[1060] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[1061] Examples:

[1062] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[1063] Remote monitoring function processing steps

[1064] Step 1:

[1065] The user touches the robot (for example, stroking its back).

[1066] Step 2:

[1067] The device's contact sensor detects contact and collects the data.

[1068] Step 3:

[1069] The device periodically transmits the acquired contact data to the server.

[1070] Step 4:

[1071] The server stores the received interaction logs in a database.

[1072] Step 5:

[1073] The server analyzes the stored data to detect abnormalities in the elderly person's behavior and physical condition, and if an abnormality is detected, it generates an alert.

[1074] Step 6:

[1075] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1076] Examples:

[1077] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, which family members can view on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[1078] Emotion Recognition Processing Steps

[1079] Step 1:

[1080] The user speaks to the robot or shows facial expressions.

[1081] The device's camera captures the user's facial expressions, and a highly sensitive microphone captures their voice.

[1082] Step 2:

[1083] The terminal transmits the user's voice data and facial expression data to the server in real time.

[1084] Step 3:

[1085] The server analyzes the received voice data and facial expression data and recognizes the user's emotions (e.g., joy, sadness, anger) using an emotion engine.

[1086] Step 4:

[1087] The server generates appropriate responses and actions based on the emotion recognition results and sends the data to the terminal.

[1088] Step 5:

[1089] The terminal executes the received response data and gives a natural response that matches the user's emotions.

[1090] Examples:

[1091] If the user says, "I've been lonely at home all day today," the emotion engine recognizes the emotion "lonely." The server generates a response, "I see you've been lonely. Let's play together!", and the robot conveys this to the user in a cheerful tone.

[1092] The specific processing steps for each function, including the emotion engine, have been explained above. The specific actions performed at each step enable this pet robot to alleviate the sense of loneliness felt by the elderly and provide daily monitoring.

[1093] Example 2

[1094] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1095] There is a need for systems that can prevent elderly people from feeling lonely and developing dementia, and that can also monitor them remotely. To alleviate the feelings of loneliness and anxiety that elderly people experience in their daily lives, natural dialogue that responds to emotions and interactive responses through gesture recognition are necessary. Remote monitoring functions are also required to make it easier for family members and caregivers to understand the condition of the elderly. Existing systems lack comprehensive solutions to meet these needs.

[1096] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1097] In this invention, the server includes means for enabling natural dialogue with the user using an artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and situation information to the remote server, means for recognizing the user's voice and facial expressions and analyzing emotions, and means for generating appropriate responses based on the analyzed emotional data and providing them to the user. This makes it possible to reduce loneliness among the elderly, prevent dementia, and even monitor them remotely.

[1098] An "artificial intelligence model" is a program designed to naturally interact with users, using speech recognition and natural language processing technology to understand the meaning of what the user says and generate appropriate responses.

[1099] "Gesture recognition" is a technology that captures a user's actions (such as waving a hand) using a sensor such as a camera, and analyzes and determines the meaning of the action.

[1100] A "remote camera" is a device that captures video from a remote location and transmits it to a remote server in real time, and is used to monitor users.

[1101] An "interaction log" is a record of physical contact between the user and the robot (such as a back pat) and is sent to a remote server.

[1102] An "emotion engine" is a technology that analyzes a user's voice and facial expression data to recognize the user's emotions at that time (for example, joy, sadness, loneliness, etc.).

[1103] "Response generation based on analysis results" is a technology that generates appropriate responses and actions based on the results of analyzing the user's statements, gestures, and emotional data.

[1104] "Alert generation" is a technology that notifies family members or caregivers from a remote server when an abnormality is detected in the user's behavior or condition.

[1105] "Users" refer to the elderly people who use the system, as well as their families and caregivers, who interact with the robot through voice and gestures.

[1106] This invention is a system that combines an emotion engine with a pet robot for elderly care, alleviating the sense of loneliness of the elderly, preventing dementia, and remotely monitoring them. The components of this system include the pet robot itself, a remote server, users (elderly people and their families), and the emotion engine.

[1107] System Configuration

[1108] 1. Pet robot main body (terminal)

[1109] High-sensitivity microphone: Captures user voice in real time.

[1110] Speaker: Replies sent from the server are spoken to the user.

[1111] Camera: Captures user gestures and facial expressions and provides data for analysis.

[1112] Contact sensor: Acquires contact data when the user touches the robot.

[1113] Processor equipped with generative artificial intelligence model: Performs voice analysis, gesture recognition, and emotion analysis.

[1114] Emotion engine: Technology for recognizing and analyzing user emotions.

[1115] 2. Remote Server

[1116] Operation of generative artificial intelligence models: Analyze acquired data and generate appropriate responses and actions.

[1117] Database: Stores interaction data and emotion analysis results.

[1118] Alert generation function: If an abnormality is detected, an alert is generated to notify family members or caregivers.

[1119] Dashboard (for family and caregivers): Displays analysis results and interaction logs, allowing users to check their condition.

[1120] 3. Users (elderly people and their families)

[1121] User Interface: Interact with the pet robot through natural dialogue and gestures.

[1122] 4. Emotion Engine

[1123] Recognizing emotions from voice and facial expressions: Analyze the user's voice tone and facial expressions to generate emotional data.

[1124] System program processing overview

[1125] The system works to recognize the user's voice and gestures, generate appropriate responses and actions based on them, and realize two-way dialogue according to emotions.

[1126] For example, when a user asks, "How was your day?", the device's high-sensitivity microphone captures the voice and sends the voice data to the server. The server uses a generative artificial intelligence model to analyze the voice data, and an emotion engine recognizes the user's emotion. The server then generates an appropriate response and sends it to the device. The response, "I'm feeling great today!", is heard through the device's speaker.

[1127] When a user makes a waving gesture, the device's camera captures the gesture and sends it to the server. The server analyzes the gesture, generates the necessary response action, and sends it to the device. The device then performs the waving action.

[1128] Furthermore, when the user touches the robot, the device's contact sensor captures the data and sends it to the server, where it is stored in a database and can be viewed by family members or caregivers via a dashboard. If an abnormality is detected, an alert is generated and a remote notification is sent.

[1129] Examples of prompt statements

[1130] "Please explain in detail the process by which an elderly care robot recognizes emotions from the user's voice and facial expressions and returns an appropriate response."

[1131] In this way, each function of this system works together to reduce loneliness and provide care for the elderly. In addition, the addition of an emotion recognition function improves emotional care for the elderly and allows family members and caregivers to remotely monitor them with peace of mind.

[1132] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1133] Handling two-way interactions

[1134] Step 1: Capture audio

[1135] Input: The user speaks to the robot (e.g., "How was your day?").

[1136] The device uses a high-sensitivity microphone to capture the user's voice in real time.

[1137] Output: The audio data is captured on the device.

[1138] Step 2: Sending audio data

[1139] Input: Audio data captured on the device.

[1140] The device transmits the captured audio data to the server in real time.

[1141] Output: The audio data is sent to the server.

[1142] Step 3: Voice analysis and emotion recognition

[1143] Input: The audio data sent to the server.

[1144] The server uses a generative artificial intelligence model to analyze the voice data and understand what the user is saying.

[1145] The server uses an emotion engine to recognize the user's emotions (e.g., joy, sadness, loneliness) from the voice.

[1146] Output: The analyzed speech content and emotion data are obtained.

[1147] Step 4: Response generation and speech output

[1148] Input: Parsed speech and sentiment data.

[1149] The server generates an appropriate response based on the analysis results.

[1150] The server generates a response and sends it to the terminal.

[1151] The device provides a response to the user through the speaker (e.g., "I'm feeling great today!").

[1152] Output: A voice is output in response to the user.

[1153] Gesture Recognition Processing

[1154] Step 1: Gesture Capture

[1155] Input: The user makes a gesture towards the robot (e.g., waving).

[1156] The device uses a camera to capture the user's gestures in real time.

[1157] Output: Gesture data is captured on the device.

[1158] Step 2: Sending gesture data

[1159] Input: Gesture data captured on the device.

[1160] The device transmits the captured gesture data to the server.

[1161] Output: Gesture data is sent to the server.

[1162] Step 3: Gesture analysis

[1163] Input: The gesture data sent to the server.

[1164] The server analyzes the received gesture data and understands its meaning (e.g., greeting, request).

[1165] Output: The parsed meaning of the gesture is obtained.

[1166] Step 4: Generate and execute response actions

[1167] Input: Parsed gesture meaning.

[1168] Based on the analysis results, the server generates an appropriate response action and sends it to the device (e.g., waving back).

[1169] The terminal performs an action based on the received response action data.

[1170] Output: The action corresponding to the gesture is performed.

[1171] Remote monitoring processing

[1172] Step 1: Touch detection

[1173] Input: The user touches the robot (e.g., pats its back).

[1174] The device's contact sensor detects this contact and captures the data.

[1175] Output: Interaction data is captured on the device.

[1176] Step 2: Sending log data

[1177] Input: Interaction data captured on the device.

[1178] The device periodically transmits the acquired contact data to the server.

[1179] Output: Interaction data is sent to the server.

[1180] Step 3: Data storage and analysis

[1181] Input: Interaction data sent to the server.

[1182] The server stores the received interaction data in a database.

[1183] The server analyzes the stored data and generates alerts if anomalies are detected.

[1184] Output: Saved interaction data and analysis results are obtained.

[1185] Step 4: Viewing analysis results and generating alerts

[1186] Input: Saved interaction data and its analysis results.

[1187] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1188] The server generates an alert and notifies you if an abnormality is detected.

[1189] Output: Data is displayed on the dashboard and alerts are sent.

[1190] Emotion recognition processing

[1191] Step 1: Capture your voice and facial expressions

[1192] Input: User's voice and facial expressions.

[1193] The device captures the user's voice and facial expressions in real time using a camera and microphone.

[1194] Output: Voice data and facial expression data are acquired on the terminal.

[1195] Step 2: Send and analyze emotion data

[1196] Input: Voice data and facial expression data acquired on the device.

[1197] The terminal transmits the acquired data to the server.

[1198] The server analyzes the received data using an emotion engine and recognizes the user's emotions.

[1199] Output: The analyzed emotion data is obtained.

[1200] Step 3: Generate and execute a response

[1201] Input: Parsed emotion data.

[1202] The server generates appropriate responses and actions based on the emotional data and sends them to the terminal.

[1203] The terminal executes the received response data and expresses an appropriate emotion to the user.

[1204] Output: A response according to the emotion is provided to the user.

[1205] The above is the specific flow of the program processing for this system. At each step, appropriate data processing and calculations are performed based on the input data, and the output is used in the processing of the next step. This enables interaction according to the user's condition and emotions, realizing a system that can alleviate the loneliness of elderly people while also allowing family members and caregivers to monitor them remotely.

[1206] (Application example 2)

[1207] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1208] The goal is to eliminate the difficulties that elderly people face when using food delivery services independently, ease their sense of loneliness, and enable them to order meals with peace of mind, check delivery status, and easily manage meal records. Another goal is to provide emotional care for elderly people by utilizing emotion recognition functions and realize appropriate remote monitoring.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1210] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for assisting the user in selecting a menu when ordering a meal, means for presenting the user with the progress of delivery, and means for automatically managing meal records and sending the record information to the remote server. This makes food delivery services easier and more secure for the elderly, alleviating their sense of loneliness and enabling remote monitoring.

[1211] "Two-way dialogue" is a mechanism in which users and systems communicate with each other in real time using voice and text, enabling natural conversation.

[1212] A "generative artificial intelligence model" is a type of artificial intelligence that learns from large amounts of data and has the ability to engage in natural conversations, recognize emotions, and generate responses.

[1213] "Gesture recognition" is a technology that uses cameras and sensors to detect a user's body movements and hand gestures and respond accordingly.

[1214] A "remote camera" is a camera installed in a physically separate location, and is a device that allows you to view images from a remote location.

[1215] "User monitoring" is a function in which the system monitors the user's behavior and situation and provides necessary support and alerts.

[1216] An "interaction log" is recorded data of contacts and interactions that occur between a user and a system.

[1217] "Menu selection assistance" is a function that shows the options presented when the user orders a meal and helps the user select an appropriate menu from among them.

[1218] "Delivery progress" is information that indicates the current status and progress of the delivery process of the ordered product.

[1219] "Dietary logging" is the process by which the system automatically saves and manages the food and drink intake of the user.

[1220] The system according to the present invention comprises the following major hardware and software components:

[1221] Hardware Configuration

[1222] 1. Device (smartphone)

[1223] High-sensitivity microphone: Captures the user's voice.

[1224] Speaker: Provides audio responses to the user.

[1225] Camera: Captures user gestures and facial expressions.

[1226] Display: Presents visual information and interfaces.

[1227] Processor: Responsible for processing information and running applications.

[1228] 2. Remote Server

[1229] High-performance processor: Responsible for running generative artificial intelligence models and emotion engines.

[1230] Database: Stores user data and log information.

[1231] Alert function: Detects and notifies abnormalities.

[1232] Software Configuration

[1233] 1. Generative AI models (e.g., OpenAI GPT-4)

[1234] It performs natural language analysis of the user's voice instructions and generates an appropriate response.

[1235] Determine an action based on the user's gesture.

[1236] 2. Emotion engine (e.g. Affectiva SDK)

[1237] It analyzes the user's facial expressions and tone of voice to recognize emotions.

[1238] 3. Admin Dashboard

[1239] An interface that allows family members and caregivers to check the user's condition remotely.

[1240] 4. Alert notification system (e.g. Twilio API)

[1241] A remote server sends a notification to family members / caregivers when an abnormality is detected.

[1242] Processing Overview

[1243] Food ordering support

[1244] When a user wants to order a meal, they launch an application on their device and give their voice command. The device's sensitive microphone captures the voice, and a generative artificial intelligence model analyzes the command. The device displays the appropriate menu and transmits the user's selection to the server.

[1245] Check delivery status

[1246] If the user wants to check the delivery status, the generative artificial intelligence model generates a voice response through the terminal, providing the user with the delivery progress status through voice and display.

[1247] Food log

[1248] After the user has finished eating, the device application automatically sends the record to the server and stores it in a database. This information is then periodically sent to family members or caregivers.

[1249] Emotion recognition function

[1250] The user's voice and facial expressions are captured by the device's camera and microphone, and analyzed by the emotion engine. Based on the analysis results, the generative AI model generates appropriate responses and actions, which are then provided to the user.

[1251] Specific examples

[1252] For example, if a user says, "What should I eat today?", the device's microphone captures the voice and sends it to the server. A generative artificial intelligence model installed on the server analyzes it and generates a response such as, "The recommended menu items are curry rice, pasta, and sushi. Which would you like?" If the user answers, "Maybe I'll have sushi," the system confirms this and completes the order.

[1253] Example prompts for generative AI models

[1254] User input:

[1255] "What should I eat today?"

[1256] Pet robot response generation:

[1257] "My recommended menu items are curry rice, pasta, and sushi. Which would you like?"

[1258] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1259] Step 1:

[1260] A user launches an application on a device to order a meal and issues a voice command such as, "What should I eat today?" The input is the user's voice command, and the device's high-sensitivity microphone captures the voice. The output is the captured voice data.

[1261] Step 2:

[1262] The device transmits captured audio data to a remote server in real time. The input is audio data, which is transmitted to the server via data communication. The output is the audio data received by the server.

[1263] Step 3:

[1264] The server receives the voice data and analyzes it using a generative artificial intelligence model (OpenAI GPT-4) to generate an appropriate response. The input is the voice data, and data analysis and natural language processing are performed. The output is the generated response text.

[1265] Step 4:

[1266] The server sends the generated response text to the terminal. The input is the generated response text, which is sent to the terminal using data communication. The output is the response text that arrives at the terminal.

[1267] Step 5:

[1268] The device converts the received response text into speech and provides a response to the user through a speaker. The input is the response text, converts the text to speech, and plays the response speech. The output is the audio response that the user hears.

[1269] Step 6:

[1270] The menu selection made by the user (e.g., "Shall I have sushi?") is captured again by the device's high-sensitivity microphone and sent to the remote server. The input is the user's secondary voice instruction, and the voice data is captured and sent. The output is the voice data received by the server.

[1271] Step 7:

[1272] The server then uses the artificial intelligence model to analyze the voice data, confirm the selected menu, and generate the order. The input is the voice data, and data analysis and order generation are performed. The output is the generated order information.

[1273] Step 8:

[1274] The server sends the generated order information to the online delivery service to complete the order. The input is the order information and data transmission. The output is the completed order.

[1275] Step 9:

[1276] As soon as delivery status progress information arrives at the server, it is sent to the terminal in real time. The input is delivery progress information, which is sent to the terminal using data communication. The output is the delivery progress information that has arrived at the terminal.

[1277] Step 10:

[1278] The terminal provides delivery progress information to the user through voice and screen display. The input is delivery progress information, which is played back as voice and displayed on the screen. The output is information on the delivery progress status provided to the user.

[1279] Step 11:

[1280] The device automatically records the user's eating behavior and sends it to a remote server. The input is the automatically collected eating behavior data, which is sent to the server via data communication. The output is the meal record data received by the server.

[1281] Step 12:

[1282] The server stores the food record data in a database and notifies family members and caregivers at regular intervals. The input is the food record data, and the server stores the data and sends notifications. The output is the stored data and sent notifications.

[1283] Step 13:

[1284] The server periodically analyzes the user's emotional state using an emotion engine and generates appropriate responses and suggestions as needed. The input is the user's voice and facial expression data, and emotion analysis is performed. The output is the generated responses and suggestions.

[1285] Step 14:

[1286] The generated response or suggestion is sent to the terminal and provided to the user. The input is the generated response or suggestion, which is sent to the terminal using data communication. The output is the response or suggestion provided to the user.

[1287] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1288] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1289] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1290] [Third embodiment]

[1291] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1292] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1293] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1294] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1295] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1296] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1297] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1298] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1299] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1300] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1301] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1302] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1303] This invention relates to a pet robot that monitors the elderly. This pet robot has three functions: two-way communication, gesture recognition, and remote camera monitoring. It can help relieve the elderly's sense of loneliness, prevent dementia, and provide remote monitoring. Details on how to implement this system are described below.

[1304] System Configuration

[1305] The system has three main components:

[1306] 1. Pet robot main body (terminal)

[1307] High-sensitivity microphone

[1308] speaker

[1309] camera

[1310] Contact Sensor

[1311] Processor equipped with generative artificial intelligence model

[1312] 2. Remote Server

[1313] Operation of generative artificial intelligence models

[1314] Database

[1315] Alert generation function

[1316] Dashboard (for families and caregivers)

[1317] 3. Users (elderly people and their families)

[1318] User interface (natural interaction with pet robots, gestures)

[1319] Program processing overview

[1320] The system programs work together to realize the functions of two-way dialogue, gesture recognition, and remote monitoring. Specifically, the following processes are performed:

[1321] Two-way dialogue

[1322] 1. The user speaks

[1323] The user talks to the robot, asking questions such as "How was your day?"

[1324] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[1325] 2. Sending and analyzing audio data

[1326] The device transmits the captured audio data to the server in real time.

[1327] The server analyzes the voice data, generates an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[1328] 3. Executing the response

[1329] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[1330] Example: When a user asks, "How are you today?", the robot responds, based on the analysis results from the server, with, "I'm feeling great today!" At this time, the robot also performs actions such as shaking its body.

[1331] Gesture Recognition

[1332] 1. Performing a gesture

[1333] The user makes a gesture such as waving or pointing.

[1334] The device's camera captures the user's gestures in real time.

[1335] 2. Sending and analyzing gesture data

[1336] The device transmits the captured gesture data to the server.

[1337] The server analyzes the gesture, understands the user's intention, generates an appropriate response, and sends the resulting data to the device.

[1338] 3. Response Execution

[1339] The device performs an appropriate action (for example, waving back) based on the received response data.

[1340] Example: When a user waves their hand, the robot recognizes the movement with a camera, and based on the analysis results of the server, the robot waves back.

[1341] Remote monitoring

[1342] 1. Touch detection

[1343] When the user touches the robot, the device's contact sensor detects the touch.

[1344] 2. Sending log data

[1345] The device collects interaction data and periodically sends the log to a server.

[1346] 3. Data storage and analysis

[1347] The server stores the received interaction logs in a database and analyzes them. If an abnormality is detected, an alert is generated.

[1348] 4. Display on the dashboard

[1349] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers, allowing them to check on the elderly person's condition remotely.

[1350] Example: When a user strokes the robot's back, the interaction data is sent to a server and can be viewed by family members on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[1351] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. This also allows family members and caregivers to remotely monitor the elderly with peace of mind.

[1352] The processing flow will be explained below.

[1353] Two-way interaction processing steps

[1354] Step 1:

[1355] The user talks to the robot, asking questions such as "How was your day?"

[1356] Step 2:

[1357] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[1358] Step 3:

[1359] The device transmits the acquired voice data to the server in real time.

[1360] Step 4:

[1361] The server analyzes the received voice data and uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue.

[1362] Step 5:

[1363] The server transmits the generated response data to the terminal.

[1364] Step 6:

[1365] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[1366] Examples:

[1367] When a user says, "How are you today?", the device captures the audio and sends it to the server. The server generates a reply, "I'm feeling great today!", which the device then relays to the user.

[1368] Gesture Recognizer Processing Steps

[1369] Step 1:

[1370] The user makes a gesture (e.g., a wave) to the robot.

[1371] Step 2:

[1372] The device's camera captures the user's gestures in real time.

[1373] Step 3:

[1374] The device transmits the captured gesture data to the server.

[1375] Step 4:

[1376] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[1377] Step 5:

[1378] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[1379] Step 6:

[1380] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[1381] Examples:

[1382] When a user waves to the robot, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[1383] Remote monitoring function processing steps

[1384] Step 1:

[1385] The user touches the robot (for example, stroking its back).

[1386] Step 2:

[1387] The device's contact sensor detects contact and collects the data.

[1388] Step 3:

[1389] The device periodically transmits the acquired contact data to the server.

[1390] Step 4:

[1391] The server stores the received interaction logs in a database.

[1392] Step 5:

[1393] The server analyzes the stored data and generates an alert if an anomaly is detected.

[1394] Step 6:

[1395] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1396] Examples:

[1397] When a user strokes the robot's back, the device detects the touch with a contact sensor and sends the data to a server, which analyzes and stores the data and displays it on a dashboard that family members can view remotely.

[1398] The specific processing steps for each function have been explained above. The specific operations performed in each step enable this pet robot to alleviate the sense of loneliness of elderly people and provide daily monitoring.

[1399] Example 1

[1400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1401] Current technology has not yet fully addressed the challenges of simultaneously alleviating loneliness among the elderly, preventing dementia, and remotely monitoring them. In particular, there is a lack of systems that allow elderly people to enjoy natural conversations and interact through gestures, while also allowing remote family members and caregivers to monitor the elderly's status in real time. Current technology has difficulty simultaneously meeting these multiple needs, and therefore has not yet succeeded in increasing the sense of security for the elderly and their families.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1403] In this invention, the server includes a two-way dialogue means for enabling natural dialogue with the user, a gesture recognition means for recognizing the user's gestures and performing corresponding actions, a remote monitoring means for monitoring the user's status using a remote camera and transmitting interaction logs and situation information to a remote server, a data transmission and response means for transmitting voice data and gesture data to the server and executing appropriate responses and response actions based on the analysis results, a voice and gesture analysis means equipped with a generative artificial intelligence model for analyzing the data received by the remote server, generating responses based on the analysis results, and transmitting the responses to the terminal, an alert generation means for analyzing the accumulated interaction logs and situation information and generating an alert if an abnormality is detected, and a dashboard means for displaying the analysis results and alert information. This makes it possible to simultaneously alleviate loneliness in the elderly, prevent dementia, and remotely monitor them.

[1404] The "two-way dialogue means" is a technology that uses voice recognition and voice synthesis to achieve two-way communication with the user in order to realize natural dialogue.

[1405] A "gesture recognition means" is a technology that uses a camera or other sensor to capture a user's physical movements, such as hand gestures or pointing, and analyzes those movements.

[1406] "Remote monitoring means" is a technology that uses remote cameras and other sensors to monitor the user's situation and transmits the data to a remote server.

[1407] "Data transmission and response means" refers to a technology that transmits voice data and gesture data collected from the terminal to a server in real time, receives analysis results from the server, and executes appropriate responses or response actions.

[1408] "Voice and gesture analysis means" refers to a technology that uses a generative artificial intelligence model installed on the server to analyze received voice and gesture data and generate a response based on the analysis results.

[1409] The "alert generation means" is a technology that analyzes accumulated interaction logs and situational information, and generates warning messages or notifications when an abnormality is detected.

[1410] "Dashboard means" is a technology that visually displays analysis results and alert information so that family members and caregivers can check them in real time.

[1411] This invention is a pet robot system for monitoring elderly people, which has three main functions: two-way interaction, gesture recognition, and remote camera monitoring. The system uses hardware such as a high-sensitivity microphone, speaker, camera, contact sensor, and processor equipped with a generative artificial intelligence model. Details for implementing this system are described below.

[1412] System Configuration

[1413] The system consists of three main components:

[1414] 1. Pet robot main body (terminal)

[1415] - High sensitivity microphone

[1416] - Speaker

[1417] - Camera

[1418] - Contact Sensor

[1419] - Processor equipped with generative artificial intelligence model

[1420] 2. Remote Server

[1421] - Operation of generative artificial intelligence models

[1422] - Database

[1423] - Alert generation function

[1424] - Dashboard (for families and caregivers)

[1425] 3. Users (elderly people and their families)

[1426] - User interface (natural interaction with pet robots, gestures)

[1427] Two-way dialogue

[1428] When a user speaks to the robot, the device's high-sensitivity microphone captures the voice. For example, if the user asks, "How was your day today?", the voice data is sent to the server in real time. The server analyzes the voice data, generates an appropriate response based on the context of the conversation, and sends this response data to the device. The device then plays back the received response data with natural voice and movements to respond to the user. For example, if the user asks, "How are you feeling today?" the robot will respond, "I'm feeling great today!" At this time, the robot will also perform movements such as shaking its body appropriately.

[1429] Gesture Recognition

[1430] When a user makes a gesture such as waving or pointing, the device's camera captures the user's gesture in real time. The device sends the captured gesture data to a server, which then analyzes the gesture. Based on the analysis results, the server understands the user's intention, generates an appropriate response action, and sends the resulting data to the device. The device then performs an appropriate action (for example, waving back) based on the received response data. As a specific example, when a user waves their hand, a robot recognizes the movement and waves back.

[1431] Remote monitoring

[1432] When a user touches the robot, the device's contact sensor detects the contact. The device collects the detected contact data and periodically sends the log to the server. The server stores the received contact log in a database and analyzes it. If an abnormality is detected, an alert is generated and family members or caregivers are notified. The server also displays the analysis results and contact log on a dashboard in real time, allowing the elderly person's condition to be checked remotely. For example, when a user strokes the robot's back, the contact data is sent to the server, and family members can check it on the remote dashboard. If an abnormality is detected, an alert is generated and a notification is sent.

[1433] Specific hardware and software names to be used

[1434] High-sensitivity microphone: General high-sensitivity microphone

[1435] Speaker: General speaker

[1436] Camera: General camera

[1437] Contact sensor: General contact sensor

[1438] Processor with generative artificial intelligence model: General processor

[1439] Remote Server: General Server

[1440] Generative Artificial Intelligence Model: General AI Model

[1441] Database: General database

[1442] Dashboard: General dashboard software

[1443] Specific examples of prompts to input to generative AI models

[1444] Speech analysis prompt: "What's the weather like tomorrow?"

[1445] Gesture analysis prompt: "Your hand wave has been recognized."

[1446] Interaction Log Prompt: "Back-pat sequence occurred"

[1447] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1448] Two-way dialogue

[1449] Step 1: User speaks

[1450] The user can ask the robot a question or start a conversation, for example, "How was your day today?"

[1451] Input: User's voice

[1452] Output: Audio data

[1453] Specific operation: When the user speaks to the pet robot, the voice is captured by a highly sensitive microphone.

[1454] Step 2: The device receives and transmits the audio data

[1455] The device collects the captured audio data.

[1456] Input: Captured audio data

[1457] Output: Audio data for transmission

[1458] What it does: It compresses audio data in real time and sends it securely to the server using TLS encryption.

[1459] Step 3: The server analyzes the voice data and generates a response

[1460] The server analyzes the received voice data, performs speech recognition, and converts the speech to text using a generative AI model.

[1461] Input: Received audio data

[1462] Output: Contextual response text

[1463] Specific operation: After converting the voice data into text, it performs context analysis and generates an appropriate response, such as "I'm feeling great today!"

[1464] Step 4: The device responds

[1465] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[1466] Input: Reply text

[1467] Output: Voice and action responses

[1468] Specific behavior: The response data is converted into voice and played back to the user. At the same time, the robot performs natural movements such as shaking its body.

[1469] Gesture Recognition

[1470] Step 1: The user makes a gesture

[1471] The user makes a gesture such as waving or pointing.

[1472] Input: User's physical movements

[1473] Output: Camera video data

[1474] Specific actions: When a user makes a gesture, the action is captured in real time by the camera.

[1475] Step 2: The device receives and sends gesture data

[1476] The device collects the captured gesture data.

[1477] Input: Captured video data

[1478] Output: Gesture data to send

[1479] How it works: Video data is compressed in real time and sent securely to the server using TLS encryption.

[1480] Step 3: The server analyzes the gesture data and generates a response

[1481] The server analyzes the received gesture data and understands the user's intention using a machine learning model.

[1482] Input: Received gesture data

[1483] Output: Response behavior data based on user intention

[1484] Specific Actions: Analyze gesture data and generate appropriate response actions, such as waving back.

[1485] Step 4: The device performs the response action

[1486] The terminal performs an appropriate operation based on the received response operation data.

[1487] Input: Response action data

[1488] Output: Actual device behavior

[1489] Specific actions: Based on the received action data, the robot will, for example, wave its hand back.

[1490] Remote monitoring

[1491] Step 1: Touch detection

[1492] When the user touches the robot, the device's contact sensor detects the touch.

[1493] Input: Physical touch

[1494] Output: Interaction data

[1495] Specific operation: When a user touches the robot, the contact sensor detects the touch and generates data.

[1496] Step 2: Device collects and sends log data

[1497] The device collects detected contact data and periodically sends the log data to a server.

[1498] Input: Interaction data

[1499] Output: Log data for sending

[1500] What it does: Collected log data is compressed in real time and sent securely to the server using TLS encryption.

[1501] Step 3: The server stores and analyzes the data

[1502] The server stores the received interaction logs in a database and analyzes them.

[1503] Input: Received log data

[1504] Output: Analysis results and anomaly detection results

[1505] Specific operation: The saved log data is analyzed periodically to check for any abnormalities. For example, if the frequency of contact decreases drastically, it will be recognized as an abnormality.

[1506] Step 4: View the server in the dashboard

[1507] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers.

[1508] Input: Analysis results and log data

[1509] Output: Visual information on a dashboard

[1510] Specific operation: Analysis results and log data are displayed on the dashboard, and if an abnormality is detected, an alert is displayed in real time. Visual information is also provided so that family members and caregivers can check the user's condition.

[1511] Specific examples of prompts to input to generative AI models

[1512] Speech analysis prompt: "What's the weather like tomorrow?"

[1513] Gesture analysis prompt: "Your hand wave has been recognized."

[1514] Interaction Log Prompt: "Back-pat sequence occurred"

[1515] (Application example 1)

[1516] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1517] Problems faced by elderly people when using food delivery services include the burden of completing the complicated ordering process on their own, the difficulty of responding quickly and safely to the delivery person's arrival, and the uncertainty of having family members or caregivers supervise the delivery remotely. There is a need to solve these problems and provide a more secure and comfortable food delivery experience.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1519] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for supporting food delivery orders, and means for confirming delivery and generating alerts to notify the remote server. This reduces the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[1520] "Two-way dialogue" is a communication method that uses a generative artificial intelligence model that recognizes speech and generates responses to enable natural dialogue with the user.

[1521] A "generative artificial intelligence model" is an artificial intelligence algorithm that analyzes voice data and generates appropriate responses.

[1522] "Gesture recognition" is a technology that analyzes a user's actions captured by a camera, understands their meaning, and performs a corresponding action.

[1523] A "remote camera" is a camera device used to monitor the user's situation from a remote location.

[1524] An "interaction log" is information that records data when a user interacts with a robot and is used for later analysis.

[1525] A "remote server" is a server device in a remote location that has high-performance computing capabilities, analyzes data in real time, and stores necessary information.

[1526] "Food delivery" is a service that delivers food to a specific location.

[1527] "Ordering support" is a function that provides assistance through voice dialogue, etc., to make it easier for users to order food delivery.

[1528] "Delivery confirmation" is a way to confirm that the delivery person has arrived and to assist with safe pickup.

[1529] "Alert generation" is a function that generates a notification when an abnormality is detected.

[1530] This invention provides a pet robot system for supporting food delivery services for the elderly. The system includes functions such as two-way interaction, gesture recognition, and remote monitoring, allowing users to use food delivery services more safely and comfortably.

[1531] System Configuration

[1532] The system has three main components:

[1533] 1. Pet robot main body (terminal):

[1534] High-sensitivity microphone

[1535] speaker

[1536] camera

[1537] Contact Sensor

[1538] Processor equipped with generative artificial intelligence model

[1539] 2. Remote Server:

[1540] Operation of generative artificial intelligence models

[1541] Database

[1542] Alert generation function

[1543] Dashboard (for families and caregivers)

[1544] 3. Users (elderly people and their families):

[1545] User interface (natural interaction with pet robots, gestures)

[1546] Program processing overview

[1547] The system's programs work together to realize the functions of food delivery order support, delivery confirmation, and safety confirmation. Specifically, the following hardware and software are used to perform the processing.

[1548] Food delivery ordering assistance

[1549] 1. User: An elderly person speaks to their pet robot, saying, "I want to order dinner."

[1550] 2. Terminal: Captures audio using a sensitive microphone and transmits the data to a remote server.

[1551] 3. Remote server: A generative AI model analyzes the speech and generates an appropriate response, which is returned to the device.

[1552] 4. Terminal: Asks "Which restaurant would you like?" The user responds to proceed with the order.

[1553] Delivery Confirmation

[1554] 1. Terminal: When the delivery person arrives, the camera recognizes their arrival and sends the data to a remote server.

[1555] 2. Remote server: Notifies family members or caregivers based on the received data.

[1556] 3. Terminal: Respond appropriately to the delivery person by saying "Thank you for your hard work."

[1557] Safety confirmation

[1558] 1. User: When receiving meals, the robot will provide support to ensure that the elderly person receives the meal safely.

[1559] 2. Terminal: Capture the situation after receiving the item with a camera and send it to a remote server.

[1560] 3. Remote server: Based on the analysis results, if an abnormality is detected, an alert is generated to notify family members or caregivers.

[1561] Examples of concrete examples and prompts

[1562] As a specific scenario for operating this system, consider the case where elderly people use food delivery services.

[1563] Examples:

[1564] When an elderly person says to the robot, "I'd like to order sushi," the robot responds, "Which sushi restaurant would you like?" If the elderly person answers, "Sushida," the robot responds, "I'll open the Sushida menu," and proceeds with the order.

[1565] Example prompt sentence:

[1566] User: I'd like to order sushi.

[1567] Robot: Which sushi restaurant would you like?

[1568] User: Sushida.

[1569] Robot: Open the Sushida menu.

[1570] This will reduce the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[1571] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1572] Step 1:

[1573] The user speaks to the robot, saying, "I'd like to order dinner." The input data includes the user's voice data. The device uses a highly sensitive microphone to capture this voice data, and the voice data is acquired as input data.

[1574] Step 2:

[1575] The device transmits the captured voice data in real time to a remote server, where the voice data is processed for transmission. The remote server analyzes the voice data received as input. A generative AI model converts the voice to text, analyzes the context, and generates an appropriate response. In this process, the voice data is converted to text data and analyzed.

[1576] Step 3:

[1577] The remote server sends the generated response data to the device. The device then synthesizes the received response data into voice and responds to the user. Voice synthesis technology is used here, and text data is converted back into voice data. Specifically, the device asks the user, "Which restaurant would you like?"

[1578] Step 4:

[1579] The user responds with "Sushida." The user's voice data is again captured as input data, and the device captures and sends it to the remote server. As in the previous step, the server converts the voice data into text data and generates an analysis result.

[1580] Step 5:

[1581] The remote server generates an appropriate response, such as "Sushida's menu will be displayed," and sends it to the device. The device then synthesizes this response and responds to the user, for example, "Sushida's menu will be displayed."

[1582] Step 6:

[1583] When the delivery person arrives, the device's camera captures the video data, which is then sent as input data to a remote server, which analyzes the video data and confirms the delivery person's arrival.

[1584] Step 7:

[1585] The remote server then notifies the family or caregiver of the results of the check. The notification includes information about the delivery person's arrival and status. At the same time, the device responds appropriately to the delivery person by saying, "Thank you for your hard work."

[1586] Step 8:

[1587] When the user receives the meal, the device's camera and contact sensors monitor the situation. The video data and contact log at the time of receipt are sent as input data to a remote server. The remote server analyzes this data and checks for any abnormalities.

[1588] Step 9:

[1589] If the remote server detects any abnormalities based on the analysis results, it generates an alert and notifies the family or caregiver, allowing them to confirm safety. If everything is normal, it notifies them that the delivery has been completed successfully.

[1590] These steps will create a system that allows seniors to use food delivery services safely and easily.

[1591] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1592] This invention relates to a pet robot for elderly care that combines an emotion engine. This pet robot is equipped with two-way communication, gesture recognition, remote camera monitoring, and emotion recognition functions, helping to alleviate feelings of loneliness in elderly people, preventing dementia, and remotely monitoring them. Details for implementing this system are described below.

[1593] System Configuration

[1594] The system comprises four main components:

[1595] 1. Pet robot main body (terminal)

[1596] High-sensitivity microphone

[1597] speaker

[1598] camera

[1599] Contact Sensor

[1600] Processor equipped with generative artificial intelligence model

[1601] Emotion Engine

[1602] 2. Remote Server

[1603] Operation of generative artificial intelligence models

[1604] Database

[1605] Alert generation function

[1606] Dashboard (for families and caregivers)

[1607] 3. Users (elderly people and their families)

[1608] User interface (natural interaction with pet robots, gestures)

[1609] 4. Emotion Engine

[1610] Recognizing emotions from the user's voice and facial expressions

[1611] Program processing overview

[1612] The system programs work together to realize the functions of two-way dialogue, gesture recognition, remote monitoring, and emotion recognition. Specifically, the following processes are performed:

[1613] Two-way dialogue

[1614] 1. The user speaks

[1615] The user talks to the robot, asking questions such as "How was your day?"

[1616] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[1617] 2. Sending and analyzing audio data

[1618] The device transmits the acquired voice data to the server in real time.

[1619] The server analyzes the voice data and recognizes the user's emotions using an emotion engine.

[1620] 3. Response Generation and Execution

[1621] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[1622] The device responds to the user by expressing the received response data in voice and motion, thereby realizing a dialogue that responds to the user's emotions.

[1623] Examples:

[1624] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" If the emotion engine recognizes the emotion "lonely" from the tone of the user's voice, the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[1625] Gesture Recognition

[1626] 1. Performing a gesture

[1627] The user makes a gesture (e.g., a wave) to the robot.

[1628] The device's camera captures the user's gestures in real time.

[1629] 2. Sending and analyzing gesture data

[1630] The device transmits the captured gesture data to the server.

[1631] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[1632] 3. Response Execution

[1633] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[1634] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[1635] Examples:

[1636] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[1637] Remote monitoring

[1638] 1. Touch detection

[1639] The user touches the robot (for example, stroking its back).

[1640] The device's contact sensor detects contact and collects the data.

[1641] 2. Sending log data

[1642] The device periodically transmits the acquired contact data to the server.

[1643] 3. Data storage and analysis

[1644] The server stores the received interaction logs in a database.

[1645] 4. Displaying analysis results and generating alerts

[1646] The server analyzes the stored data and generates an alert if an anomaly is detected.

[1647] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1648] Examples:

[1649] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, and displays it on a dashboard that family members can view remotely. If an abnormality is detected, an alert is generated and notified.

[1650] Emotion recognition function

[1651] 1. Recognizing the user's voice and facial expressions

[1652] The device captures the user's voice and facial expressions through a camera and microphone.

[1653] The emotion engine recognizes the user's emotions from the captured data.

[1654] 2. Transmission and analysis of emotional data

[1655] The device transmits the recognized emotion data to the server.

[1656] The server analyzes the emotional data to adjust dialogue and behavior.

[1657] 3. Generating and executing responses according to emotions

[1658] The server generates appropriate responses and actions based on the emotion data and sends that data to the terminal.

[1659] The terminal executes the received response data and expresses an appropriate emotion to the user.

[1660] Examples:

[1661] If a user says, "I'm feeling lonely after being at home all day today," the emotion engine recognizes the emotion of "lonely." The server generates data to respond in a cheerful tone, saying, "I see you're lonely. Let's play together!", and the device executes that response.

[1662] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. In addition, the addition of an emotion recognition function further improves emotional care for the elderly, allowing family members and caregivers to remotely monitor the elderly with peace of mind.

[1663] The processing flow will be explained below.

[1664] Two-way interaction processing steps

[1665] Step 1:

[1666] The user talks to the robot, for example, asking, "How was your day?"

[1667] Step 2:

[1668] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[1669] Step 3:

[1670] The device transmits the acquired voice data to the server in real time.

[1671] Step 4:

[1672] The server analyzes the received voice data using a generative artificial intelligence model to understand the content and context of the speech.

[1673] Step 5:

[1674] The server uses an emotion engine to analyze the user's tone of voice and phrasing to recognize the user's emotions.

[1675] Step 6:

[1676] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the conversation and the emotion recognition results, and sends the response data to the terminal.

[1677] Step 7:

[1678] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[1679] Examples:

[1680] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" At the same time, the emotion engine recognizes "lonely" from the tone of the user's voice, and the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[1681] Gesture Recognizer Processing Steps

[1682] Step 1:

[1683] The user makes a gesture (e.g., a wave) to the robot.

[1684] Step 2:

[1685] The device's camera captures the user's gestures in real time.

[1686] Step 3:

[1687] The device transmits the captured gesture data to the server.

[1688] Step 4:

[1689] The server analyzes the received gesture data using machine learning models to understand the meaning of the gesture.

[1690] Step 5:

[1691] The server generates an appropriate response action based on the user's gesture and transmits the response data to the terminal.

[1692] Step 6:

[1693] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[1694] Examples:

[1695] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[1696] Remote monitoring function processing steps

[1697] Step 1:

[1698] The user touches the robot (for example, stroking its back).

[1699] Step 2:

[1700] The device's contact sensor detects contact and collects the data.

[1701] Step 3:

[1702] The device periodically transmits the acquired contact data to the server.

[1703] Step 4:

[1704] The server stores the received interaction logs in a database.

[1705] Step 5:

[1706] The server analyzes the stored data to detect abnormalities in the elderly person's behavior and physical condition, and if an abnormality is detected, it generates an alert.

[1707] Step 6:

[1708] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1709] Examples:

[1710] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, which family members can view on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[1711] Emotion Recognition Processing Steps

[1712] Step 1:

[1713] The user speaks to the robot or shows facial expressions.

[1714] The device's camera captures the user's facial expressions, and a highly sensitive microphone captures their voice.

[1715] Step 2:

[1716] The terminal transmits the user's voice data and facial expression data to the server in real time.

[1717] Step 3:

[1718] The server analyzes the received voice data and facial expression data and recognizes the user's emotions (e.g., joy, sadness, anger) using an emotion engine.

[1719] Step 4:

[1720] The server generates appropriate responses and actions based on the emotion recognition results and sends the data to the terminal.

[1721] Step 5:

[1722] The terminal executes the received response data and gives a natural response that matches the user's emotions.

[1723] Examples:

[1724] If the user says, "I've been lonely at home all day today," the emotion engine recognizes the emotion "lonely." The server generates a response, "I see you've been lonely. Let's play together!", and the robot conveys this to the user in a cheerful tone.

[1725] The specific processing steps for each function, including the emotion engine, have been explained above. The specific actions performed at each step enable this pet robot to alleviate the sense of loneliness felt by the elderly and provide daily monitoring.

[1726] Example 2

[1727] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1728] There is a need for systems that can prevent elderly people from feeling lonely and developing dementia, and that can also monitor them remotely. To alleviate the feelings of loneliness and anxiety that elderly people experience in their daily lives, natural dialogue that responds to emotions and interactive responses through gesture recognition are necessary. Remote monitoring functions are also required to make it easier for family members and caregivers to understand the condition of the elderly. Existing systems lack comprehensive solutions to meet these needs.

[1729] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1730] In this invention, the server includes means for enabling natural dialogue with the user using an artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and situation information to the remote server, means for recognizing the user's voice and facial expressions and analyzing emotions, and means for generating appropriate responses based on the analyzed emotional data and providing them to the user. This makes it possible to reduce loneliness among the elderly, prevent dementia, and even monitor them remotely.

[1731] An "artificial intelligence model" is a program designed to naturally interact with users, using speech recognition and natural language processing technology to understand the meaning of what the user says and generate appropriate responses.

[1732] "Gesture recognition" is a technology that captures a user's actions (such as waving a hand) using a sensor such as a camera, and analyzes and determines the meaning of the action.

[1733] A "remote camera" is a device that captures video from a remote location and transmits it to a remote server in real time, and is used to monitor users.

[1734] An "interaction log" is a record of physical contact between the user and the robot (such as a back pat) and is sent to a remote server.

[1735] An "emotion engine" is a technology that analyzes a user's voice and facial expression data to recognize the user's emotions at that time (for example, joy, sadness, loneliness, etc.).

[1736] "Response generation based on analysis results" is a technology that generates appropriate responses and actions based on the results of analyzing the user's statements, gestures, and emotional data.

[1737] "Alert generation" is a technology that notifies family members or caregivers from a remote server when an abnormality is detected in the user's behavior or condition.

[1738] "Users" refer to the elderly people who use the system, as well as their families and caregivers, who interact with the robot through voice and gestures.

[1739] This invention is a system that combines an emotion engine with a pet robot for elderly care, alleviating the sense of loneliness of the elderly, preventing dementia, and remotely monitoring them. The components of this system include the pet robot itself, a remote server, users (elderly people and their families), and the emotion engine.

[1740] System Configuration

[1741] 1. Pet robot main body (terminal)

[1742] High-sensitivity microphone: Captures user voice in real time.

[1743] Speaker: Replies sent from the server are spoken to the user.

[1744] Camera: Captures user gestures and facial expressions and provides data for analysis.

[1745] Contact sensor: Acquires contact data when the user touches the robot.

[1746] Processor equipped with generative artificial intelligence model: Performs voice analysis, gesture recognition, and emotion analysis.

[1747] Emotion engine: Technology for recognizing and analyzing user emotions.

[1748] 2. Remote Server

[1749] Operation of generative artificial intelligence models: Analyze acquired data and generate appropriate responses and actions.

[1750] Database: Stores interaction data and emotion analysis results.

[1751] Alert generation function: If an abnormality is detected, an alert is generated to notify family members or caregivers.

[1752] Dashboard (for family and caregivers): Displays analysis results and interaction logs, allowing users to check their condition.

[1753] 3. Users (elderly people and their families)

[1754] User Interface: Interact with the pet robot through natural dialogue and gestures.

[1755] 4. Emotion Engine

[1756] Recognizing emotions from voice and facial expressions: Analyze the user's voice tone and facial expressions to generate emotional data.

[1757] System program processing overview

[1758] The system works to recognize the user's voice and gestures, generate appropriate responses and actions based on them, and realize two-way dialogue according to emotions.

[1759] For example, when a user asks, "How was your day?", the device's high-sensitivity microphone captures the voice and sends the voice data to the server. The server uses a generative artificial intelligence model to analyze the voice data, and an emotion engine recognizes the user's emotion. The server then generates an appropriate response and sends it to the device. The response, "I'm feeling great today!", is heard through the device's speaker.

[1760] When a user makes a waving gesture, the device's camera captures the gesture and sends it to the server. The server analyzes the gesture, generates the necessary response action, and sends it to the device. The device then performs the waving action.

[1761] Furthermore, when the user touches the robot, the device's contact sensor captures the data and sends it to the server, where it is stored in a database and can be viewed by family members or caregivers via a dashboard. If an abnormality is detected, an alert is generated and a remote notification is sent.

[1762] Examples of prompt statements

[1763] "Please explain in detail the process by which an elderly care robot recognizes emotions from the user's voice and facial expressions and returns an appropriate response."

[1764] In this way, each function of this system works together to reduce loneliness and provide care for the elderly. In addition, the addition of an emotion recognition function improves emotional care for the elderly and allows family members and caregivers to remotely monitor them with peace of mind.

[1765] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1766] Handling two-way interactions

[1767] Step 1: Capture audio

[1768] Input: The user speaks to the robot (e.g., "How was your day?").

[1769] The device uses a high-sensitivity microphone to capture the user's voice in real time.

[1770] Output: The audio data is captured on the device.

[1771] Step 2: Sending audio data

[1772] Input: Audio data captured on the device.

[1773] The device transmits the captured audio data to the server in real time.

[1774] Output: The audio data is sent to the server.

[1775] Step 3: Voice analysis and emotion recognition

[1776] Input: The audio data sent to the server.

[1777] The server uses a generative artificial intelligence model to analyze the voice data and understand what the user is saying.

[1778] The server uses an emotion engine to recognize the user's emotions (e.g., joy, sadness, loneliness) from the voice.

[1779] Output: The analyzed speech content and emotion data are obtained.

[1780] Step 4: Response generation and speech output

[1781] Input: Parsed speech and sentiment data.

[1782] The server generates an appropriate response based on the analysis results.

[1783] The server generates a response and sends it to the terminal.

[1784] The device provides a response to the user through the speaker (e.g., "I'm feeling great today!").

[1785] Output: A voice is output in response to the user.

[1786] Gesture Recognition Processing

[1787] Step 1: Gesture Capture

[1788] Input: The user makes a gesture towards the robot (e.g., waving).

[1789] The device uses a camera to capture the user's gestures in real time.

[1790] Output: Gesture data is captured on the device.

[1791] Step 2: Sending gesture data

[1792] Input: Gesture data captured on the device.

[1793] The device transmits the captured gesture data to the server.

[1794] Output: Gesture data is sent to the server.

[1795] Step 3: Gesture analysis

[1796] Input: The gesture data sent to the server.

[1797] The server analyzes the received gesture data and understands its meaning (e.g., greeting, request).

[1798] Output: The parsed meaning of the gesture is obtained.

[1799] Step 4: Generate and execute response actions

[1800] Input: Parsed gesture meaning.

[1801] Based on the analysis results, the server generates an appropriate response action and sends it to the device (e.g., waving back).

[1802] The terminal performs an action based on the received response action data.

[1803] Output: The action corresponding to the gesture is performed.

[1804] Remote monitoring processing

[1805] Step 1: Touch detection

[1806] Input: The user touches the robot (e.g., pats its back).

[1807] The device's contact sensor detects this contact and captures the data.

[1808] Output: Interaction data is captured on the device.

[1809] Step 2: Sending log data

[1810] Input: Interaction data captured on the device.

[1811] The device periodically transmits the acquired contact data to the server.

[1812] Output: Interaction data is sent to the server.

[1813] Step 3: Data storage and analysis

[1814] Input: Interaction data sent to the server.

[1815] The server stores the received interaction data in a database.

[1816] The server analyzes the stored data and generates alerts if anomalies are detected.

[1817] Output: Saved interaction data and analysis results are obtained.

[1818] Step 4: Viewing analysis results and generating alerts

[1819] Input: Saved interaction data and its analysis results.

[1820] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[1821] The server generates an alert and notifies you if an abnormality is detected.

[1822] Output: Data is displayed on the dashboard and alerts are sent.

[1823] Emotion recognition processing

[1824] Step 1: Capture your voice and facial expressions

[1825] Input: User's voice and facial expressions.

[1826] The device captures the user's voice and facial expressions in real time using a camera and microphone.

[1827] Output: Voice data and facial expression data are acquired on the terminal.

[1828] Step 2: Send and analyze emotion data

[1829] Input: Voice data and facial expression data acquired on the device.

[1830] The terminal transmits the acquired data to the server.

[1831] The server analyzes the received data using an emotion engine and recognizes the user's emotions.

[1832] Output: The analyzed emotion data is obtained.

[1833] Step 3: Generate and execute a response

[1834] Input: Parsed emotion data.

[1835] The server generates appropriate responses and actions based on the emotional data and sends them to the terminal.

[1836] The terminal executes the received response data and expresses an appropriate emotion to the user.

[1837] Output: A response according to the emotion is provided to the user.

[1838] The above is the specific flow of the program processing for this system. At each step, appropriate data processing and calculations are performed based on the input data, and the output is used in the processing of the next step. This enables interaction according to the user's condition and emotions, realizing a system that can alleviate the loneliness of elderly people while also allowing family members and caregivers to monitor them remotely.

[1839] (Application example 2)

[1840] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1841] The goal is to eliminate the difficulties that elderly people face when using food delivery services independently, ease their sense of loneliness, and enable them to order meals with peace of mind, check delivery status, and easily manage meal records. Another goal is to provide emotional care for elderly people by utilizing emotion recognition functions and realize appropriate remote monitoring.

[1842] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1843] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for assisting the user in selecting a menu when ordering a meal, means for presenting the user with the progress of delivery, and means for automatically managing meal records and sending the record information to the remote server. This makes food delivery services easier and more secure for the elderly, alleviating their sense of loneliness and enabling remote monitoring.

[1844] "Two-way dialogue" is a mechanism in which users and systems communicate with each other in real time using voice and text, enabling natural conversation.

[1845] A "generative artificial intelligence model" is a type of artificial intelligence that learns from large amounts of data and has the ability to engage in natural conversations, recognize emotions, and generate responses.

[1846] "Gesture recognition" is a technology that uses cameras and sensors to detect a user's body movements and hand gestures and respond accordingly.

[1847] A "remote camera" is a camera installed in a physically separate location, and is a device that allows you to view images from a remote location.

[1848] "User monitoring" is a function in which the system monitors the user's behavior and situation and provides necessary support and alerts.

[1849] An "interaction log" is recorded data of contacts and interactions that occur between a user and a system.

[1850] "Menu selection assistance" is a function that shows the options presented when the user orders a meal and helps the user select an appropriate menu from among them.

[1851] "Delivery progress" is information that indicates the current status and progress of the delivery process of the ordered product.

[1852] "Dietary logging" is the process by which the system automatically saves and manages the food and drink intake of the user.

[1853] The system according to the present invention comprises the following major hardware and software components:

[1854] Hardware Configuration

[1855] 1. Device (smartphone)

[1856] High-sensitivity microphone: Captures the user's voice.

[1857] Speaker: Provides audio responses to the user.

[1858] Camera: Captures user gestures and facial expressions.

[1859] Display: Presents visual information and interfaces.

[1860] Processor: Responsible for processing information and running applications.

[1861] 2. Remote Server

[1862] High-performance processor: Responsible for running generative artificial intelligence models and emotion engines.

[1863] Database: Stores user data and log information.

[1864] Alert function: Detects and notifies abnormalities.

[1865] Software Configuration

[1866] 1. Generative AI models (e.g., OpenAI GPT-4)

[1867] It performs natural language analysis of the user's voice instructions and generates an appropriate response.

[1868] Determine an action based on the user's gesture.

[1869] 2. Emotion engine (e.g. Affectiva SDK)

[1870] It analyzes the user's facial expressions and tone of voice to recognize emotions.

[1871] 3. Admin Dashboard

[1872] An interface that allows family members and caregivers to check the user's condition remotely.

[1873] 4. Alert notification system (e.g. Twilio API)

[1874] A remote server sends a notification to family members / caregivers when an abnormality is detected.

[1875] Processing Overview

[1876] Food ordering support

[1877] When a user wants to order a meal, they launch an application on their device and give their voice command. The device's sensitive microphone captures the voice, and a generative artificial intelligence model analyzes the command. The device displays the appropriate menu and transmits the user's selection to the server.

[1878] Check delivery status

[1879] If the user wants to check the delivery status, the generative artificial intelligence model generates a voice response through the terminal, providing the user with the delivery progress status through voice and display.

[1880] Food log

[1881] After the user has finished eating, the device application automatically sends the record to the server and stores it in a database. This information is then periodically sent to family members or caregivers.

[1882] Emotion recognition function

[1883] The user's voice and facial expressions are captured by the device's camera and microphone, and analyzed by the emotion engine. Based on the analysis results, the generative AI model generates appropriate responses and actions, which are then provided to the user.

[1884] Specific examples

[1885] For example, if a user says, "What should I eat today?", the device's microphone captures the voice and sends it to the server. A generative artificial intelligence model installed on the server analyzes it and generates a response such as, "The recommended menu items are curry rice, pasta, and sushi. Which would you like?" If the user answers, "Maybe I'll have sushi," the system confirms this and completes the order.

[1886] Example prompts for generative AI models

[1887] User input:

[1888] "What should I eat today?"

[1889] Pet robot response generation:

[1890] "My recommended menu items are curry rice, pasta, and sushi. Which would you like?"

[1891] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1892] Step 1:

[1893] A user launches an application on a device to order a meal and issues a voice command such as, "What should I eat today?" The input is the user's voice command, and the device's high-sensitivity microphone captures the voice. The output is the captured voice data.

[1894] Step 2:

[1895] The device transmits captured audio data to a remote server in real time. The input is audio data, which is transmitted to the server via data communication. The output is the audio data received by the server.

[1896] Step 3:

[1897] The server receives the voice data and analyzes it using a generative artificial intelligence model (OpenAI GPT-4) to generate an appropriate response. The input is the voice data, and data analysis and natural language processing are performed. The output is the generated response text.

[1898] Step 4:

[1899] The server sends the generated response text to the terminal. The input is the generated response text, which is sent to the terminal using data communication. The output is the response text that arrives at the terminal.

[1900] Step 5:

[1901] The device converts the received response text into speech and provides a response to the user through a speaker. The input is the response text, converts the text to speech, and plays the response speech. The output is the audio response that the user hears.

[1902] Step 6:

[1903] The menu selection made by the user (e.g., "Shall I have sushi?") is captured again by the device's high-sensitivity microphone and sent to the remote server. The input is the user's secondary voice instruction, and the voice data is captured and sent. The output is the voice data received by the server.

[1904] Step 7:

[1905] The server then uses the artificial intelligence model to analyze the voice data, confirm the selected menu, and generate the order. The input is the voice data, and data analysis and order generation are performed. The output is the generated order information.

[1906] Step 8:

[1907] The server sends the generated order information to the online delivery service to complete the order. The input is the order information and data transmission. The output is the completed order.

[1908] Step 9:

[1909] As soon as delivery status progress information arrives at the server, it is sent to the terminal in real time. The input is delivery progress information, which is sent to the terminal using data communication. The output is the delivery progress information that has arrived at the terminal.

[1910] Step 10:

[1911] The terminal provides delivery progress information to the user through voice and screen display. The input is delivery progress information, which is played back as voice and displayed on the screen. The output is information on the delivery progress status provided to the user.

[1912] Step 11:

[1913] The device automatically records the user's eating behavior and sends it to a remote server. The input is the automatically collected eating behavior data, which is sent to the server via data communication. The output is the meal record data received by the server.

[1914] Step 12:

[1915] The server stores the food record data in a database and notifies family members and caregivers at regular intervals. The input is the food record data, and the server stores the data and sends notifications. The output is the stored data and sent notifications.

[1916] Step 13:

[1917] The server periodically analyzes the user's emotional state using an emotion engine and generates appropriate responses and suggestions as needed. The input is the user's voice and facial expression data, and emotion analysis is performed. The output is the generated responses and suggestions.

[1918] Step 14:

[1919] The generated response or suggestion is sent to the terminal and provided to the user. The input is the generated response or suggestion, which is sent to the terminal using data communication. The output is the response or suggestion provided to the user.

[1920] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1921] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1922] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1923] [Fourth embodiment]

[1924] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1925] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1926] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1927] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1928] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1929] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1930] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1931] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1932] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1933] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1934] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1935] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1936] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1937] This invention relates to a pet robot that monitors the elderly. This pet robot has three functions: two-way communication, gesture recognition, and remote camera monitoring. It can help relieve the elderly's sense of loneliness, prevent dementia, and provide remote monitoring. Details on how to implement this system are described below.

[1938] System Configuration

[1939] The system has three main components:

[1940] 1. Pet robot main body (terminal)

[1941] High-sensitivity microphone

[1942] speaker

[1943] camera

[1944] Contact Sensor

[1945] Processor equipped with generative artificial intelligence model

[1946] 2. Remote Server

[1947] Operation of generative artificial intelligence models

[1948] Database

[1949] Alert generation function

[1950] Dashboard (for families and caregivers)

[1951] 3. Users (elderly people and their families)

[1952] User interface (natural interaction with pet robots, gestures)

[1953] Program processing overview

[1954] The system programs work together to realize the functions of two-way dialogue, gesture recognition, and remote monitoring. Specifically, the following processes are performed:

[1955] Two-way dialogue

[1956] 1. The user speaks

[1957] The user talks to the robot, asking questions such as "How was your day?"

[1958] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[1959] 2. Sending and analyzing audio data

[1960] The device transmits the captured audio data to the server in real time.

[1961] The server analyzes the voice data, generates an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[1962] 3. Executing the response

[1963] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[1964] Example: When a user asks, "How are you today?", the robot responds, based on the analysis results from the server, with, "I'm feeling great today!" At this time, the robot also performs actions such as shaking its body.

[1965] Gesture Recognition

[1966] 1. Performing a gesture

[1967] The user makes a gesture such as waving or pointing.

[1968] The device's camera captures the user's gestures in real time.

[1969] 2. Sending and analyzing gesture data

[1970] The device transmits the captured gesture data to the server.

[1971] The server analyzes the gesture, understands the user's intention, generates an appropriate response, and sends the resulting data to the device.

[1972] 3. Response Execution

[1973] The device performs an appropriate action (for example, waving back) based on the received response data.

[1974] Example: When a user waves their hand, the robot recognizes the movement with a camera, and based on the analysis results of the server, the robot waves back.

[1975] Remote monitoring

[1976] 1. Touch detection

[1977] When the user touches the robot, the device's contact sensor detects the touch.

[1978] 2. Sending log data

[1979] The device collects interaction data and periodically sends the log to a server.

[1980] 3. Data storage and analysis

[1981] The server stores the received interaction logs in a database and analyzes them. If an abnormality is detected, an alert is generated.

[1982] 4. Display on the dashboard

[1983] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers, allowing them to check on the elderly person's condition remotely.

[1984] Example: When a user strokes the robot's back, the interaction data is sent to a server and can be viewed by family members on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[1985] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. This also allows family members and caregivers to remotely monitor the elderly with peace of mind.

[1986] The processing flow will be explained below.

[1987] Two-way interaction processing steps

[1988] Step 1:

[1989] The user talks to the robot, asking questions such as "How was your day?"

[1990] Step 2:

[1991] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[1992] Step 3:

[1993] The device transmits the acquired voice data to the server in real time.

[1994] Step 4:

[1995] The server analyzes the received voice data and uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue.

[1996] Step 5:

[1997] The server transmits the generated response data to the terminal.

[1998] Step 6:

[1999] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[2000] Examples:

[2001] When a user says, "How are you today?", the device captures the audio and sends it to the server. The server generates a reply, "I'm feeling great today!", which the device then relays to the user.

[2002] Gesture Recognizer Processing Steps

[2003] Step 1:

[2004] The user makes a gesture (e.g., a wave) to the robot.

[2005] Step 2:

[2006] The device's camera captures the user's gestures in real time.

[2007] Step 3:

[2008] The device transmits the captured gesture data to the server.

[2009] Step 4:

[2010] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[2011] Step 5:

[2012] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[2013] Step 6:

[2014] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[2015] Examples:

[2016] When a user waves to the robot, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[2017] Remote monitoring function processing steps

[2018] Step 1:

[2019] The user touches the robot (for example, stroking its back).

[2020] Step 2:

[2021] The device's contact sensor detects contact and collects the data.

[2022] Step 3:

[2023] The device periodically transmits the acquired contact data to the server.

[2024] Step 4:

[2025] The server stores the received interaction logs in a database.

[2026] Step 5:

[2027] The server analyzes the stored data and generates an alert if an anomaly is detected.

[2028] Step 6:

[2029] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[2030] Examples:

[2031] When a user strokes the robot's back, the device detects the touch with a contact sensor and sends the data to a server, which analyzes and stores the data and displays it on a dashboard that family members can view remotely.

[2032] The specific processing steps for each function have been explained above. The specific operations performed in each step enable this pet robot to alleviate the sense of loneliness of elderly people and provide daily monitoring.

[2033] Example 1

[2034] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2035] Current technology has not yet fully addressed the challenges of simultaneously alleviating loneliness among the elderly, preventing dementia, and remotely monitoring them. In particular, there is a lack of systems that allow elderly people to enjoy natural conversations and interact through gestures, while also allowing remote family members and caregivers to monitor the elderly's status in real time. Current technology has difficulty simultaneously meeting these multiple needs, and therefore has not yet succeeded in increasing the sense of security for the elderly and their families.

[2036] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2037] In this invention, the server includes a two-way dialogue means for enabling natural dialogue with the user, a gesture recognition means for recognizing the user's gestures and performing corresponding actions, a remote monitoring means for monitoring the user's status using a remote camera and transmitting interaction logs and situation information to a remote server, a data transmission and response means for transmitting voice data and gesture data to the server and executing appropriate responses and response actions based on the analysis results, a voice and gesture analysis means equipped with a generative artificial intelligence model for analyzing the data received by the remote server, generating responses based on the analysis results, and transmitting the responses to the terminal, an alert generation means for analyzing the accumulated interaction logs and situation information and generating an alert if an abnormality is detected, and a dashboard means for displaying the analysis results and alert information. This makes it possible to simultaneously alleviate loneliness in the elderly, prevent dementia, and remotely monitor them.

[2038] The "two-way dialogue means" is a technology that uses voice recognition and voice synthesis to achieve two-way communication with the user in order to realize natural dialogue.

[2039] A "gesture recognition means" is a technology that uses a camera or other sensor to capture a user's physical movements, such as hand gestures or pointing, and analyzes those movements.

[2040] "Remote monitoring means" is a technology that uses remote cameras and other sensors to monitor the user's situation and transmits the data to a remote server.

[2041] "Data transmission and response means" refers to a technology that transmits voice data and gesture data collected from the terminal to a server in real time, receives analysis results from the server, and executes appropriate responses or response actions.

[2042] "Voice and gesture analysis means" refers to a technology that uses a generative artificial intelligence model installed on the server to analyze received voice and gesture data and generate a response based on the analysis results.

[2043] The "alert generation means" is a technology that analyzes accumulated interaction logs and situational information, and generates warning messages or notifications when an abnormality is detected.

[2044] "Dashboard means" is a technology that visually displays analysis results and alert information so that family members and caregivers can check them in real time.

[2045] This invention is a pet robot system for monitoring elderly people, which has three main functions: two-way interaction, gesture recognition, and remote camera monitoring. The system uses hardware such as a high-sensitivity microphone, speaker, camera, contact sensor, and processor equipped with a generative artificial intelligence model. Details for implementing this system are described below.

[2046] System Configuration

[2047] The system consists of three main components:

[2048] 1. Pet robot main body (terminal)

[2049] - High sensitivity microphone

[2050] - Speaker

[2051] - Camera

[2052] - Contact Sensor

[2053] - Processor equipped with generative artificial intelligence model

[2054] 2. Remote Server

[2055] - Operation of generative artificial intelligence models

[2056] - Database

[2057] - Alert generation function

[2058] - Dashboard (for families and caregivers)

[2059] 3. Users (elderly people and their families)

[2060] - User interface (natural interaction with pet robots, gestures)

[2061] Two-way dialogue

[2062] When a user speaks to the robot, the device's high-sensitivity microphone captures the voice. For example, if the user asks, "How was your day today?", the voice data is sent to the server in real time. The server analyzes the voice data, generates an appropriate response based on the context of the conversation, and sends this response data to the device. The device then plays back the received response data with natural voice and movements to respond to the user. For example, if the user asks, "How are you feeling today?" the robot will respond, "I'm feeling great today!" At this time, the robot will also perform movements such as shaking its body appropriately.

[2063] Gesture Recognition

[2064] When a user makes a gesture such as waving or pointing, the device's camera captures the user's gesture in real time. The device sends the captured gesture data to a server, which then analyzes the gesture. Based on the analysis results, the server understands the user's intention, generates an appropriate response action, and sends the resulting data to the device. The device then performs an appropriate action (for example, waving back) based on the received response data. As a specific example, when a user waves their hand, a robot recognizes the movement and waves back.

[2065] Remote monitoring

[2066] When a user touches the robot, the device's contact sensor detects the contact. The device collects the detected contact data and periodically sends the log to the server. The server stores the received contact log in a database and analyzes it. If an abnormality is detected, an alert is generated and family members or caregivers are notified. The server also displays the analysis results and contact log on a dashboard in real time, allowing the elderly person's condition to be checked remotely. For example, when a user strokes the robot's back, the contact data is sent to the server, and family members can check it on the remote dashboard. If an abnormality is detected, an alert is generated and a notification is sent.

[2067] Specific hardware and software names to be used

[2068] High-sensitivity microphone: General high-sensitivity microphone

[2069] Speaker: General speaker

[2070] Camera: General camera

[2071] Contact sensor: General contact sensor

[2072] Processor with generative artificial intelligence model: General processor

[2073] Remote Server: General Server

[2074] Generative Artificial Intelligence Model: General AI Model

[2075] Database: General database

[2076] Dashboard: General dashboard software

[2077] Specific examples of prompts to input to generative AI models

[2078] Speech analysis prompt: "What's the weather like tomorrow?"

[2079] Gesture analysis prompt: "Your hand wave has been recognized."

[2080] Interaction Log Prompt: "Back-pat sequence occurred"

[2081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2082] Two-way dialogue

[2083] Step 1: User speaks

[2084] The user can ask the robot a question or start a conversation, for example, "How was your day today?"

[2085] Input: User's voice

[2086] Output: Audio data

[2087] Specific operation: When the user speaks to the pet robot, the voice is captured by a highly sensitive microphone.

[2088] Step 2: The device receives and transmits the audio data

[2089] The device collects the captured audio data.

[2090] Input: Captured audio data

[2091] Output: Audio data for transmission

[2092] What it does: It compresses audio data in real time and sends it securely to the server using TLS encryption.

[2093] Step 3: The server analyzes the voice data and generates a response

[2094] The server analyzes the received voice data, performs speech recognition, and converts the speech to text using a generative AI model.

[2095] Input: Received audio data

[2096] Output: Contextual response text

[2097] Specific operation: After converting the voice data into text, it performs context analysis and generates an appropriate response, such as "I'm feeling great today!"

[2098] Step 4: The device responds

[2099] The terminal reproduces the received response data with natural voice and movements to respond to the user.

[2100] Input: Reply text

[2101] Output: Voice and action responses

[2102] Specific behavior: The response data is converted into voice and played back to the user. At the same time, the robot performs natural movements such as shaking its body.

[2103] Gesture Recognition

[2104] Step 1: The user makes a gesture

[2105] The user makes a gesture such as waving or pointing.

[2106] Input: User's physical movements

[2107] Output: Camera video data

[2108] Specific actions: When a user makes a gesture, the action is captured in real time by the camera.

[2109] Step 2: The device receives and sends gesture data

[2110] The device collects the captured gesture data.

[2111] Input: Captured video data

[2112] Output: Gesture data to send

[2113] How it works: Video data is compressed in real time and sent securely to the server using TLS encryption.

[2114] Step 3: The server analyzes the gesture data and generates a response

[2115] The server analyzes the received gesture data and understands the user's intention using a machine learning model.

[2116] Input: Received gesture data

[2117] Output: Response behavior data based on user intention

[2118] Specific Actions: Analyze gesture data and generate appropriate response actions, such as waving back.

[2119] Step 4: The device performs the response action

[2120] The terminal performs an appropriate operation based on the received response operation data.

[2121] Input: Response action data

[2122] Output: Actual device behavior

[2123] Specific actions: Based on the received action data, the robot will, for example, wave its hand back.

[2124] Remote monitoring

[2125] Step 1: Touch detection

[2126] When the user touches the robot, the device's contact sensor detects the touch.

[2127] Input: Physical touch

[2128] Output: Interaction data

[2129] Specific operation: When a user touches the robot, the contact sensor detects the touch and generates data.

[2130] Step 2: Device collects and sends log data

[2131] The device collects detected contact data and periodically sends the log data to a server.

[2132] Input: Interaction data

[2133] Output: Log data for sending

[2134] What it does: Collected log data is compressed in real time and sent securely to the server using TLS encryption.

[2135] Step 3: The server stores and analyzes the data

[2136] The server stores the received interaction logs in a database and analyzes them.

[2137] Input: Received log data

[2138] Output: Analysis results and anomaly detection results

[2139] Specific operation: The saved log data is analyzed periodically to check for any abnormalities. For example, if the frequency of contact decreases drastically, it will be recognized as an abnormality.

[2140] Step 4: View the server in the dashboard

[2141] The server displays the analysis results and interaction logs in real time on a dashboard for family members and caregivers.

[2142] Input: Analysis results and log data

[2143] Output: Visual information on a dashboard

[2144] Specific operation: Analysis results and log data are displayed on the dashboard, and if an abnormality is detected, an alert is displayed in real time. Visual information is also provided so that family members and caregivers can check the user's condition.

[2145] Specific examples of prompts to input to generative AI models

[2146] Speech analysis prompt: "What's the weather like tomorrow?"

[2147] Gesture analysis prompt: "Your hand wave has been recognized."

[2148] Interaction Log Prompt: "Back-pat sequence occurred"

[2149] (Application example 1)

[2150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2151] Problems faced by elderly people when using food delivery services include the burden of completing the complicated ordering process on their own, the difficulty of responding quickly and safely to the delivery person's arrival, and the uncertainty of having family members or caregivers supervise the delivery remotely. There is a need to solve these problems and provide a more secure and comfortable food delivery experience.

[2152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2153] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for supporting food delivery orders, and means for confirming delivery and generating alerts to notify the remote server. This reduces the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[2154] "Two-way dialogue" is a communication method that uses a generative artificial intelligence model that recognizes speech and generates responses to enable natural dialogue with the user.

[2155] A "generative artificial intelligence model" is an artificial intelligence algorithm that analyzes voice data and generates appropriate responses.

[2156] "Gesture recognition" is a technology that analyzes a user's actions captured by a camera, understands their meaning, and performs a corresponding action.

[2157] A "remote camera" is a camera device used to monitor the user's situation from a remote location.

[2158] An "interaction log" is information that records data when a user interacts with a robot and is used for later analysis.

[2159] A "remote server" is a server device in a remote location that has high-performance computing capabilities, analyzes data in real time, and stores necessary information.

[2160] "Food delivery" is a service that delivers food to a specific location.

[2161] "Ordering support" is a function that provides assistance through voice dialogue, etc., to make it easier for users to order food delivery.

[2162] "Delivery confirmation" is a way to confirm that the delivery person has arrived and to assist with safe pickup.

[2163] "Alert generation" is a function that generates a notification when an abnormality is detected.

[2164] This invention provides a pet robot system for supporting food delivery services for the elderly. The system includes functions such as two-way interaction, gesture recognition, and remote monitoring, allowing users to use food delivery services more safely and comfortably.

[2165] System Configuration

[2166] The system has three main components:

[2167] 1. Pet robot main body (terminal):

[2168] High-sensitivity microphone

[2169] speaker

[2170] camera

[2171] Contact Sensor

[2172] Processor equipped with generative artificial intelligence model

[2173] 2. Remote Server:

[2174] Operation of generative artificial intelligence models

[2175] Database

[2176] Alert generation function

[2177] Dashboard (for families and caregivers)

[2178] 3. Users (elderly people and their families):

[2179] User interface (natural interaction with pet robots, gestures)

[2180] Program processing overview

[2181] The system's programs work together to realize the functions of food delivery order support, delivery confirmation, and safety confirmation. Specifically, the following hardware and software are used to perform the processing.

[2182] Food delivery ordering assistance

[2183] 1. User: An elderly person speaks to their pet robot, saying, "I want to order dinner."

[2184] 2. Terminal: Captures audio using a sensitive microphone and transmits the data to a remote server.

[2185] 3. Remote server: A generative AI model analyzes the speech and generates an appropriate response, which is returned to the device.

[2186] 4. Terminal: Asks "Which restaurant would you like?" The user responds to proceed with the order.

[2187] Delivery Confirmation

[2188] 1. Terminal: When the delivery person arrives, the camera recognizes their arrival and sends the data to a remote server.

[2189] 2. Remote server: Notifies family members or caregivers based on the received data.

[2190] 3. Terminal: Respond appropriately to the delivery person by saying "Thank you for your hard work."

[2191] Safety confirmation

[2192] 1. User: When receiving meals, the robot will provide support to ensure that the elderly person receives the meal safely.

[2193] 2. Terminal: Capture the situation after receiving the item with a camera and send it to a remote server.

[2194] 3. Remote server: Based on the analysis results, if an abnormality is detected, an alert is generated to notify family members or caregivers.

[2195] Examples of concrete examples and prompts

[2196] As a specific scenario for operating this system, consider the case where elderly people use food delivery services.

[2197] Examples:

[2198] When an elderly person says to the robot, "I'd like to order sushi," the robot responds, "Which sushi restaurant would you like?" If the elderly person answers, "Sushida," the robot responds, "I'll open the Sushida menu," and proceeds with the order.

[2199] Example prompt sentence:

[2200] User: I'd like to order sushi.

[2201] Robot: Which sushi restaurant would you like?

[2202] User: Sushida.

[2203] Robot: Open the Sushida menu.

[2204] This will reduce the burden on elderly people when using food delivery services, allowing them to use the service safely and with peace of mind.

[2205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2206] Step 1:

[2207] The user speaks to the robot, saying, "I'd like to order dinner." The input data includes the user's voice data. The device uses a highly sensitive microphone to capture this voice data, and the voice data is acquired as input data.

[2208] Step 2:

[2209] The device transmits the captured voice data in real time to a remote server, where the voice data is processed for transmission. The remote server analyzes the voice data received as input. A generative AI model converts the voice to text, analyzes the context, and generates an appropriate response. In this process, the voice data is converted to text data and analyzed.

[2210] Step 3:

[2211] The remote server sends the generated response data to the device. The device then synthesizes the received response data into voice and responds to the user. Voice synthesis technology is used here, and text data is converted back into voice data. Specifically, the device asks the user, "Which restaurant would you like?"

[2212] Step 4:

[2213] The user responds with "Sushida." The user's voice data is again captured as input data, and the device captures and sends it to the remote server. As in the previous step, the server converts the voice data into text data and generates an analysis result.

[2214] Step 5:

[2215] The remote server generates an appropriate response, such as "Sushida's menu will be displayed," and sends it to the device. The device then synthesizes this response and responds to the user, for example, "Sushida's menu will be displayed."

[2216] Step 6:

[2217] When the delivery person arrives, the device's camera captures the video data, which is then sent as input data to a remote server, which analyzes the video data and confirms the delivery person's arrival.

[2218] Step 7:

[2219] The remote server then notifies the family or caregiver of the results of the check. The notification includes information about the delivery person's arrival and status. At the same time, the device responds appropriately to the delivery person by saying, "Thank you for your hard work."

[2220] Step 8:

[2221] When the user receives the meal, the device's camera and contact sensors monitor the situation. The video data and contact log at the time of receipt are sent as input data to a remote server. The remote server analyzes this data and checks for any abnormalities.

[2222] Step 9:

[2223] If the remote server detects any abnormalities based on the analysis results, it generates an alert and notifies the family or caregiver, allowing them to confirm safety. If everything is normal, it notifies them that the delivery has been completed successfully.

[2224] These steps will create a system that allows seniors to use food delivery services safely and easily.

[2225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2226] This invention relates to a pet robot for elderly care that combines an emotion engine. This pet robot is equipped with two-way communication, gesture recognition, remote camera monitoring, and emotion recognition functions, helping to alleviate feelings of loneliness in elderly people, preventing dementia, and remotely monitoring them. Details for implementing this system are described below.

[2227] System Configuration

[2228] The system comprises four main components:

[2229] 1. Pet robot main body (terminal)

[2230] High-sensitivity microphone

[2231] speaker

[2232] camera

[2233] Contact Sensor

[2234] Processor equipped with generative artificial intelligence model

[2235] Emotion Engine

[2236] 2. Remote Server

[2237] Operation of generative artificial intelligence models

[2238] Database

[2239] Alert generation function

[2240] Dashboard (for families and caregivers)

[2241] 3. Users (elderly people and their families)

[2242] User interface (natural interaction with pet robots, gestures)

[2243] 4. Emotion Engine

[2244] Recognizing emotions from the user's voice and facial expressions

[2245] Program processing overview

[2246] The system programs work together to realize the functions of two-way dialogue, gesture recognition, remote monitoring, and emotion recognition. Specifically, the following processes are performed:

[2247] Two-way dialogue

[2248] 1. The user speaks

[2249] The user talks to the robot, asking questions such as "How was your day?"

[2250] The terminal (robot) uses a highly sensitive microphone to capture audio and obtain the audio data.

[2251] 2. Sending and analyzing audio data

[2252] The device transmits the acquired voice data to the server in real time.

[2253] The server analyzes the voice data and recognizes the user's emotions using an emotion engine.

[2254] 3. Response Generation and Execution

[2255] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the dialogue, and sends the response data to the terminal.

[2256] The device responds to the user by expressing the received response data in voice and motion, thereby realizing a dialogue that responds to the user's emotions.

[2257] Examples:

[2258] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" If the emotion engine recognizes the emotion "lonely" from the tone of the user's voice, the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[2259] Gesture Recognition

[2260] 1. Performing a gesture

[2261] The user makes a gesture (e.g., a wave) to the robot.

[2262] The device's camera captures the user's gestures in real time.

[2263] 2. Sending and analyzing gesture data

[2264] The device transmits the captured gesture data to the server.

[2265] The server analyzes the received gesture data and understands the meaning of the user's gesture.

[2266] 3. Response Execution

[2267] The server generates an appropriate response action based on the analysis result and transmits the data to the terminal.

[2268] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[2269] Examples:

[2270] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[2271] Remote monitoring

[2272] 1. Touch detection

[2273] The user touches the robot (for example, stroking its back).

[2274] The device's contact sensor detects contact and collects the data.

[2275] 2. Sending log data

[2276] The device periodically transmits the acquired contact data to the server.

[2277] 3. Data storage and analysis

[2278] The server stores the received interaction logs in a database.

[2279] 4. Displaying analysis results and generating alerts

[2280] The server analyzes the stored data and generates an alert if an anomaly is detected.

[2281] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[2282] Examples:

[2283] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, and displays it on a dashboard that family members can view remotely. If an abnormality is detected, an alert is generated and notified.

[2284] Emotion recognition function

[2285] 1. Recognizing the user's voice and facial expressions

[2286] The device captures the user's voice and facial expressions through a camera and microphone.

[2287] The emotion engine recognizes the user's emotions from the captured data.

[2288] 2. Transmission and analysis of emotional data

[2289] The device transmits the recognized emotion data to the server.

[2290] The server analyzes the emotional data to adjust dialogue and behavior.

[2291] 3. Generating and executing responses according to emotions

[2292] The server generates appropriate responses and actions based on the emotion data and sends that data to the terminal.

[2293] The terminal executes the received response data and expresses an appropriate emotion to the user.

[2294] Examples:

[2295] If a user says, "I'm feeling lonely after being at home all day today," the emotion engine recognizes the emotion of "lonely." The server generates data to respond in a cheerful tone, saying, "I see you're lonely. Let's play together!", and the device executes that response.

[2296] In this way, each function works together to provide a system that alleviates the sense of loneliness felt by the elderly and enables them to be monitored. In addition, the addition of an emotion recognition function further improves emotional care for the elderly, allowing family members and caregivers to remotely monitor the elderly with peace of mind.

[2297] The processing flow will be explained below.

[2298] Two-way interaction processing steps

[2299] Step 1:

[2300] The user talks to the robot, for example, asking, "How was your day?"

[2301] Step 2:

[2302] The terminal (robot) uses a highly sensitive microphone to capture the user's voice and obtains the voice data.

[2303] Step 3:

[2304] The device transmits the acquired voice data to the server in real time.

[2305] Step 4:

[2306] The server analyzes the received voice data using a generative artificial intelligence model to understand the content and context of the speech.

[2307] Step 5:

[2308] The server uses an emotion engine to analyze the user's tone of voice and phrasing to recognize the user's emotions.

[2309] Step 6:

[2310] The server uses a generative artificial intelligence model to generate an appropriate response based on the context of the conversation and the emotion recognition results, and sends the response data to the terminal.

[2311] Step 7:

[2312] The terminal outputs the received response data by voice, and the robot responds to the user using natural movements.

[2313] Examples:

[2314] When a user says, "How are you today?", the device captures the voice and sends it to the server. The server generates a reply saying, "I'm feeling great today!" At the same time, the emotion engine recognizes "lonely" from the tone of the user's voice, and the robot responds in a cheerful voice, "I'm feeling great today! Let's play!"

[2315] Gesture Recognizer Processing Steps

[2316] Step 1:

[2317] The user makes a gesture (e.g., a wave) to the robot.

[2318] Step 2:

[2319] The device's camera captures the user's gestures in real time.

[2320] Step 3:

[2321] The device transmits the captured gesture data to the server.

[2322] Step 4:

[2323] The server analyzes the received gesture data using machine learning models to understand the meaning of the gesture.

[2324] Step 5:

[2325] The server generates an appropriate response action based on the user's gesture and transmits the response data to the terminal.

[2326] Step 6:

[2327] Based on the received response data, the device performs an action corresponding to the gesture (e.g., waving back).

[2328] Examples:

[2329] When a user waves, the device's camera captures the gesture and sends it to the server. The server then analyzes it, recognizes it as a "greeting," and the device waves back.

[2330] Remote monitoring function processing steps

[2331] Step 1:

[2332] The user touches the robot (for example, stroking its back).

[2333] Step 2:

[2334] The device's contact sensor detects contact and collects the data.

[2335] Step 3:

[2336] The device periodically transmits the acquired contact data to the server.

[2337] Step 4:

[2338] The server stores the received interaction logs in a database.

[2339] Step 5:

[2340] The server analyzes the stored data to detect abnormalities in the elderly person's behavior and physical condition, and if an abnormality is detected, it generates an alert.

[2341] Step 6:

[2342] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[2343] Examples:

[2344] When a user strokes the robot's back, the device detects the contact with a contact sensor and sends the data to the server. The server analyzes and stores the data, which family members can view on a remote dashboard. If an abnormality is detected, an alert is generated and notified.

[2345] Emotion Recognition Processing Steps

[2346] Step 1:

[2347] The user speaks to the robot or shows facial expressions.

[2348] The device's camera captures the user's facial expressions, and a highly sensitive microphone captures their voice.

[2349] Step 2:

[2350] The terminal transmits the user's voice data and facial expression data to the server in real time.

[2351] Step 3:

[2352] The server analyzes the received voice data and facial expression data and recognizes the user's emotions (e.g., joy, sadness, anger) using an emotion engine.

[2353] Step 4:

[2354] The server generates appropriate responses and actions based on the emotion recognition results and sends the data to the terminal.

[2355] Step 5:

[2356] The terminal executes the received response data and gives a natural response that matches the user's emotions.

[2357] Examples:

[2358] If the user says, "I've been lonely at home all day today," the emotion engine recognizes the emotion "lonely." The server generates a response, "I see you've been lonely. Let's play together!", and the robot conveys this to the user in a cheerful tone.

[2359] The specific processing steps for each function, including the emotion engine, have been explained above. The specific actions performed at each step enable this pet robot to alleviate the sense of loneliness felt by the elderly and provide daily monitoring.

[2360] Example 2

[2361] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2362] There is a need for systems that can prevent elderly people from feeling lonely and developing dementia, and that can also monitor them remotely. To alleviate the feelings of loneliness and anxiety that elderly people experience in their daily lives, natural dialogue that responds to emotions and interactive responses through gesture recognition are necessary. Remote monitoring functions are also required to make it easier for family members and caregivers to understand the condition of the elderly. Existing systems lack comprehensive solutions to meet these needs.

[2363] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2364] In this invention, the server includes means for enabling natural dialogue with the user using an artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and situation information to the remote server, means for recognizing the user's voice and facial expressions and analyzing emotions, and means for generating appropriate responses based on the analyzed emotional data and providing them to the user. This makes it possible to reduce loneliness among the elderly, prevent dementia, and even monitor them remotely.

[2365] An "artificial intelligence model" is a program designed to naturally interact with users, using speech recognition and natural language processing technology to understand the meaning of what the user says and generate appropriate responses.

[2366] "Gesture recognition" is a technology that captures a user's actions (such as waving a hand) using a sensor such as a camera, and analyzes and determines the meaning of the action.

[2367] A "remote camera" is a device that captures video from a remote location and transmits it to a remote server in real time, and is used to monitor users.

[2368] An "interaction log" is a record of physical contact between the user and the robot (such as a back pat) and is sent to a remote server.

[2369] An "emotion engine" is a technology that analyzes a user's voice and facial expression data to recognize the user's emotions at that time (for example, joy, sadness, loneliness, etc.).

[2370] "Response generation based on analysis results" is a technology that generates appropriate responses and actions based on the results of analyzing the user's statements, gestures, and emotional data.

[2371] "Alert generation" is a technology that notifies family members or caregivers from a remote server when an abnormality is detected in the user's behavior or condition.

[2372] "Users" refer to the elderly people who use the system, as well as their families and caregivers, who interact with the robot through voice and gestures.

[2373] This invention is a system that combines an emotion engine with a pet robot for elderly care, alleviating the sense of loneliness of the elderly, preventing dementia, and remotely monitoring them. The components of this system include the pet robot itself, a remote server, users (elderly people and their families), and the emotion engine.

[2374] System Configuration

[2375] 1. Pet robot main body (terminal)

[2376] High-sensitivity microphone: Captures user voice in real time.

[2377] Speaker: Replies sent from the server are spoken to the user.

[2378] Camera: Captures user gestures and facial expressions and provides data for analysis.

[2379] Contact sensor: Acquires contact data when the user touches the robot.

[2380] Processor equipped with generative artificial intelligence model: Performs voice analysis, gesture recognition, and emotion analysis.

[2381] Emotion engine: Technology for recognizing and analyzing user emotions.

[2382] 2. Remote Server

[2383] Operation of generative artificial intelligence models: Analyze acquired data and generate appropriate responses and actions.

[2384] Database: Stores interaction data and emotion analysis results.

[2385] Alert generation function: If an abnormality is detected, an alert is generated to notify family members or caregivers.

[2386] Dashboard (for family and caregivers): Displays analysis results and interaction logs, allowing users to check their condition.

[2387] 3. Users (elderly people and their families)

[2388] User Interface: Interact with the pet robot through natural dialogue and gestures.

[2389] 4. Emotion Engine

[2390] Recognizing emotions from voice and facial expressions: Analyze the user's voice tone and facial expressions to generate emotional data.

[2391] System program processing overview

[2392] The system works to recognize the user's voice and gestures, generate appropriate responses and actions based on them, and realize two-way dialogue according to emotions.

[2393] For example, when a user asks, "How was your day?", the device's high-sensitivity microphone captures the voice and sends the voice data to the server. The server uses a generative artificial intelligence model to analyze the voice data, and an emotion engine recognizes the user's emotion. The server then generates an appropriate response and sends it to the device. The response, "I'm feeling great today!", is heard through the device's speaker.

[2394] When a user makes a waving gesture, the device's camera captures the gesture and sends it to the server. The server analyzes the gesture, generates the necessary response action, and sends it to the device. The device then performs the waving action.

[2395] Furthermore, when the user touches the robot, the device's contact sensor captures the data and sends it to the server, where it is stored in a database and can be viewed by family members or caregivers via a dashboard. If an abnormality is detected, an alert is generated and a remote notification is sent.

[2396] Examples of prompt statements

[2397] "Please explain in detail the process by which an elderly care robot recognizes emotions from the user's voice and facial expressions and returns an appropriate response."

[2398] In this way, each function of this system works together to reduce loneliness and provide care for the elderly. In addition, the addition of an emotion recognition function improves emotional care for the elderly and allows family members and caregivers to remotely monitor them with peace of mind.

[2399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2400] Handling two-way interactions

[2401] Step 1: Capture audio

[2402] Input: The user speaks to the robot (e.g., "How was your day?").

[2403] The device uses a high-sensitivity microphone to capture the user's voice in real time.

[2404] Output: The audio data is captured on the device.

[2405] Step 2: Sending audio data

[2406] Input: Audio data captured on the device.

[2407] The device transmits the captured audio data to the server in real time.

[2408] Output: The audio data is sent to the server.

[2409] Step 3: Voice analysis and emotion recognition

[2410] Input: The audio data sent to the server.

[2411] The server uses a generative artificial intelligence model to analyze the voice data and understand what the user is saying.

[2412] The server uses an emotion engine to recognize the user's emotions (e.g., joy, sadness, loneliness) from the voice.

[2413] Output: The analyzed speech content and emotion data are obtained.

[2414] Step 4: Response generation and speech output

[2415] Input: Parsed speech and sentiment data.

[2416] The server generates an appropriate response based on the analysis results.

[2417] The server generates a response and sends it to the terminal.

[2418] The device provides a response to the user through the speaker (e.g., "I'm feeling great today!").

[2419] Output: A voice is output in response to the user.

[2420] Gesture Recognition Processing

[2421] Step 1: Gesture Capture

[2422] Input: The user makes a gesture towards the robot (e.g., waving).

[2423] The device uses a camera to capture the user's gestures in real time.

[2424] Output: Gesture data is captured on the device.

[2425] Step 2: Sending gesture data

[2426] Input: Gesture data captured on the device.

[2427] The device transmits the captured gesture data to the server.

[2428] Output: Gesture data is sent to the server.

[2429] Step 3: Gesture analysis

[2430] Input: The gesture data sent to the server.

[2431] The server analyzes the received gesture data and understands its meaning (e.g., greeting, request).

[2432] Output: The parsed meaning of the gesture is obtained.

[2433] Step 4: Generate and execute response actions

[2434] Input: Parsed gesture meaning.

[2435] Based on the analysis results, the server generates an appropriate response action and sends it to the device (e.g., waving back).

[2436] The terminal performs an action based on the received response action data.

[2437] Output: The action corresponding to the gesture is performed.

[2438] Remote monitoring processing

[2439] Step 1: Touch detection

[2440] Input: The user touches the robot (e.g., pats its back).

[2441] The device's contact sensor detects this contact and captures the data.

[2442] Output: Interaction data is captured on the device.

[2443] Step 2: Sending log data

[2444] Input: Interaction data captured on the device.

[2445] The device periodically transmits the acquired contact data to the server.

[2446] Output: Interaction data is sent to the server.

[2447] Step 3: Data storage and analysis

[2448] Input: Interaction data sent to the server.

[2449] The server stores the received interaction data in a database.

[2450] The server analyzes the stored data and generates alerts if anomalies are detected.

[2451] Output: Saved interaction data and analysis results are obtained.

[2452] Step 4: Viewing analysis results and generating alerts

[2453] Input: Saved interaction data and its analysis results.

[2454] The server displays the analysis results and interaction logs on a dashboard that can be viewed by family members or caregivers.

[2455] The server generates an alert and notifies you if an abnormality is detected.

[2456] Output: Data is displayed on the dashboard and alerts are sent.

[2457] Emotion recognition processing

[2458] Step 1: Capture your voice and facial expressions

[2459] Input: User's voice and facial expressions.

[2460] The device captures the user's voice and facial expressions in real time using a camera and microphone.

[2461] Output: Voice data and facial expression data are acquired on the terminal.

[2462] Step 2: Send and analyze emotion data

[2463] Input: Voice data and facial expression data acquired on the device.

[2464] The terminal transmits the acquired data to the server.

[2465] The server analyzes the received data using an emotion engine and recognizes the user's emotions.

[2466] Output: The analyzed emotion data is obtained.

[2467] Step 3: Generate and execute a response

[2468] Input: Parsed emotion data.

[2469] The server generates appropriate responses and actions based on the emotional data and sends them to the terminal.

[2470] The terminal executes the received response data and expresses an appropriate emotion to the user.

[2471] Output: A response according to the emotion is provided to the user.

[2472] The above is the specific flow of the program processing for this system. At each step, appropriate data processing and calculations are performed based on the input data, and the output is used in the processing of the next step. This enables interaction according to the user's condition and emotions, realizing a system that can alleviate the loneliness of elderly people while also allowing family members and caregivers to monitor them remotely.

[2473] (Application example 2)

[2474] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2475] The goal is to eliminate the difficulties that elderly people face when using food delivery services independently, ease their sense of loneliness, and enable them to order meals with peace of mind, check delivery status, and easily manage meal records. Another goal is to provide emotional care for elderly people by utilizing emotion recognition functions and realize appropriate remote monitoring.

[2476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2477] In this invention, the server includes means for enabling natural dialogue with the user using a generative artificial intelligence model that performs two-way dialogue, means for recognizing the user's gestures and performing corresponding actions, means for monitoring the user using a remote camera and sending interaction logs and status information to the remote server, means for assisting the user in selecting a menu when ordering a meal, means for presenting the user with the progress of delivery, and means for automatically managing meal records and sending the record information to the remote server. This makes food delivery services easier and more secure for the elderly, alleviating their sense of loneliness and enabling remote monitoring.

[2478] "Two-way dialogue" is a mechanism in which users and systems communicate with each other in real time using voice and text, enabling natural conversation.

[2479] A "generative artificial intelligence model" is a type of artificial intelligence that learns from large amounts of data and has the ability to engage in natural conversations, recognize emotions, and generate responses.

[2480] "Gesture recognition" is a technology that uses cameras and sensors to detect a user's body movements and hand gestures and respond accordingly.

[2481] A "remote camera" is a camera installed in a physically separate location, and is a device that allows you to view images from a remote location.

[2482] "User monitoring" is a function in which the system monitors the user's behavior and situation and provides necessary support and alerts.

[2483] An "interaction log" is recorded data of contacts and interactions that occur between a user and a system.

[2484] "Menu selection assistance" is a function that shows the options presented when the user orders a meal and helps the user select an appropriate menu from among them.

[2485] "Delivery progress" is information that indicates the current status and progress of the delivery process of the ordered product.

[2486] "Dietary logging" is the process by which the system automatically saves and manages the food and drink intake of the user.

[2487] The system according to the present invention comprises the following major hardware and software components:

[2488] Hardware Configuration

[2489] 1. Device (smartphone)

[2490] High-sensitivity microphone: Captures the user's voice.

[2491] Speaker: Provides audio responses to the user.

[2492] Camera: Captures user gestures and facial expressions.

[2493] Display: Presents visual information and interfaces.

[2494] Processor: Responsible for processing information and running applications.

[2495] 2. Remote Server

[2496] High-performance processor: Responsible for running generative artificial intelligence models and emotion engines.

[2497] Database: Stores user data and log information.

[2498] Alert function: Detects and notifies abnormalities.

[2499] Software Configuration

[2500] 1. Generative AI models (e.g., OpenAI GPT-4)

[2501] It performs natural language analysis of the user's voice instructions and generates an appropriate response.

[2502] Determine an action based on the user's gesture.

[2503] 2. Emotion engine (e.g. Affectiva SDK)

[2504] It analyzes the user's facial expressions and tone of voice to recognize emotions.

[2505] 3. Admin Dashboard

[2506] An interface that allows family members and caregivers to check the user's condition remotely.

[2507] 4. Alert notification system (e.g. Twilio API)

[2508] A remote server sends a notification to family members / caregivers when an abnormality is detected.

[2509] Processing Overview

[2510] Food ordering support

[2511] When a user wants to order a meal, they launch an application on their device and give their voice command. The device's sensitive microphone captures the voice, and a generative artificial intelligence model analyzes the command. The device displays the appropriate menu and transmits the user's selection to the server.

[2512] Check delivery status

[2513] If the user wants to check the delivery status, the generative artificial intelligence model generates a voice response through the terminal, providing the user with the delivery progress status through voice and display.

[2514] Food log

[2515] After the user has finished eating, the device application automatically sends the record to the server and stores it in a database. This information is then periodically sent to family members or caregivers.

[2516] Emotion recognition function

[2517] The user's voice and facial expressions are captured by the device's camera and microphone, and analyzed by the emotion engine. Based on the analysis results, the generative AI model generates appropriate responses and actions, which are then provided to the user.

[2518] Specific examples

[2519] For example, if a user says, "What should I eat today?", the device's microphone captures the voice and sends it to the server. A generative artificial intelligence model installed on the server analyzes it and generates a response such as, "The recommended menu items are curry rice, pasta, and sushi. Which would you like?" If the user answers, "Maybe I'll have sushi," the system confirms this and completes the order.

[2520] Example prompts for generative AI models

[2521] User input:

[2522] "What should I eat today?"

[2523] Pet robot response generation:

[2524] "My recommended menu items are curry rice, pasta, and sushi. Which would you like?"

[2525] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2526] Step 1:

[2527] A user launches an application on a device to order a meal and issues a voice command such as, "What should I eat today?" The input is the user's voice command, and the device's high-sensitivity microphone captures the voice. The output is the captured voice data.

[2528] Step 2:

[2529] The device transmits captured audio data to a remote server in real time. The input is audio data, which is transmitted to the server via data communication. The output is the audio data received by the server.

[2530] Step 3:

[2531] The server receives the voice data and analyzes it using a generative artificial intelligence model (OpenAI GPT-4) to generate an appropriate response. The input is the voice data, and data analysis and natural language processing are performed. The output is the generated response text.

[2532] Step 4:

[2533] The server sends the generated response text to the terminal. The input is the generated response text, which is sent to the terminal using data communication. The output is the response text that arrives at the terminal.

[2534] Step 5:

[2535] The device converts the received response text into speech and provides a response to the user through a speaker. The input is the response text, converts the text to speech, and plays the response speech. The output is the audio response that the user hears.

[2536] Step 6:

[2537] The menu selection made by the user (e.g., "Shall I have sushi?") is captured again by the device's high-sensitivity microphone and sent to the remote server. The input is the user's secondary voice instruction, and the voice data is captured and sent. The output is the voice data received by the server.

[2538] Step 7:

[2539] The server then uses the artificial intelligence model to analyze the voice data, confirm the selected menu, and generate the order. The input is the voice data, and data analysis and order generation are performed. The output is the generated order information.

[2540] Step 8:

[2541] The server sends the generated order information to the online delivery service to complete the order. The input is the order information and data transmission. The output is the completed order.

[2542] Step 9:

[2543] As soon as delivery status progress information arrives at the server, it is sent to the terminal in real time. The input is delivery progress information, which is sent to the terminal using data communication. The output is the delivery progress information that has arrived at the terminal.

[2544] Step 10:

[2545] The terminal provides delivery progress information to the user through voice and screen display. The input is delivery progress information, which is played back as voice and displayed on the screen. The output is information on the delivery progress status provided to the user.

[2546] Step 11:

[2547] The device automatically records the user's eating behavior and sends it to a remote server. The input is the automatically collected eating behavior data, which is sent to the server via data communication. The output is the meal record data received by the server.

[2548] Step 12:

[2549] The server stores the food record data in a database and notifies family members and caregivers at regular intervals. The input is the food record data, and the server stores the data and sends notifications. The output is the stored data and sent notifications.

[2550] Step 13:

[2551] The server periodically analyzes the user's emotional state using an emotion engine and generates appropriate responses and suggestions as needed. The input is the user's voice and facial expression data, and emotion analysis is performed. The output is the generated responses and suggestions.

[2552] Step 14:

[2553] The generated response or suggestion is sent to the terminal and provided to the user. The input is the generated response or suggestion, which is sent to the terminal using data communication. The output is the response or suggestion provided to the user.

[2554] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2556] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2557] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2558] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other. 【...

Claims

1. A means for enabling natural dialogue with a user by utilizing a generative artificial intelligence model that performs two-way dialogue; means for recognizing a user's gesture and performing a corresponding action; A means for monitoring the user using a remote camera and transmitting interaction logs and status information to a remote server; A system including:

2. 10. The system of claim 1, further comprising means for utilizing a generative artificial intelligence model to recognize a user's speech and generate appropriate responses based on the context.

3. The system according to claim 1, further comprising means for detecting abnormalities in the behavior or physical condition of the elderly person based on the analysis results transmitted from the remote server and generating an alert.

Images