System

Abstract

A system is provided.SOLUTION: A system including a means for collecting physical motion and voice of a support target person by using a wearable device that captures the behavior of the support target person, a means for transmitting data collected from the wearable device to a server and preprocessing the data, a means for analyzing the preprocessed data using a generative AI and estimating an intention of the support target person, a means for converting the estimated intention into voice data and transmitting the voice data to a device used by a support target person, and a means for reproducing the transmitted voice data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In an aging society, it is predicted that the number of people with cognitive problems will increase. Such people have difficulty effectively expressing their intentions and have problems communicating with their relatives and caregivers. As a result, they are unable to receive appropriate support, leading to a decline in their quality of life. The present invention aims to solve this problem by providing a system that accurately infers the intentions of the person being supported from their words, actions, and behavior, and conveys these to the supporter. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including: means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported; means for sending the data collected from the wearable device to a server and preprocessing the data; means for analyzing the preprocessed data using generative AI and inferring the intentions of the person being supported; means for converting the inferred intentions into voice data and sending it to a device used by the supporter; and means for playing back the transmitted voice data.

[0006] "Support recipients" refer to people who have cognitive problems and have difficulty communicating their intentions effectively.

[0007] A "wearable device" is an electronic device that can be worn on the body and is equipped with sensors that have the ability to collect physical movements and sounds.

[0008] "Physical movements" refer to movements and actions performed by the person being supported, such as standing up, walking, or raising one's arms.

[0009] "Voice" refers to the voice emitted by the person being supported, including sound signals that include conversation and other expressions of intent.

[0010] "Server" refers to a computer system that receives data sent from a wearable device and performs preprocessing and analysis.

[0011] "Preprocessing" refers to the process of converting raw data received from a wearable device into a format that is easier to analyze.

[0012] "Generative AI" refers to artificial intelligence that analyzes data based on natural language processing and machine learning, and infers the intentions of the person receiving support.

[0013] "Inference" refers to the process by which the generative AI analyzes and determines the intentions of the person being supported.

[0014] "Speech data" refers to the digital speech signals generated to vocally convey the estimated intent.

[0015] "Device used by the supporter" refers to a device that has the function of receiving and playing audio data, such as audio glasses. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to the supporter. This system includes a wearable device, a server, a generative AI, and devices such as audio glasses.

[0038] 1. Wearable devices that capture the behavior of those receiving support

[0039] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0040] 2. Data preprocessing and transmission

[0041] The device sends the collected data to the server, where it is encoded and converted into an appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and voice data recorded by the microphone are sent to the server.

[0042] 3. Analysis of Inferred Intent

[0043] The server preprocesses the received data and formats it into a unified format before inputting it into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generation AI will interpret this as, "It's cold, so I want to go back indoors."

[0044] 4. Speech conversion and output of estimated intention

[0045] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device such as audio glasses worn by the support person.

[0046] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0047] Specific examples

[0048] For example, the following scenario is possible:

[0049] Step 1: The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's speech and collects movement data.

[0050] Step 2: The wearable device sends voice and motion data to the server.

[0051] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0052] Step 4: The server converts this intent into audio data and sends it to the audio glasses.

[0053] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0054] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported and enables the supporter to respond appropriately.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The device collects the movements and voice of the person being supported. Specifically, the wearable device uses an accelerometer and gyroscope to capture physical movement data, and a microphone to record voice.

[0058] Step 2:

[0059] The device preprocesses and organizes this data, for example, by recognizing walking movements from accelerometer and gyroscope data and converting recorded audio data into text using voice recognition technology.

[0060] Step 3:

[0061] The device sends the preprocessed data to the server, where it is encoded according to a communication protocol and transmitted using a secure communication method (e.g., HTTPS).

[0062] Step 4:

[0063] The server further preprocesses the received data for analysis, specifically by converting different data formats into a unified and consistent format.

[0064] Step 5:

[0065] The server inputs the formatted data into the generation AI for analysis. The generation AI uses a natural language processing model (e.g., GPT-4) to infer the intention of the person being supported.

[0066] Step 6:

[0067] The server receives the analysis results of the generation AI in text format and inputs the text into a speech synthesis engine, which converts the text into voice data.

[0068] Step 7:

[0069] The server then transmits the generated audio data to the device (audio glasses) worn by the supporter, where the data is again encoded and transmitted via a secure means.

[0070] Step 8:

[0071] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The audio data is played and transmitted to the support person.

[0072] Step 9:

[0073] The user (support person) listens to the audio through the audio glasses and understands the intention of the person being supported. For example, if the person being supported says, "It's cold, so I want to go back indoors," the support person will guide the person indoors.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] In modern society, it is extremely important to accurately grasp the intentions of those receiving support who have cognitive disabilities and to convey them quickly and accurately to those providing support. However, conventional technology has difficulty accurately grasping the intentions of those receiving support, resulting in delayed responses and often misinterpretations. Furthermore, when analyzing data collected from multiple sensors and devices, inefficiencies in integration and preprocessing make it difficult to respond in real time. There is a need to provide a system that solves these issues and facilitates smooth communication between those receiving support and those providing support.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for encoding the data collected from the wearable device, converting it into an appropriate data format, and transmitting it to the server, means for the server to preprocess the data received and estimate the intention of the person being supported using a generative AI model, means for generating the estimated intention in text format, inputting it into a speech synthesis engine to convert it into voice data, and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp the intention of the person being supported in real time and convey it to the supporter quickly and accurately.

[0079] A "wearable device" is a device that is worn on the body and is equipped with sensors and communication functions to collect the physical movements and voices of the person being supported.

[0080] An "accelerometer" is a type of sensor installed in wearable devices, and is a device that measures the acceleration of an object.

[0081] A "gyroscope" is a device installed in wearable devices that measures angular velocity to detect changes in the rotation or tilt of an object.

[0082] A "microphone" is a device that converts sound into an electrical signal, and is a component installed in wearable devices to collect the speech of the person being supported.

[0083] "GPS" stands for Global Positioning System, a system for measuring positions on Earth with high precision.

[0084] "Encoding" is the process of converting data into a particular format, with the goal of optimizing it for transmission or storage.

[0085] A "generative AI model" is an artificial intelligence model that analyzes data and generates new information or results based on that information, and is used here to infer the intentions of the person being supported.

[0086] A "speech synthesis engine" is a system that converts text data into speech data, and is a device used to output the estimated intention as speech.

[0087] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to a supporter. This system includes devices such as a wearable device, a server, a generative AI, and an audio output device.

[0088] First, a wearable device is used to capture the behavior of the person being supported. A specific example is a smartwatch. This device is equipped with an accelerometer, gyroscope, microphone, and GPS, and collects the physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") of the person being supported in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0089] Next, when the device sends the collected data to the server, the data is encoded and converted into the appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and audio data recorded by the microphone are sent to the server. The audio data is encoded into MP3 format, and the sensor data is converted into JSON format.

[0090] The server preprocesses the received data. This preprocessing includes formatting data from different sensors into a unified format. The formatted data is then input into a generative AI model (e.g., GPT-4). The generative AI model analyzes the data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generative AI will interpret this as, "It's cold, so I want to go back indoors."

[0091] The server generates the estimated intention in text format. This text is input into a speech synthesis engine and converted into voice data. The generated voice data is sent to an audio output device (such as audio glasses) worn by the support person. The device (audio output device) plays the received voice data and conveys it to the support person. This ensures that the intention of the person being supported is accurately conveyed. For example, the audio glasses could say, "It's cold, so I'd like to go back indoors," allowing the support person to take appropriate action.

[0092] To illustrate, consider the following scenario:

[0093] 1. The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's words and collects movement data.

[0094] 2. The wearable device sends voice and movement data to the server.

[0095] 3. The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0096] 4. The server converts this intent into audio data and sends it to the audio glasses.

[0097] 5. The audio glasses will tell the support person, "It's cold, I want to go back indoors."

[0098] Below is an example of a prompt sentence to input to the generative AI model.

[0099] Example 1: "The person you are supporting walks away saying, 'It's cold.' Guess what he wants to do."

[0100] Example 2: "The user stands up and says 'I'm thirsty.' Please infer the intention of this action."

[0101] As described above, the present invention makes it possible to accurately grasp the intentions of the person being supported in real time and to convey them to the person providing support quickly and accurately.

[0102] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0103] Step 1:

[0104] The terminal uses a wearable device to capture the behavior of the person being supported. Specifically, it uses an accelerometer, gyroscope, microphone, and GPS to collect the physical movements and voice of the person being supported in real time. The input is the movements and speech of the person being supported, and the output is to store these as digital data.

[0105] As a specific example of operation, the accelerometer and gyroscope detect the movement of the person being supported standing up, and at the same time, the microphone collects the voice recording of the person saying "It's cold."

[0106] Step 2:

[0107] The device encodes the collected data and converts it into the appropriate data format. For example, audio data is encoded into MP3 format, and sensor data is converted into JSON format. The converted data is then sent to the server via a communication module. The input is the raw data collected by the wearable device, and the output is the encoded data.

[0108] Specific examples of operations include compressing audio data and converting it to MP3 format, and converting acceleration and gyroscope data to JSON format.

[0109] Step 3:

[0110] The server preprocesses the received data. This preprocessing involves shaping the heterogeneous data into a unified format. Data from different sensors is integrated into a single format. The input is encoded data, and the output is data in a unified format that can be fed into a generative AI model.

[0111] A specific example of the operation includes converting voice data into text using a voice recognition engine and combining it with sensor data.

[0112] Step 4:

[0113] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes and infers the intention of the person receiving support. The input is preprocessed data in a unified format, and the output is text data of the inferred intention of the person receiving support.

[0114] As a specific example of operation, the generative AI model is input with a prompt statement such as, "The person being supported started walking away saying, 'It's cold.' Please guess what he wants to do.", and an intent such as, "It's cold, so I want to go back indoors" is generated.

[0115] Step 5:

[0116] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to convert it into voice data. The input is the text data of the estimated intention, and the output is voice data. The generated voice data is sent to the audio output device used by the supporter.

[0117] As a specific example of operation, text such as "It's cold, so I want to go back indoors" is input into a speech synthesis engine to generate voice data.

[0118] Step 6:

[0119] The terminal (audio output device) plays the received audio data and transmits it to the device that supports it. The input is the audio data sent from the server, and the output is the played audio.

[0120] As a specific example of how it works, the audio glasses will play a voice message saying, "It's cold, so I want to go back indoors," and the support person will hear this and take appropriate action.

[0121] This enables the system to accurately grasp the intentions of the person being supported in real time and convey them to the person providing support quickly and accurately.

[0122] (Application example 1)

[0123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] The present invention solves the problem that customers with cognitive disabilities have difficulty accurately communicating their intentions and requests to store staff. In particular, when customers are unable to express their intentions well, it becomes difficult for store staff to respond appropriately, which can lead to a decline in the quality of the customer experience. In such situations, there is a need for a means to accurately estimate a customer's intentions and requests and quickly communicate them to store staff.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0126] In this invention, the server includes: means for collecting the physical movements and voice of the support recipient using a wearable device that captures the support recipient's behavior; means for transmitting the data collected from the wearable device to the server and preprocessing the data; means for analyzing the preprocessed data using a generative AI model and estimating the support recipient's intention; means for converting the estimated intention into voice data and transmitting it to a device used by a store employee; means for playing the transmitted voice data and notifying the employee; and means for the employee to prompt an appropriate response based on the notification. This enables customers with cognitive disabilities to accurately communicate their intentions in the store to employees, allowing the employees to respond quickly and appropriately.

[0127] "Support recipients" refer to people who have cognitive problems and require external support.

[0128] A "wearable device" is an electronic device that is worn on the body and is equipped with sensors and communication functions.

[0129] "Physical actions" refer to bodily movements such as walking, standing up, and raising your hands.

[0130] "Voice" refers to the words and sounds made by the person being supported.

[0131] "Server" refers to a computer system for processing and storing data over a network.

[0132] "Preprocessing" refers to the process of converting collected raw data into a format that is easy to analyze.

[0133] "Generative AI model" refers to the artificial intelligence algorithm used to analyze collected data and infer the intent of the person being supported.

[0134] "Brick and mortar store" refers to a commercial establishment that offers goods or services in a physical location.

[0135] "Employees" refers to staff who work in physical stores and deal with customers.

[0136] "Voice data" refers to data that has been converted into voice format from intent analyzed by a generative AI model.

[0137] "Notification" refers to information that communicates the support recipient's intentions to the employee.

[0138] The present invention relates to a system that enables customers with cognitive disabilities to accurately communicate their intentions and requests to store staff in a physical store. The system includes a wearable device (e.g., a smartwatch), a server, a generative AI model, and a notification device (e.g., a smartphone).

[0139] Data collection using wearable devices

[0140] The terminal uses a wearable device to collect the physical movements (e.g., walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported. Specifically, the wearable device is equipped with an accelerometer, gyroscope, microphone, and GPS function, and collects data in real time using these sensors. The collected data is either stored in the wearable device or sent directly to a server.

[0141] Data preprocessing and transmission

[0142] The server receives the data sent by the wearable device and preprocesses it, converting voice data to text and analyzing sensor data to understand the physical movements of the person being supported, then formats the data appropriately and prepares it for input into the generative AI model.

[0143] Intent Analysis

[0144] The server analyzes the preprocessed data using a generative AI model (e.g., GPT-4). The generative AI model infers the intention of the person being supported based on the voice text and movement data. For example, if the person being supported starts walking while saying, "I want to go to the toilet," the generative AI model will interpret this as the intention, "I want to go to the toilet."

[0145] Notice of presumed intent

[0146] The server generates the estimated intent in text format and inputs that text into a speech synthesis engine to generate voice data. This voice data is then sent to a device used by an employee in the physical store. The device then notifies the employee that it has received the notification and prompts them to take an appropriate response. This allows the employee to accurately understand the intent of the person being supported and respond quickly.

[0147] Specific examples

[0148] Consider the following scenario: If a person being supported walks through a physical store saying, "I want to go to the restroom," the wearable device collects this statement and movement data. The collected data is sent to a server, where it is preprocessed and a generative AI model analyzes the person's intent. As a result, the intention of "I want to go to the restroom" is estimated, and this intent is notified via voice to the employee's smartphone, allowing the employee to immediately guide the person being supported to the restroom.

[0149] Prompt Sentence Examples

[0150] An example of a prompt for the generative AI model is, "Analyze the behavior and voice data of the customer who says that the employee wants to go to the restroom, and infer their intent." This allows the generative AI model to infer the appropriate intent based on the voice and movement data of the person being supported.

[0151] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0152] Step 1:

[0153] The terminal (wearable device) collects the physical movements (walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported in real time. The wearable device's accelerometer, gyroscope, microphone, and GPS are used to acquire movement and voice data, allowing the current actions and speech of the person being supported to be collected.

[0154] Step 2:

[0155] The device temporarily stores the collected motion and voice data or transmits it to a server in real time. The data includes motion data from the accelerometer and gyroscope, and voice data from the microphone. This allows the raw data collected by the wearable device to be transmitted to the server.

[0156] Step 3:

[0157] The server receives the data sent from the device and preprocesses it. In the preprocessing step, the voice data is converted into text (e.g., "I want to go to the toilet"), and the motion data is simultaneously analyzed to identify physical movements (e.g., while walking). Other processes include standardizing the data format and removing noise. This results in preprocessed data that is ready to be input into the generative AI model.

[0158] Step 4:

[0159] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes the voice text and behavioral data to infer the intention of the person being supported. A prompt such as "Analyze the behavior and voice data of a customer who says that an employee wants to go to the restroom, and infer their intention" is used. This allows the generative AI model to infer the intention of "I want to go to the restroom" from the input data.

[0160] Step 5:

[0161] The server generates the intent estimated by the generative AI model in text format. This text is then input into a speech synthesis engine to generate voice data. The speech synthesis engine converts the text into natural-sounding speech and creates an audio file. This outputs the estimated intent as voice data.

[0162] Step 6:

[0163] The server then sends the generated voice data to a device (e.g., a smartphone) used by an employee at the physical store. The employee's device receives and plays the voice data, notifying the employee of the intention of the person receiving support. This allows the employee to accurately understand the intention of the person receiving support and respond quickly.

[0164] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0165] This invention is a system that estimates the intentions of support recipients with cognitive problems, improves the accuracy of the estimation by combining it with an emotion engine, and accurately conveys those intentions to the supporter. This system includes devices such as a wearable device, a server, a generative AI, an emotion engine, and audio glasses.

[0166] 1. Wearable devices that capture the behavior of those receiving support

[0167] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0168] 2. Data preprocessing and transmission

[0169] The device preprocesses and organizes this data. For example, it recognizes walking movements from data from the accelerometer and gyroscope, and converts recorded voice data into text using voice recognition technology. In addition, an emotion engine identifies the emotions of the person being supported from facial expressions, tone of voice, and body movements. The identified emotion data is also sent to the server.

[0170] 3. Data analysis and intent estimation

[0171] The server further preprocesses the received data for analysis. Specifically, it converts data from different formats into a unified format to ensure consistency. The converted data is then input into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person being supported.

[0172] 4. Emotional Data Integration

[0173] The server combines the analysis results of the generation AI with the emotional data from the emotion engine. This combined data is used to improve the accuracy of estimating the intention of the person receiving support. For example, if the person receiving support says "It's cold" and stands up, and the emotional data indicates feelings such as "anxiety" or "discomfort," the generation AI will more accurately estimate the intention as "I want to go back indoors."

[0174] 5. Speech conversion and output of estimated intention

[0175] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device (audio glasses) worn by the support person.

[0176] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0177] Specific examples

[0178] For example, the following scenario is possible:

[0179] Step 1: The person being supported begins walking outdoors, saying "It's cold." The wearable device records the person's speech and collects movement data. At the same time, the emotion engine identifies the emotion "anxiety" from the person's tone of voice and facial expression.

[0180] Step 2: The wearable device sends voice data, movement data, and emotion data to the server.

[0181] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0182] Step 4: The server integrates the analysis results of the generative AI with the emotional data to improve the accuracy of the intention estimation. The server converts this intention into voice data and sends it to the audio glasses.

[0183] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0184] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported, improves the accuracy by combining it with an emotion engine, and enables the supporter to respond appropriately.

[0185] The processing flow will be explained below.

[0186] Step 1:

[0187] The device collects the actions and voice of the person receiving support. A wearable device (e.g., a smartwatch) uses an accelerometer, gyroscope, microphone, and GPS to record the physical movements and voice of the person receiving support in real time. For example, it captures the action of the person saying, "It's cold," and starting to walk away.

[0188] Step 2:

[0189] The device preprocesses the collected motion data and recognizes specific movements such as walking and standing up. At the same time, it converts the recorded voice data into text using voice recognition technology. For example, the voice data can be converted into text data such as "It's cold."

[0190] Step 3:

[0191] The device sends the collected and pre-processed data to the server, where the motion data, voice text, and location data are encoded using a secure communication protocol (e.g., HTTPS).

[0192] Step 4:

[0193] The server further preprocesses the data it receives and formats it into a unified format. Data in different formats is processed into a form suitable for analysis. For example, the action data for "standing up" and the voice text "It's cold" are unified.

[0194] Step 5:

[0195] The server inputs the formatted data into the generation AI for analysis. The generation AI (e.g., GPT-4) analyzes the movement data and voice data and infers the intention of the person being supported. For example, the generation AI infers the intention, "It's cold, so I want to go back indoors."

[0196] Step 6:

[0197] The device recognizes the emotions of the person receiving support. Using the camera and microphone installed in the wearable device, emotions are identified from the facial expressions and tone of voice of the person receiving support. For example, emotional data such as "anxiety" or "discomfort" is generated.

[0198] Step 7:

[0199] The server integrates the emotion data received from the emotion engine with the analysis results of the generative AI. This improves the accuracy of estimating the intention of the person being supported. For example, the intention of "I want to go back indoors because it's cold" can be further confirmed by the emotion data of "anxiety."

[0200] Step 8:

[0201] The server converts the estimated intent into voice data. The generated text is then input into a speech synthesis engine to generate voice data. For example, the generated voice data might say, "It's cold, so I want to go back indoors."

[0202] Step 9:

[0203] The server sends the generated audio data to the device (audio glasses) worn by the supporter. The audio data is encoded and transmitted using a secure communication protocol.

[0204] Step 10:

[0205] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The built-in speaker of the Audio Glasses plays the audio data and conveys it to the support person. For example, the voice may say, "It's cold, so I want to go back indoors."

[0206] Step 11:

[0207] The user (support person) listens to the audio through the audio glasses and understands the intentions of the person being supported. For example, if the user understands that the person wants to go back indoors, they can take action to guide the person indoors.

[0208] Example 2

[0209] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0210] In situations where support recipients with cognitive disabilities have difficulty accurately communicating their intentions, there is a need to accurately estimate their intentions and communicate them quickly and accurately to supporters. However, conventional systems lack the accuracy of data collection and intention estimation, and in particular lack a mechanism to improve estimation accuracy by combining emotional data. As a result, the intentions of support recipients are often misunderstood, making it difficult to provide appropriate support.

[0211] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for further preprocessing the transmitted data and shaping data of different formats into a unified format, means for analyzing the preprocessed data using a generative AI model and estimating the intention of the person being supported, and means for integrating emotion data acquired from an emotion engine with the estimated intention to improve the accuracy of intention estimation. In this way, by integrating and analyzing not only the behavior and voice of the person being supported but also the emotion data, the accuracy of intention estimation improves, making it possible to quickly and accurately convey the intention to the person providing support.

[0212] - "Support recipient" refers to a person who has cognitive problems and requires the support of others.

[0213] A "wearable device" is an electronic device that can be worn by the user and is equipped with sensors and microphones to collect data.

[0214] "Physical movement" refers to data showing changes in the body movements and posture of the person being supported.

[0215] "Audio" refers to data that indicates the statements and tone of voice made by the person being supported.

[0216] "Preprocessing" refers to the process of formatting and organizing collected data into a form that is easy to analyze.

[0217] A "server" refers to a computer system that stores and analyzes data and communicates with external devices.

[0218] "Generative AI model" refers to an artificial intelligence model that analyzes collected data and infers the intentions of the person being supported.

[0219] An "emotion engine" refers to a system that identifies emotions from a person's facial expressions and tone of voice and outputs them as data.

[0220] "Intention" refers to what the person receiving support wants or intends to do.

[0221] "Unified format" refers to the formatting of data of different formats into one consistent format.

[0222] "Audio data" refers to data that has been converted from text information into a digital format that can be played as sound.

[0223] "Audio glasses" are devices that provide information through sight and hearing, especially those worn by a support person.

[0224] "Intention estimation accuracy" refers to an indicator of how accurately the intention of the person being supported can be estimated.

[0225] This invention is a system that estimates the intentions of the person being supported with high accuracy and quickly and accurately communicates those intentions to the supporter. The system includes devices such as a wearable device, a server, a generative AI model, an emotion engine, and audio glasses.

[0226] First, the terminal (wearable device) collects the behavioral and voice data of the person being supported. The wearable device is equipped with an accelerometer, gyroscope, microphone, and location information acquisition function, which allows it to record the person's actions (e.g., standing up, walking, raising their hand, etc.) and voice (e.g., utterances such as "It's cold") in real time. This collected data is either temporarily stored within the device or sent directly to a server.

[0227] The device then preprocesses and organizes this data. This involves identifying walking and other movements from accelerometer and gyroscope data, converting recorded voice data into text using speech recognition technology, and using an emotion engine to identify emotions from voice tone and facial expressions, generating emotion data such as "anxiety." This organized data is then sent to a server.

[0228] The server further preprocesses the transmitted data. Specifically, it formats the data in different formats into a unified format to ensure consistency. This formatted data is then input into a generative AI model (e.g., GPT-4). The server analyzes the data using the generative AI model and infers the intention of the person receiving support. For example, by inputting a prompt such as, "The person receiving support started walking while saying 'I'm cold,' and their emotion was identified as 'anxious.' Please infer the person's intention," the generative AI model performs the analysis.

[0229] Furthermore, the server integrates the analysis results of the generative AI model with the emotional data from the emotion engine. This integration improves the accuracy of intent estimation. For example, if a subject says "It's cold" while standing up and the emotional data indicates "anxiety," the generative AI model will more accurately estimate the intent as "I want to go back indoors."

[0230] Finally, the server generates the estimated intention in text format and inputs that text into a speech synthesis engine to generate voice data. The generated voice data is sent to the device (e.g., audio glasses) used by the support person. The device (audio glasses) plays back the received voice data and conveys it to the support person, accurately conveying the support person's intention. For example, the audio glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0231] In this way, this invention improves the accuracy of estimating intentions by integrating and analyzing not only the behavior and voice of the person being supported, but also emotional data, making it possible to quickly and accurately convey intentions to the person providing support.

[0232] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0233] Step 1:

[0234] The terminal (wearable device) collects behavioral and voice data from the person receiving support. The wearable device's accelerometer, gyroscope, microphone, and location information acquisition function are used to record the person receiving support's movements (e.g., standing up, walking) and voice (e.g., saying "it's cold") in real time. The input is the person receiving support's physical movements and voice, and the output is the collected raw data. As a specific example, data is collected when the person receiving support starts walking while saying "it's cold."

[0235] Step 2:

[0236] The data collected by the device is preprocessed and organized. First, data from the accelerometer and gyroscope is analyzed to identify walking and other movements. Next, voice data is converted into text using speech recognition technology. An emotion engine then identifies emotions from voice tone and facial expressions, generating emotion data such as "anxiety." The input is the collected raw data, and the output is organized data (movement data, textual voice data, and emotion data). For example, the statement "It's cold" is converted into text, and data identified as "walking movements" is generated.

[0237] Step 3:

[0238] The device sends the preprocessed data to the server. The input is the organized data, and the output is the data sent to the server. For example, the textual utterance "It's cold" and the emotional data "I feel anxious" are sent to the server.

[0239] Step 4:

[0240] The server further preprocesses the transmitted data, shaping data of different formats into a unified format and making it consistent. The input is the organized data transmitted from the device, and the output is unified format data. For example, text data, action data, and emotion data are converted into a unified format.

[0241] Step 5:

[0242] The server uses a generative AI model to analyze the preprocessed data and infer the intention of the person being supported. Data is input into the generative AI model using a prompt. The input is unified format data and a prompt, and the output is the inferred intention. As a specific example, the prompt is "The person being supported starts walking while saying 'It's cold', and the emotion is identified as 'anxiety'. Please infer the person's intention," and the inferred intention is "It's cold, so I want to go back indoors."

[0243] Step 6:

[0244] The server integrates the analysis results of the generative AI model with the emotion data obtained from the emotion engine. The input is the estimated intention and emotion data, and the output is the integrated intention data. For example, by integrating the emotion data of "anxiety" with the intention of "wanting to go back indoors," the accuracy of the intent estimation improves.

[0245] Step 7:

[0246] The server generates the estimated intention in text format and inputs it into a speech synthesis engine to generate speech data. The input is the intention data in text format, and the output is speech data. For example, the text "It's cold, so I want to go back indoors" is generated as speech data.

[0247] Step 8:

[0248] The server sends the generated voice data to the audio glasses. The input is the generated voice data, and the output is the voice data sent to the audio glasses. For example, voice data saying "It's cold, so I want to go back indoors" is sent to the audio glasses.

[0249] Step 9:

[0250] The device (audio glasses) plays the received voice data and conveys it to the support person. The input is the voice data sent to the audio glasses, and the output is the played voice data. Specifically, the audio glasses play the voice "It's cold, I want to go back indoors," allowing the support person to take appropriate action.

[0251] (Application example 2)

[0252] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0253] Food delivery services are required to accurately understand customer intent and emotions to reduce misunderstandings and troubles about orders and improve customer satisfaction. However, conventional systems have difficulty accurately analyzing customer intent and emotions, leading to a high likelihood of misunderstandings. Furthermore, there is a lack of a mechanism to integrate emotion analysis and intent estimation to improve accuracy.

[0254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0255] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for transmitting the data collected from the wearable device to the server and preprocessing the data, means for analyzing the preprocessed data using a generative AI and estimating the intention of the person being supported, means for analyzing the emotions of the person being supported by combining an emotion engine and integrating it with the generative AI to improve the accuracy of the intention estimation, means for converting the estimated intention into voice data and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp customer intentions and emotions even in food delivery services, reducing misunderstandings about order details and problems.

[0256] A "supported person" is a user or customer of a food delivery service who provides specific actions or voice data.

[0257] A "wearable device" is an electronic device worn by an individual on the body, which includes an accelerometer, gyroscope, microphone, and GPS, and has the ability to collect physical movements and sounds.

[0258] The "server" is a computer system that preprocesses, analyzes, and infers intent from data. It utilizes generative AI and an emotion engine to accurately infer the intent of the person being supported, and processes and transmits the necessary data.

[0259] "Generative AI" refers to technology that uses natural language processing models to analyze data and infer the intentions of the person being supported, and specifically includes generative artificial intelligence models such as GPT-4.

[0260] The "emotion engine" refers to technology that analyzes the tone of voice, facial expressions, and movements of the person being supported to identify their emotional state, and is integrated with generative AI to improve the accuracy of intent estimation.

[0261] "Intention estimation accuracy" refers to the high level of ability to accurately analyze and judge the intentions of the person being supported, and by integrating with the emotion engine, intentions can be identified with greater accuracy.

[0262] "Voice data" refers to voice information generated by a voice synthesis engine based on analyzed text information, and is used to convey the intentions of the person being supported to the person providing support.

[0263] "Supporter" refers to the delivery person who delivers food to customers in a food delivery service, who receives and plays back the transmitted voice data and takes appropriate action based on the customer's intentions.

[0264] This invention is a system for accurately understanding customer intentions and emotions in food delivery services, reducing misunderstandings and troubles about orders and improving customer satisfaction. This system includes wearable devices (smart glasses, smart watches, etc.), a server, generative AI, an emotion engine, and an audio device (audio glasses).

[0265] 1. Wearable devices that capture the behavior of those receiving support

[0266] First, the customer wears smart glasses. The glasses are equipped with an accelerometer, gyroscope, microphone, and GPS to collect the customer's physical movements (e.g., waving, walking, stopping, etc.) and voice (order details, preferences, and requests) in real time. The collected data is either temporarily stored on the device or sent directly to a server.

[0267] 2. Data preprocessing and transmission

[0268] After receiving this data, the server performs initial preprocessing. Specifically, it recognizes movements from data from the accelerometer and gyroscope, converts recorded voice data into text using voice recognition technology, and uses an emotion engine to identify emotions from the voice tone and facial expressions of the person receiving support. The identified emotion data is also integrated for analysis.

[0269] 3. Data analysis and intent estimation

[0270] The server inputs the received data into a generation AI (e.g., GPT-4) for analysis. The generation AI uses text data, behavioral data, and emotional data to infer the intention of the person being supported. For example, when a customer says, "Please avoid spicy food," and the emotion engine provides information that the face is smiling, the generation AI infers that "the customer does not want spicy food."

[0271] 4. Emotional Data Integration

[0272] The server integrates the analysis results of the generative AI with data from the emotion engine to improve the accuracy of intent estimation. Based on this integrated data, the customer's intent becomes clear and the order details can be accurately understood.

[0273] 5. Speech conversion and output of estimated intention

[0274] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is sent to the delivery person's audio device (Audio Glasses). The Audio Glasses play back the received voice data and accurately convey the intended intention of the support person to the delivery person.

[0275] Specific examples

[0276] For example, consider the following scenario:

[0277] 1. A customer puts on smart glasses and says, "Please avoid spicy food."

[0278] 2. The smart glasses' sensors collect movement data, the microphone records voice data, and the emotion engine analyzes smiles (positive).

[0279] 3. The data is sent to the server, where it undergoes preprocessing and is then analyzed by the generation AI.

[0280] 4. The generative AI understands that the customer wants to avoid spicy food and infers their intention.

[0281] 5. The server converts this intention into voice data and transmits it to the delivery person's audio glasses, saying, "Please avoid spicy food."

[0282] Prompt Sentence Examples

[0283] The user says, "I don't like spicy food." The face is smiling (positive). Please infer the user's intention from this statement.

[0284] This system makes it possible to accurately understand the complex intentions and emotions of customers, significantly improving customer satisfaction in delivery services.

[0285] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0286] Step 1:

[0287] The server collects physical movement and voice data from a wearable device (smart glasses) that captures the behavior of the person being supported.

[0288] Input: Real-time movement data (acceleration, gyro) and audio data from the supported individual.

[0289] Output: Recording files of motion data (e.g. accelerometer and gyroscope data) and audio data.

[0290] Specific operation: The smart glasses detect the customer's movements and speech and record them as digital data.

[0291] Step 2:

[0292] The server pre-processes the collected data.

[0293] Input: Motion data and audio data (recorded files).

[0294] Output: Preprocessed action data (action classification results) and audio-text data.

[0295] Specific operation: Recognizes actions such as walking and standing from data from the accelerometer and gyroscope, and converts the voice data into text using voice recognition software. Example: Converts the voice data "I don't like spicy food" into text "I don't like spicy food."

[0296] Step 3:

[0297] The server uses an emotion engine to analyze the emotions of the person being supported from the collected data.

[0298] Input: Audio and text data.

[0299] Output: Emotion data (e.g., emotional state, such as positive or negative).

[0300] Specific operation: The emotion engine analyzes the tone of voice and facial expression data (facial recognition by camera) to identify the emotional state. Example: Because the person is smiling, it is judged to be "positive."

[0301] Step 4:

[0302] The server uses generative AI to analyze the preprocessed data and emotional data to infer the intentions of the person being supported.

[0303] Input: Preprocessed text data and sentiment data.

[0304] Output: The inferred intent (e.g., "Avoid spicy food").

[0305] Specific operation: A generative AI model (e.g., GPT-4) analyzes the preprocessed data and derives the intention of the person being supported using the prompt: "The user says, 'I don't like spicy food.' Their face is smiling (positive). Please infer the user's intention from this statement."

[0306] Step 5:

[0307] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate speech data.

[0308] Input: Inferred intent (in text format).

[0309] Output: The generated audio data.

[0310] Specific behavior: Use a speech synthesis engine (e.g., Text-to-Speech engine) to convert the text "Avoid spicy food" into audio data.

[0311] Step 6:

[0312] The server transmits the generated voice data to the delivery person's audio device (audio glasses) and plays the voice data.

[0313] Input: The generated audio data.

[0314] Output: Playback of the audio data sent to the delivery person.

[0315] Specific operation: The audio glasses receive the voice data and play back "Please avoid spicy food," allowing the delivery person to accurately understand the customer's intentions.

[0316] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0317] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0318] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0319] [Second embodiment]

[0320] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0321] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0322] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0323] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0324] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0325] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0326] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0327] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0328] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0329] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0330] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0331] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0332] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to the supporter. This system includes a wearable device, a server, a generative AI, and devices such as audio glasses.

[0333] 1. Wearable devices that capture the behavior of those receiving support

[0334] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0335] 2. Data preprocessing and transmission

[0336] The device sends the collected data to the server, where it is encoded and converted into an appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and voice data recorded by the microphone are sent to the server.

[0337] 3. Analysis of Inferred Intent

[0338] The server preprocesses the received data and formats it into a unified format before inputting it into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generation AI will interpret this as, "It's cold, so I want to go back indoors."

[0339] 4. Speech conversion and output of estimated intention

[0340] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device such as audio glasses worn by the support person.

[0341] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0342] Specific examples

[0343] For example, the following scenario is possible:

[0344] Step 1: The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's speech and collects movement data.

[0345] Step 2: The wearable device sends voice and motion data to the server.

[0346] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0347] Step 4: The server converts this intent into audio data and sends it to the audio glasses.

[0348] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0349] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported and enables the supporter to respond appropriately.

[0350] The processing flow will be explained below.

[0351] Step 1:

[0352] The device collects the movements and voice of the person being supported. Specifically, the wearable device uses an accelerometer and gyroscope to capture physical movement data, and a microphone to record voice.

[0353] Step 2:

[0354] The device preprocesses and organizes this data, for example, by recognizing walking movements from accelerometer and gyroscope data and converting recorded audio data into text using voice recognition technology.

[0355] Step 3:

[0356] The device sends the preprocessed data to the server, where it is encoded according to a communication protocol and transmitted using a secure communication method (e.g., HTTPS).

[0357] Step 4:

[0358] The server further preprocesses the received data for analysis, specifically by converting different data formats into a unified and consistent format.

[0359] Step 5:

[0360] The server inputs the formatted data into the generation AI for analysis. The generation AI uses a natural language processing model (e.g., GPT-4) to infer the intention of the person being supported.

[0361] Step 6:

[0362] The server receives the analysis results of the generation AI in text format and inputs the text into a speech synthesis engine, which converts the text into voice data.

[0363] Step 7:

[0364] The server then transmits the generated audio data to the device (audio glasses) worn by the supporter, where the data is again encoded and transmitted via a secure means.

[0365] Step 8:

[0366] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The audio data is played and transmitted to the support person.

[0367] Step 9:

[0368] The user (support person) listens to the audio through the audio glasses and understands the intention of the person being supported. For example, if the person being supported says, "It's cold, so I want to go back indoors," the support person will guide the person indoors.

[0369] Example 1

[0370] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0371] In modern society, it is extremely important to accurately grasp the intentions of those receiving support who have cognitive disabilities and to convey them quickly and accurately to those providing support. However, conventional technology has difficulty accurately grasping the intentions of those receiving support, resulting in delayed responses and often misinterpretations. Furthermore, when analyzing data collected from multiple sensors and devices, inefficiencies in integration and preprocessing make it difficult to respond in real time. There is a need to provide a system that solves these issues and facilitates smooth communication between those receiving support and those providing support.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0373] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for encoding the data collected from the wearable device, converting it into an appropriate data format, and transmitting it to the server, means for the server to preprocess the data received and estimate the intention of the person being supported using a generative AI model, means for generating the estimated intention in text format, inputting it into a speech synthesis engine to convert it into voice data, and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp the intention of the person being supported in real time and convey it to the supporter quickly and accurately.

[0374] A "wearable device" is a device that is worn on the body and is equipped with sensors and communication functions to collect the physical movements and voices of the person being supported.

[0375] An "accelerometer" is a type of sensor installed in wearable devices, and is a device that measures the acceleration of an object.

[0376] A "gyroscope" is a device installed in wearable devices that measures angular velocity to detect changes in the rotation or tilt of an object.

[0377] A "microphone" is a device that converts sound into an electrical signal, and is a component installed in wearable devices to collect the speech of the person being supported.

[0378] "GPS" stands for Global Positioning System, a system for measuring positions on Earth with high precision.

[0379] "Encoding" is the process of converting data into a particular format, with the goal of optimizing it for transmission or storage.

[0380] A "generative AI model" is an artificial intelligence model that analyzes data and generates new information or results based on that information, and is used here to infer the intentions of the person being supported.

[0381] A "speech synthesis engine" is a system that converts text data into speech data, and is a device used to output the estimated intention as speech.

[0382] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to a supporter. This system includes devices such as a wearable device, a server, a generative AI, and an audio output device.

[0383] First, a wearable device is used to capture the behavior of the person being supported. A specific example is a smartwatch. This device is equipped with an accelerometer, gyroscope, microphone, and GPS, and collects the physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") of the person being supported in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0384] Next, when the device sends the collected data to the server, the data is encoded and converted into the appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and audio data recorded by the microphone are sent to the server. The audio data is encoded into MP3 format, and the sensor data is converted into JSON format.

[0385] The server preprocesses the received data. This preprocessing includes formatting data from different sensors into a unified format. The formatted data is then input into a generative AI model (e.g., GPT-4). The generative AI model analyzes the data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generative AI will interpret this as, "It's cold, so I want to go back indoors."

[0386] The server generates the estimated intention in text format. This text is input into a speech synthesis engine and converted into voice data. The generated voice data is sent to an audio output device (such as audio glasses) worn by the support person. The device (audio output device) plays the received voice data and conveys it to the support person. This ensures that the intention of the person being supported is accurately conveyed. For example, the audio glasses could say, "It's cold, so I'd like to go back indoors," allowing the support person to take appropriate action.

[0387] To illustrate, consider the following scenario:

[0388] 1. The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's words and collects movement data.

[0389] 2. The wearable device sends voice and movement data to the server.

[0390] 3. The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0391] 4. The server converts this intent into audio data and sends it to the audio glasses.

[0392] 5. The audio glasses will tell the support person, "It's cold, I want to go back indoors."

[0393] Below is an example of a prompt sentence to input to the generative AI model.

[0394] Example 1: "The person you are supporting walks away saying, 'It's cold.' Guess what he wants to do."

[0395] Example 2: "The user stands up and says 'I'm thirsty.' Please infer the intention of this action."

[0396] As described above, the present invention makes it possible to accurately grasp the intentions of the person being supported in real time and to convey them to the person providing support quickly and accurately.

[0397] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0398] Step 1:

[0399] The terminal uses a wearable device to capture the behavior of the person being supported. Specifically, it uses an accelerometer, gyroscope, microphone, and GPS to collect the physical movements and voice of the person being supported in real time. The input is the movements and speech of the person being supported, and the output is to store these as digital data.

[0400] As a specific example of operation, the accelerometer and gyroscope detect the movement of the person being supported standing up, and at the same time, the microphone collects the voice recording of the person saying "It's cold."

[0401] Step 2:

[0402] The device encodes the collected data and converts it into the appropriate data format. For example, audio data is encoded into MP3 format, and sensor data is converted into JSON format. The converted data is then sent to the server via a communication module. The input is the raw data collected by the wearable device, and the output is the encoded data.

[0403] Specific examples of operations include compressing audio data and converting it to MP3 format, and converting acceleration and gyroscope data to JSON format.

[0404] Step 3:

[0405] The server preprocesses the received data. This preprocessing involves shaping the heterogeneous data into a unified format. Data from different sensors is integrated into a single format. The input is encoded data, and the output is data in a unified format that can be fed into a generative AI model.

[0406] A specific example of the operation includes converting voice data into text using a voice recognition engine and combining it with sensor data.

[0407] Step 4:

[0408] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes and infers the intention of the person receiving support. The input is preprocessed data in a unified format, and the output is text data of the inferred intention of the person receiving support.

[0409] As a specific example of operation, the generative AI model is input with a prompt statement such as, "The person being supported started walking away saying, 'It's cold.' Please guess what he wants to do.", and an intent such as, "It's cold, so I want to go back indoors" is generated.

[0410] Step 5:

[0411] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to convert it into voice data. The input is the text data of the estimated intention, and the output is voice data. The generated voice data is sent to the audio output device used by the supporter.

[0412] As a specific example of operation, text such as "It's cold, so I want to go back indoors" is input into a speech synthesis engine to generate voice data.

[0413] Step 6:

[0414] The terminal (audio output device) plays the received audio data and transmits it to the device that supports it. The input is the audio data sent from the server, and the output is the played audio.

[0415] As a specific example of how it works, the audio glasses will play a voice message saying, "It's cold, so I want to go back indoors," and the support person will hear this and take appropriate action.

[0416] This enables the system to accurately grasp the intentions of the person being supported in real time and convey them to the person providing support quickly and accurately.

[0417] (Application example 1)

[0418] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0419] The present invention solves the problem that customers with cognitive disabilities have difficulty accurately communicating their intentions and requests to store staff. In particular, when customers are unable to express their intentions well, it becomes difficult for store staff to respond appropriately, which can lead to a decline in the quality of the customer experience. In such situations, there is a need for a means to accurately estimate a customer's intentions and requests and quickly communicate them to store staff.

[0420] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0421] In this invention, the server includes: means for collecting the physical movements and voice of the support recipient using a wearable device that captures the support recipient's behavior; means for transmitting the data collected from the wearable device to the server and preprocessing the data; means for analyzing the preprocessed data using a generative AI model and estimating the support recipient's intention; means for converting the estimated intention into voice data and transmitting it to a device used by a store employee; means for playing the transmitted voice data and notifying the employee; and means for the employee to prompt an appropriate response based on the notification. This enables customers with cognitive disabilities to accurately communicate their intentions in the store to employees, allowing the employees to respond quickly and appropriately.

[0422] "Support recipients" refer to people who have cognitive problems and require external support.

[0423] A "wearable device" is an electronic device that is worn on the body and is equipped with sensors and communication functions.

[0424] "Physical actions" refer to bodily movements such as walking, standing up, and raising your hands.

[0425] "Voice" refers to the words and sounds made by the person being supported.

[0426] "Server" refers to a computer system for processing and storing data over a network.

[0427] "Preprocessing" refers to the process of converting collected raw data into a format that is easy to analyze.

[0428] "Generative AI model" refers to the artificial intelligence algorithm used to analyze collected data and infer the intent of the person being supported.

[0429] "Brick and mortar store" refers to a commercial establishment that offers goods or services in a physical location.

[0430] "Employees" refers to staff who work in physical stores and deal with customers.

[0431] "Voice data" refers to data that has been converted into voice format from intent analyzed by a generative AI model.

[0432] "Notification" refers to information that communicates the support recipient's intentions to the employee.

[0433] The present invention relates to a system that enables customers with cognitive disabilities to accurately communicate their intentions and requests to store staff in a physical store. The system includes a wearable device (e.g., a smartwatch), a server, a generative AI model, and a notification device (e.g., a smartphone).

[0434] Data collection using wearable devices

[0435] The terminal uses a wearable device to collect the physical movements (e.g., walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported. Specifically, the wearable device is equipped with an accelerometer, gyroscope, microphone, and GPS function, and collects data in real time using these sensors. The collected data is either stored in the wearable device or sent directly to a server.

[0436] Data preprocessing and transmission

[0437] The server receives the data sent by the wearable device and preprocesses it, converting voice data to text and analyzing sensor data to understand the physical movements of the person being supported, then formats the data appropriately and prepares it for input into the generative AI model.

[0438] Intent Analysis

[0439] The server analyzes the preprocessed data using a generative AI model (e.g., GPT-4). The generative AI model infers the intention of the person being supported based on the voice text and movement data. For example, if the person being supported starts walking while saying, "I want to go to the toilet," the generative AI model will interpret this as the intention, "I want to go to the toilet."

[0440] Notice of presumed intent

[0441] The server generates the estimated intent in text format and inputs that text into a speech synthesis engine to generate voice data. This voice data is then sent to a device used by an employee in the physical store. The device then notifies the employee that it has received the notification and prompts them to take an appropriate response. This allows the employee to accurately understand the intent of the person being supported and respond quickly.

[0442] Specific examples

[0443] Consider the following scenario: If a person being supported walks through a physical store saying, "I want to go to the restroom," the wearable device collects this statement and movement data. The collected data is sent to a server, where it is preprocessed and a generative AI model analyzes the person's intent. As a result, the intention of "I want to go to the restroom" is estimated, and this intent is notified via voice to the employee's smartphone, allowing the employee to immediately guide the person being supported to the restroom.

[0444] Prompt Sentence Examples

[0445] An example of a prompt for the generative AI model is, "Analyze the behavior and voice data of the customer who says that the employee wants to go to the restroom, and infer their intent." This allows the generative AI model to infer the appropriate intent based on the voice and movement data of the person being supported.

[0446] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0447] Step 1:

[0448] The terminal (wearable device) collects the physical movements (walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported in real time. The wearable device's accelerometer, gyroscope, microphone, and GPS are used to acquire movement and voice data, allowing the current actions and speech of the person being supported to be collected.

[0449] Step 2:

[0450] The device temporarily stores the collected motion and voice data or transmits it to a server in real time. The data includes motion data from the accelerometer and gyroscope, and voice data from the microphone. This allows the raw data collected by the wearable device to be transmitted to the server.

[0451] Step 3:

[0452] The server receives the data sent from the device and preprocesses it. In the preprocessing step, the voice data is converted into text (e.g., "I want to go to the toilet"), and the motion data is simultaneously analyzed to identify physical movements (e.g., while walking). Other processes include standardizing the data format and removing noise. This results in preprocessed data that is ready to be input into the generative AI model.

[0453] Step 4:

[0454] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes the voice text and behavioral data to infer the intention of the person being supported. A prompt such as "Analyze the behavior and voice data of a customer who says that an employee wants to go to the restroom, and infer their intention" is used. This allows the generative AI model to infer the intention of "I want to go to the restroom" from the input data.

[0455] Step 5:

[0456] The server generates the intent estimated by the generative AI model in text format. This text is then input into a speech synthesis engine to generate voice data. The speech synthesis engine converts the text into natural-sounding speech and creates an audio file. This outputs the estimated intent as voice data.

[0457] Step 6:

[0458] The server then sends the generated voice data to a device (e.g., a smartphone) used by an employee at the physical store. The employee's device receives and plays the voice data, notifying the employee of the intention of the person receiving support. This allows the employee to accurately understand the intention of the person receiving support and respond quickly.

[0459] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0460] This invention is a system that estimates the intentions of support recipients with cognitive problems, improves the accuracy of the estimation by combining it with an emotion engine, and accurately conveys those intentions to the supporter. This system includes devices such as a wearable device, a server, a generative AI, an emotion engine, and audio glasses.

[0461] 1. Wearable devices that capture the behavior of those receiving support

[0462] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0463] 2. Data preprocessing and transmission

[0464] The device preprocesses and organizes this data. For example, it recognizes walking movements from data from the accelerometer and gyroscope, and converts recorded voice data into text using voice recognition technology. In addition, an emotion engine identifies the emotions of the person being supported from facial expressions, tone of voice, and body movements. The identified emotion data is also sent to the server.

[0465] 3. Data analysis and intent estimation

[0466] The server further preprocesses the received data for analysis. Specifically, it converts data from different formats into a unified format to ensure consistency. The converted data is then input into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person being supported.

[0467] 4. Emotional Data Integration

[0468] The server combines the analysis results of the generation AI with the emotional data from the emotion engine. This combined data is used to improve the accuracy of estimating the intention of the person receiving support. For example, if the person receiving support says "It's cold" and stands up, and the emotional data indicates feelings such as "anxiety" or "discomfort," the generation AI will more accurately estimate the intention as "I want to go back indoors."

[0469] 5. Speech conversion and output of estimated intention

[0470] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device (audio glasses) worn by the support person.

[0471] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0472] Specific examples

[0473] For example, the following scenario is possible:

[0474] Step 1: The person being supported begins walking outdoors, saying "It's cold." The wearable device records the person's speech and collects movement data. At the same time, the emotion engine identifies the emotion "anxiety" from the person's tone of voice and facial expression.

[0475] Step 2: The wearable device sends voice data, movement data, and emotion data to the server.

[0476] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0477] Step 4: The server integrates the analysis results of the generative AI with the emotional data to improve the accuracy of the intention estimation. The server converts this intention into voice data and sends it to the audio glasses.

[0478] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0479] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported, improves the accuracy by combining it with an emotion engine, and enables the supporter to respond appropriately.

[0480] The processing flow will be explained below.

[0481] Step 1:

[0482] The device collects the actions and voice of the person receiving support. A wearable device (e.g., a smartwatch) uses an accelerometer, gyroscope, microphone, and GPS to record the physical movements and voice of the person receiving support in real time. For example, it captures the action of the person saying, "It's cold," and starting to walk away.

[0483] Step 2:

[0484] The device preprocesses the collected motion data and recognizes specific movements such as walking and standing up. At the same time, it converts the recorded voice data into text using voice recognition technology. For example, the voice data can be converted into text data such as "It's cold."

[0485] Step 3:

[0486] The device sends the collected and pre-processed data to the server, where the motion data, voice text, and location data are encoded using a secure communication protocol (e.g., HTTPS).

[0487] Step 4:

[0488] The server further preprocesses the data it receives and formats it into a unified format. Data in different formats is processed into a form suitable for analysis. For example, the action data for "standing up" and the voice text "It's cold" are unified.

[0489] Step 5:

[0490] The server inputs the formatted data into the generation AI for analysis. The generation AI (e.g., GPT-4) analyzes the movement data and voice data and infers the intention of the person being supported. For example, the generation AI infers the intention, "It's cold, so I want to go back indoors."

[0491] Step 6:

[0492] The device recognizes the emotions of the person receiving support. Using the camera and microphone installed in the wearable device, emotions are identified from the facial expressions and tone of voice of the person receiving support. For example, emotional data such as "anxiety" or "discomfort" is generated.

[0493] Step 7:

[0494] The server integrates the emotion data received from the emotion engine with the analysis results of the generative AI. This improves the accuracy of estimating the intention of the person being supported. For example, the intention of "I want to go back indoors because it's cold" can be further confirmed by the emotion data of "anxiety."

[0495] Step 8:

[0496] The server converts the estimated intent into voice data. The generated text is then input into a speech synthesis engine to generate voice data. For example, the generated voice data might say, "It's cold, so I want to go back indoors."

[0497] Step 9:

[0498] The server sends the generated audio data to the device (audio glasses) worn by the supporter. The audio data is encoded and transmitted using a secure communication protocol.

[0499] Step 10:

[0500] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The built-in speaker of the Audio Glasses plays the audio data and conveys it to the support person. For example, the voice may say, "It's cold, so I want to go back indoors."

[0501] Step 11:

[0502] The user (support person) listens to the audio through the audio glasses and understands the intentions of the person being supported. For example, if the user understands that the person wants to go back indoors, they can take action to guide the person indoors.

[0503] Example 2

[0504] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] In situations where support recipients with cognitive disabilities have difficulty accurately communicating their intentions, there is a need to accurately estimate their intentions and communicate them quickly and accurately to supporters. However, conventional systems lack the accuracy of data collection and intention estimation, and in particular lack a mechanism to improve estimation accuracy by combining emotional data. As a result, the intentions of support recipients are often misunderstood, making it difficult to provide appropriate support.

[0506] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for further preprocessing the transmitted data and shaping data of different formats into a unified format, means for analyzing the preprocessed data using a generative AI model and estimating the intention of the person being supported, and means for integrating emotion data acquired from an emotion engine with the estimated intention to improve the accuracy of intention estimation. In this way, by integrating and analyzing not only the behavior and voice of the person being supported but also the emotion data, the accuracy of intention estimation improves, making it possible to quickly and accurately convey the intention to the person providing support.

[0507] - "Support recipient" refers to a person who has cognitive problems and requires the support of others.

[0508] A "wearable device" is an electronic device that can be worn by the user and is equipped with sensors and microphones to collect data.

[0509] "Physical movement" refers to data showing changes in the body movements and posture of the person being supported.

[0510] "Audio" refers to data that indicates the statements and tone of voice made by the person being supported.

[0511] "Preprocessing" refers to the process of formatting and organizing collected data into a form that is easy to analyze.

[0512] A "server" refers to a computer system that stores and analyzes data and communicates with external devices.

[0513] "Generative AI model" refers to an artificial intelligence model that analyzes collected data and infers the intentions of the person being supported.

[0514] An "emotion engine" refers to a system that identifies emotions from a person's facial expressions and tone of voice and outputs them as data.

[0515] "Intention" refers to what the person receiving support wants or intends to do.

[0516] "Unified format" refers to the formatting of data of different formats into one consistent format.

[0517] "Audio data" refers to data that has been converted from text information into a digital format that can be played as sound.

[0518] "Audio glasses" are devices that provide information through sight and hearing, especially those worn by a support person.

[0519] "Intention estimation accuracy" refers to an indicator of how accurately the intention of the person being supported can be estimated.

[0520] This invention is a system that estimates the intentions of the person being supported with high accuracy and quickly and accurately communicates those intentions to the supporter. The system includes devices such as a wearable device, a server, a generative AI model, an emotion engine, and audio glasses.

[0521] First, the terminal (wearable device) collects the behavioral and voice data of the person being supported. The wearable device is equipped with an accelerometer, gyroscope, microphone, and location information acquisition function, which allows it to record the person's actions (e.g., standing up, walking, raising their hand, etc.) and voice (e.g., utterances such as "It's cold") in real time. This collected data is either temporarily stored within the device or sent directly to a server.

[0522] The device then preprocesses and organizes this data. This involves identifying walking and other movements from accelerometer and gyroscope data, converting recorded voice data into text using speech recognition technology, and using an emotion engine to identify emotions from voice tone and facial expressions, generating emotion data such as "anxiety." This organized data is then sent to a server.

[0523] The server further preprocesses the transmitted data. Specifically, it formats the data in different formats into a unified format to ensure consistency. This formatted data is then input into a generative AI model (e.g., GPT-4). The server analyzes the data using the generative AI model and infers the intention of the person receiving support. For example, by inputting a prompt such as, "The person receiving support started walking while saying 'I'm cold,' and their emotion was identified as 'anxious.' Please infer the person's intention," the generative AI model performs the analysis.

[0524] Furthermore, the server integrates the analysis results of the generative AI model with the emotional data from the emotion engine. This integration improves the accuracy of intent estimation. For example, if a subject says "It's cold" while standing up and the emotional data indicates "anxiety," the generative AI model will more accurately estimate the intent as "I want to go back indoors."

[0525] Finally, the server generates the estimated intention in text format and inputs that text into a speech synthesis engine to generate voice data. The generated voice data is sent to the device (e.g., audio glasses) used by the support person. The device (audio glasses) plays back the received voice data and conveys it to the support person, accurately conveying the support person's intention. For example, the audio glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0526] In this way, this invention improves the accuracy of estimating intentions by integrating and analyzing not only the behavior and voice of the person being supported, but also emotional data, making it possible to quickly and accurately convey intentions to the person providing support.

[0527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0528] Step 1:

[0529] The terminal (wearable device) collects behavioral and voice data from the person receiving support. The wearable device's accelerometer, gyroscope, microphone, and location information acquisition function are used to record the person receiving support's movements (e.g., standing up, walking) and voice (e.g., saying "it's cold") in real time. The input is the person receiving support's physical movements and voice, and the output is the collected raw data. As a specific example, data is collected when the person receiving support starts walking while saying "it's cold."

[0530] Step 2:

[0531] The data collected by the device is preprocessed and organized. First, data from the accelerometer and gyroscope is analyzed to identify walking and other movements. Next, voice data is converted into text using speech recognition technology. An emotion engine then identifies emotions from voice tone and facial expressions, generating emotion data such as "anxiety." The input is the collected raw data, and the output is organized data (movement data, textual voice data, and emotion data). For example, the statement "It's cold" is converted into text, and data identified as "walking movements" is generated.

[0532] Step 3:

[0533] The device sends the preprocessed data to the server. The input is the organized data, and the output is the data sent to the server. For example, the textual utterance "It's cold" and the emotional data "I feel anxious" are sent to the server.

[0534] Step 4:

[0535] The server further preprocesses the transmitted data, shaping data of different formats into a unified format and making it consistent. The input is the organized data transmitted from the device, and the output is unified format data. For example, text data, action data, and emotion data are converted into a unified format.

[0536] Step 5:

[0537] The server uses a generative AI model to analyze the preprocessed data and infer the intention of the person being supported. Data is input into the generative AI model using a prompt. The input is unified format data and a prompt, and the output is the inferred intention. As a specific example, the prompt is "The person being supported starts walking while saying 'It's cold', and the emotion is identified as 'anxiety'. Please infer the person's intention," and the inferred intention is "It's cold, so I want to go back indoors."

[0538] Step 6:

[0539] The server integrates the analysis results of the generative AI model with the emotion data obtained from the emotion engine. The input is the estimated intention and emotion data, and the output is the integrated intention data. For example, by integrating the emotion data of "anxiety" with the intention of "wanting to go back indoors," the accuracy of the intent estimation improves.

[0540] Step 7:

[0541] The server generates the estimated intention in text format and inputs it into a speech synthesis engine to generate speech data. The input is the intention data in text format, and the output is speech data. For example, the text "It's cold, so I want to go back indoors" is generated as speech data.

[0542] Step 8:

[0543] The server sends the generated voice data to the audio glasses. The input is the generated voice data, and the output is the voice data sent to the audio glasses. For example, voice data saying "It's cold, so I want to go back indoors" is sent to the audio glasses.

[0544] Step 9:

[0545] The device (audio glasses) plays the received voice data and conveys it to the support person. The input is the voice data sent to the audio glasses, and the output is the played voice data. Specifically, the audio glasses play the voice "It's cold, I want to go back indoors," allowing the support person to take appropriate action.

[0546] (Application example 2)

[0547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0548] Food delivery services are required to accurately understand customer intent and emotions to reduce misunderstandings and troubles about orders and improve customer satisfaction. However, conventional systems have difficulty accurately analyzing customer intent and emotions, leading to a high likelihood of misunderstandings. Furthermore, there is a lack of a mechanism to integrate emotion analysis and intent estimation to improve accuracy.

[0549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0550] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for transmitting the data collected from the wearable device to the server and preprocessing the data, means for analyzing the preprocessed data using a generative AI and estimating the intention of the person being supported, means for analyzing the emotions of the person being supported by combining an emotion engine and integrating it with the generative AI to improve the accuracy of the intention estimation, means for converting the estimated intention into voice data and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp customer intentions and emotions even in food delivery services, reducing misunderstandings about order details and problems.

[0551] A "supported person" is a user or customer of a food delivery service who provides specific actions or voice data.

[0552] A "wearable device" is an electronic device worn by an individual on the body, which includes an accelerometer, gyroscope, microphone, and GPS, and has the ability to collect physical movements and sounds.

[0553] The "server" is a computer system that preprocesses, analyzes, and infers intent from data. It utilizes generative AI and an emotion engine to accurately infer the intent of the person being supported, and processes and transmits the necessary data.

[0554] "Generative AI" refers to technology that uses natural language processing models to analyze data and infer the intentions of the person being supported, and specifically includes generative artificial intelligence models such as GPT-4.

[0555] The "emotion engine" refers to technology that analyzes the tone of voice, facial expressions, and movements of the person being supported to identify their emotional state, and is integrated with generative AI to improve the accuracy of intent estimation.

[0556] "Intention estimation accuracy" refers to the high level of ability to accurately analyze and judge the intentions of the person being supported, and by integrating with the emotion engine, intentions can be identified with greater accuracy.

[0557] "Voice data" refers to voice information generated by a voice synthesis engine based on analyzed text information, and is used to convey the intentions of the person being supported to the person providing support.

[0558] "Supporter" refers to the delivery person who delivers food to customers in a food delivery service, who receives and plays back the transmitted voice data and takes appropriate action based on the customer's intentions.

[0559] This invention is a system for accurately understanding customer intentions and emotions in food delivery services, reducing misunderstandings and troubles about orders and improving customer satisfaction. This system includes wearable devices (smart glasses, smart watches, etc.), a server, generative AI, an emotion engine, and an audio device (audio glasses).

[0560] 1. Wearable devices that capture the behavior of those receiving support

[0561] First, the customer wears smart glasses. The glasses are equipped with an accelerometer, gyroscope, microphone, and GPS to collect the customer's physical movements (e.g., waving, walking, stopping, etc.) and voice (order details, preferences, and requests) in real time. The collected data is either temporarily stored on the device or sent directly to a server.

[0562] 2. Data preprocessing and transmission

[0563] After receiving this data, the server performs initial preprocessing. Specifically, it recognizes movements from data from the accelerometer and gyroscope, converts recorded voice data into text using voice recognition technology, and uses an emotion engine to identify emotions from the voice tone and facial expressions of the person receiving support. The identified emotion data is also integrated for analysis.

[0564] 3. Data analysis and intent estimation

[0565] The server inputs the received data into a generation AI (e.g., GPT-4) for analysis. The generation AI uses text data, behavioral data, and emotional data to infer the intention of the person being supported. For example, when a customer says, "Please avoid spicy food," and the emotion engine provides information that the face is smiling, the generation AI infers that "the customer does not want spicy food."

[0566] 4. Emotional Data Integration

[0567] The server integrates the analysis results of the generative AI with data from the emotion engine to improve the accuracy of intent estimation. Based on this integrated data, the customer's intent becomes clear and the order details can be accurately understood.

[0568] 5. Speech conversion and output of estimated intention

[0569] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is sent to the delivery person's audio device (Audio Glasses). The Audio Glasses play back the received voice data and accurately convey the intended intention of the support person to the delivery person.

[0570] Specific examples

[0571] For example, consider the following scenario:

[0572] 1. A customer puts on smart glasses and says, "Please avoid spicy food."

[0573] 2. The smart glasses' sensors collect movement data, the microphone records voice data, and the emotion engine analyzes smiles (positive).

[0574] 3. The data is sent to the server, where it undergoes preprocessing and is then analyzed by the generation AI.

[0575] 4. The generative AI understands that the customer wants to avoid spicy food and infers their intention.

[0576] 5. The server converts this intention into voice data and transmits it to the delivery person's audio glasses, saying, "Please avoid spicy food."

[0577] Prompt Sentence Examples

[0578] The user says, "I don't like spicy food." The face is smiling (positive). Please infer the user's intention from this statement.

[0579] This system makes it possible to accurately understand the complex intentions and emotions of customers, significantly improving customer satisfaction in delivery services.

[0580] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0581] Step 1:

[0582] The server collects physical movement and voice data from a wearable device (smart glasses) that captures the behavior of the person being supported.

[0583] Input: Real-time movement data (acceleration, gyro) and audio data from the supported individual.

[0584] Output: Recording files of motion data (e.g. accelerometer and gyroscope data) and audio data.

[0585] Specific operation: The smart glasses detect the customer's movements and speech and record them as digital data.

[0586] Step 2:

[0587] The server pre-processes the collected data.

[0588] Input: Motion data and audio data (recorded files).

[0589] Output: Preprocessed action data (action classification results) and audio-text data.

[0590] Specific operation: Recognizes actions such as walking and standing from data from the accelerometer and gyroscope, and converts the voice data into text using voice recognition software. Example: Converts the voice data "I don't like spicy food" into text "I don't like spicy food."

[0591] Step 3:

[0592] The server uses an emotion engine to analyze the emotions of the person being supported from the collected data.

[0593] Input: Audio and text data.

[0594] Output: Emotion data (e.g., emotional state, such as positive or negative).

[0595] Specific operation: The emotion engine analyzes the tone of voice and facial expression data (facial recognition by camera) to identify the emotional state. Example: Because the person is smiling, it is judged to be "positive."

[0596] Step 4:

[0597] The server uses generative AI to analyze the preprocessed data and emotional data to infer the intentions of the person being supported.

[0598] Input: Preprocessed text data and sentiment data.

[0599] Output: The inferred intent (e.g., "Avoid spicy food").

[0600] Specific operation: A generative AI model (e.g., GPT-4) analyzes the preprocessed data and derives the intention of the person being supported using the prompt: "The user says, 'I don't like spicy food.' Their face is smiling (positive). Please infer the user's intention from this statement."

[0601] Step 5:

[0602] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate speech data.

[0603] Input: Inferred intent (in text format).

[0604] Output: The generated audio data.

[0605] Specific behavior: Use a speech synthesis engine (e.g., Text-to-Speech engine) to convert the text "Avoid spicy food" into audio data.

[0606] Step 6:

[0607] The server transmits the generated voice data to the delivery person's audio device (audio glasses) and plays the voice data.

[0608] Input: The generated audio data.

[0609] Output: Playback of the audio data sent to the delivery person.

[0610] Specific operation: The audio glasses receive the voice data and play back "Please avoid spicy food," allowing the delivery person to accurately understand the customer's intentions.

[0611] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0613] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0614] [Third embodiment]

[0615] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0616] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0617] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0618] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0619] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0620] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0621] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0622] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0623] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0624] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0625] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0626] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0627] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to the supporter. This system includes a wearable device, a server, a generative AI, and devices such as audio glasses.

[0628] 1. Wearable devices that capture the behavior of those receiving support

[0629] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0630] 2. Data preprocessing and transmission

[0631] The device sends the collected data to the server, where it is encoded and converted into an appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and voice data recorded by the microphone are sent to the server.

[0632] 3. Analysis of Inferred Intent

[0633] The server preprocesses the received data and formats it into a unified format before inputting it into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generation AI will interpret this as, "It's cold, so I want to go back indoors."

[0634] 4. Speech conversion and output of estimated intention

[0635] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device such as audio glasses worn by the support person.

[0636] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0637] Specific examples

[0638] For example, the following scenario is possible:

[0639] Step 1: The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's speech and collects movement data.

[0640] Step 2: The wearable device sends voice and motion data to the server.

[0641] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0642] Step 4: The server converts this intent into audio data and sends it to the audio glasses.

[0643] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0644] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported and enables the supporter to respond appropriately.

[0645] The processing flow will be explained below.

[0646] Step 1:

[0647] The device collects the movements and voice of the person being supported. Specifically, the wearable device uses an accelerometer and gyroscope to capture physical movement data, and a microphone to record voice.

[0648] Step 2:

[0649] The device preprocesses and organizes this data, for example, by recognizing walking movements from accelerometer and gyroscope data and converting recorded audio data into text using voice recognition technology.

[0650] Step 3:

[0651] The device sends the preprocessed data to the server, where it is encoded according to a communication protocol and transmitted using a secure communication method (e.g., HTTPS).

[0652] Step 4:

[0653] The server further preprocesses the received data for analysis, specifically by converting different data formats into a unified and consistent format.

[0654] Step 5:

[0655] The server inputs the formatted data into the generation AI for analysis. The generation AI uses a natural language processing model (e.g., GPT-4) to infer the intention of the person being supported.

[0656] Step 6:

[0657] The server receives the analysis results of the generation AI in text format and inputs the text into a speech synthesis engine, which converts the text into voice data.

[0658] Step 7:

[0659] The server then transmits the generated audio data to the device (audio glasses) worn by the supporter, where the data is again encoded and transmitted via a secure means.

[0660] Step 8:

[0661] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The audio data is played and transmitted to the support person.

[0662] Step 9:

[0663] The user (support person) listens to the audio through the audio glasses and understands the intention of the person being supported. For example, if the person being supported says, "It's cold, so I want to go back indoors," the support person will guide the person indoors.

[0664] Example 1

[0665] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0666] In modern society, it is extremely important to accurately grasp the intentions of those receiving support who have cognitive disabilities and to convey them quickly and accurately to those providing support. However, conventional technology has difficulty accurately grasping the intentions of those receiving support, resulting in delayed responses and often misinterpretations. Furthermore, when analyzing data collected from multiple sensors and devices, inefficiencies in integration and preprocessing make it difficult to respond in real time. There is a need to provide a system that solves these issues and facilitates smooth communication between those receiving support and those providing support.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0668] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for encoding the data collected from the wearable device, converting it into an appropriate data format, and transmitting it to the server, means for the server to preprocess the data received and estimate the intention of the person being supported using a generative AI model, means for generating the estimated intention in text format, inputting it into a speech synthesis engine to convert it into voice data, and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp the intention of the person being supported in real time and convey it to the supporter quickly and accurately.

[0669] A "wearable device" is a device that is worn on the body and is equipped with sensors and communication functions to collect the physical movements and voices of the person being supported.

[0670] An "accelerometer" is a type of sensor installed in wearable devices, and is a device that measures the acceleration of an object.

[0671] A "gyroscope" is a device installed in wearable devices that measures angular velocity to detect changes in the rotation or tilt of an object.

[0672] A "microphone" is a device that converts sound into an electrical signal, and is a component installed in wearable devices to collect the speech of the person being supported.

[0673] "GPS" stands for Global Positioning System, a system for measuring positions on Earth with high precision.

[0674] "Encoding" is the process of converting data into a particular format, with the goal of optimizing it for transmission or storage.

[0675] A "generative AI model" is an artificial intelligence model that analyzes data and generates new information or results based on that information, and is used here to infer the intentions of the person being supported.

[0676] A "speech synthesis engine" is a system that converts text data into speech data, and is a device used to output the estimated intention as speech.

[0677] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to a supporter. This system includes devices such as a wearable device, a server, a generative AI, and an audio output device.

[0678] First, a wearable device is used to capture the behavior of the person being supported. A specific example is a smartwatch. This device is equipped with an accelerometer, gyroscope, microphone, and GPS, and collects the physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") of the person being supported in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0679] Next, when the device sends the collected data to the server, the data is encoded and converted into the appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and audio data recorded by the microphone are sent to the server. The audio data is encoded into MP3 format, and the sensor data is converted into JSON format.

[0680] The server preprocesses the received data. This preprocessing includes formatting data from different sensors into a unified format. The formatted data is then input into a generative AI model (e.g., GPT-4). The generative AI model analyzes the data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generative AI will interpret this as, "It's cold, so I want to go back indoors."

[0681] The server generates the estimated intention in text format. This text is input into a speech synthesis engine and converted into voice data. The generated voice data is sent to an audio output device (such as audio glasses) worn by the support person. The device (audio output device) plays the received voice data and conveys it to the support person. This ensures that the intention of the person being supported is accurately conveyed. For example, the audio glasses could say, "It's cold, so I'd like to go back indoors," allowing the support person to take appropriate action.

[0682] To illustrate, consider the following scenario:

[0683] 1. The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's words and collects movement data.

[0684] 2. The wearable device sends voice and movement data to the server.

[0685] 3. The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0686] 4. The server converts this intent into audio data and sends it to the audio glasses.

[0687] 5. The audio glasses will tell the support person, "It's cold, I want to go back indoors."

[0688] Below is an example of a prompt sentence to input to the generative AI model.

[0689] Example 1: "The person you are supporting walks away saying, 'It's cold.' Guess what he wants to do."

[0690] Example 2: "The user stands up and says 'I'm thirsty.' Please infer the intention of this action."

[0691] As described above, the present invention makes it possible to accurately grasp the intentions of the person being supported in real time and to convey them to the person providing support quickly and accurately.

[0692] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0693] Step 1:

[0694] The terminal uses a wearable device to capture the behavior of the person being supported. Specifically, it uses an accelerometer, gyroscope, microphone, and GPS to collect the physical movements and voice of the person being supported in real time. The input is the movements and speech of the person being supported, and the output is to store these as digital data.

[0695] As a specific example of operation, the accelerometer and gyroscope detect the movement of the person being supported standing up, and at the same time, the microphone collects the voice recording of the person saying "It's cold."

[0696] Step 2:

[0697] The device encodes the collected data and converts it into the appropriate data format. For example, audio data is encoded into MP3 format, and sensor data is converted into JSON format. The converted data is then sent to the server via a communication module. The input is the raw data collected by the wearable device, and the output is the encoded data.

[0698] Specific examples of operations include compressing audio data and converting it to MP3 format, and converting acceleration and gyroscope data to JSON format.

[0699] Step 3:

[0700] The server preprocesses the received data. This preprocessing involves shaping the heterogeneous data into a unified format. Data from different sensors is integrated into a single format. The input is encoded data, and the output is data in a unified format that can be fed into a generative AI model.

[0701] A specific example of the operation includes converting voice data into text using a voice recognition engine and combining it with sensor data.

[0702] Step 4:

[0703] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes and infers the intention of the person receiving support. The input is preprocessed data in a unified format, and the output is text data of the inferred intention of the person receiving support.

[0704] As a specific example of operation, the generative AI model is input with a prompt statement such as, "The person being supported started walking away saying, 'It's cold.' Please guess what he wants to do.", and an intent such as, "It's cold, so I want to go back indoors" is generated.

[0705] Step 5:

[0706] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to convert it into voice data. The input is the text data of the estimated intention, and the output is voice data. The generated voice data is sent to the audio output device used by the supporter.

[0707] As a specific example of operation, text such as "It's cold, so I want to go back indoors" is input into a speech synthesis engine to generate voice data.

[0708] Step 6:

[0709] The terminal (audio output device) plays the received audio data and transmits it to the device that supports it. The input is the audio data sent from the server, and the output is the played audio.

[0710] As a specific example of how it works, the audio glasses will play a voice message saying, "It's cold, so I want to go back indoors," and the support person will hear this and take appropriate action.

[0711] This enables the system to accurately grasp the intentions of the person being supported in real time and convey them to the person providing support quickly and accurately.

[0712] (Application example 1)

[0713] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0714] The present invention solves the problem that customers with cognitive disabilities have difficulty accurately communicating their intentions and requests to store staff. In particular, when customers are unable to express their intentions well, it becomes difficult for store staff to respond appropriately, which can lead to a decline in the quality of the customer experience. In such situations, there is a need for a means to accurately estimate a customer's intentions and requests and quickly communicate them to store staff.

[0715] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0716] In this invention, the server includes: means for collecting the physical movements and voice of the support recipient using a wearable device that captures the support recipient's behavior; means for transmitting the data collected from the wearable device to the server and preprocessing the data; means for analyzing the preprocessed data using a generative AI model and estimating the support recipient's intention; means for converting the estimated intention into voice data and transmitting it to a device used by a store employee; means for playing the transmitted voice data and notifying the employee; and means for the employee to prompt an appropriate response based on the notification. This enables customers with cognitive disabilities to accurately communicate their intentions in the store to employees, allowing the employees to respond quickly and appropriately.

[0717] "Support recipients" refer to people who have cognitive problems and require external support.

[0718] A "wearable device" is an electronic device that is worn on the body and is equipped with sensors and communication functions.

[0719] "Physical actions" refer to bodily movements such as walking, standing up, and raising your hands.

[0720] "Voice" refers to the words and sounds made by the person being supported.

[0721] "Server" refers to a computer system for processing and storing data over a network.

[0722] "Preprocessing" refers to the process of converting collected raw data into a format that is easy to analyze.

[0723] "Generative AI model" refers to the artificial intelligence algorithm used to analyze collected data and infer the intent of the person being supported.

[0724] "Brick and mortar store" refers to a commercial establishment that offers goods or services in a physical location.

[0725] "Employees" refers to staff who work in physical stores and deal with customers.

[0726] "Voice data" refers to data that has been converted into voice format from intent analyzed by a generative AI model.

[0727] "Notification" refers to information that communicates the support recipient's intentions to the employee.

[0728] The present invention relates to a system that enables customers with cognitive disabilities to accurately communicate their intentions and requests to store staff in a physical store. The system includes a wearable device (e.g., a smartwatch), a server, a generative AI model, and a notification device (e.g., a smartphone).

[0729] Data collection using wearable devices

[0730] The terminal uses a wearable device to collect the physical movements (e.g., walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported. Specifically, the wearable device is equipped with an accelerometer, gyroscope, microphone, and GPS function, and collects data in real time using these sensors. The collected data is either stored in the wearable device or sent directly to a server.

[0731] Data preprocessing and transmission

[0732] The server receives the data sent by the wearable device and preprocesses it, converting voice data to text and analyzing sensor data to understand the physical movements of the person being supported, then formats the data appropriately and prepares it for input into the generative AI model.

[0733] Intent Analysis

[0734] The server analyzes the preprocessed data using a generative AI model (e.g., GPT-4). The generative AI model infers the intention of the person being supported based on the voice text and movement data. For example, if the person being supported starts walking while saying, "I want to go to the toilet," the generative AI model will interpret this as the intention, "I want to go to the toilet."

[0735] Notice of presumed intent

[0736] The server generates the estimated intent in text format and inputs that text into a speech synthesis engine to generate voice data. This voice data is then sent to a device used by an employee in the physical store. The device then notifies the employee that it has received the notification and prompts them to take an appropriate response. This allows the employee to accurately understand the intent of the person being supported and respond quickly.

[0737] Specific examples

[0738] Consider the following scenario: If a person being supported walks through a physical store saying, "I want to go to the restroom," the wearable device collects this statement and movement data. The collected data is sent to a server, where it is preprocessed and a generative AI model analyzes the person's intent. As a result, the intention of "I want to go to the restroom" is estimated, and this intent is notified via voice to the employee's smartphone, allowing the employee to immediately guide the person being supported to the restroom.

[0739] Prompt Sentence Examples

[0740] An example of a prompt for the generative AI model is, "Analyze the behavior and voice data of the customer who says that the employee wants to go to the restroom, and infer their intent." This allows the generative AI model to infer the appropriate intent based on the voice and movement data of the person being supported.

[0741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0742] Step 1:

[0743] The terminal (wearable device) collects the physical movements (walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported in real time. The wearable device's accelerometer, gyroscope, microphone, and GPS are used to acquire movement and voice data, allowing the current actions and speech of the person being supported to be collected.

[0744] Step 2:

[0745] The device temporarily stores the collected motion and voice data or transmits it to a server in real time. The data includes motion data from the accelerometer and gyroscope, and voice data from the microphone. This allows the raw data collected by the wearable device to be transmitted to the server.

[0746] Step 3:

[0747] The server receives the data sent from the device and preprocesses it. In the preprocessing step, the voice data is converted into text (e.g., "I want to go to the toilet"), and the motion data is simultaneously analyzed to identify physical movements (e.g., while walking). Other processes include standardizing the data format and removing noise. This results in preprocessed data that is ready to be input into the generative AI model.

[0748] Step 4:

[0749] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes the voice text and behavioral data to infer the intention of the person being supported. A prompt such as "Analyze the behavior and voice data of a customer who says that an employee wants to go to the restroom, and infer their intention" is used. This allows the generative AI model to infer the intention of "I want to go to the restroom" from the input data.

[0750] Step 5:

[0751] The server generates the intent estimated by the generative AI model in text format. This text is then input into a speech synthesis engine to generate voice data. The speech synthesis engine converts the text into natural-sounding speech and creates an audio file. This outputs the estimated intent as voice data.

[0752] Step 6:

[0753] The server then sends the generated voice data to a device (e.g., a smartphone) used by an employee at the physical store. The employee's device receives and plays the voice data, notifying the employee of the intention of the person receiving support. This allows the employee to accurately understand the intention of the person receiving support and respond quickly.

[0754] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0755] This invention is a system that estimates the intentions of support recipients with cognitive problems, improves the accuracy of the estimation by combining it with an emotion engine, and accurately conveys those intentions to the supporter. This system includes devices such as a wearable device, a server, a generative AI, an emotion engine, and audio glasses.

[0756] 1. Wearable devices that capture the behavior of those receiving support

[0757] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0758] 2. Data preprocessing and transmission

[0759] The device preprocesses and organizes this data. For example, it recognizes walking movements from data from the accelerometer and gyroscope, and converts recorded voice data into text using voice recognition technology. In addition, an emotion engine identifies the emotions of the person being supported from facial expressions, tone of voice, and body movements. The identified emotion data is also sent to the server.

[0760] 3. Data analysis and intent estimation

[0761] The server further preprocesses the received data for analysis. Specifically, it converts data from different formats into a unified format to ensure consistency. The converted data is then input into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person being supported.

[0762] 4. Emotional Data Integration

[0763] The server combines the analysis results of the generation AI with the emotional data from the emotion engine. This combined data is used to improve the accuracy of estimating the intention of the person receiving support. For example, if the person receiving support says "It's cold" and stands up, and the emotional data indicates feelings such as "anxiety" or "discomfort," the generation AI will more accurately estimate the intention as "I want to go back indoors."

[0764] 5. Speech conversion and output of estimated intention

[0765] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device (audio glasses) worn by the support person.

[0766] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0767] Specific examples

[0768] For example, the following scenario is possible:

[0769] Step 1: The person being supported begins walking outdoors, saying "It's cold." The wearable device records the person's speech and collects movement data. At the same time, the emotion engine identifies the emotion "anxiety" from the person's tone of voice and facial expression.

[0770] Step 2: The wearable device sends voice data, movement data, and emotion data to the server.

[0771] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0772] Step 4: The server integrates the analysis results of the generative AI with the emotional data to improve the accuracy of the intention estimation. The server converts this intention into voice data and sends it to the audio glasses.

[0773] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0774] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported, improves the accuracy by combining it with an emotion engine, and enables the supporter to respond appropriately.

[0775] The processing flow will be explained below.

[0776] Step 1:

[0777] The device collects the actions and voice of the person receiving support. A wearable device (e.g., a smartwatch) uses an accelerometer, gyroscope, microphone, and GPS to record the physical movements and voice of the person receiving support in real time. For example, it captures the action of the person saying, "It's cold," and starting to walk away.

[0778] Step 2:

[0779] The device preprocesses the collected motion data and recognizes specific movements such as walking and standing up. At the same time, it converts the recorded voice data into text using voice recognition technology. For example, the voice data can be converted into text data such as "It's cold."

[0780] Step 3:

[0781] The device sends the collected and pre-processed data to the server, where the motion data, voice text, and location data are encoded using a secure communication protocol (e.g., HTTPS).

[0782] Step 4:

[0783] The server further preprocesses the data it receives and formats it into a unified format. Data in different formats is processed into a form suitable for analysis. For example, the action data for "standing up" and the voice text "It's cold" are unified.

[0784] Step 5:

[0785] The server inputs the formatted data into the generation AI for analysis. The generation AI (e.g., GPT-4) analyzes the movement data and voice data and infers the intention of the person being supported. For example, the generation AI infers the intention, "It's cold, so I want to go back indoors."

[0786] Step 6:

[0787] The device recognizes the emotions of the person receiving support. Using the camera and microphone installed in the wearable device, emotions are identified from the facial expressions and tone of voice of the person receiving support. For example, emotional data such as "anxiety" or "discomfort" is generated.

[0788] Step 7:

[0789] The server integrates the emotion data received from the emotion engine with the analysis results of the generative AI. This improves the accuracy of estimating the intention of the person being supported. For example, the intention of "I want to go back indoors because it's cold" can be further confirmed by the emotion data of "anxiety."

[0790] Step 8:

[0791] The server converts the estimated intent into voice data. The generated text is then input into a speech synthesis engine to generate voice data. For example, the generated voice data might say, "It's cold, so I want to go back indoors."

[0792] Step 9:

[0793] The server sends the generated audio data to the device (audio glasses) worn by the supporter. The audio data is encoded and transmitted using a secure communication protocol.

[0794] Step 10:

[0795] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The built-in speaker of the Audio Glasses plays the audio data and conveys it to the support person. For example, the voice may say, "It's cold, so I want to go back indoors."

[0796] Step 11:

[0797] The user (support person) listens to the audio through the audio glasses and understands the intentions of the person being supported. For example, if the user understands that the person wants to go back indoors, they can take action to guide the person indoors.

[0798] Example 2

[0799] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0800] In situations where support recipients with cognitive disabilities have difficulty accurately communicating their intentions, there is a need to accurately estimate their intentions and communicate them quickly and accurately to supporters. However, conventional systems lack the accuracy of data collection and intention estimation, and in particular lack a mechanism to improve estimation accuracy by combining emotional data. As a result, the intentions of support recipients are often misunderstood, making it difficult to provide appropriate support.

[0801] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for further preprocessing the transmitted data and shaping data of different formats into a unified format, means for analyzing the preprocessed data using a generative AI model and estimating the intention of the person being supported, and means for integrating emotion data acquired from an emotion engine with the estimated intention to improve the accuracy of intention estimation. In this way, by integrating and analyzing not only the behavior and voice of the person being supported but also the emotion data, the accuracy of intention estimation improves, making it possible to quickly and accurately convey the intention to the person providing support.

[0802] - "Support recipient" refers to a person who has cognitive problems and requires the support of others.

[0803] A "wearable device" is an electronic device that can be worn by the user and is equipped with sensors and microphones to collect data.

[0804] "Physical movement" refers to data showing changes in the body movements and posture of the person being supported.

[0805] "Audio" refers to data that indicates the statements and tone of voice made by the person being supported.

[0806] "Preprocessing" refers to the process of formatting and organizing collected data into a form that is easy to analyze.

[0807] A "server" refers to a computer system that stores and analyzes data and communicates with external devices.

[0808] "Generative AI model" refers to an artificial intelligence model that analyzes collected data and infers the intentions of the person being supported.

[0809] An "emotion engine" refers to a system that identifies emotions from a person's facial expressions and tone of voice and outputs them as data.

[0810] "Intention" refers to what the person receiving support wants or intends to do.

[0811] "Unified format" refers to the formatting of data of different formats into one consistent format.

[0812] "Audio data" refers to data that has been converted from text information into a digital format that can be played as sound.

[0813] "Audio glasses" are devices that provide information through sight and hearing, especially those worn by a support person.

[0814] "Intention estimation accuracy" refers to an indicator of how accurately the intention of the person being supported can be estimated.

[0815] This invention is a system that estimates the intentions of the person being supported with high accuracy and quickly and accurately communicates those intentions to the supporter. The system includes devices such as a wearable device, a server, a generative AI model, an emotion engine, and audio glasses.

[0816] First, the terminal (wearable device) collects the behavioral and voice data of the person being supported. The wearable device is equipped with an accelerometer, gyroscope, microphone, and location information acquisition function, which allows it to record the person's actions (e.g., standing up, walking, raising their hand, etc.) and voice (e.g., utterances such as "It's cold") in real time. This collected data is either temporarily stored within the device or sent directly to a server.

[0817] The device then preprocesses and organizes this data. This involves identifying walking and other movements from accelerometer and gyroscope data, converting recorded voice data into text using speech recognition technology, and using an emotion engine to identify emotions from voice tone and facial expressions, generating emotion data such as "anxiety." This organized data is then sent to a server.

[0818] The server further preprocesses the transmitted data. Specifically, it formats the data in different formats into a unified format to ensure consistency. This formatted data is then input into a generative AI model (e.g., GPT-4). The server analyzes the data using the generative AI model and infers the intention of the person receiving support. For example, by inputting a prompt such as, "The person receiving support started walking while saying 'I'm cold,' and their emotion was identified as 'anxious.' Please infer the person's intention," the generative AI model performs the analysis.

[0819] Furthermore, the server integrates the analysis results of the generative AI model with the emotional data from the emotion engine. This integration improves the accuracy of intent estimation. For example, if a subject says "It's cold" while standing up and the emotional data indicates "anxiety," the generative AI model will more accurately estimate the intent as "I want to go back indoors."

[0820] Finally, the server generates the estimated intention in text format and inputs that text into a speech synthesis engine to generate voice data. The generated voice data is sent to the device (e.g., audio glasses) used by the support person. The device (audio glasses) plays back the received voice data and conveys it to the support person, accurately conveying the support person's intention. For example, the audio glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0821] In this way, this invention improves the accuracy of estimating intentions by integrating and analyzing not only the behavior and voice of the person being supported, but also emotional data, making it possible to quickly and accurately convey intentions to the person providing support.

[0822] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0823] Step 1:

[0824] The terminal (wearable device) collects behavioral and voice data from the person receiving support. The wearable device's accelerometer, gyroscope, microphone, and location information acquisition function are used to record the person receiving support's movements (e.g., standing up, walking) and voice (e.g., saying "it's cold") in real time. The input is the person receiving support's physical movements and voice, and the output is the collected raw data. As a specific example, data is collected when the person receiving support starts walking while saying "it's cold."

[0825] Step 2:

[0826] The data collected by the device is preprocessed and organized. First, data from the accelerometer and gyroscope is analyzed to identify walking and other movements. Next, voice data is converted into text using speech recognition technology. An emotion engine then identifies emotions from voice tone and facial expressions, generating emotion data such as "anxiety." The input is the collected raw data, and the output is organized data (movement data, textual voice data, and emotion data). For example, the statement "It's cold" is converted into text, and data identified as "walking movements" is generated.

[0827] Step 3:

[0828] The device sends the preprocessed data to the server. The input is the organized data, and the output is the data sent to the server. For example, the textual utterance "It's cold" and the emotional data "I feel anxious" are sent to the server.

[0829] Step 4:

[0830] The server further preprocesses the transmitted data, shaping data of different formats into a unified format and making it consistent. The input is the organized data transmitted from the device, and the output is unified format data. For example, text data, action data, and emotion data are converted into a unified format.

[0831] Step 5:

[0832] The server uses a generative AI model to analyze the preprocessed data and infer the intention of the person being supported. Data is input into the generative AI model using a prompt. The input is unified format data and a prompt, and the output is the inferred intention. As a specific example, the prompt is "The person being supported starts walking while saying 'It's cold', and the emotion is identified as 'anxiety'. Please infer the person's intention," and the inferred intention is "It's cold, so I want to go back indoors."

[0833] Step 6:

[0834] The server integrates the analysis results of the generative AI model with the emotion data obtained from the emotion engine. The input is the estimated intention and emotion data, and the output is the integrated intention data. For example, by integrating the emotion data of "anxiety" with the intention of "wanting to go back indoors," the accuracy of the intent estimation improves.

[0835] Step 7:

[0836] The server generates the estimated intention in text format and inputs it into a speech synthesis engine to generate speech data. The input is the intention data in text format, and the output is speech data. For example, the text "It's cold, so I want to go back indoors" is generated as speech data.

[0837] Step 8:

[0838] The server sends the generated voice data to the audio glasses. The input is the generated voice data, and the output is the voice data sent to the audio glasses. For example, voice data saying "It's cold, so I want to go back indoors" is sent to the audio glasses.

[0839] Step 9:

[0840] The device (audio glasses) plays the received voice data and conveys it to the support person. The input is the voice data sent to the audio glasses, and the output is the played voice data. Specifically, the audio glasses play the voice "It's cold, I want to go back indoors," allowing the support person to take appropriate action.

[0841] (Application example 2)

[0842] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0843] Food delivery services are required to accurately understand customer intent and emotions to reduce misunderstandings and troubles about orders and improve customer satisfaction. However, conventional systems have difficulty accurately analyzing customer intent and emotions, leading to a high likelihood of misunderstandings. Furthermore, there is a lack of a mechanism to integrate emotion analysis and intent estimation to improve accuracy.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0845] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for transmitting the data collected from the wearable device to the server and preprocessing the data, means for analyzing the preprocessed data using a generative AI and estimating the intention of the person being supported, means for analyzing the emotions of the person being supported by combining an emotion engine and integrating it with the generative AI to improve the accuracy of the intention estimation, means for converting the estimated intention into voice data and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp customer intentions and emotions even in food delivery services, reducing misunderstandings about order details and problems.

[0846] A "supported person" is a user or customer of a food delivery service who provides specific actions or voice data.

[0847] A "wearable device" is an electronic device worn by an individual on the body, which includes an accelerometer, gyroscope, microphone, and GPS, and has the ability to collect physical movements and sounds.

[0848] The "server" is a computer system that preprocesses, analyzes, and infers intent from data. It utilizes generative AI and an emotion engine to accurately infer the intent of the person being supported, and processes and transmits the necessary data.

[0849] "Generative AI" refers to technology that uses natural language processing models to analyze data and infer the intentions of the person being supported, and specifically includes generative artificial intelligence models such as GPT-4.

[0850] The "emotion engine" refers to technology that analyzes the tone of voice, facial expressions, and movements of the person being supported to identify their emotional state, and is integrated with generative AI to improve the accuracy of intent estimation.

[0851] "Intention estimation accuracy" refers to the high level of ability to accurately analyze and judge the intentions of the person being supported, and by integrating with the emotion engine, intentions can be identified with greater accuracy.

[0852] "Voice data" refers to voice information generated by a voice synthesis engine based on analyzed text information, and is used to convey the intentions of the person being supported to the person providing support.

[0853] "Supporter" refers to the delivery person who delivers food to customers in a food delivery service, who receives and plays back the transmitted voice data and takes appropriate action based on the customer's intentions.

[0854] This invention is a system for accurately understanding customer intentions and emotions in food delivery services, reducing misunderstandings and troubles about orders and improving customer satisfaction. This system includes wearable devices (smart glasses, smart watches, etc.), a server, generative AI, an emotion engine, and an audio device (audio glasses).

[0855] 1. Wearable devices that capture the behavior of those receiving support

[0856] First, the customer wears smart glasses. The glasses are equipped with an accelerometer, gyroscope, microphone, and GPS to collect the customer's physical movements (e.g., waving, walking, stopping, etc.) and voice (order details, preferences, and requests) in real time. The collected data is either temporarily stored on the device or sent directly to a server.

[0857] 2. Data preprocessing and transmission

[0858] After receiving this data, the server performs initial preprocessing. Specifically, it recognizes movements from data from the accelerometer and gyroscope, converts recorded voice data into text using voice recognition technology, and uses an emotion engine to identify emotions from the voice tone and facial expressions of the person receiving support. The identified emotion data is also integrated for analysis.

[0859] 3. Data analysis and intent estimation

[0860] The server inputs the received data into a generation AI (e.g., GPT-4) for analysis. The generation AI uses text data, behavioral data, and emotional data to infer the intention of the person being supported. For example, when a customer says, "Please avoid spicy food," and the emotion engine provides information that the face is smiling, the generation AI infers that "the customer does not want spicy food."

[0861] 4. Emotional Data Integration

[0862] The server integrates the analysis results of the generative AI with data from the emotion engine to improve the accuracy of intent estimation. Based on this integrated data, the customer's intent becomes clear and the order details can be accurately understood.

[0863] 5. Speech conversion and output of estimated intention

[0864] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is sent to the delivery person's audio device (Audio Glasses). The Audio Glasses play back the received voice data and accurately convey the intended intention of the support person to the delivery person.

[0865] Specific examples

[0866] For example, consider the following scenario:

[0867] 1. A customer puts on smart glasses and says, "Please avoid spicy food."

[0868] 2. The smart glasses' sensors collect movement data, the microphone records voice data, and the emotion engine analyzes smiles (positive).

[0869] 3. The data is sent to the server, where it undergoes preprocessing and is then analyzed by the generation AI.

[0870] 4. The generative AI understands that the customer wants to avoid spicy food and infers their intention.

[0871] 5. The server converts this intention into voice data and transmits it to the delivery person's audio glasses, saying, "Please avoid spicy food."

[0872] Prompt Sentence Examples

[0873] The user says, "I don't like spicy food." The face is smiling (positive). Please infer the user's intention from this statement.

[0874] This system makes it possible to accurately understand the complex intentions and emotions of customers, significantly improving customer satisfaction in delivery services.

[0875] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0876] Step 1:

[0877] The server collects physical movement and voice data from a wearable device (smart glasses) that captures the behavior of the person being supported.

[0878] Input: Real-time movement data (acceleration, gyro) and audio data from the supported individual.

[0879] Output: Recording files of motion data (e.g. accelerometer and gyroscope data) and audio data.

[0880] Specific operation: The smart glasses detect the customer's movements and speech and record them as digital data.

[0881] Step 2:

[0882] The server pre-processes the collected data.

[0883] Input: Motion data and audio data (recorded files).

[0884] Output: Preprocessed action data (action classification results) and audio-text data.

[0885] Specific operation: Recognizes actions such as walking and standing from data from the accelerometer and gyroscope, and converts the voice data into text using voice recognition software. Example: Converts the voice data "I don't like spicy food" into text "I don't like spicy food."

[0886] Step 3:

[0887] The server uses an emotion engine to analyze the emotions of the person being supported from the collected data.

[0888] Input: Audio and text data.

[0889] Output: Emotion data (e.g., emotional state, such as positive or negative).

[0890] Specific operation: The emotion engine analyzes the tone of voice and facial expression data (facial recognition by camera) to identify the emotional state. Example: Because the person is smiling, it is judged to be "positive."

[0891] Step 4:

[0892] The server uses generative AI to analyze the preprocessed data and emotional data to infer the intentions of the person being supported.

[0893] Input: Preprocessed text data and sentiment data.

[0894] Output: The inferred intent (e.g., "Avoid spicy food").

[0895] Specific operation: A generative AI model (e.g., GPT-4) analyzes the preprocessed data and derives the intention of the person being supported using the prompt: "The user says, 'I don't like spicy food.' Their face is smiling (positive). Please infer the user's intention from this statement."

[0896] Step 5:

[0897] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate speech data.

[0898] Input: Inferred intent (in text format).

[0899] Output: The generated audio data.

[0900] Specific behavior: Use a speech synthesis engine (e.g., Text-to-Speech engine) to convert the text "Avoid spicy food" into audio data.

[0901] Step 6:

[0902] The server transmits the generated voice data to the delivery person's audio device (audio glasses) and plays the voice data.

[0903] Input: The generated audio data.

[0904] Output: Playback of the audio data sent to the delivery person.

[0905] Specific operation: The audio glasses receive the voice data and play back "Please avoid spicy food," allowing the delivery person to accurately understand the customer's intentions.

[0906] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0907] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0908] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0909] [Fourth embodiment]

[0910] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0911] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0912] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0913] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0914] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0915] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0916] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0917] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0918] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0919] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0920] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0921] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0922] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0923] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to the supporter. This system includes a wearable device, a server, a generative AI, and devices such as audio glasses.

[0924] 1. Wearable devices that capture the behavior of those receiving support

[0925] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0926] 2. Data preprocessing and transmission

[0927] The device sends the collected data to the server, where it is encoded and converted into an appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and voice data recorded by the microphone are sent to the server.

[0928] 3. Analysis of Inferred Intent

[0929] The server preprocesses the received data and formats it into a unified format before inputting it into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generation AI will interpret this as, "It's cold, so I want to go back indoors."

[0930] 4. Speech conversion and output of estimated intention

[0931] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device such as audio glasses worn by the support person.

[0932] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[0933] Specific examples

[0934] For example, the following scenario is possible:

[0935] Step 1: The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's speech and collects movement data.

[0936] Step 2: The wearable device sends voice and motion data to the server.

[0937] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0938] Step 4: The server converts this intent into audio data and sends it to the audio glasses.

[0939] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[0940] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported and enables the supporter to respond appropriately.

[0941] The processing flow will be explained below.

[0942] Step 1:

[0943] The device collects the movements and voice of the person being supported. Specifically, the wearable device uses an accelerometer and gyroscope to capture physical movement data, and a microphone to record voice.

[0944] Step 2:

[0945] The device preprocesses and organizes this data, for example, by recognizing walking movements from accelerometer and gyroscope data and converting recorded audio data into text using voice recognition technology.

[0946] Step 3:

[0947] The device sends the preprocessed data to the server, where it is encoded according to a communication protocol and transmitted using a secure communication method (e.g., HTTPS).

[0948] Step 4:

[0949] The server further preprocesses the received data for analysis, specifically by converting different data formats into a unified and consistent format.

[0950] Step 5:

[0951] The server inputs the formatted data into the generation AI for analysis. The generation AI uses a natural language processing model (e.g., GPT-4) to infer the intention of the person being supported.

[0952] Step 6:

[0953] The server receives the analysis results of the generation AI in text format and inputs the text into a speech synthesis engine, which converts the text into voice data.

[0954] Step 7:

[0955] The server then transmits the generated audio data to the device (audio glasses) worn by the supporter, where the data is again encoded and transmitted via a secure means.

[0956] Step 8:

[0957] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The audio data is played and transmitted to the support person.

[0958] Step 9:

[0959] The user (support person) listens to the audio through the audio glasses and understands the intention of the person being supported. For example, if the person being supported says, "It's cold, so I want to go back indoors," the support person will guide the person indoors.

[0960] Example 1

[0961] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0962] In modern society, it is extremely important to accurately grasp the intentions of those receiving support who have cognitive disabilities and to convey them quickly and accurately to those providing support. However, conventional technology has difficulty accurately grasping the intentions of those receiving support, resulting in delayed responses and often misinterpretations. Furthermore, when analyzing data collected from multiple sensors and devices, inefficiencies in integration and preprocessing make it difficult to respond in real time. There is a need to provide a system that solves these issues and facilitates smooth communication between those receiving support and those providing support.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0964] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for encoding the data collected from the wearable device, converting it into an appropriate data format, and transmitting it to the server, means for the server to preprocess the data received and estimate the intention of the person being supported using a generative AI model, means for generating the estimated intention in text format, inputting it into a speech synthesis engine to convert it into voice data, and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp the intention of the person being supported in real time and convey it to the supporter quickly and accurately.

[0965] A "wearable device" is a device that is worn on the body and is equipped with sensors and communication functions to collect the physical movements and voices of the person being supported.

[0966] An "accelerometer" is a type of sensor installed in wearable devices, and is a device that measures the acceleration of an object.

[0967] A "gyroscope" is a device installed in wearable devices that measures angular velocity to detect changes in the rotation or tilt of an object.

[0968] A "microphone" is a device that converts sound into an electrical signal, and is a component installed in wearable devices to collect the speech of the person being supported.

[0969] "GPS" stands for Global Positioning System, a system for measuring positions on Earth with high precision.

[0970] "Encoding" is the process of converting data into a particular format, with the goal of optimizing it for transmission or storage.

[0971] A "generative AI model" is an artificial intelligence model that analyzes data and generates new information or results based on that information, and is used here to infer the intentions of the person being supported.

[0972] A "speech synthesis engine" is a system that converts text data into speech data, and is a device used to output the estimated intention as speech.

[0973] The present invention is a system that estimates the intentions of a person receiving support who has cognitive disabilities and accurately conveys those intentions to a supporter. This system includes devices such as a wearable device, a server, a generative AI, and an audio output device.

[0974] First, a wearable device is used to capture the behavior of the person being supported. A specific example is a smartwatch. This device is equipped with an accelerometer, gyroscope, microphone, and GPS, and collects the physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") of the person being supported in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[0975] Next, when the device sends the collected data to the server, the data is encoded and converted into the appropriate data format. For example, if the person receiving support stands up while saying "I'm cold," data from the accelerometer and audio data recorded by the microphone are sent to the server. The audio data is encoded into MP3 format, and the sensor data is converted into JSON format.

[0976] The server preprocesses the received data. This preprocessing includes formatting data from different sensors into a unified format. The formatted data is then input into a generative AI model (e.g., GPT-4). The generative AI model analyzes the data and infers the intention of the person receiving support. For example, if the person receiving support stands up while saying, "It's cold," the generative AI will interpret this as, "It's cold, so I want to go back indoors."

[0977] The server generates the estimated intention in text format. This text is input into a speech synthesis engine and converted into voice data. The generated voice data is sent to an audio output device (such as audio glasses) worn by the support person. The device (audio output device) plays the received voice data and conveys it to the support person. This ensures that the intention of the person being supported is accurately conveyed. For example, the audio glasses could say, "It's cold, so I'd like to go back indoors," allowing the support person to take appropriate action.

[0978] To illustrate, consider the following scenario:

[0979] 1. The person being supported begins walking outdoors, saying, "It's cold." The wearable device records the person's words and collects movement data.

[0980] 2. The wearable device sends voice and movement data to the server.

[0981] 3. The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[0982] 4. The server converts this intent into audio data and sends it to the audio glasses.

[0983] 5. The audio glasses will tell the support person, "It's cold, I want to go back indoors."

[0984] Below is an example of a prompt sentence to input to the generative AI model.

[0985] Example 1: "The person you are supporting walks away saying, 'It's cold.' Guess what he wants to do."

[0986] Example 2: "The user stands up and says 'I'm thirsty.' Please infer the intention of this action."

[0987] As described above, the present invention makes it possible to accurately grasp the intentions of the person being supported in real time and to convey them to the person providing support quickly and accurately.

[0988] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0989] Step 1:

[0990] The terminal uses a wearable device to capture the behavior of the person being supported. Specifically, it uses an accelerometer, gyroscope, microphone, and GPS to collect the physical movements and voice of the person being supported in real time. The input is the movements and speech of the person being supported, and the output is to store these as digital data.

[0991] As a specific example of operation, the accelerometer and gyroscope detect the movement of the person being supported standing up, and at the same time, the microphone collects the voice recording of the person saying "It's cold."

[0992] Step 2:

[0993] The device encodes the collected data and converts it into the appropriate data format. For example, audio data is encoded into MP3 format, and sensor data is converted into JSON format. The converted data is then sent to the server via a communication module. The input is the raw data collected by the wearable device, and the output is the encoded data.

[0994] Specific examples of operations include compressing audio data and converting it to MP3 format, and converting acceleration and gyroscope data to JSON format.

[0995] Step 3:

[0996] The server preprocesses the received data. This preprocessing involves shaping the heterogeneous data into a unified format. Data from different sensors is integrated into a single format. The input is encoded data, and the output is data in a unified format that can be fed into a generative AI model.

[0997] A specific example of the operation includes converting voice data into text using a voice recognition engine and combining it with sensor data.

[0998] Step 4:

[0999] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes and infers the intention of the person receiving support. The input is preprocessed data in a unified format, and the output is text data of the inferred intention of the person receiving support.

[1000] As a specific example of operation, the generative AI model is input with a prompt statement such as, "The person being supported started walking away saying, 'It's cold.' Please guess what he wants to do.", and an intent such as, "It's cold, so I want to go back indoors" is generated.

[1001] Step 5:

[1002] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to convert it into voice data. The input is the text data of the estimated intention, and the output is voice data. The generated voice data is sent to the audio output device used by the supporter.

[1003] As a specific example of operation, text such as "It's cold, so I want to go back indoors" is input into a speech synthesis engine to generate voice data.

[1004] Step 6:

[1005] The terminal (audio output device) plays the received audio data and transmits it to the device that supports it. The input is the audio data sent from the server, and the output is the played audio.

[1006] As a specific example of how it works, the audio glasses will play a voice message saying, "It's cold, so I want to go back indoors," and the support person will hear this and take appropriate action.

[1007] This enables the system to accurately grasp the intentions of the person being supported in real time and convey them to the person providing support quickly and accurately.

[1008] (Application example 1)

[1009] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1010] The present invention solves the problem that customers with cognitive disabilities have difficulty accurately communicating their intentions and requests to store staff. In particular, when customers are unable to express their intentions well, it becomes difficult for store staff to respond appropriately, which can lead to a decline in the quality of the customer experience. In such situations, there is a need for a means to accurately estimate a customer's intentions and requests and quickly communicate them to store staff.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1012] In this invention, the server includes: means for collecting the physical movements and voice of the support recipient using a wearable device that captures the support recipient's behavior; means for transmitting the data collected from the wearable device to the server and preprocessing the data; means for analyzing the preprocessed data using a generative AI model and estimating the support recipient's intention; means for converting the estimated intention into voice data and transmitting it to a device used by a store employee; means for playing the transmitted voice data and notifying the employee; and means for the employee to prompt an appropriate response based on the notification. This enables customers with cognitive disabilities to accurately communicate their intentions in the store to employees, allowing the employees to respond quickly and appropriately.

[1013] "Support recipients" refer to people who have cognitive problems and require external support.

[1014] A "wearable device" is an electronic device that is worn on the body and is equipped with sensors and communication functions.

[1015] "Physical actions" refer to bodily movements such as walking, standing up, and raising your hands.

[1016] "Voice" refers to the words and sounds made by the person being supported.

[1017] "Server" refers to a computer system for processing and storing data over a network.

[1018] "Preprocessing" refers to the process of converting collected raw data into a format that is easy to analyze.

[1019] "Generative AI model" refers to the artificial intelligence algorithm used to analyze collected data and infer the intent of the person being supported.

[1020] "Brick and mortar store" refers to a commercial establishment that offers goods or services in a physical location.

[1021] "Employees" refers to staff who work in physical stores and deal with customers.

[1022] "Voice data" refers to data that has been converted into voice format from intent analyzed by a generative AI model.

[1023] "Notification" refers to information that communicates the support recipient's intentions to the employee.

[1024] The present invention relates to a system that enables customers with cognitive disabilities to accurately communicate their intentions and requests to store staff in a physical store. The system includes a wearable device (e.g., a smartwatch), a server, a generative AI model, and a notification device (e.g., a smartphone).

[1025] Data collection using wearable devices

[1026] The terminal uses a wearable device to collect the physical movements (e.g., walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported. Specifically, the wearable device is equipped with an accelerometer, gyroscope, microphone, and GPS function, and collects data in real time using these sensors. The collected data is either stored in the wearable device or sent directly to a server.

[1027] Data preprocessing and transmission

[1028] The server receives the data sent by the wearable device and preprocesses it, converting voice data to text and analyzing sensor data to understand the physical movements of the person being supported, then formats the data appropriately and prepares it for input into the generative AI model.

[1029] Intent Analysis

[1030] The server analyzes the preprocessed data using a generative AI model (e.g., GPT-4). The generative AI model infers the intention of the person being supported based on the voice text and movement data. For example, if the person being supported starts walking while saying, "I want to go to the toilet," the generative AI model will interpret this as the intention, "I want to go to the toilet."

[1031] Notice of presumed intent

[1032] The server generates the estimated intent in text format and inputs that text into a speech synthesis engine to generate voice data. This voice data is then sent to a device used by an employee in the physical store. The device then notifies the employee that it has received the notification and prompts them to take an appropriate response. This allows the employee to accurately understand the intent of the person being supported and respond quickly.

[1033] Specific examples

[1034] Consider the following scenario: If a person being supported walks through a physical store saying, "I want to go to the restroom," the wearable device collects this statement and movement data. The collected data is sent to a server, where it is preprocessed and a generative AI model analyzes the person's intent. As a result, the intention of "I want to go to the restroom" is estimated, and this intent is notified via voice to the employee's smartphone, allowing the employee to immediately guide the person being supported to the restroom.

[1035] Prompt Sentence Examples

[1036] An example of a prompt for the generative AI model is, "Analyze the behavior and voice data of the customer who says that the employee wants to go to the restroom, and infer their intent." This allows the generative AI model to infer the appropriate intent based on the voice and movement data of the person being supported.

[1037] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1038] Step 1:

[1039] The terminal (wearable device) collects the physical movements (walking, standing, etc.) and voice (e.g., "I want to go to the toilet") of the person being supported in real time. The wearable device's accelerometer, gyroscope, microphone, and GPS are used to acquire movement and voice data, allowing the current actions and speech of the person being supported to be collected.

[1040] Step 2:

[1041] The device temporarily stores the collected motion and voice data or transmits it to a server in real time. The data includes motion data from the accelerometer and gyroscope, and voice data from the microphone. This allows the raw data collected by the wearable device to be transmitted to the server.

[1042] Step 3:

[1043] The server receives the data sent from the device and preprocesses it. In the preprocessing step, the voice data is converted into text (e.g., "I want to go to the toilet"), and the motion data is simultaneously analyzed to identify physical movements (e.g., while walking). Other processes include standardizing the data format and removing noise. This results in preprocessed data that is ready to be input into the generative AI model.

[1044] Step 4:

[1045] The server inputs the preprocessed data into a generative AI model. The generative AI model (e.g., GPT-4) analyzes the voice text and behavioral data to infer the intention of the person being supported. A prompt such as "Analyze the behavior and voice data of a customer who says that an employee wants to go to the restroom, and infer their intention" is used. This allows the generative AI model to infer the intention of "I want to go to the restroom" from the input data.

[1046] Step 5:

[1047] The server generates the intent estimated by the generative AI model in text format. This text is then input into a speech synthesis engine to generate voice data. The speech synthesis engine converts the text into natural-sounding speech and creates an audio file. This outputs the estimated intent as voice data.

[1048] Step 6:

[1049] The server then sends the generated voice data to a device (e.g., a smartphone) used by an employee at the physical store. The employee's device receives and plays the voice data, notifying the employee of the intention of the person receiving support. This allows the employee to accurately understand the intention of the person receiving support and respond quickly.

[1050] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1051] This invention is a system that estimates the intentions of support recipients with cognitive problems, improves the accuracy of the estimation by combining it with an emotion engine, and accurately conveys those intentions to the supporter. This system includes devices such as a wearable device, a server, a generative AI, an emotion engine, and audio glasses.

[1052] 1. Wearable devices that capture the behavior of those receiving support

[1053] A wearable device (e.g., a smartwatch) is used to capture the behavior of the person being supported. This device is equipped with an accelerometer, gyroscope, microphone, and GPS functionality, and collects the person's physical movements (e.g., standing up, walking, raising a hand) and voice (e.g., utterances such as "It's cold") in real time. The collected data is either stored temporarily within the device or sent directly to a server.

[1054] 2. Data preprocessing and transmission

[1055] The device preprocesses and organizes this data. For example, it recognizes walking movements from data from the accelerometer and gyroscope, and converts recorded voice data into text using voice recognition technology. In addition, an emotion engine identifies the emotions of the person being supported from facial expressions, tone of voice, and body movements. The identified emotion data is also sent to the server.

[1056] 3. Data analysis and intent estimation

[1057] The server further preprocesses the received data for analysis. Specifically, it converts data from different formats into a unified format to ensure consistency. The converted data is then input into the generation AI. The generation AI (e.g., GPT-4) analyzes this data and infers the intention of the person being supported.

[1058] 4. Emotional Data Integration

[1059] The server combines the analysis results of the generation AI with the emotional data from the emotion engine. This combined data is used to improve the accuracy of estimating the intention of the person receiving support. For example, if the person receiving support says "It's cold" and stands up, and the emotional data indicates feelings such as "anxiety" or "discomfort," the generation AI will more accurately estimate the intention as "I want to go back indoors."

[1060] 5. Speech conversion and output of estimated intention

[1061] The server generates the estimated intention in text format, and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is then sent to a device (audio glasses) worn by the support person.

[1062] The device (Audio Glasses) plays back the received voice data and transmits it to the support person, accurately conveying the intentions of the person being supported. For example, the Audio Glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[1063] Specific examples

[1064] For example, the following scenario is possible:

[1065] Step 1: The person being supported begins walking outdoors, saying "It's cold." The wearable device records the person's speech and collects movement data. At the same time, the emotion engine identifies the emotion "anxiety" from the person's tone of voice and facial expression.

[1066] Step 2: The wearable device sends voice data, movement data, and emotion data to the server.

[1067] Step 3: The server preprocesses the data and inputs it into the generation AI, which then infers that the user's intention is "It's cold, so I want to go back indoors."

[1068] Step 4: The server integrates the analysis results of the generative AI with the emotional data to improve the accuracy of the intention estimation. The server converts this intention into voice data and sends it to the audio glasses.

[1069] Step 5: The Audio Glasses will tell the support person via voice, "It's cold, I want to go back indoors."

[1070] In this way, the present invention provides a system that accurately grasps the intentions of the person being supported, improves the accuracy by combining it with an emotion engine, and enables the supporter to respond appropriately.

[1071] The processing flow will be explained below.

[1072] Step 1:

[1073] The device collects the actions and voice of the person receiving support. A wearable device (e.g., a smartwatch) uses an accelerometer, gyroscope, microphone, and GPS to record the physical movements and voice of the person receiving support in real time. For example, it captures the action of the person saying, "It's cold," and starting to walk away.

[1074] Step 2:

[1075] The device preprocesses the collected motion data and recognizes specific movements such as walking and standing up. At the same time, it converts the recorded voice data into text using voice recognition technology. For example, the voice data can be converted into text data such as "It's cold."

[1076] Step 3:

[1077] The device sends the collected and pre-processed data to the server, where the motion data, voice text, and location data are encoded using a secure communication protocol (e.g., HTTPS).

[1078] Step 4:

[1079] The server further preprocesses the data it receives and formats it into a unified format. Data in different formats is processed into a form suitable for analysis. For example, the action data for "standing up" and the voice text "It's cold" are unified.

[1080] Step 5:

[1081] The server inputs the formatted data into the generation AI for analysis. The generation AI (e.g., GPT-4) analyzes the movement data and voice data and infers the intention of the person being supported. For example, the generation AI infers the intention, "It's cold, so I want to go back indoors."

[1082] Step 6:

[1083] The device recognizes the emotions of the person receiving support. Using the camera and microphone installed in the wearable device, emotions are identified from the facial expressions and tone of voice of the person receiving support. For example, emotional data such as "anxiety" or "discomfort" is generated.

[1084] Step 7:

[1085] The server integrates the emotion data received from the emotion engine with the analysis results of the generative AI. This improves the accuracy of estimating the intention of the person being supported. For example, the intention of "I want to go back indoors because it's cold" can be further confirmed by the emotion data of "anxiety."

[1086] Step 8:

[1087] The server converts the estimated intent into voice data. The generated text is then input into a speech synthesis engine to generate voice data. For example, the generated voice data might say, "It's cold, so I want to go back indoors."

[1088] Step 9:

[1089] The server sends the generated audio data to the device (audio glasses) worn by the supporter. The audio data is encoded and transmitted using a secure communication protocol.

[1090] Step 10:

[1091] The device (Audio Glasses) decodes the received audio data and converts it into a playable format. The built-in speaker of the Audio Glasses plays the audio data and conveys it to the support person. For example, the voice may say, "It's cold, so I want to go back indoors."

[1092] Step 11:

[1093] The user (support person) listens to the audio through the audio glasses and understands the intentions of the person being supported. For example, if the user understands that the person wants to go back indoors, they can take action to guide the person indoors.

[1094] Example 2

[1095] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1096] In situations where support recipients with cognitive disabilities have difficulty accurately communicating their intentions, there is a need to accurately estimate their intentions and communicate them quickly and accurately to supporters. However, conventional systems lack the accuracy of data collection and intention estimation, and in particular lack a mechanism to improve estimation accuracy by combining emotional data. As a result, the intentions of support recipients are often misunderstood, making it difficult to provide appropriate support.

[1097] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for further preprocessing the transmitted data and shaping data of different formats into a unified format, means for analyzing the preprocessed data using a generative AI model and estimating the intention of the person being supported, and means for integrating emotion data acquired from an emotion engine with the estimated intention to improve the accuracy of intention estimation. In this way, by integrating and analyzing not only the behavior and voice of the person being supported but also the emotion data, the accuracy of intention estimation improves, making it possible to quickly and accurately convey the intention to the person providing support.

[1098] - "Support recipient" refers to a person who has cognitive problems and requires the support of others.

[1099] A "wearable device" is an electronic device that can be worn by the user and is equipped with sensors and microphones to collect data.

[1100] "Physical movement" refers to data showing changes in the body movements and posture of the person being supported.

[1101] "Audio" refers to data that indicates the statements and tone of voice made by the person being supported.

[1102] "Preprocessing" refers to the process of formatting and organizing collected data into a form that is easy to analyze.

[1103] A "server" refers to a computer system that stores and analyzes data and communicates with external devices.

[1104] "Generative AI model" refers to an artificial intelligence model that analyzes collected data and infers the intentions of the person being supported.

[1105] An "emotion engine" refers to a system that identifies emotions from a person's facial expressions and tone of voice and outputs them as data.

[1106] "Intention" refers to what the person receiving support wants or intends to do.

[1107] "Unified format" refers to the formatting of data of different formats into one consistent format.

[1108] "Audio data" refers to data that has been converted from text information into a digital format that can be played as sound.

[1109] "Audio glasses" are devices that provide information through sight and hearing, especially those worn by a support person.

[1110] "Intention estimation accuracy" refers to an indicator of how accurately the intention of the person being supported can be estimated.

[1111] This invention is a system that estimates the intentions of the person being supported with high accuracy and quickly and accurately communicates those intentions to the supporter. The system includes devices such as a wearable device, a server, a generative AI model, an emotion engine, and audio glasses.

[1112] First, the terminal (wearable device) collects the behavioral and voice data of the person being supported. The wearable device is equipped with an accelerometer, gyroscope, microphone, and location information acquisition function, which allows it to record the person's actions (e.g., standing up, walking, raising their hand, etc.) and voice (e.g., utterances such as "It's cold") in real time. This collected data is either temporarily stored within the device or sent directly to a server.

[1113] The device then preprocesses and organizes this data. This involves identifying walking and other movements from accelerometer and gyroscope data, converting recorded voice data into text using speech recognition technology, and using an emotion engine to identify emotions from voice tone and facial expressions, generating emotion data such as "anxiety." This organized data is then sent to a server.

[1114] The server further preprocesses the transmitted data. Specifically, it formats the data in different formats into a unified format to ensure consistency. This formatted data is then input into a generative AI model (e.g., GPT-4). The server analyzes the data using the generative AI model and infers the intention of the person receiving support. For example, by inputting a prompt such as, "The person receiving support started walking while saying 'I'm cold,' and their emotion was identified as 'anxious.' Please infer the person's intention," the generative AI model performs the analysis.

[1115] Furthermore, the server integrates the analysis results of the generative AI model with the emotional data from the emotion engine. This integration improves the accuracy of intent estimation. For example, if a subject says "It's cold" while standing up and the emotional data indicates "anxiety," the generative AI model will more accurately estimate the intent as "I want to go back indoors."

[1116] Finally, the server generates the estimated intention in text format and inputs that text into a speech synthesis engine to generate voice data. The generated voice data is sent to the device (e.g., audio glasses) used by the support person. The device (audio glasses) plays back the received voice data and conveys it to the support person, accurately conveying the support person's intention. For example, the audio glasses could say, "It's cold, so I want to go back indoors," allowing the support person to take appropriate action.

[1117] In this way, this invention improves the accuracy of estimating intentions by integrating and analyzing not only the behavior and voice of the person being supported, but also emotional data, making it possible to quickly and accurately convey intentions to the person providing support.

[1118] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1119] Step 1:

[1120] The terminal (wearable device) collects behavioral and voice data from the person receiving support. The wearable device's accelerometer, gyroscope, microphone, and location information acquisition function are used to record the person receiving support's movements (e.g., standing up, walking) and voice (e.g., saying "it's cold") in real time. The input is the person receiving support's physical movements and voice, and the output is the collected raw data. As a specific example, data is collected when the person receiving support starts walking while saying "it's cold."

[1121] Step 2:

[1122] The data collected by the device is preprocessed and organized. First, data from the accelerometer and gyroscope is analyzed to identify walking and other movements. Next, voice data is converted into text using speech recognition technology. An emotion engine then identifies emotions from voice tone and facial expressions, generating emotion data such as "anxiety." The input is the collected raw data, and the output is organized data (movement data, textual voice data, and emotion data). For example, the statement "It's cold" is converted into text, and data identified as "walking movements" is generated.

[1123] Step 3:

[1124] The device sends the preprocessed data to the server. The input is the organized data, and the output is the data sent to the server. For example, the textual utterance "It's cold" and the emotional data "I feel anxious" are sent to the server.

[1125] Step 4:

[1126] The server further preprocesses the transmitted data, shaping data of different formats into a unified format and making it consistent. The input is the organized data transmitted from the device, and the output is unified format data. For example, text data, action data, and emotion data are converted into a unified format.

[1127] Step 5:

[1128] The server uses a generative AI model to analyze the preprocessed data and infer the intention of the person being supported. Data is input into the generative AI model using a prompt. The input is unified format data and a prompt, and the output is the inferred intention. As a specific example, the prompt is "The person being supported starts walking while saying 'It's cold', and the emotion is identified as 'anxiety'. Please infer the person's intention," and the inferred intention is "It's cold, so I want to go back indoors."

[1129] Step 6:

[1130] The server integrates the analysis results of the generative AI model with the emotion data obtained from the emotion engine. The input is the estimated intention and emotion data, and the output is the integrated intention data. For example, by integrating the emotion data of "anxiety" with the intention of "wanting to go back indoors," the accuracy of the intent estimation improves.

[1131] Step 7:

[1132] The server generates the estimated intention in text format and inputs it into a speech synthesis engine to generate speech data. The input is the intention data in text format, and the output is speech data. For example, the text "It's cold, so I want to go back indoors" is generated as speech data.

[1133] Step 8:

[1134] The server sends the generated voice data to the audio glasses. The input is the generated voice data, and the output is the voice data sent to the audio glasses. For example, voice data saying "It's cold, so I want to go back indoors" is sent to the audio glasses.

[1135] Step 9:

[1136] The device (audio glasses) plays the received voice data and conveys it to the support person. The input is the voice data sent to the audio glasses, and the output is the played voice data. Specifically, the audio glasses play the voice "It's cold, I want to go back indoors," allowing the support person to take appropriate action.

[1137] (Application example 2)

[1138] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1139] Food delivery services are required to accurately understand customer intent and emotions to reduce misunderstandings and troubles about orders and improve customer satisfaction. However, conventional systems have difficulty accurately analyzing customer intent and emotions, leading to a high likelihood of misunderstandings. Furthermore, there is a lack of a mechanism to integrate emotion analysis and intent estimation to improve accuracy.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1141] In this invention, the server includes means for collecting the physical movements and voice of the person being supported using a wearable device that captures the behavior of the person being supported, means for transmitting the data collected from the wearable device to the server and preprocessing the data, means for analyzing the preprocessed data using a generative AI and estimating the intention of the person being supported, means for analyzing the emotions of the person being supported by combining an emotion engine and integrating it with the generative AI to improve the accuracy of the intention estimation, means for converting the estimated intention into voice data and transmitting it to a device used by the supporter, and means for playing back the transmitted voice data. This makes it possible to accurately grasp customer intentions and emotions even in food delivery services, reducing misunderstandings about order details and problems.

[1142] A "supported person" is a user or customer of a food delivery service who provides specific actions or voice data.

[1143] A "wearable device" is an electronic device worn by an individual on the body, which includes an accelerometer, gyroscope, microphone, and GPS, and has the ability to collect physical movements and sounds.

[1144] The "server" is a computer system that preprocesses, analyzes, and infers intent from data. It utilizes generative AI and an emotion engine to accurately infer the intent of the person being supported, and processes and transmits the necessary data.

[1145] "Generative AI" refers to technology that uses natural language processing models to analyze data and infer the intentions of the person being supported, and specifically includes generative artificial intelligence models such as GPT-4.

[1146] The "emotion engine" refers to technology that analyzes the tone of voice, facial expressions, and movements of the person being supported to identify their emotional state, and is integrated with generative AI to improve the accuracy of intent estimation.

[1147] "Intention estimation accuracy" refers to the high level of ability to accurately analyze and judge the intentions of the person being supported, and by integrating with the emotion engine, intentions can be identified with greater accuracy.

[1148] "Voice data" refers to voice information generated by a voice synthesis engine based on analyzed text information, and is used to convey the intentions of the person being supported to the person providing support.

[1149] "Supporter" refers to the delivery person who delivers food to customers in a food delivery service, who receives and plays back the transmitted voice data and takes appropriate action based on the customer's intentions.

[1150] This invention is a system for accurately understanding customer intentions and emotions in food delivery services, reducing misunderstandings and troubles about orders and improving customer satisfaction. This system includes wearable devices (smart glasses, smart watches, etc.), a server, generative AI, an emotion engine, and an audio device (audio glasses).

[1151] 1. Wearable devices that capture the behavior of those receiving support

[1152] First, the customer wears smart glasses. The glasses are equipped with an accelerometer, gyroscope, microphone, and GPS to collect the customer's physical movements (e.g., waving, walking, stopping, etc.) and voice (order details, preferences, and requests) in real time. The collected data is either temporarily stored on the device or sent directly to a server.

[1153] 2. Data preprocessing and transmission

[1154] After receiving this data, the server performs initial preprocessing. Specifically, it recognizes movements from data from the accelerometer and gyroscope, converts recorded voice data into text using voice recognition technology, and uses an emotion engine to identify emotions from the voice tone and facial expressions of the person receiving support. The identified emotion data is also integrated for analysis.

[1155] 3. Data analysis and intent estimation

[1156] The server inputs the received data into a generation AI (e.g., GPT-4) for analysis. The generation AI uses text data, behavioral data, and emotional data to infer the intention of the person being supported. For example, when a customer says, "Please avoid spicy food," and the emotion engine provides information that the face is smiling, the generation AI infers that "the customer does not want spicy food."

[1157] 4. Emotional Data Integration

[1158] The server integrates the analysis results of the generative AI with data from the emotion engine to improve the accuracy of intent estimation. Based on this integrated data, the customer's intent becomes clear and the order details can be accurately understood.

[1159] 5. Speech conversion and output of estimated intention

[1160] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate voice data. The generated voice data is sent to the delivery person's audio device (Audio Glasses). The Audio Glasses play back the received voice data and accurately convey the intended intention of the support person to the delivery person.

[1161] Specific examples

[1162] For example, consider the following scenario:

[1163] 1. A customer puts on smart glasses and says, "Please avoid spicy food."

[1164] 2. The smart glasses' sensors collect movement data, the microphone records voice data, and the emotion engine analyzes smiles (positive).

[1165] 3. The data is sent to the server, where it undergoes preprocessing and is then analyzed by the generation AI.

[1166] 4. The generative AI understands that the customer wants to avoid spicy food and infers their intention.

[1167] 5. The server converts this intention into voice data and transmits it to the delivery person's audio glasses, saying, "Please avoid spicy food."

[1168] Prompt Sentence Examples

[1169] The user says, "I don't like spicy food." The face is smiling (positive). Please infer the user's intention from this statement.

[1170] This system makes it possible to accurately understand the complex intentions and emotions of customers, significantly improving customer satisfaction in delivery services.

[1171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1172] Step 1:

[1173] The server collects physical movement and voice data from a wearable device (smart glasses) that captures the behavior of the person being supported.

[1174] Input: Real-time movement data (acceleration, gyro) and audio data from the supported individual.

[1175] Output: Recording files of motion data (e.g. accelerometer and gyroscope data) and audio data.

[1176] Specific operation: The smart glasses detect the customer's movements and speech and record them as digital data.

[1177] Step 2:

[1178] The server pre-processes the collected data.

[1179] Input: Motion data and audio data (recorded files).

[1180] Output: Preprocessed action data (action classification results) and audio-text data.

[1181] Specific operation: Recognizes actions such as walking and standing from data from the accelerometer and gyroscope, and converts the voice data into text using voice recognition software. Example: Converts the voice data "I don't like spicy food" into text "I don't like spicy food."

[1182] Step 3:

[1183] The server uses an emotion engine to analyze the emotions of the person being supported from the collected data.

[1184] Input: Audio and text data.

[1185] Output: Emotion data (e.g., emotional state, such as positive or negative).

[1186] Specific operation: The emotion engine analyzes the tone of voice and facial expression data (facial recognition by camera) to identify the emotional state. Example: Because the person is smiling, it is judged to be "positive."

[1187] Step 4:

[1188] The server uses generative AI to analyze the preprocessed data and emotional data to infer the intentions of the person being supported.

[1189] Input: Preprocessed text data and sentiment data.

[1190] Output: The inferred intent (e.g., "Avoid spicy food").

[1191] Specific operation: A generative AI model (e.g., GPT-4) analyzes the preprocessed data and derives the intention of the person being supported using the prompt: "The user says, 'I don't like spicy food.' Their face is smiling (positive). Please infer the user's intention from this statement."

[1192] Step 5:

[1193] The server generates the estimated intention in text format and inputs the text into a speech synthesis engine to generate speech data.

[1194] Input: Inferred intent (in text format).

[1195] Output: The generated audio data.

[1196] Specific behavior: Use a speech synthesis engine (e.g., Text-to-Speech engine) to convert the text "Avoid spicy food" into audio data.

[1197] Step 6:

[1198] The server transmits the generated voice data to the delivery person's audio device (audio glasses) and plays the voice data.

[1199] Input: The generated audio data.

[1200] Output: Playback of the audio data sent to the delivery person.

[1201] Specific operation: The audio glasses receive the voice data and play back "Please avoid spicy food," allowing the delivery person to accurately understand the customer's intentions.

[1202] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1203] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1204] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1205] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1206] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1207] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1208] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1209] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1210] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1211] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1212] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1213] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1214] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1215] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1216] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1217] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1218] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1219] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1220] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1221] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1222] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1223] The following is further disclosed regarding the above embodiment.

[1224] (Claim 1)

[1225] A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported;

[1226] means for transmitting the data collected from the wearable device to a server and pre-processing the data;

[1227] A means for analyzing preprocessed data using generative AI and inferring the intentions of the person being supported;

[1228] means for converting the estimated intention into voice data and transmitting the voice data to a device used by a support person;

[1229] a system including means for playing back said transmitted audio data.

[1230] (Claim 2)

[1231] 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a GPS.

[1232] (Claim 3)

[1233] The system of claim 1, wherein the generative AI uses a natural language processing model.

[1234] "Example 1"

[1235] (Claim 1)

[1236] A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported;

[1237] means for encoding the data collected from the wearable device, converting it into a suitable data format, and transmitting it to a server;

[1238] A means for preprocessing the data received by the server and estimating the intention of the person being supported using a generative AI model;

[1239] a means for generating the estimated intention in a text format, inputting the text into a speech synthesis engine to convert the text into speech data, and transmitting the speech data to a device used by a support person;

[1240] a system including means for playing back said transmitted audio data.

[1241] (Claim 2)

[1242] 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a GPS.

[1243] (Claim 3)

[1244] The system of claim 1, wherein the generative AI model uses a natural language processing model.

[1245] "Application Example 1"

[1246] (Claim 1)

[1247] A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported;

[1248] means for transmitting the data collected from the wearable device to a server and pre-processing the data;

[1249] A means for analyzing the preprocessed data using a generative AI model and inferring the intention of the person being supported;

[1250] means for converting the estimated intention into voice data and transmitting the voice data to a device used by an employee of the physical store;

[1251] means for playing back the transmitted voice data and notifying an employee;

[1252] A means to encourage employees to take appropriate action based on the notification

[1253] A system including:

[1254] (Claim 2)

[1255] 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a GPS.

[1256] (Claim 3)

[1257] The system of claim 1, wherein the generative AI model uses a natural language processing model.

[1258] "Example 2: Combining Emotion Engines"

[1259] (Claim 1)

[1260] A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported;

[1261] means for pre-processing and organizing data collected from the wearable device;

[1262] means for transmitting the preprocessed data to a server;

[1263] A means for the server to further preprocess the transmitted data and convert data of different formats into a unified format;

[1264] A means for analyzing the preprocessed data using a generative AI model and inferring the intention of the person being supported;

[1265] means for integrating emotion data acquired from an emotion engine with the estimated intention to improve the accuracy of intention estimation;

[1266] means for converting the estimated intention into voice data and transmitting the voice data to a device used by a support person;

[1267] means for reproducing the transmitted audio data;

[1268] A system including:

[1269] (Claim 2)

[1270] 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a location acquisition function.

[1271] (Claim 3)

[1272] The system of claim 1, wherein the generative AI model uses a natural language processing model.

[1273] "Application example 2 when combining emotion engines"

[1274] (Claim 1)

[1275] A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported;

[1276] means for transmitting the data collected from the wearable device to a server and pre-processing the data;

[1277] A means for analyzing preprocessed data using generative AI and inferring the intentions of the person being supported;

[1278] A method to improve the accuracy of intent estimation by combining an emotion engine to analyze the emotions of the person being supported and integrating it with generative AI.

[1279] means for converting the estimated intention into voice data and transmitting the voice data to a device used by a support person;

[1280] a system including means for playing back said transmitted audio data.

[1281] (Claim 2)

[1282] 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a GPS.

[1283] (Claim 3)

[1284] The system of claim 1, wherein the generative AI uses a natural language processing model. [Explanation of symbols]

[1285] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting physical movements and voices of the person being supported using a wearable device that captures the movements of the person being supported; means for transmitting the data collected from the wearable device to a server and pre-processing the data; A means for analyzing preprocessed data using generative AI and inferring the intentions of the person being supported; means for converting the estimated intention into voice data and transmitting the voice data to a device used by a support person; a system including means for playing back said transmitted audio data.

2. 10. The system of claim 1, wherein the wearable device includes an accelerometer, a gyroscope, a microphone, and a GPS.

3. The system of claim 1, wherein the generative AI uses a natural language processing model.

Images