system

Abstract

We provide the system. [Solution] A means for performing facial recognition, A means of performing speech recognition, A means of displaying the generated information, A system that provides memory support information by coordinating the above methods.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For dementia patients, the difficulty of daily communication is a factor that significantly reduces the quality of life. Furthermore, due to the ambiguity of memory, smooth communication with family and friends becomes impossible, increasing the mental burden. Also, in the customer service business such as hotels and restaurants, since many customers need to be served, it is difficult to continuously grasp customer information accurately. In such a situation, communication with customers may not proceed smoothly, and the quality of service may decline. This invention aims to solve these problems and support communication for dementia patients and in customer service.

Means for Solving the Problems

[0005] The present invention is a system that includes means for performing face recognition, means for performing speech recognition, and means for displaying generated information. Specifically, a camera is used to capture the face of a person within the field of view, and the image is transmitted to a face recognition server. The face recognition server compares the face image with a database and returns the recognition result to the smart glasses. The smart glasses display the name of the recognized person and capture the conversation using a microphone. The captured audio is transmitted to a speech recognition server, converted to text, and stored in a database. The generating AI server fuses past conversation data with newly acquired conversation data, generates a summary, and transmits and displays it on the smart glasses. This allows the user to sequentially supplement their memory and achieve smooth communication.

[0006] "Face recognition means" refers to a device or system that has the function of analyzing a person's face image captured by a camera and identifying the person by comparing it with records in a database.

[0007] "Speech recognition means" refers to a device or system that has the function of analyzing speech data captured by a microphone and converting it into text.

[0008] "Means for displaying generated information" refers to a device or system that has the function of displaying the results and summarized information generated by facial recognition and speech recognition to the user in real time.

[0009] "Memory support information" refers to information generated by a generating AI that summarizes and analyzes data collected through facial recognition and speech recognition, in order to assist users in supplementing their memories.

[0010] A "camera" is a device used to capture the faces of people within its field of view.

[0011] A "microphone" is an audio input device used to capture ambient sounds.

[0012] A "face recognition server" is a computer device that analyzes captured facial images and identifies individuals by comparing them with facial data in a database.

[0013] A "generating AI server" is a computer device equipped with artificial intelligence capabilities for analyzing and summarizing past and new conversational data.

[0014] A "database" is a data storage device for accumulating past facial images, audio data, and analysis results.

[0015] "Smart glasses" are wearable devices equipped with a display, camera, microphone, etc., that present information generated through facial recognition and voice recognition to the user in real time. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] It shows an emotion map on which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention relates to a system that provides memory support information by combining facial recognition means, voice recognition means, and means for displaying generated information. A specific embodiment of this system, including its program and processing, is described below.

[0038] System Configuration

[0039] This system consists of the following main elements:

[0040] 1. Smart glasses device

[0041] 2. Face recognition server

[0042] 3. Speech recognition server

[0043] 4. Generation AI Server

[0044] 5. Database

[0045] Program processing

[0046] Performing facial recognition

[0047] 1. Terminal

[0048] The smart glasses' camera captures the faces of people within the user's field of view.

[0049] 2. Terminal → Server (Facial Recognition)

[0050] The captured facial image is sent to a facial recognition server.

[0051] 3. Server (facial recognition)

[0052] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[0053] 4. Server (facial recognition) → Terminal

[0054] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[0055] 5. Terminal

[0056] The smart glasses display the name of the recognized person.

[0057] Executing speech recognition

[0058] 1. Terminal

[0059] The smart glasses' microphone continuously captures the audio of the conversation.

[0060] 2. Terminal → Server (Speech Recognition)

[0061] The captured audio data is sent to the speech recognition server.

[0062] 3. Server (Speech Recognition)

[0063] Convert audio data to text.

[0064] 4. Server (Speech Recognition)

[0065] Save text data to a database.

[0066] Execution of Generative AI

[0067] 1. Generation AI Server

[0068] The generation AI server analyzes and merges past conversation data with newly acquired conversation data.

[0069] 2. Generation AI server → Terminal

[0070] The generated summary text is sent to the smart glasses device.

[0071] 3. Terminal

[0072] The smart glasses display a summarized version of the conversation.

[0073] Specific example

[0074] Scenes involving dementia patients and their caregivers

[0075] 1. User (Patient)

[0076] A dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face.

[0077] 2. Terminal → Server (Facial Recognition)

[0078] The captured facial image is sent to the facial recognition server.

[0079] 3. Server (facial recognition)

[0080] The facial recognition server matches the helper's facial image against the database and returns the result, "This person is helper AA."

[0081] 4. Server (facial recognition) → Terminal

[0082] The helper's name and past conversation history are sent to the smart glasses.

[0083] 5. Terminal

[0084] The smart glasses display the message, "This is caregiver AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[0085] 6. Terminal → Server (Speech Recognition)

[0086] The captured audio data is sent to the speech recognition server.

[0087] 7. Server (Speech Recognition)

[0088] Convert the conversation content into text and save it to a database.

[0089] 8. Generation AI Server

[0090] The generating AI server analyzes new conversation data and past conversation data to generate a summary such as, "Last time we talked about BB, but the new topic mentioned this time is CC."

[0091] 9. Generation AI Server → Terminal

[0092] The generated summary is sent to the smart glasses.

[0093] 10. Terminal

[0094] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC," making it easier for the user to understand the flow of the conversation.

[0095] In this way, the system of the present invention can facilitate communication with dementia patients and in service industries dealing with many customers, thereby improving the quality of life for users.

[0096] The following describes the processing flow.

[0097] Step 1:

[0098] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[0099] Step 2:

[0100] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[0101] Step 3:

[0102] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[0103] Step 4:

[0104] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[0105] Step 5:

[0106] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[0107] Step 6:

[0108] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[0109] Step 7:

[0110] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[0111] Step 8:

[0112] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[0113] Step 9:

[0114] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[0115] Step 10:

[0116] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[0117] Step 11:

[0118] The server (speech recognition) saves the transcribed conversation to a database.

[0119] Step 12:

[0120] The generating AI server retrieves newly saved text data and past conversation data from the database and analyzes both.

[0121] Step 13:

[0122] The generation AI server merges past and new conversation data to generate a summary. This summary includes key points from the current conversation and references to past information.

[0123] Step 14:

[0124] The generated summary text is sent from the generation AI server to the terminal.

[0125] Step 15:

[0126] The device displays a summary it has received on the smart glasses' screen. This allows the user to understand, "In our last conversation, we talked about AA. This time, we're talking about CC."

[0127] This series of processes enables smooth communication in service industries dealing with dementia patients and large numbers of customers by utilizing facial recognition, voice recognition, and generative AI.

[0128] (Example 1)

[0129] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0130] Conventional memory support systems have a drawback: they lack features to help users remember people and conversation content. In particular, there was a need for effective support in situations where memory assistance is required, such as for dementia patients and service industries that involve interaction with many customers. Furthermore, existing systems sometimes suffered from low accuracy in facial and voice recognition, resulting in inaccurate memory support information.

[0131] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0132] In this invention, the server includes means for the terminal to capture a person's face, means for the terminal to transmit the captured face image to a face recognition server, means for the face recognition server to analyze the face image and compare it with a database, means for the face recognition server to return identified person information to the terminal, means for the terminal to display the identified person information, means for the terminal to capture the audio of a conversation, means for the terminal to transmit the captured audio data to a speech recognition server, means for the speech recognition server to convert the audio data into text, means for the speech recognition server to store the text data in a database, means for the generating AI server to analyze and merge past conversation data and new conversation data, means for the generating AI server to transmit the generated summary text to the terminal, and means for the terminal to display the summarized conversation content. As a result, the user can instantly identify people and check past conversation content, enabling the conversation to proceed smoothly.

[0133] A "terminal" is a device that a user wears and operates, and in this invention, this specifically refers to smart glasses, etc.

[0134] A "face recognition server" is a server that analyzes facial images transmitted from a terminal, compares them with a database, and identifies the corresponding person.

[0135] A "speech recognition server" is a server that converts speech data sent from a terminal into text.

[0136] A "generative AI server" is a server that analyzes and merges past conversation data with new conversation data to generate summarized text.

[0137] A "database" is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[0138] A "camera" is a device built into a terminal that captures the faces of people within the user's field of view.

[0139] A "microphone" is an audio input device built into a terminal that captures the sound of conversations.

[0140] A "face image" is image data of a person's face captured by the device's camera.

[0141] "Audio data" refers to audio data of a conversation captured by the device's microphone.

[0142] "Text data" refers to character data converted from speech data by a speech recognition server.

[0143] A "summary text" is a summary that integrates past conversation content and new conversation content, generated by a generation AI server.

[0144] This invention relates to a system for providing memory support information to a user. This system combines facial recognition means, voice recognition means, and means for displaying generated information to provide effective memory support to the user.

[0145] System Configuration

[0146] This system consists of the following main elements:

[0147] 1. Terminal

[0148] This includes wearable devices such as smart glasses. These devices are equipped with cameras and microphones that capture the faces of people and the audio of conversations within the user's field of view.

[0149] 2. Face recognition server

[0150] The face recognition server analyzes face images sent from the terminal and matches them against a database to identify the corresponding person. It uses face recognition libraries such as OpenCV and AWS® Rekognition.

[0151] 3. Speech recognition server

[0152] The speech recognition server converts the audio data sent from the terminal into text. It uses speech recognition services such as Google® Cloud Speech-to-Text or IBM Watson® Speech to Text.

[0153] 4. Generation AI Server

[0154] The generative AI server analyzes and merges past conversation data with newly acquired conversation data to generate summarized text. Examples of generative AI models include GPT-3(registered trademark) and BERT.

[0155] 5. Database

[0156] This is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[0157] Program processing

[0158] In this system, when the user wears smart glasses, a series of processes are performed, from facial recognition of a person to speech recognition of conversations, and then the generation and display of summarized text by a generative AI.

[0159] Facial recognition means

[0160] The device's camera captures the faces of people within the user's field of view and sends the facial images to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database to identify the corresponding individuals, and sends the results back to the device. The device then displays the returned person information on its screen.

[0161] Speech recognition means

[0162] The device's microphone captures the conversation audio and sends the audio data to a speech recognition server. The speech recognition server converts the audio data into text and stores the resulting text data in a database. This makes searching and analysis easier later on.

[0163] Generation AI means

[0164] The AI ​​server analyzes and merges previously stored conversation data with newly acquired conversation data to generate a summary text. The generated summary text is sent to the device and displayed on the device's screen. This allows the user to smoothly understand the content of past and current conversations.

[0165] Specific example

[0166] Consider a scenario where a dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face and sends the information to a server. The server sends back the relevant information, and the device displays, "This is caregiver AA. We talked about BB during the last visit." When a conversation begins, the smart glasses' microphone captures the audio and sends the audio data to the server. The server converts the audio to text and stores it in a database. Later, a generative AI server analyzes this data, generates a summary text, and displays it on the device.

[0167] An example of a prompt message would be: "Generate a program that analyzes facial images captured by smart glasses and displays information about the recognized person. Also, include a function to convert spoken conversation into text and summarize it."

[0168] Thus, by providing memory support information, the present invention can facilitate communication in service industries that interact with dementia patients and many customers, thereby improving the quality of life for users.

[0169] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0170] Step 1:

[0171] terminal

[0172] The smart glasses' camera captures the faces of people within the user's field of view in high resolution. The input is a person's face, which is saved as a still image using the camera. The output is the captured face image. This face image data is used in the next step.

[0173] Step 2:

[0174] Terminal → Server (Facial Recognition)

[0175] The terminal captures a face image and sends it to a face recognition server using a secure communication protocol (e.g., HTTPS). The input is the face image data obtained in step 1. The output is the face image data sent to the face recognition server.

[0176] Step 3:

[0177] Server (facial recognition)

[0178] The face recognition server analyzes the received face image and extracts feature points. Specifically, the relevant system (e.g., OpenCV or AWS Rekognition) receives face image data as input and outputs the coordinates of feature points and other identification information. Next, these feature points are compared with existing face data stored in the database to identify the corresponding person. As a result of the comparison, person information is output.

[0179] Step 4:

[0180] Server (facial recognition) → Terminal

[0181] The facial recognition server sends back relevant information to the terminal, such as the name of the identified person and past conversation content. The input is the identified person information data, and the output is the person information data sent to the terminal.

[0182] Step 5:

[0183] terminal

[0184] The device displays the person's information, which has been returned, on the smart glasses' display. The input is person information data received from the server, and the output is display information that the user can visually confirm (e.g., "This person is helper AA. We talked about BB during the last visit").

[0185] Step 6:

[0186] terminal

[0187] The smart glasses' microphone continuously captures the conversation audio. The input is the conversational audio, which is saved as audio data using the microphone. The output is the captured audio data, which is used in the next step.

[0188] Step 7:

[0189] Terminal → Server (Speech Recognition)

[0190] The terminal transmits the captured audio data to the speech recognition server in real time. The input is the audio data obtained in step 6, which is transmitted via a secure communication protocol. The output is the audio data sent to the speech recognition server.

[0191] Step 8:

[0192] Server (Speech Recognition)

[0193] The speech recognition server converts the audio data into text. Specifically, it uses Google Cloud Speech-to-Text or IBM Watson Speech to Text to receive audio data as input and generate text data as output.

[0194] Step 9:

[0195] Server (Speech Recognition)

[0196] The converted text data is saved to a database. The input is the generated text data, and the output is the text data saved in the database.

[0197] Step 10:

[0198] AI generation server

[0199] The generative AI server analyzes and fuses past conversation data with newly acquired conversation data. The input is stored text data, which is analyzed using generative AI models such as GPT-3 and BERT to generate a summarized text. The output is the generated summarized text.

[0200] Step 11:

[0201] AI generation server → Terminal

[0202] The AI ​​server generates a summary text and sends it to the terminal. The input is the summary text, which is sent via a secure communication protocol. The output is the summary text that was sent to the terminal.

[0203] Step 12:

[0204] terminal

[0205] The device displays the summarized text on the smart glasses' display. The input is the summarized text received from the generating AI server, and the output is summarized information that the user can visually confirm (e.g., "Last time we talked about BB, but the new topic mentioned this time is CC"). This display allows the user to smoothly understand past and current conversations.

[0206] (Application Example 1)

[0207] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0208] In traditional brick-and-mortar stores, it was difficult for employees to instantly recognize customers' faces and access their past interactions and purchase history. This could lead to communication breakdowns and decreased customer satisfaction. Furthermore, a lack of information to appropriately suggest new products was also a problem.

[0209] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0210] In this invention, the server includes means for facial recognition, means for voice recognition, means for displaying generated information, means for providing memory support information, means for customer recognition, means for providing a summary of conversation content, and means for suggesting recommended products. This enables the recognition of the customer's face, instant display of past conversations and purchase history, and facilitates effective communication and appropriate product suggestions.

[0211] "Face recognition means" refers to a technology that uses a camera to capture the faces of people within its field of view and transmits them to a server.

[0212] "Speech recognition means" refers to a technology that captures conversations using a microphone and transmits them to a recognition server.

[0213] "Means for displaying generated information" refers to technologies that display information obtained from facial recognition or voice recognition on a display.

[0214] "Methods for providing memory support information" refers to technologies that provide memory support to users based on information such as past conversation data and purchase history.

[0215] "Customer recognition means" refers to technology that captures a customer's face and identifies and displays their past interactions and purchase history.

[0216] A "means for providing summaries of conversation content" is a technology that analyzes past conversation data and new conversation data and displays it as a summary.

[0217] "Recommended product suggestions" refers to techniques that suggest the most suitable products based on a customer's past purchase history and conversations.

[0218] The system for carrying out this invention consists of the following main elements.

[0219] System Configuration

[0220] 1. Smart Glasses

[0221] 2. Face recognition server

[0222] 3. Speech recognition server

[0223] 4. Generation AI Server

[0224] 5. Database

[0225] Details of the embodiment

[0226] Hardware configuration

[0227] 1. Smart Glasses

[0228] It has a built-in camera and microphone, and captures the faces of people within the user's field of view.

[0229] The system displays information generated through the display to the user.

[0230] 2. Server cluster

[0231] Face recognition server: Analyzes captured facial images and matches them against a database to identify individuals.

[0232] Speech recognition server: Converts captured audio data into text.

[0233] Generation AI Server: Analyzes and merges past and new conversation data to generate summarized information.

[0234] Database: Stores customer facial images, past conversation data, purchase history, etc.

[0235] System program

[0236] 1. Customer awareness

[0237] The smart glasses' camera captures the faces of people within the user's field of vision.

[0238] The facial image is sent to a facial recognition server and compared against a database.

[0239] The relevant person's information (name and past purchase history) is sent to the smart glasses and displayed on the screen.

[0240] 2. Speech Recognition

[0241] The microphone in the smart glasses captures the audio of the conversation.

[0242] The audio data is sent to a speech recognition server and converted into text.

[0243] The converted text data is stored in the database.

[0244] 3. Providing a summary of the conversation.

[0245] The generation AI server analyzes past and new conversation data to generate a summary text.

[0246] The generated summary text is sent to the smart glasses and displayed on the screen.

[0247] 4. Suggestions for recommended products

[0248] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[0249] Information about the suggested products is displayed on the smart glasses.

[0250] Specific example

[0251] 1. Customer service scenarios

[0252] Employees wear smart glasses to capture the faces of customers who come into the store.

[0253] A facial image is sent to a facial recognition server, and the server returns information such as, "This person is customer A. They purchased a vacuum cleaner last time."

[0254] The smart glasses display the message, "This person is Customer A. They purchased a vacuum cleaner last time."

[0255] The conversation audio is captured and sent to a speech recognition server.

[0256] The audio data is converted to text and stored in a database.

[0257] The AI ​​server analyzes the old and new conversation data and generates a summary such as, "Last time we talked about vacuum cleaners. The new topic is washing machines."

[0258] The generated summary is displayed on smart glasses.

[0259] Examples of prompts to input into a generative AI model

[0260] Customer data: {"Name": "Customer A", "Previously purchased items": ["Vacuum cleaner"], "Previous conversation content": ["About the vacuum cleaner's functions"]}

[0261] New conversation topic: "I'd like to ask for advice about washing machines."

[0262] Output: "Last time we talked about vacuum cleaners, but the new topic mentioned this time is washing machines."

[0263] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0264] Step 1:

[0265] The smart glasses' camera captures the faces of people within the user's field of vision.

[0266] Input: Camera video (face image)

[0267] Output: Captured facial image

[0268] Specific operation: The smart glasses detect the faces of people within the user's field of vision and capture those faces with the camera.

[0269] Step 2:

[0270] The captured facial image is sent to a facial recognition server.

[0271] Input: Captured facial image

[0272] Output: Notification of successful image transmission to face recognition server

[0273] Specific operation: The smart glasses use wireless communication to send the captured facial image to a facial recognition server.

[0274] Step 3:

[0275] The face recognition server analyzes the face image and matches it with the database to identify the corresponding person.

[0276] Input: Face image

[0277] Output: Information of the identified person (name, past purchase history)

[0278] Specific operation: The face recognition server analyzes the face image and matches it with the person records stored in the database.

[0279] Step 4:

[0280] Send the information of the corresponding person to the smart glasses.

[0281] Input: Information of the identified person

[0282] Output: Information display instruction to the smart glasses

[0283] Specific operation: The face recognition server sends the information of the identified person (name, past purchase history) to the smart glasses.

[0284] Step 5:

[0285] Display the information of the recognized person on the display of the smart glasses.

[0286] Input: Information of the identified person

[0287] Output: Person information displayed on the display

[0288] Specific operation: The smart glasses display the received person information on the display so that the user can confirm it.

[0289] Step 6:

[0290] The microphone in the smart glasses captures the audio of the conversation.

[0291] Input: Conversation audio

[0292] Output: Captured audio data

[0293] Specific operation: The smart glasses use the microphone to record the audio of the conversation.

[0294] Step 7:

[0295] The captured audio data is sent to the speech recognition server.

[0296] Input: Captured audio data

[0297] Output: Notification of successful transmission of voice data to the speech recognition server

[0298] Specific operation: The smart glasses use wireless communication to send the captured audio data to the speech recognition server.

[0299] Step 8:

[0300] The speech recognition server converts the speech data into text and stores it in a database.

[0301] Input: Audio data

[0302] Output: Text data

[0303] Specific operation: The speech recognition server converts speech data into text using speech recognition technology and saves the converted text data to a database.

[0304] Step 9:

[0305] The generation AI server analyzes past and new conversation data to generate a summary text.

[0306] Input: Past conversation data, new conversation data

[0307] Output: Summary text

[0308] Specific operation: The generation AI server uses the generation AI model to analyze the past and latest conversation data, extracts important points, and generates a summary text.

[0309] Step 10:

[0310] Send the generated summary text to the smart glasses.

[0311] Input: Summary text

[0312] Output: Instruction to display the summary text on the smart glasses

[0313] Specific operation: The generation AI server sends the generated summary text to the smart glasses, and the smart glasses display it on the display.

[0314] Step 11:

[0315] Display the summarized conversation content on the display of the smart glasses.

[0316] Input: Summary text

[0317] Output: Summarized information displayed on the display

[0318] Specific operation: The smart glasses display the summary text on the display so that the user can view it.

[0319] Step 12:

[0320] The generation AI server analyzes the customer's past purchase history and conversation content and proposes the optimal product.

[0321] Input: Past purchase history, conversation content

[0322] Output: Proposed product information

[0323] Specific operation: The generation AI server uses a generation AI model based on purchase history and conversation content to suggest the most suitable products.

[0324] Step 13:

[0325] Information about the suggested products is displayed on smart glasses.

[0326] Input: Proposed product information

[0327] Output: Suggested product information displayed on the screen

[0328] Specific operation: The smart glasses display suggested product information on the screen for the user to review.

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] This invention relates to a system combining facial recognition means, voice recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. A specific embodiment of this system, including its program and processing, is described below.

[0331] System Configuration

[0332] This system consists of the following main elements:

[0333] 1. Smart glasses device

[0334] 2. Face recognition server

[0335] 3. Speech recognition server

[0336] 4. Generation AI Server

[0337] 5. Emotional Engine

[0338] 6. Database

[0339] Program processing

[0340] Performing facial recognition

[0341] 1. Terminal

[0342] The smart glasses' camera captures the faces of people within the user's field of view.

[0343] 2. Terminal → Server (Facial Recognition)

[0344] The captured facial image is sent to a facial recognition server.

[0345] 3. Server (facial recognition)

[0346] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[0347] 4. Server (facial recognition) → Terminal

[0348] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[0349] 5. Terminal

[0350] The smart glasses display the name of the recognized person.

[0351] Executing speech recognition

[0352] 1. Terminal

[0353] The smart glasses' microphone captures the conversation audio in real time and temporarily stores that audio data.

[0354] 2. Terminal → Server (Speech Recognition)

[0355] The captured audio data is sent to the speech recognition server.

[0356] 3. Server (Speech Recognition)

[0357] Analyzes audio data and converts it to text.

[0358] 4. Server (Speech Recognition)

[0359] Save text data to a database.

[0360] Execution of the Emotion Engine

[0361] 1. Emotional Engine

[0362] The system analyzes voice data and facial image data to determine the user's emotional state (e.g., joy, anger, sadness, etc.).

[0363] 2. Emotional Engine

[0364] The determined emotional state is generated as emotional information and stored in a database.

[0365] Execution of Generative AI

[0366] 1. Generation AI Server

[0367] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes both.

[0368] 2. Generation AI server → Terminal

[0369] The generated summary text and sentiment information are sent to the smart glasses device.

[0370] 3. Terminal

[0371] The smart glasses display a summary of the conversation and emotional information.

[0372] Specific example

[0373] Scenes involving dementia patients and their families

[0374] 1. User (Patient)

[0375] A dementia patient is wearing smart glasses. When a family member enters the room, the smart glasses' camera captures the family member's face.

[0376] 2. Terminal → Server (Facial Recognition)

[0377] The captured facial image is sent to the facial recognition server.

[0378] 3. Server (facial recognition)

[0379] The facial recognition server matches the family's facial images against a database and returns the result, "This person is family member AA."

[0380] 4. Server (facial recognition) → Terminal

[0381] Family members' names and past conversation history are sent to the smart glasses.

[0382] 5. Terminal

[0383] The smart glasses display the message, "This is family member AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[0384] 6. Terminal → Server (Speech Recognition)

[0385] The captured audio data is sent to the speech recognition server.

[0386] 7. Server (Speech Recognition)

[0387] Convert the conversation content into text and save it to a database.

[0388] 8. Emotional Engine

[0389] The system analyzes captured audio and facial expression data to determine the user's emotional state. For example, it might generate emotional information such as, "The user is currently excited."

[0390] 9. Emotion Engine → Database

[0391] Emotional information is stored in a database.

[0392] 10. Generation AI Server

[0393] The generating AI server analyzes new conversation data, past conversation data, and sentiment information to generate summaries such as, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited."

[0394] 11. Generation AI Server → Terminal

[0395] The generated summary is sent to the smart glasses.

[0396] 12. Terminal

[0397] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," making it easier for the user to understand the flow of the conversation.

[0398] In this way, the system of the present invention can enable smooth communication in service industries dealing with dementia patients and many customers, thereby improving the quality of life for users. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, enabling more appropriate responses.

[0399] The following describes the processing flow.

[0400] Step 1:

[0401] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[0402] Step 2:

[0403] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[0404] Step 3:

[0405] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[0406] Step 4:

[0407] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[0408] Step 5:

[0409] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[0410] Step 6:

[0411] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[0412] Step 7:

[0413] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[0414] Step 8:

[0415] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[0416] Step 9:

[0417] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[0418] Step 10:

[0419] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[0420] Step 11:

[0421] The server (speech recognition) saves the transcribed conversation to a database.

[0422] Step 12:

[0423] The emotion engine retrieves voice data and facial expression data from the database. The emotion engine then begins analyzing the voice and facial expressions.

[0424] Step 13:

[0425] The emotion engine determines the user's emotional state based on the analysis results. For example, it identifies one of the following emotional states: joy, anger, sadness, or pleasure.

[0426] Step 14:

[0427] The emotion engine generates emotion information based on the determined emotional state and stores that information in a database.

[0428] Step 15:

[0429] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them.

[0430] Step 16:

[0431] The generation AI server fuses past conversation data, new conversation data, and sentiment information to generate a summary. This summary includes key points from the current conversation, references to past information, and the user's emotional state.

[0432] Step 17:

[0433] The generated summary text and sentiment information are sent from the generating AI server to the terminal.

[0434] Step 18:

[0435] The device displays the summary and sentiment information it receives on the smart glasses' display. This allows the user to understand, "In our last conversation we talked about AA. This time we're talking about CC, and the user is currently excited."

[0436] This series of processes utilizes facial recognition, voice recognition, generative AI, and an emotion engine to enable smooth communication in service industries dealing with dementia patients and large customer bases.

[0437] (Example 2)

[0438] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0439] Conventional systems provided limited memory support information and lacked information based on the user's emotional state. This made it difficult to improve the user's quality of life and facilitate smooth communication. Furthermore, it was challenging to provide appropriate support for dementia patients and service industries dealing with large customer bases.

[0440] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0441] In this invention, the server includes means for capturing the faces of people within its field of view using a camera and transmitting them to the server, means including a microphone for capturing and transmitting audio to the server, means equipped with a display for displaying the generated information, server means for analyzing face data and audio data and returning the recognized information, emotion engine means for analyzing audio data and face image data and determining the user's emotional state, and generation AI server means for retrieving newly stored text data, past conversation data, and emotional information from a database and generating summary text and emotional information. This makes it possible to provide memory support information and information based on emotional state.

[0442] A "camera" is a device that captures images and acquires that digital data.

[0443] A "server" is a computer system that operates on a network and provides data processing and storage services.

[0444] A "microphone" is a device that captures sound and acquires its digital data.

[0445] A "display" is a device that displays visual information.

[0446] "Facial recognition" is a technology that analyzes captured facial images to identify specific individuals.

[0447] "Speech recognition" is a technology that analyzes captured audio and converts it into text data.

[0448] An "emotion engine" is a technology that analyzes voice data and facial image data to determine the user's emotional state.

[0449] A "database" is a system for systematically storing and managing data.

[0450] A "generating AI server" is a server equipped with artificial intelligence that generates summarized text and sentiment information using stored data.

[0451] "Memory support information" refers to information that supports a user's memory by referencing specific records or histories.

[0452] A "summary text" is a document that concisely summarizes multiple pieces of information.

[0453] This invention relates to a system combining facial recognition means, speech recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. The system consists of smart glasses terminals, a facial recognition server, a speech recognition server, a generation AI server, an emotion engine, and a database as its main components.

[0454] Performing facial recognition

[0455] Hardware and software to be used

[0456] Smart Glasses Device

[0457] Face recognition server

[0458] Camera (high resolution)

[0459] OpenCV (face recognition library)

[0460] operation

[0461] The smart glasses device captures the faces of people within the user's field of view via its camera and sends them to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database, and identifies specific individuals. The results are sent back to the smart glasses device, and the person's name and related information are displayed on the screen.

[0462] Executing speech recognition

[0463] Hardware and software to be used

[0464] Smart Glasses Device

[0465] Speech recognition server

[0466] microphone

[0467] Google Speech-to-Text API

[0468] operation

[0469] The microphone on the smart glasses device captures the conversation audio in real time and sends it to a speech recognition server. The speech recognition server converts the audio data into text and stores it in a database.

[0470] Execution of the Emotion Engine

[0471] Hardware and software to be used

[0472] Emotional Engine

[0473] Emotion analysis algorithms (voice tone analysis, facial expression analysis)

[0474] operation

[0475] The emotion engine analyzes voice data and facial image data to determine the user's emotional state. The determined emotional state is then stored in a database as emotion information.

[0476] Execution of Generative AI

[0477] Hardware and software to be used

[0478] AI generation server

[0479] GPT-4 (registered trademark) (natural language processing model)

[0480] operation

[0481] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes them, and generates summarized text and sentiment information. This generated information is sent to the smart glasses terminal and displayed on the screen.

[0482] Specific example

[0483] For example, if a dementia patient is wearing smart glasses, when a family member enters the room, the smart glasses' camera captures the family member's face and sends it to a facial recognition server. The facial recognition server compares the family member's face image with a database and sends the result, "This person is family member AA," back to the smart glasses terminal. The smart glasses then display, "This person is family member AA. We talked about BB during the last visit."

[0484] Additionally, once a conversation begins, the smart glasses' microphone captures the audio and sends it to a speech recognition server. The speech recognition server converts the conversation into text and stores it in a database. The emotion engine determines the user's emotional state from the audio and facial expression data, generates emotion information such as "The user is currently excited," and stores it in the database. The generating AI server analyzes the new conversation data, past conversation data, and emotion information to generate a summary such as "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," and sends it to the smart glasses for display.

[0485] Example of a prompt

[0486] "Last time we talked about BB, but the new topic we discussed this time is CC. Users are currently very excited about it."

[0487] This system enables smooth communication in service industries dealing with dementia patients and large customer bases, thereby improving the user's quality of life. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, leading to more appropriate responses.

[0488] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0489] Step 1: Face Capture

[0490] terminal

[0491] The smart glasses' camera captures the faces of people within the user's field of view in real time. The input data is images of people's faces within the field of view, and a high-resolution camera is used to obtain clear images. The output is the captured face image.

[0492] Step 2: Send your face image

[0493] Terminal → Server (Facial Recognition)

[0494] The captured facial image is sent to a facial recognition server. The facial image data is compressed and transferred to the facial recognition server in real time using a secure communication protocol (e.g., HTTPS). The input data is the captured facial image, and the output is the facial image data sent to the facial recognition server.

[0495] Step 3: Analysis of facial data

[0496] Server (facial recognition)

[0497] The face recognition server analyzes the received face image. The software used is a face recognition library such as OpenCV, and the input data is the transmitted face image. This analysis extracts facial feature points. The output is face feature data.

[0498] Step 4: Match with database

[0499] Server (facial recognition)

[0500] The analyzed facial data is compared with a database. A specific matching algorithm is used to identify individuals whose facial features match those in the existing database. The input data is facial feature data, and the output is information about the identified individuals.

[0501] Step 5: Submit the identification result

[0502] Server (facial recognition) → Terminal

[0503] The server sends the name of the identified person and past conversation details back to the smart glasses device. The server encrypts and transmits this information. The input data is the identified person's information, and the output is the identification information sent to the device.

[0504] Step 6: Displaying Identification Information

[0505] terminal

[0506] The smart glasses display the name and related information of the recognized person. The input data is identification information, and the output is the information displayed on the screen.

[0507] Step 7: Capture audio

[0508] User

[0509] When a user starts a conversation, the smart glasses' microphone captures the audio in real time. The input data is the conversation audio, and the output is the captured audio data.

[0510] Step 8: Sending audio data

[0511] Terminal → Server (Speech Recognition)

[0512] The captured audio data is sent to the speech recognition server. Before transmission, the data is converted to an appropriate format (e.g., WAV format). The input data is the audio data, and the output is the data sent to the speech recognition server.

[0513] Step 9: Analyzing the audio data

[0514] Server (Speech Recognition)

[0515] A speech recognition server analyzes the audio data and converts it to text. The input data is audio, and the software used is the Google Speech-to-Text API. The output is text data.

[0516] Step 10: Saving the text data

[0517] Server (Speech Recognition)

[0518] The data is converted to text and saved to a database. The input data is the converted text data, which is saved with a timestamp. The output is the text data saved in the database.

[0519] Step 11: Analysis of emotional state

[0520] Emotional Engine

[0521] This system analyzes voice and facial image data to determine the user's emotional state. Based on an emotion analysis algorithm, the input data consists of voice and facial image data. The output is the determined emotion information.

[0522] Step 12: Preserving emotional information

[0523] Emotional Engine

[0524] The determined emotional state is stored in a database as emotional information. The input data is emotional information, and the output is the emotional information stored in the database.

[0525] Step 13: Summarizing the conversation

[0526] AI generation server

[0527] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them. The technology used is GPT-4. The input data consists of text data, past conversation data, and sentiment information, and the output is summarized text and sentiment information.

[0528] Step 14: Sending generated information

[0529] AI generation server → Terminal

[0530] The generated summary text and sentiment information are sent to the smart glasses device. Encryption is used to maintain security. The input data consists of the summary text and sentiment information; the output data is the data sent to the smart glasses device.

[0531] Step 15: Display summary information

[0532] terminal

[0533] The smart glasses display a summarized conversation and sentiment information. The input data is the summarized text and sentiment information, and the output is the information displayed on the screen.

[0534] (Application Example 2)

[0535] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0536] Traditional in-store customer service systems only used facial and voice recognition, resulting in insufficient personalized service. Furthermore, it was difficult to grasp customers' emotional states in real time, hindering the ability to take prompt and appropriate action in situations requiring special attention.

[0537] In Application Example 2, the identification processing performed by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for analyzing emotional states, means for generating summaries based on past conversation content and emotional information, means for capturing the faces of people within the field of view using a camera and transmitting them to the server, and means for capturing conversations using a microphone and transmitting them to the recognition server. This enables the provision of information based on the customer's emotional state and past conversation history, allowing for the rapid and appropriate provision of personalized services.

[0538] "Means of performing facial recognition" refers to technology that analyzes and identifies a person's face image captured by a camera.

[0539] "Means of performing speech recognition" refers to technology for converting audio data captured by a microphone into text format and then analyzing it.

[0540] "Means for displaying generated information" refers to display and interface technologies for visually providing analyzed and processed data to the user.

[0541] "Means for analyzing emotional states" refers to technologies that use voice data and facial image data to determine the emotional state of a subject (such as joy, anger, sadness, etc.).

[0542] "Means for generating summaries based on past conversation content and sentiment information" refers to a generative AI model that analyzes past conversation content and sentiment information in a database to generate important information and summaries.

[0543] A "system that provides personalized service information" is a set of technologies that provide information and services to individual customers based on their individual needs and past history.

[0544] To carry out this invention, a system comprising the following main elements is required. The system comprises the following components.

[0545] System Configuration Details

[0546] 1. Smart glasses device

[0547] 2. Face recognition server

[0548] 3. Speech recognition server

[0549] 4. Generation AI Server

[0550] 5. Emotional Engine

[0551] 6. Database

[0552] Hardware and software

[0553] Hardware: Smart glasses (e.g., Microsoft HoloLens®), Servers (high-performance cloud servers)

[0554] Software: Face recognition software (e.g., Amazon Rekognition), speech recognition software (e.g., Google Speech-to-Text), sentiment analysis software (e.g., Microsoft Azure®'s Text Analytics for sentiment analysis), generative AI models (e.g., OpenAI®'s GPT-4)

[0555] System program processing details

[0556] Face recognition processing

[0557] The smart glasses device captures the faces of people within the user's field of view using its camera. The captured facial images are sent to a facial recognition server in real time or in batch processing. This facial recognition server performs image analysis and identifies specific individuals by comparing them with existing data in its database. The person's name and related information are sent back to the smart glasses device and displayed to the user.

[0558] Speech recognition processing

[0559] The smart glasses' microphone captures conversations between the user and customers in real time and sends the audio data to a speech recognition server. This server converts the audio data into text and stores the text data in a database.

[0560] Emotional Engine Processing

[0561] The emotion engine analyzes voice and facial image data to determine the user's emotional state. The determined emotional state is stored in a database and used by the generating AI server.

[0562] Processing of generated AI models

[0563] The AI ​​server analyzes newly stored text data, past conversation data, and sentiment information to generate conversation summaries and key information. The generated information is sent to the smart glasses terminal and provided to the user.

[0564] Specific example

[0565] Examples of customer service in physical stores

[0566] When a user wears smart glasses and captures the face of a customer entering the store, a facial recognition server identifies the customer's name and past visit history. For example, information such as, "This customer is Mr. / Ms. Tanaka. During their last visit, they showed interest in allergy-free products," is displayed. When the customer and the user begin a conversation, the conversation is recorded, and after it ends, a summary such as, "Mr. / Ms. Tanaka is interested in new organic products," is generated. Furthermore, an emotion engine analyzes the customer's emotional state in real time and provides information such as, "The customer is excited," enabling the user to respond appropriately.

[0567] Example of a prompt

[0568] Examples of prompts for a generative AI model are as follows:

[0569] prompt:

[0570] Past conversation data: {conversation data}

[0571] Latest conversation: {New conversation}

[0572] Past emotional information: {Past emotional information}

[0573] New emotional state: {New emotional state}

[0574] output:

[0575] 1. Summary of the latest conversation

[0576] 2. Customer concerns and proposed solutions

[0577] Through this format, personalized services can be provided in physical stores, significantly improving the quality of customer service.

[0578] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0579] Step 1:

[0580] The smart glasses' camera captures the faces of people within the user's field of view. The captured face images are sent from the smart glasses device to a face recognition server. At this point, the captured image serves as the input, and the transmission process is initiated.

[0581] Step 2:

[0582] The facial recognition server analyzes the received facial image and compares it with existing data in the database to identify a specific person. The input is a facial image, and the output is the name of the identified person and related information. The process involves facial image analysis and database matching.

[0583] Step 3:

[0584] The facial recognition server sends the name and related information of the identified person back to the smart glasses terminal. The output information includes the identified person's name and past conversation content. The user then receives the information displayed on the screen.

[0585] Step 4:

[0586] The smart glasses' microphone captures the conversation between the user and the customer in real time. The captured audio data is sent from the smart glasses device to the speech recognition server. At this point, the captured audio data is the input, and the transmission process is performed.

[0587] Step 5:

[0588] The speech recognition server analyzes the received audio data and converts it into text format. The input is audio data, and the output is text data. Audio analysis and text conversion processes are performed.

[0589] Step 6:

[0590] The text data generated by the speech recognition server is saved to the database. The text data is used as input for the database registration process.

[0591] Step 7:

[0592] The emotion engine analyzes voice data and facial image data in the database to determine the user's emotional state. The input is voice data and facial image data, and the output is the emotional state. Emotion analysis processing is then performed.

[0593] Step 8:

[0594] The emotion engine determines the emotional state, generates emotion information, and stores it in the database. The emotional state is the input, and the process of generating emotion information and registering it in the database is performed.

[0595] Step 9:

[0596] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes both, and generates a summary. The input consists of text data, past conversation data, and sentiment information, while the output is summarized text information. Data analysis and summary generation processes are performed.

[0597] Step 10:

[0598] The generation AI server sends the generated summary text and sentiment information to the smart glasses terminal. The output consists of the summary text and sentiment information, and the process of sending them to the smart glasses is performed.

[0599] Step 11:

[0600] The smart glasses display shows a summarized conversation and emotional information. Based on this, the user can provide personalized service information to the customer. The displayed information is the output.

[0601] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0602] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0604] [Second Embodiment]

[0605] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0606] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0607] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0609] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0611] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0612] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0613] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0614] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0615] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0616] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0617] This invention relates to a system that provides memory support information by combining facial recognition means, voice recognition means, and means for displaying generated information. A specific embodiment of this system, including its program and processing, is described below.

[0618] System Configuration

[0619] This system consists of the following main elements:

[0620] 1. Smart glasses device

[0621] 2. Face recognition server

[0622] 3. Speech recognition server

[0623] 4. Generation AI Server

[0624] 5. Database

[0625] Program processing

[0626] Performing facial recognition

[0627] 1. Terminal

[0628] The smart glasses' camera captures the faces of people within the user's field of view.

[0629] 2. Terminal → Server (Facial Recognition)

[0630] The captured facial image is sent to a facial recognition server.

[0631] 3. Server (facial recognition)

[0632] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[0633] 4. Server (facial recognition) → Terminal

[0634] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[0635] 5. Terminal

[0636] The smart glasses display the name of the recognized person.

[0637] Executing speech recognition

[0638] 1. Terminal

[0639] The smart glasses' microphone continuously captures the audio of the conversation.

[0640] 2. Terminal → Server (Speech Recognition)

[0641] The captured audio data is sent to the speech recognition server.

[0642] 3. Server (Speech Recognition)

[0643] Convert audio data to text.

[0644] 4. Server (Speech Recognition)

[0645] Save text data to a database.

[0646] Execution of Generative AI

[0647] 1. Generation AI Server

[0648] The generation AI server analyzes and merges past conversation data with newly acquired conversation data.

[0649] 2. Generation AI server → Terminal

[0650] The generated summary text is sent to the smart glasses device.

[0651] 3. Terminal

[0652] The smart glasses display a summarized version of the conversation.

[0653] Specific example

[0654] Scenes involving dementia patients and their caregivers

[0655] 1. User (Patient)

[0656] A dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face.

[0657] 2. Terminal → Server (Facial Recognition)

[0658] The captured facial image is sent to the facial recognition server.

[0659] 3. Server (facial recognition)

[0660] The facial recognition server matches the helper's facial image against the database and returns the result, "This person is helper AA."

[0661] 4. Server (facial recognition) → Terminal

[0662] The helper's name and past conversation history are sent to the smart glasses.

[0663] 5. Terminal

[0664] The smart glasses display the message, "This is caregiver AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[0665] 6. Terminal → Server (Speech Recognition)

[0666] The captured audio data is sent to the speech recognition server.

[0667] 7. Server (Speech Recognition)

[0668] Convert the conversation content into text and save it to a database.

[0669] 8. Generation AI Server

[0670] The generating AI server analyzes new conversation data and past conversation data to generate a summary such as, "Last time we talked about BB, but the new topic mentioned this time is CC."

[0671] 9. Generation AI Server → Terminal

[0672] The generated summary is sent to the smart glasses.

[0673] 10. Terminal

[0674] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC," making it easier for the user to understand the flow of the conversation.

[0675] In this way, the system of the present invention can facilitate communication with dementia patients and in service industries dealing with many customers, thereby improving the quality of life for users.

[0676] The following describes the processing flow.

[0677] Step 1:

[0678] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[0679] Step 2:

[0680] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[0681] Step 3:

[0682] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[0683] Step 4:

[0684] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[0685] Step 5:

[0686] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[0687] Step 6:

[0688] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[0689] Step 7:

[0690] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[0691] Step 8:

[0692] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[0693] Step 9:

[0694] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[0695] Step 10:

[0696] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[0697] Step 11:

[0698] The server (speech recognition) saves the transcribed conversation to a database.

[0699] Step 12:

[0700] The generating AI server retrieves newly saved text data and past conversation data from the database and analyzes both.

[0701] Step 13:

[0702] The generation AI server merges past and new conversation data to generate a summary. This summary includes key points from the current conversation and references to past information.

[0703] Step 14:

[0704] The generated summary text is sent from the generation AI server to the terminal.

[0705] Step 15:

[0706] The device displays a summary it has received on the smart glasses' screen. This allows the user to understand, "In our last conversation, we talked about AA. This time, we're talking about CC."

[0707] This series of processes enables smooth communication in service industries dealing with dementia patients and large numbers of customers by utilizing facial recognition, voice recognition, and generative AI.

[0708] (Example 1)

[0709] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0710] Conventional memory support systems have a drawback: they lack features to help users remember people and conversation content. In particular, there was a need for effective support in situations where memory assistance is required, such as for dementia patients and service industries that involve interaction with many customers. Furthermore, existing systems sometimes suffered from low accuracy in facial and voice recognition, resulting in inaccurate memory support information.

[0711] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0712] In this invention, the server includes means for the terminal to capture a person's face, means for the terminal to transmit the captured face image to a face recognition server, means for the face recognition server to analyze the face image and compare it with a database, means for the face recognition server to return identified person information to the terminal, means for the terminal to display the identified person information, means for the terminal to capture the audio of a conversation, means for the terminal to transmit the captured audio data to a speech recognition server, means for the speech recognition server to convert the audio data into text, means for the speech recognition server to store the text data in a database, means for the generating AI server to analyze and merge past conversation data and new conversation data, means for the generating AI server to transmit the generated summary text to the terminal, and means for the terminal to display the summarized conversation content. As a result, the user can instantly identify people and check past conversation content, enabling the conversation to proceed smoothly.

[0713] A "terminal" is a device that a user wears and operates, and in this invention, this specifically refers to smart glasses, etc.

[0714] A "face recognition server" is a server that analyzes facial images transmitted from a terminal, compares them with a database, and identifies the corresponding person.

[0715] A "speech recognition server" is a server that converts speech data sent from a terminal into text.

[0716] A "generative AI server" is a server that analyzes and merges past conversation data with new conversation data to generate summarized text.

[0717] A "database" is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[0718] A "camera" is a device built into a terminal that captures the faces of people within the user's field of view.

[0719] A "microphone" is an audio input device built into a terminal that captures the sound of conversations.

[0720] A "face image" is image data of a person's face captured by the device's camera.

[0721] "Audio data" refers to audio data of a conversation captured by the device's microphone.

[0722] "Text data" refers to character data converted from speech data by a speech recognition server.

[0723] A "summary text" is a summary that integrates past conversation content and new conversation content, generated by a generation AI server.

[0724] This invention relates to a system for providing memory support information to a user. This system combines facial recognition means, voice recognition means, and means for displaying generated information to provide effective memory support to the user.

[0725] System Configuration

[0726] This system consists of the following main elements:

[0727] 1. Terminal

[0728] This includes wearable devices such as smart glasses. These devices are equipped with cameras and microphones that capture the faces of people and the audio of conversations within the user's field of view.

[0729] 2. Face recognition server

[0730] The face recognition server analyzes face images sent from the terminal and matches them against a database to identify the corresponding person. It uses face recognition libraries such as OpenCV or AWS Rekognition.

[0731] 3. Speech recognition server

[0732] The speech recognition server converts the audio data sent from the terminal into text. It uses speech recognition services such as Google Cloud Speech-to-Text or IBM Watson Speech to Text.

[0733] 4. Generation AI Server

[0734] The generative AI server analyzes and merges past conversation data with newly acquired conversation data to generate summarized text. Examples of generative AI models include GPT-3 and BERT.

[0735] 5. Database

[0736] This is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[0737] Program processing

[0738] In this system, when the user wears smart glasses, a series of processes are performed, from facial recognition of a person to speech recognition of conversations, and then the generation and display of summarized text by a generative AI.

[0739] Facial recognition means

[0740] The device's camera captures the faces of people within the user's field of view and sends the facial images to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database to identify the corresponding individuals, and sends the results back to the device. The device then displays the returned person information on its screen.

[0741] Speech recognition means

[0742] The device's microphone captures the conversation audio and sends the audio data to a speech recognition server. The speech recognition server converts the audio data into text and stores the resulting text data in a database. This makes searching and analysis easier later on.

[0743] Generation AI means

[0744] The AI ​​server analyzes and merges previously stored conversation data with newly acquired conversation data to generate a summary text. The generated summary text is sent to the device and displayed on the device's screen. This allows the user to smoothly understand the content of past and current conversations.

[0745] Specific example

[0746] Consider a scenario where a dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face and sends the information to a server. The server sends back the relevant information, and the device displays, "This is caregiver AA. We talked about BB during the last visit." When a conversation begins, the smart glasses' microphone captures the audio and sends the audio data to the server. The server converts the audio to text and stores it in a database. Later, a generative AI server analyzes this data, generates a summary text, and displays it on the device.

[0747] An example of a prompt message would be: "Generate a program that analyzes facial images captured by smart glasses and displays information about the recognized person. Also, include a function to convert spoken conversation into text and summarize it."

[0748] Thus, by providing memory support information, the present invention can facilitate communication in service industries that interact with dementia patients and many customers, thereby improving the quality of life for users.

[0749] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0750] Step 1:

[0751] terminal

[0752] The smart glasses' camera captures the faces of people within the user's field of view in high resolution. The input is a person's face, which is saved as a still image using the camera. The output is the captured face image. This face image data is used in the next step.

[0753] Step 2:

[0754] Terminal → Server (Facial Recognition)

[0755] The terminal captures a face image and sends it to a face recognition server using a secure communication protocol (e.g., HTTPS). The input is the face image data obtained in step 1. The output is the face image data sent to the face recognition server.

[0756] Step 3:

[0757] Server (facial recognition)

[0758] The face recognition server analyzes the received face image and extracts feature points. Specifically, the relevant system (e.g., OpenCV or AWS Rekognition) receives face image data as input and outputs the coordinates of feature points and other identification information. Next, these feature points are compared with existing face data stored in the database to identify the corresponding person. As a result of the comparison, person information is output.

[0759] Step 4:

[0760] Server (facial recognition) → Terminal

[0761] The facial recognition server sends back relevant information to the terminal, such as the name of the identified person and past conversation content. The input is the identified person information data, and the output is the person information data sent to the terminal.

[0762] Step 5:

[0763] terminal

[0764] The device displays the person's information, which has been returned, on the smart glasses' display. The input is person information data received from the server, and the output is display information that the user can visually confirm (e.g., "This person is helper AA. We talked about BB during the last visit").

[0765] Step 6:

[0766] terminal

[0767] The smart glasses' microphone continuously captures the conversation audio. The input is the conversational audio, which is saved as audio data using the microphone. The output is the captured audio data, which is used in the next step.

[0768] Step 7:

[0769] Terminal → Server (Speech Recognition)

[0770] The terminal transmits the captured audio data to the speech recognition server in real time. The input is the audio data obtained in step 6, which is transmitted via a secure communication protocol. The output is the audio data sent to the speech recognition server.

[0771] Step 8:

[0772] Server (Speech Recognition)

[0773] The speech recognition server converts the audio data into text. Specifically, it uses Google Cloud Speech-to-Text or IBM Watson Speech to Text to receive audio data as input and generate text data as output.

[0774] Step 9:

[0775] Server (Speech Recognition)

[0776] The converted text data is saved to a database. The input is the generated text data, and the output is the text data saved in the database.

[0777] Step 10:

[0778] AI generation server

[0779] The generative AI server analyzes and fuses past conversation data with newly acquired conversation data. The input is stored text data, which is analyzed using generative AI models such as GPT-3 and BERT to generate a summarized text. The output is the generated summarized text.

[0780] Step 11:

[0781] AI generation server → Terminal

[0782] The AI ​​server generates a summary text and sends it to the terminal. The input is the summary text, which is sent via a secure communication protocol. The output is the summary text that was sent to the terminal.

[0783] Step 12:

[0784] terminal

[0785] The device displays the summarized text on the smart glasses' display. The input is the summarized text received from the generating AI server, and the output is summarized information that the user can visually confirm (e.g., "Last time we talked about BB, but the new topic mentioned this time is CC"). This display allows the user to smoothly understand past and current conversations.

[0786] (Application Example 1)

[0787] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0788] In traditional brick-and-mortar stores, it was difficult for employees to instantly recognize customers' faces and access their past interactions and purchase history. This could lead to communication breakdowns and decreased customer satisfaction. Furthermore, a lack of information to appropriately suggest new products was also a problem.

[0789] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0790] In this invention, the server includes means for facial recognition, means for voice recognition, means for displaying generated information, means for providing memory support information, means for customer recognition, means for providing a summary of conversation content, and means for suggesting recommended products. This enables the recognition of the customer's face, instant display of past conversations and purchase history, and facilitates effective communication and appropriate product suggestions.

[0791] "Face recognition means" refers to a technology that uses a camera to capture the faces of people within its field of view and transmits them to a server.

[0792] "Speech recognition means" refers to a technology that captures conversations using a microphone and transmits them to a recognition server.

[0793] "Means for displaying generated information" refers to technologies that display information obtained from facial recognition or voice recognition on a display.

[0794] "Methods for providing memory support information" refers to technologies that provide memory support to users based on information such as past conversation data and purchase history.

[0795] "Customer recognition means" refers to technology that captures a customer's face and identifies and displays their past interactions and purchase history.

[0796] A "means for providing summaries of conversation content" is a technology that analyzes past conversation data and new conversation data and displays it as a summary.

[0797] "Recommended product suggestions" refers to techniques that suggest the most suitable products based on a customer's past purchase history and conversations.

[0798] The system for carrying out this invention consists of the following main elements.

[0799] System Configuration

[0800] 1. Smart Glasses

[0801] 2. Face recognition server

[0802] 3. Speech recognition server

[0803] 4. Generation AI Server

[0804] 5. Database

[0805] Details of the embodiment

[0806] Hardware configuration

[0807] 1. Smart Glasses

[0808] It has a built-in camera and microphone, and captures the faces of people within the user's field of view.

[0809] The system displays information generated through the display to the user.

[0810] 2. Server cluster

[0811] Face recognition server: Analyzes captured facial images and matches them against a database to identify individuals.

[0812] Speech recognition server: Converts captured audio data into text.

[0813] Generation AI Server: Analyzes and merges past and new conversation data to generate summarized information.

[0814] Database: Stores customer facial images, past conversation data, purchase history, etc.

[0815] System program

[0816] 1. Customer awareness

[0817] The smart glasses' camera captures the faces of people within the user's field of vision.

[0818] The facial image is sent to a facial recognition server and compared against a database.

[0819] The relevant person's information (name and past purchase history) is sent to the smart glasses and displayed on the screen.

[0820] 2. Speech Recognition

[0821] The microphone in the smart glasses captures the audio of the conversation.

[0822] The audio data is sent to a speech recognition server and converted into text.

[0823] The converted text data is stored in the database.

[0824] 3. Providing a summary of the conversation.

[0825] The generation AI server analyzes past and new conversation data to generate a summary text.

[0826] The generated summary text is sent to the smart glasses and displayed on the screen.

[0827] 4. Suggestions for recommended products

[0828] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[0829] Information about the suggested products is displayed on the smart glasses.

[0830] Specific example

[0831] 1. Customer service scenarios

[0832] Employees wear smart glasses to capture the faces of customers who come into the store.

[0833] A facial image is sent to a facial recognition server, and the server returns information such as, "This person is customer A. They purchased a vacuum cleaner last time."

[0834] The smart glasses display the message, "This person is Customer A. They purchased a vacuum cleaner last time."

[0835] The conversation audio is captured and sent to a speech recognition server.

[0836] The audio data is converted to text and stored in a database.

[0837] The AI ​​server analyzes the old and new conversation data and generates a summary such as, "Last time we talked about vacuum cleaners. The new topic is washing machines."

[0838] The generated summary is displayed on smart glasses.

[0839] Examples of prompts to input into a generative AI model

[0840] Customer data: {"Name": "Customer A", "Previously purchased items": ["Vacuum cleaner"], "Previous conversation content": ["About the vacuum cleaner's functions"]}

[0841] New conversation topic: "I'd like to ask for advice about washing machines."

[0842] Output: "Last time we talked about vacuum cleaners, but the new topic mentioned this time is washing machines."

[0843] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0844] Step 1:

[0845] The smart glasses' camera captures the faces of people within the user's field of vision.

[0846] Input: Camera video (face image)

[0847] Output: Captured facial image

[0848] Specific operation: The smart glasses detect the faces of people within the user's field of vision and capture those faces with the camera.

[0849] Step 2:

[0850] The captured facial image is sent to a facial recognition server.

[0851] Input: Captured facial image

[0852] Output: Notification of successful image transmission to face recognition server

[0853] Specific operation: The smart glasses use wireless communication to send the captured facial image to a facial recognition server.

[0854] Step 3:

[0855] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[0856] Input: Face image

[0857] Output: Information about the identified person (name, past purchase history)

[0858] Specific operation: The face recognition server analyzes the face image and compares it with the person's record stored in the database.

[0859] Step 4:

[0860] The information of the person in question is sent to the smart glasses.

[0861] Input: Information of an identified person

[0862] Output: Instructions for displaying information on smart glasses

[0863] Specific operation: The facial recognition server sends information about the identified person (name, past purchase history) to the smart glasses.

[0864] Step 5:

[0865] The smart glasses display information about the recognized person.

[0866] Input: Information of an identified person

[0867] Output: Person information displayed on the screen

[0868] Specific operation: The smart glasses display the person information they receive on the screen, allowing the user to verify it.

[0869] Step 6:

[0870] The microphone in the smart glasses captures the audio of the conversation.

[0871] Input: Conversation audio

[0872] Output: Captured audio data

[0873] Specific operation: The smart glasses use the microphone to record the audio of the conversation.

[0874] Step 7:

[0875] The captured audio data is sent to the speech recognition server.

[0876] Input: Captured audio data

[0877] Output: Notification of successful transmission of voice data to the speech recognition server

[0878] Specific operation: The smart glasses use wireless communication to send the captured audio data to the speech recognition server.

[0879] Step 8:

[0880] The speech recognition server converts the speech data into text and stores it in a database.

[0881] Input: Audio data

[0882] Output: Text data

[0883] Specific operation: The speech recognition server converts speech data into text using speech recognition technology and saves the converted text data to a database.

[0884] Step 9:

[0885] The generation AI server analyzes past and new conversation data to generate a summary text.

[0886] Input: Past conversation data, new conversation data

[0887] Output: Summary text

[0888] Specific operation: The generative AI server uses a generative AI model to analyze past and recent conversation data, extract key points, and generate a summarized text.

[0889] Step 10:

[0890] The generated summary text is sent to smart glasses.

[0891] Input: Summary text

[0892] Output: Instructions to display summarized text on smart glasses

[0893] Specific operation: The generation AI server sends the generated summary text to the smart glasses, and the smart glasses display it on the screen.

[0894] Step 11:

[0895] The smart glasses display a summarized version of the conversation.

[0896] Input: Summary text

[0897] Output: Summary information displayed on the screen

[0898] Specific action: The smart glasses display a summary text on the screen for the user to review.

[0899] Step 12:

[0900] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[0901] Input: Past purchase history, conversation content

[0902] Output: Proposed product information

[0903] Specific operation: The generation AI server uses a generation AI model based on purchase history and conversation content to suggest the most suitable products.

[0904] Step 13:

[0905] Information about the suggested products is displayed on smart glasses.

[0906] Input: Proposed product information

[0907] Output: Suggested product information displayed on the screen

[0908] Specific operation: The smart glasses display suggested product information on the screen for the user to review.

[0909] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0910] This invention relates to a system combining facial recognition means, voice recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. A specific embodiment of this system, including its program and processing, is described below.

[0911] System Configuration

[0912] This system consists of the following main elements:

[0913] 1. Smart glasses device

[0914] 2. Face recognition server

[0915] 3. Speech recognition server

[0916] 4. Generation AI Server

[0917] 5. Emotional Engine

[0918] 6. Database

[0919] Program processing

[0920] Performing facial recognition

[0921] 1. Terminal

[0922] The smart glasses' camera captures the faces of people within the user's field of view.

[0923] 2. Terminal → Server (Facial Recognition)

[0924] The captured facial image is sent to a facial recognition server.

[0925] 3. Server (facial recognition)

[0926] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[0927] 4. Server (facial recognition) → Terminal

[0928] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[0929] 5. Terminal

[0930] The smart glasses display the name of the recognized person.

[0931] Executing speech recognition

[0932] 1. Terminal

[0933] The smart glasses' microphone captures the conversation audio in real time and temporarily stores that audio data.

[0934] 2. Terminal → Server (Speech Recognition)

[0935] The captured audio data is sent to the speech recognition server.

[0936] 3. Server (Speech Recognition)

[0937] Analyzes audio data and converts it to text.

[0938] 4. Server (Speech Recognition)

[0939] Save text data to a database.

[0940] Execution of the Emotion Engine

[0941] 1. Emotional Engine

[0942] The system analyzes voice data and facial image data to determine the user's emotional state (e.g., joy, anger, sadness, etc.).

[0943] 2. Emotional Engine

[0944] The determined emotional state is generated as emotional information and stored in a database.

[0945] Execution of Generative AI

[0946] 1. Generation AI Server

[0947] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes both.

[0948] 2. Generation AI server → Terminal

[0949] The generated summary text and sentiment information are sent to the smart glasses device.

[0950] 3. Terminal

[0951] The smart glasses display a summary of the conversation and emotional information.

[0952] Specific example

[0953] Scenes involving dementia patients and their families

[0954] 1. User (Patient)

[0955] A dementia patient is wearing smart glasses. When a family member enters the room, the smart glasses' camera captures the family member's face.

[0956] 2. Terminal → Server (Facial Recognition)

[0957] The captured facial image is sent to the facial recognition server.

[0958] 3. Server (facial recognition)

[0959] The facial recognition server matches the family's facial images against a database and returns the result, "This person is family member AA."

[0960] 4. Server (facial recognition) → Terminal

[0961] Family members' names and past conversation history are sent to the smart glasses.

[0962] 5. Terminal

[0963] The smart glasses display the message, "This is family member AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[0964] 6. Terminal → Server (Speech Recognition)

[0965] The captured audio data is sent to the speech recognition server.

[0966] 7. Server (Speech Recognition)

[0967] Convert the conversation content into text and save it to a database.

[0968] 8. Emotional Engine

[0969] The system analyzes captured audio and facial expression data to determine the user's emotional state. For example, it might generate emotional information such as, "The user is currently excited."

[0970] 9. Emotion Engine → Database

[0971] Emotional information is stored in a database.

[0972] 10. Generation AI Server

[0973] The generating AI server analyzes new conversation data, past conversation data, and sentiment information to generate summaries such as, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited."

[0974] 11. Generation AI Server → Terminal

[0975] The generated summary is sent to the smart glasses.

[0976] 12. Terminal

[0977] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," making it easier for the user to understand the flow of the conversation.

[0978] In this way, the system of the present invention can enable smooth communication in service industries dealing with dementia patients and many customers, thereby improving the quality of life for users. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, enabling more appropriate responses.

[0979] The following describes the processing flow.

[0980] Step 1:

[0981] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[0982] Step 2:

[0983] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[0984] Step 3:

[0985] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[0986] Step 4:

[0987] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[0988] Step 5:

[0989] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[0990] Step 6:

[0991] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[0992] Step 7:

[0993] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[0994] Step 8:

[0995] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[0996] Step 9:

[0997] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[0998] Step 10:

[0999] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[1000] Step 11:

[1001] The server (speech recognition) saves the transcribed conversation to a database.

[1002] Step 12:

[1003] The emotion engine retrieves voice data and facial expression data from the database. The emotion engine then begins analyzing the voice and facial expressions.

[1004] Step 13:

[1005] The emotion engine determines the user's emotional state based on the analysis results. For example, it identifies one of the following emotional states: joy, anger, sadness, or pleasure.

[1006] Step 14:

[1007] The emotion engine generates emotion information based on the determined emotional state and stores that information in a database.

[1008] Step 15:

[1009] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them.

[1010] Step 16:

[1011] The generation AI server fuses past conversation data, new conversation data, and sentiment information to generate a summary. This summary includes key points from the current conversation, references to past information, and the user's emotional state.

[1012] Step 17:

[1013] The generated summary text and sentiment information are sent from the generating AI server to the terminal.

[1014] Step 18:

[1015] The device displays the summary and sentiment information it receives on the smart glasses' display. This allows the user to understand, "In our last conversation we talked about AA. This time we're talking about CC, and the user is currently excited."

[1016] This series of processes utilizes facial recognition, voice recognition, generative AI, and an emotion engine to enable smooth communication in service industries dealing with dementia patients and large customer bases.

[1017] (Example 2)

[1018] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1019] Conventional systems provided limited memory support information and lacked information based on the user's emotional state. This made it difficult to improve the user's quality of life and facilitate smooth communication. Furthermore, it was challenging to provide appropriate support for dementia patients and service industries dealing with large customer bases.

[1020] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1021] In this invention, the server includes means for capturing the faces of people within its field of view using a camera and transmitting them to the server, means including a microphone for capturing and transmitting audio to the server, means equipped with a display for displaying the generated information, server means for analyzing face data and audio data and returning the recognized information, emotion engine means for analyzing audio data and face image data and determining the user's emotional state, and generation AI server means for retrieving newly stored text data, past conversation data, and emotional information from a database and generating summary text and emotional information. This makes it possible to provide memory support information and information based on emotional state.

[1022] A "camera" is a device that captures images and acquires that digital data.

[1023] A "server" is a computer system that operates on a network and provides data processing and storage services.

[1024] A "microphone" is a device that captures sound and acquires its digital data.

[1025] A "display" is a device that displays visual information.

[1026] "Facial recognition" is a technology that analyzes captured facial images to identify specific individuals.

[1027] "Speech recognition" is a technology that analyzes captured audio and converts it into text data.

[1028] An "emotion engine" is a technology that analyzes voice data and facial image data to determine the user's emotional state.

[1029] A "database" is a system for systematically storing and managing data.

[1030] A "generating AI server" is a server equipped with artificial intelligence that generates summarized text and sentiment information using stored data.

[1031] "Memory support information" refers to information that supports a user's memory by referencing specific records or histories.

[1032] A "summary text" is a document that concisely summarizes multiple pieces of information.

[1033] This invention relates to a system combining facial recognition means, speech recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. The system consists of smart glasses terminals, a facial recognition server, a speech recognition server, a generation AI server, an emotion engine, and a database as its main components.

[1034] Performing facial recognition

[1035] Hardware and software to be used

[1036] Smart Glasses Device

[1037] Face recognition server

[1038] Camera (high resolution)

[1039] OpenCV (face recognition library)

[1040] operation

[1041] The smart glasses device captures the faces of people within the user's field of view via its camera and sends them to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database, and identifies specific individuals. The results are sent back to the smart glasses device, and the person's name and related information are displayed on the screen.

[1042] Executing speech recognition

[1043] Hardware and software to be used

[1044] Smart Glasses Device

[1045] Speech recognition server

[1046] microphone

[1047] Google Speech-to-Text API

[1048] operation

[1049] The microphone on the smart glasses device captures the conversation audio in real time and sends it to a speech recognition server. The speech recognition server converts the audio data into text and stores it in a database.

[1050] Execution of the Emotion Engine

[1051] Hardware and software to be used

[1052] Emotional Engine

[1053] Emotion analysis algorithms (voice tone analysis, facial expression analysis)

[1054] operation

[1055] The emotion engine analyzes voice data and facial image data to determine the user's emotional state. The determined emotional state is then stored in a database as emotion information.

[1056] Execution of Generative AI

[1057] Hardware and software to be used

[1058] AI generation server

[1059] GPT-4 (Natural Language Processing Model)

[1060] operation

[1061] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes them, and generates summarized text and sentiment information. This generated information is sent to the smart glasses terminal and displayed on the screen.

[1062] Specific example

[1063] For example, if a dementia patient is wearing smart glasses, when a family member enters the room, the smart glasses' camera captures the family member's face and sends it to a facial recognition server. The facial recognition server compares the family member's face image with a database and sends the result, "This person is family member AA," back to the smart glasses terminal. The smart glasses then display, "This person is family member AA. We talked about BB during the last visit."

[1064] Additionally, once a conversation begins, the smart glasses' microphone captures the audio and sends it to a speech recognition server. The speech recognition server converts the conversation into text and stores it in a database. The emotion engine determines the user's emotional state from the audio and facial expression data, generates emotion information such as "The user is currently excited," and stores it in the database. The generating AI server analyzes the new conversation data, past conversation data, and emotion information to generate a summary such as "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," and sends it to the smart glasses for display.

[1065] Example of a prompt

[1066] "Last time we talked about BB, but the new topic we discussed this time is CC. Users are currently very excited about it."

[1067] This system enables smooth communication in service industries dealing with dementia patients and large customer bases, thereby improving the user's quality of life. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, leading to more appropriate responses.

[1068] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1069] Step 1: Face Capture

[1070] terminal

[1071] The smart glasses' camera captures the faces of people within the user's field of view in real time. The input data is images of people's faces within the field of view, and a high-resolution camera is used to obtain clear images. The output is the captured face image.

[1072] Step 2: Send your face image

[1073] Terminal → Server (Facial Recognition)

[1074] The captured facial image is sent to a facial recognition server. The facial image data is compressed and transferred to the facial recognition server in real time using a secure communication protocol (e.g., HTTPS). The input data is the captured facial image, and the output is the facial image data sent to the facial recognition server.

[1075] Step 3: Analysis of facial data

[1076] Server (facial recognition)

[1077] The face recognition server analyzes the received face image. The software used is a face recognition library such as OpenCV, and the input data is the transmitted face image. This analysis extracts facial feature points. The output is face feature data.

[1078] Step 4: Match with database

[1079] Server (facial recognition)

[1080] The analyzed facial data is compared with a database. A specific matching algorithm is used to identify individuals whose facial features match those in the existing database. The input data is facial feature data, and the output is information about the identified individuals.

[1081] Step 5: Submit the identification result

[1082] Server (facial recognition) → Terminal

[1083] The server sends the name of the identified person and past conversation details back to the smart glasses device. The server encrypts and transmits this information. The input data is the identified person's information, and the output is the identification information sent to the device.

[1084] Step 6: Displaying Identification Information

[1085] terminal

[1086] The smart glasses display the name and related information of the recognized person. The input data is identification information, and the output is the information displayed on the screen.

[1087] Step 7: Capture audio

[1088] User

[1089] When a user starts a conversation, the smart glasses' microphone captures the audio in real time. The input data is the conversation audio, and the output is the captured audio data.

[1090] Step 8: Sending audio data

[1091] Terminal → Server (Speech Recognition)

[1092] The captured audio data is sent to the speech recognition server. Before transmission, the data is converted to an appropriate format (e.g., WAV format). The input data is the audio data, and the output is the data sent to the speech recognition server.

[1093] Step 9: Analyzing the audio data

[1094] Server (Speech Recognition)

[1095] A speech recognition server analyzes the audio data and converts it to text. The input data is audio, and the software used is the Google Speech-to-Text API. The output is text data.

[1096] Step 10: Saving the text data

[1097] Server (Speech Recognition)

[1098] The data is converted to text and saved to a database. The input data is the converted text data, which is saved with a timestamp. The output is the text data saved in the database.

[1099] Step 11: Analysis of emotional state

[1100] Emotional Engine

[1101] This system analyzes voice and facial image data to determine the user's emotional state. Based on an emotion analysis algorithm, the input data consists of voice and facial image data. The output is the determined emotion information.

[1102] Step 12: Preserving emotional information

[1103] Emotional Engine

[1104] The determined emotional state is stored in a database as emotional information. The input data is emotional information, and the output is the emotional information stored in the database.

[1105] Step 13: Summarizing the conversation

[1106] AI generation server

[1107] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them. The technology used is GPT-4. The input data consists of text data, past conversation data, and sentiment information, and the output is summarized text and sentiment information.

[1108] Step 14: Sending generated information

[1109] AI generation server → Terminal

[1110] The generated summary text and sentiment information are sent to the smart glasses device. Encryption is used to maintain security. The input data consists of the summary text and sentiment information; the output data is the data sent to the smart glasses device.

[1111] Step 15: Display summary information

[1112] terminal

[1113] The smart glasses display a summarized conversation and sentiment information. The input data is the summarized text and sentiment information, and the output is the information displayed on the screen.

[1114] (Application Example 2)

[1115] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1116] Traditional in-store customer service systems only used facial and voice recognition, resulting in insufficient personalized service. Furthermore, it was difficult to grasp customers' emotional states in real time, hindering the ability to take prompt and appropriate action in situations requiring special attention.

[1117] In Application Example 2, the identification processing performed by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for analyzing emotional states, means for generating summaries based on past conversation content and emotional information, means for capturing the faces of people within the field of view using a camera and transmitting them to the server, and means for capturing conversations using a microphone and transmitting them to the recognition server. This enables the provision of information based on the customer's emotional state and past conversation history, allowing for the rapid and appropriate provision of personalized services.

[1118] "Means of performing facial recognition" refers to technology that analyzes and identifies a person's face image captured by a camera.

[1119] "Means of performing speech recognition" refers to technology for converting audio data captured by a microphone into text format and then analyzing it.

[1120] "Means for displaying generated information" refers to display and interface technologies for visually providing analyzed and processed data to the user.

[1121] "Means for analyzing emotional states" refers to technologies that use voice data and facial image data to determine the emotional state of a subject (such as joy, anger, sadness, etc.).

[1122] "Means for generating summaries based on past conversation content and sentiment information" refers to a generative AI model that analyzes past conversation content and sentiment information in a database to generate important information and summaries.

[1123] A "system that provides personalized service information" is a set of technologies that provide information and services to individual customers based on their individual needs and past history.

[1124] To carry out this invention, a system comprising the following main elements is required. The system comprises the following components.

[1125] System Configuration Details

[1126] 1. Smart glasses device

[1127] 2. Face recognition server

[1128] 3. Speech recognition server

[1129] 4. Generation AI Server

[1130] 5. Emotional Engine

[1131] 6. Database

[1132] Hardware and software

[1133] Hardware: Smart glasses (e.g., Microsoft HoloLens), Servers (high-performance cloud servers)

[1134] Software: Face recognition software (e.g., Amazon Rekognition), speech recognition software (e.g., Google Speech-to-Text), sentiment analysis software (e.g., Microsoft Azure's Text Analytics for sentiment analysis), generative AI models (e.g., OpenAI's GPT-4)

[1135] System program processing details

[1136] Face recognition processing

[1137] The smart glasses device captures the faces of people within the user's field of view using its camera. The captured facial images are sent to a facial recognition server in real time or in batch processing. This facial recognition server performs image analysis and identifies specific individuals by comparing them with existing data in its database. The person's name and related information are sent back to the smart glasses device and displayed to the user.

[1138] Speech recognition processing

[1139] The smart glasses' microphone captures conversations between the user and customers in real time and sends the audio data to a speech recognition server. This server converts the audio data into text and stores the text data in a database.

[1140] Emotional Engine Processing

[1141] The emotion engine analyzes voice and facial image data to determine the user's emotional state. The determined emotional state is stored in a database and used by the generating AI server.

[1142] Processing of generated AI models

[1143] The AI ​​server analyzes newly stored text data, past conversation data, and sentiment information to generate conversation summaries and key information. The generated information is sent to the smart glasses terminal and provided to the user.

[1144] Specific example

[1145] Examples of customer service in physical stores

[1146] When a user wears smart glasses and captures the face of a customer entering the store, a facial recognition server identifies the customer's name and past visit history. For example, information such as, "This customer is Mr. / Ms. Tanaka. During their last visit, they showed interest in allergy-free products," is displayed. When the customer and the user begin a conversation, the conversation is recorded, and after it ends, a summary such as, "Mr. / Ms. Tanaka is interested in new organic products," is generated. Furthermore, an emotion engine analyzes the customer's emotional state in real time and provides information such as, "The customer is excited," enabling the user to respond appropriately.

[1147] Example of a prompt

[1148] Examples of prompts for a generative AI model are as follows:

[1149] prompt:

[1150] Past conversation data: {conversation data}

[1151] Latest conversation: {New conversation}

[1152] Past emotional information: {Past emotional information}

[1153] New emotional state: {New emotional state}

[1154] output:

[1155] 1. Summary of the latest conversation

[1156] 2. Customer concerns and proposed solutions

[1157] Through this format, personalized services can be provided in physical stores, significantly improving the quality of customer service.

[1158] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1159] Step 1:

[1160] The smart glasses' camera captures the faces of people within the user's field of view. The captured face images are sent from the smart glasses device to a face recognition server. At this point, the captured image serves as the input, and the transmission process is initiated.

[1161] Step 2:

[1162] The facial recognition server analyzes the received facial image and compares it with existing data in the database to identify a specific person. The input is a facial image, and the output is the name of the identified person and related information. The process involves facial image analysis and database matching.

[1163] Step 3:

[1164] The facial recognition server sends the name and related information of the identified person back to the smart glasses terminal. The output information includes the identified person's name and past conversation content. The user then receives the information displayed on the screen.

[1165] Step 4:

[1166] The smart glasses' microphone captures the conversation between the user and the customer in real time. The captured audio data is sent from the smart glasses device to the speech recognition server. At this point, the captured audio data is the input, and the transmission process is performed.

[1167] Step 5:

[1168] The speech recognition server analyzes the received audio data and converts it into text format. The input is audio data, and the output is text data. Audio analysis and text conversion processes are performed.

[1169] Step 6:

[1170] The text data generated by the speech recognition server is saved to the database. The text data is used as input for the database registration process.

[1171] Step 7:

[1172] The emotion engine analyzes voice data and facial image data in the database to determine the user's emotional state. The input is voice data and facial image data, and the output is the emotional state. Emotion analysis processing is then performed.

[1173] Step 8:

[1174] The emotion engine determines the emotional state, generates emotion information, and stores it in the database. The emotional state is the input, and the process of generating emotion information and registering it in the database is performed.

[1175] Step 9:

[1176] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes both, and generates a summary. The input consists of text data, past conversation data, and sentiment information, while the output is summarized text information. Data analysis and summary generation processes are performed.

[1177] Step 10:

[1178] The generation AI server sends the generated summary text and sentiment information to the smart glasses terminal. The output consists of the summary text and sentiment information, and the process of sending them to the smart glasses is performed.

[1179] Step 11:

[1180] The smart glasses display shows a summarized conversation and emotional information. Based on this, the user can provide personalized service information to the customer. The displayed information is the output.

[1181] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1182] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1183] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1184] [Third Embodiment]

[1185] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1186] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1187] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1188] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1189] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1190] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1191] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1192] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1193] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1194] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1195] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1196] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1197] This invention relates to a system that provides memory support information by combining facial recognition means, voice recognition means, and means for displaying generated information. A specific embodiment of this system, including its program and processing, is described below.

[1198] System Configuration

[1199] This system consists of the following main elements:

[1200] 1. Smart glasses device

[1201] 2. Face recognition server

[1202] 3. Speech recognition server

[1203] 4. Generation AI Server

[1204] 5. Database

[1205] Program processing

[1206] Performing facial recognition

[1207] 1. Terminal

[1208] The smart glasses' camera captures the faces of people within the user's field of view.

[1209] 2. Terminal → Server (Facial Recognition)

[1210] The captured facial image is sent to a facial recognition server.

[1211] 3. Server (facial recognition)

[1212] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[1213] 4. Server (facial recognition) → Terminal

[1214] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[1215] 5. Terminal

[1216] The smart glasses display the name of the recognized person.

[1217] Executing speech recognition

[1218] 1. Terminal

[1219] The smart glasses' microphone continuously captures the audio of the conversation.

[1220] 2. Terminal → Server (Speech Recognition)

[1221] The captured audio data is sent to the speech recognition server.

[1222] 3. Server (Speech Recognition)

[1223] Convert audio data to text.

[1224] 4. Server (Speech Recognition)

[1225] Save text data to a database.

[1226] Execution of Generative AI

[1227] 1. Generation AI Server

[1228] The generation AI server analyzes and merges past conversation data with newly acquired conversation data.

[1229] 2. Generation AI server → Terminal

[1230] The generated summary text is sent to the smart glasses device.

[1231] 3. Terminal

[1232] The smart glasses display a summarized version of the conversation.

[1233] Specific example

[1234] Scenes involving dementia patients and their caregivers

[1235] 1. User (Patient)

[1236] A dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face.

[1237] 2. Terminal → Server (Facial Recognition)

[1238] The captured facial image is sent to the facial recognition server.

[1239] 3. Server (facial recognition)

[1240] The facial recognition server matches the helper's facial image against the database and returns the result, "This person is helper AA."

[1241] 4. Server (facial recognition) → Terminal

[1242] The helper's name and past conversation history are sent to the smart glasses.

[1243] 5. Terminal

[1244] The smart glasses display the message, "This is caregiver AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[1245] 6. Terminal → Server (Speech Recognition)

[1246] The captured audio data is sent to the speech recognition server.

[1247] 7. Server (Speech Recognition)

[1248] Convert the conversation content into text and save it to a database.

[1249] 8. Generation AI Server

[1250] The generating AI server analyzes new conversation data and past conversation data to generate a summary such as, "Last time we talked about BB, but the new topic mentioned this time is CC."

[1251] 9. Generation AI Server → Terminal

[1252] The generated summary is sent to the smart glasses.

[1253] 10. Terminal

[1254] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC," making it easier for the user to understand the flow of the conversation.

[1255] In this way, the system of the present invention can facilitate communication with dementia patients and in service industries dealing with many customers, thereby improving the quality of life for users.

[1256] The following describes the processing flow.

[1257] Step 1:

[1258] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[1259] Step 2:

[1260] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[1261] Step 3:

[1262] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[1263] Step 4:

[1264] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[1265] Step 5:

[1266] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[1267] Step 6:

[1268] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[1269] Step 7:

[1270] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[1271] Step 8:

[1272] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[1273] Step 9:

[1274] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[1275] Step 10:

[1276] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[1277] Step 11:

[1278] The server (speech recognition) saves the transcribed conversation to a database.

[1279] Step 12:

[1280] The generating AI server retrieves newly saved text data and past conversation data from the database and analyzes both.

[1281] Step 13:

[1282] The generation AI server merges past and new conversation data to generate a summary. This summary includes key points from the current conversation and references to past information.

[1283] Step 14:

[1284] The generated summary text is sent from the generation AI server to the terminal.

[1285] Step 15:

[1286] The device displays a summary it has received on the smart glasses' screen. This allows the user to understand, "In our last conversation, we talked about AA. This time, we're talking about CC."

[1287] This series of processes enables smooth communication in service industries dealing with dementia patients and large numbers of customers by utilizing facial recognition, voice recognition, and generative AI.

[1288] (Example 1)

[1289] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1290] Conventional memory support systems have a drawback: they lack features to help users remember people and conversation content. In particular, there was a need for effective support in situations where memory assistance is required, such as for dementia patients and service industries that involve interaction with many customers. Furthermore, existing systems sometimes suffered from low accuracy in facial and voice recognition, resulting in inaccurate memory support information.

[1291] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1292] In this invention, the server includes means for the terminal to capture a person's face, means for the terminal to transmit the captured face image to a face recognition server, means for the face recognition server to analyze the face image and compare it with a database, means for the face recognition server to return identified person information to the terminal, means for the terminal to display the identified person information, means for the terminal to capture the audio of a conversation, means for the terminal to transmit the captured audio data to a speech recognition server, means for the speech recognition server to convert the audio data into text, means for the speech recognition server to store the text data in a database, means for the generating AI server to analyze and merge past conversation data and new conversation data, means for the generating AI server to transmit the generated summary text to the terminal, and means for the terminal to display the summarized conversation content. As a result, the user can instantly identify people and check past conversation content, enabling the conversation to proceed smoothly.

[1293] A "terminal" is a device that a user wears and operates, and in this invention, this specifically refers to smart glasses, etc.

[1294] A "face recognition server" is a server that analyzes facial images transmitted from a terminal, compares them with a database, and identifies the corresponding person.

[1295] A "speech recognition server" is a server that converts speech data sent from a terminal into text.

[1296] A "generative AI server" is a server that analyzes and merges past conversation data with new conversation data to generate summarized text.

[1297] A "database" is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[1298] A "camera" is a device built into a terminal that captures the faces of people within the user's field of view.

[1299] A "microphone" is an audio input device built into a terminal that captures the sound of conversations.

[1300] A "face image" is image data of a person's face captured by the device's camera.

[1301] "Audio data" refers to audio data of a conversation captured by the device's microphone.

[1302] "Text data" refers to character data converted from speech data by a speech recognition server.

[1303] A "summary text" is a summary that integrates past conversation content and new conversation content, generated by a generation AI server.

[1304] This invention relates to a system for providing memory support information to a user. This system combines facial recognition means, voice recognition means, and means for displaying generated information to provide effective memory support to the user.

[1305] System Configuration

[1306] This system consists of the following main elements:

[1307] 1. Terminal

[1308] This includes wearable devices such as smart glasses. These devices are equipped with cameras and microphones that capture the faces of people and the audio of conversations within the user's field of view.

[1309] 2. Face recognition server

[1310] The face recognition server analyzes face images sent from the terminal and matches them against a database to identify the corresponding person. It uses face recognition libraries such as OpenCV or AWS Rekognition.

[1311] 3. Speech recognition server

[1312] The speech recognition server converts the audio data sent from the terminal into text. It uses speech recognition services such as Google Cloud Speech-to-Text or IBM Watson Speech to Text.

[1313] 4. Generation AI Server

[1314] The generative AI server analyzes and merges past conversation data with newly acquired conversation data to generate summarized text. Examples of generative AI models include GPT-3 and BERT.

[1315] 5. Database

[1316] This is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[1317] Program processing

[1318] In this system, when the user wears smart glasses, a series of processes are performed, from facial recognition of a person to speech recognition of conversations, and then the generation and display of summarized text by a generative AI.

[1319] Facial recognition means

[1320] The device's camera captures the faces of people within the user's field of view and sends the facial images to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database to identify the corresponding individuals, and sends the results back to the device. The device then displays the returned person information on its screen.

[1321] Speech recognition means

[1322] The device's microphone captures the conversation audio and sends the audio data to a speech recognition server. The speech recognition server converts the audio data into text and stores the resulting text data in a database. This makes searching and analysis easier later on.

[1323] Generation AI means

[1324] The AI ​​server analyzes and merges previously stored conversation data with newly acquired conversation data to generate a summary text. The generated summary text is sent to the device and displayed on the device's screen. This allows the user to smoothly understand the content of past and current conversations.

[1325] Specific example

[1326] Consider a scenario where a dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face and sends the information to a server. The server sends back the relevant information, and the device displays, "This is caregiver AA. We talked about BB during the last visit." When a conversation begins, the smart glasses' microphone captures the audio and sends the audio data to the server. The server converts the audio to text and stores it in a database. Later, a generative AI server analyzes this data, generates a summary text, and displays it on the device.

[1327] An example of a prompt message would be: "Generate a program that analyzes facial images captured by smart glasses and displays information about the recognized person. Also, include a function to convert spoken conversation into text and summarize it."

[1328] Thus, by providing memory support information, the present invention can facilitate communication in service industries that interact with dementia patients and many customers, thereby improving the quality of life for users.

[1329] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1330] Step 1:

[1331] terminal

[1332] The smart glasses' camera captures the faces of people within the user's field of view in high resolution. The input is a person's face, which is saved as a still image using the camera. The output is the captured face image. This face image data is used in the next step.

[1333] Step 2:

[1334] Terminal → Server (Facial Recognition)

[1335] The terminal captures a face image and sends it to a face recognition server using a secure communication protocol (e.g., HTTPS). The input is the face image data obtained in step 1. The output is the face image data sent to the face recognition server.

[1336] Step 3:

[1337] Server (facial recognition)

[1338] The face recognition server analyzes the received face image and extracts feature points. Specifically, the relevant system (e.g., OpenCV or AWS Rekognition) receives face image data as input and outputs the coordinates of feature points and other identification information. Next, these feature points are compared with existing face data stored in the database to identify the corresponding person. As a result of the comparison, person information is output.

[1339] Step 4:

[1340] Server (facial recognition) → Terminal

[1341] The facial recognition server sends back relevant information to the terminal, such as the name of the identified person and past conversation content. The input is the identified person information data, and the output is the person information data sent to the terminal.

[1342] Step 5:

[1343] terminal

[1344] The device displays the person's information, which has been returned, on the smart glasses' display. The input is person information data received from the server, and the output is display information that the user can visually confirm (e.g., "This person is helper AA. We talked about BB during the last visit").

[1345] Step 6:

[1346] terminal

[1347] The smart glasses' microphone continuously captures the conversation audio. The input is the conversational audio, which is saved as audio data using the microphone. The output is the captured audio data, which is used in the next step.

[1348] Step 7:

[1349] Terminal → Server (Speech Recognition)

[1350] The terminal transmits the captured audio data to the speech recognition server in real time. The input is the audio data obtained in step 6, which is transmitted via a secure communication protocol. The output is the audio data sent to the speech recognition server.

[1351] Step 8:

[1352] Server (Speech Recognition)

[1353] The speech recognition server converts the audio data into text. Specifically, it uses Google Cloud Speech-to-Text or IBM Watson Speech to Text to receive audio data as input and generate text data as output.

[1354] Step 9:

[1355] Server (Speech Recognition)

[1356] The converted text data is saved to a database. The input is the generated text data, and the output is the text data saved in the database.

[1357] Step 10:

[1358] AI generation server

[1359] The generative AI server analyzes and fuses past conversation data with newly acquired conversation data. The input is stored text data, which is analyzed using generative AI models such as GPT-3 and BERT to generate a summarized text. The output is the generated summarized text.

[1360] Step 11:

[1361] AI generation server → Terminal

[1362] The AI ​​server generates a summary text and sends it to the terminal. The input is the summary text, which is sent via a secure communication protocol. The output is the summary text that was sent to the terminal.

[1363] Step 12:

[1364] terminal

[1365] The device displays the summarized text on the smart glasses' display. The input is the summarized text received from the generating AI server, and the output is summarized information that the user can visually confirm (e.g., "Last time we talked about BB, but the new topic mentioned this time is CC"). This display allows the user to smoothly understand past and current conversations.

[1366] (Application Example 1)

[1367] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1368] In traditional brick-and-mortar stores, it was difficult for employees to instantly recognize customers' faces and access their past interactions and purchase history. This could lead to communication breakdowns and decreased customer satisfaction. Furthermore, a lack of information to appropriately suggest new products was also a problem.

[1369] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1370] In this invention, the server includes means for facial recognition, means for voice recognition, means for displaying generated information, means for providing memory support information, means for customer recognition, means for providing a summary of conversation content, and means for suggesting recommended products. This enables the recognition of the customer's face, instant display of past conversations and purchase history, and facilitates effective communication and appropriate product suggestions.

[1371] "Face recognition means" refers to a technology that uses a camera to capture the faces of people within its field of view and transmits them to a server.

[1372] "Speech recognition means" refers to a technology that captures conversations using a microphone and transmits them to a recognition server.

[1373] "Means for displaying generated information" refers to technologies that display information obtained from facial recognition or voice recognition on a display.

[1374] "Methods for providing memory support information" refers to technologies that provide memory support to users based on information such as past conversation data and purchase history.

[1375] "Customer recognition means" refers to technology that captures a customer's face and identifies and displays their past interactions and purchase history.

[1376] A "means for providing summaries of conversation content" is a technology that analyzes past conversation data and new conversation data and displays it as a summary.

[1377] "Recommended product suggestions" refers to techniques that suggest the most suitable products based on a customer's past purchase history and conversations.

[1378] The system for carrying out this invention consists of the following main elements.

[1379] System Configuration

[1380] 1. Smart Glasses

[1381] 2. Face recognition server

[1382] 3. Speech recognition server

[1383] 4. Generation AI Server

[1384] 5. Database

[1385] Details of the embodiment

[1386] Hardware configuration

[1387] 1. Smart Glasses

[1388] It has a built-in camera and microphone, and captures the faces of people within the user's field of view.

[1389] The system displays information generated through the display to the user.

[1390] 2. Server cluster

[1391] Face recognition server: Analyzes captured facial images and matches them against a database to identify individuals.

[1392] Speech recognition server: Converts captured audio data into text.

[1393] Generation AI Server: Analyzes and merges past and new conversation data to generate summarized information.

[1394] Database: Stores customer facial images, past conversation data, purchase history, etc.

[1395] System program

[1396] 1. Customer awareness

[1397] The smart glasses' camera captures the faces of people within the user's field of vision.

[1398] The facial image is sent to a facial recognition server and compared against a database.

[1399] The relevant person's information (name and past purchase history) is sent to the smart glasses and displayed on the screen.

[1400] 2. Speech Recognition

[1401] The microphone in the smart glasses captures the audio of the conversation.

[1402] The audio data is sent to a speech recognition server and converted into text.

[1403] The converted text data is stored in the database.

[1404] 3. Providing a summary of the conversation.

[1405] The generation AI server analyzes past and new conversation data to generate a summary text.

[1406] The generated summary text is sent to the smart glasses and displayed on the screen.

[1407] 4. Suggestions for recommended products

[1408] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[1409] Information about the suggested products is displayed on the smart glasses.

[1410] Specific example

[1411] 1. Customer service scenarios

[1412] Employees wear smart glasses to capture the faces of customers who come into the store.

[1413] A facial image is sent to a facial recognition server, and the server returns information such as, "This person is customer A. They purchased a vacuum cleaner last time."

[1414] The smart glasses display the message, "This person is Customer A. They purchased a vacuum cleaner last time."

[1415] The conversation audio is captured and sent to a speech recognition server.

[1416] The audio data is converted to text and stored in a database.

[1417] The AI ​​server analyzes the old and new conversation data and generates a summary such as, "Last time we talked about vacuum cleaners. The new topic is washing machines."

[1418] The generated summary is displayed on smart glasses.

[1419] Examples of prompts to input into a generative AI model

[1420] Customer data: {"Name": "Customer A", "Previously purchased items": ["Vacuum cleaner"], "Previous conversation content": ["About the vacuum cleaner's functions"]}

[1421] New conversation topic: "I'd like to ask for advice about washing machines."

[1422] Output: "Last time we talked about vacuum cleaners, but the new topic mentioned this time is washing machines."

[1423] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1424] Step 1:

[1425] The smart glasses' camera captures the faces of people within the user's field of vision.

[1426] Input: Camera video (face image)

[1427] Output: Captured facial image

[1428] Specific operation: The smart glasses detect the faces of people within the user's field of vision and capture those faces with the camera.

[1429] Step 2:

[1430] The captured facial image is sent to a facial recognition server.

[1431] Input: Captured facial image

[1432] Output: Notification of successful image transmission to face recognition server

[1433] Specific operation: The smart glasses use wireless communication to send the captured facial image to a facial recognition server.

[1434] Step 3:

[1435] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[1436] Input: Face image

[1437] Output: Information about the identified person (name, past purchase history)

[1438] Specific operation: The face recognition server analyzes the face image and compares it with the person's record stored in the database.

[1439] Step 4:

[1440] The information of the person in question is sent to the smart glasses.

[1441] Input: Information of an identified person

[1442] Output: Instructions for displaying information on smart glasses

[1443] Specific operation: The facial recognition server sends information about the identified person (name, past purchase history) to the smart glasses.

[1444] Step 5:

[1445] The smart glasses display information about the recognized person.

[1446] Input: Information of an identified person

[1447] Output: Person information displayed on the screen

[1448] Specific operation: The smart glasses display the person information they receive on the screen, allowing the user to verify it.

[1449] Step 6:

[1450] The microphone in the smart glasses captures the audio of the conversation.

[1451] Input: Conversation audio

[1452] Output: Captured audio data

[1453] Specific operation: The smart glasses use the microphone to record the audio of the conversation.

[1454] Step 7:

[1455] The captured audio data is sent to the speech recognition server.

[1456] Input: Captured audio data

[1457] Output: Notification of successful transmission of voice data to the speech recognition server

[1458] Specific operation: The smart glasses use wireless communication to send the captured audio data to the speech recognition server.

[1459] Step 8:

[1460] The speech recognition server converts the speech data into text and stores it in a database.

[1461] Input: Audio data

[1462] Output: Text data

[1463] Specific operation: The speech recognition server converts speech data into text using speech recognition technology and saves the converted text data to a database.

[1464] Step 9:

[1465] The generation AI server analyzes past and new conversation data to generate a summary text.

[1466] Input: Past conversation data, new conversation data

[1467] Output: Summary text

[1468] Specific operation: The generative AI server uses a generative AI model to analyze past and recent conversation data, extract key points, and generate a summarized text.

[1469] Step 10:

[1470] The generated summary text is sent to smart glasses.

[1471] Input: Summary text

[1472] Output: Instructions to display summarized text on smart glasses

[1473] Specific operation: The generation AI server sends the generated summary text to the smart glasses, and the smart glasses display it on the screen.

[1474] Step 11:

[1475] The smart glasses display a summarized version of the conversation.

[1476] Input: Summary text

[1477] Output: Summary information displayed on the screen

[1478] Specific action: The smart glasses display a summary text on the screen for the user to review.

[1479] Step 12:

[1480] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[1481] Input: Past purchase history, conversation content

[1482] Output: Proposed product information

[1483] Specific operation: The generation AI server uses a generation AI model based on purchase history and conversation content to suggest the most suitable products.

[1484] Step 13:

[1485] Information about the suggested products is displayed on smart glasses.

[1486] Input: Proposed product information

[1487] Output: Suggested product information displayed on the screen

[1488] Specific operation: The smart glasses display suggested product information on the screen for the user to review.

[1489] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1490] This invention relates to a system combining facial recognition means, voice recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. A specific embodiment of this system, including its program and processing, is described below.

[1491] System Configuration

[1492] This system consists of the following main elements:

[1493] 1. Smart glasses device

[1494] 2. Face recognition server

[1495] 3. Speech recognition server

[1496] 4. Generation AI Server

[1497] 5. Emotional Engine

[1498] 6. Database

[1499] Program processing

[1500] Performing facial recognition

[1501] 1. Terminal

[1502] The smart glasses' camera captures the faces of people within the user's field of view.

[1503] 2. Terminal → Server (Facial Recognition)

[1504] The captured facial image is sent to a facial recognition server.

[1505] 3. Server (facial recognition)

[1506] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[1507] 4. Server (facial recognition) → Terminal

[1508] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[1509] 5. Terminal

[1510] The smart glasses display the name of the recognized person.

[1511] Executing speech recognition

[1512] 1. Terminal

[1513] The smart glasses' microphone captures the conversation audio in real time and temporarily stores that audio data.

[1514] 2. Terminal → Server (Speech Recognition)

[1515] The captured audio data is sent to the speech recognition server.

[1516] 3. Server (Speech Recognition)

[1517] Analyzes audio data and converts it to text.

[1518] 4. Server (Speech Recognition)

[1519] Save text data to a database.

[1520] Execution of the Emotion Engine

[1521] 1. Emotional Engine

[1522] The system analyzes voice data and facial image data to determine the user's emotional state (e.g., joy, anger, sadness, etc.).

[1523] 2. Emotional Engine

[1524] The determined emotional state is generated as emotional information and stored in a database.

[1525] Execution of Generative AI

[1526] 1. Generation AI Server

[1527] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes both.

[1528] 2. Generation AI server → Terminal

[1529] The generated summary text and sentiment information are sent to the smart glasses device.

[1530] 3. Terminal

[1531] The smart glasses display a summary of the conversation and emotional information.

[1532] Specific example

[1533] Scenes involving dementia patients and their families

[1534] 1. User (Patient)

[1535] A dementia patient is wearing smart glasses. When a family member enters the room, the smart glasses' camera captures the family member's face.

[1536] 2. Terminal → Server (Facial Recognition)

[1537] The captured facial image is sent to the facial recognition server.

[1538] 3. Server (facial recognition)

[1539] The facial recognition server matches the family's facial images against a database and returns the result, "This person is family member AA."

[1540] 4. Server (facial recognition) → Terminal

[1541] Family members' names and past conversation history are sent to the smart glasses.

[1542] 5. Terminal

[1543] The smart glasses display the message, "This is family member AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[1544] 6. Terminal → Server (Speech Recognition)

[1545] The captured audio data is sent to the speech recognition server.

[1546] 7. Server (Speech Recognition)

[1547] Convert the conversation content into text and save it to a database.

[1548] 8. Emotional Engine

[1549] The system analyzes captured audio and facial expression data to determine the user's emotional state. For example, it might generate emotional information such as, "The user is currently excited."

[1550] 9. Emotion Engine → Database

[1551] Emotional information is stored in a database.

[1552] 10. Generation AI Server

[1553] The generating AI server analyzes new conversation data, past conversation data, and sentiment information to generate summaries such as, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited."

[1554] 11. Generation AI Server → Terminal

[1555] The generated summary is sent to the smart glasses.

[1556] 12. Terminal

[1557] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," making it easier for the user to understand the flow of the conversation.

[1558] In this way, the system of the present invention can enable smooth communication in service industries dealing with dementia patients and many customers, thereby improving the quality of life for users. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, enabling more appropriate responses.

[1559] The following describes the processing flow.

[1560] Step 1:

[1561] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[1562] Step 2:

[1563] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[1564] Step 3:

[1565] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[1566] Step 4:

[1567] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[1568] Step 5:

[1569] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[1570] Step 6:

[1571] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[1572] Step 7:

[1573] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[1574] Step 8:

[1575] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[1576] Step 9:

[1577] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[1578] Step 10:

[1579] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[1580] Step 11:

[1581] The server (speech recognition) saves the transcribed conversation to a database.

[1582] Step 12:

[1583] The emotion engine retrieves voice data and facial expression data from the database. The emotion engine then begins analyzing the voice and facial expressions.

[1584] Step 13:

[1585] The emotion engine determines the user's emotional state based on the analysis results. For example, it identifies one of the following emotional states: joy, anger, sadness, or pleasure.

[1586] Step 14:

[1587] The emotion engine generates emotion information based on the determined emotional state and stores that information in a database.

[1588] Step 15:

[1589] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them.

[1590] Step 16:

[1591] The generation AI server fuses past conversation data, new conversation data, and sentiment information to generate a summary. This summary includes key points from the current conversation, references to past information, and the user's emotional state.

[1592] Step 17:

[1593] The generated summary text and sentiment information are sent from the generating AI server to the terminal.

[1594] Step 18:

[1595] The device displays the summary and sentiment information it receives on the smart glasses' display. This allows the user to understand, "In our last conversation we talked about AA. This time we're talking about CC, and the user is currently excited."

[1596] This series of processes utilizes facial recognition, voice recognition, generative AI, and an emotion engine to enable smooth communication in service industries dealing with dementia patients and large customer bases.

[1597] (Example 2)

[1598] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1599] Conventional systems provided limited memory support information and lacked information based on the user's emotional state. This made it difficult to improve the user's quality of life and facilitate smooth communication. Furthermore, it was challenging to provide appropriate support for dementia patients and service industries dealing with large customer bases.

[1600] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1601] In this invention, the server includes means for capturing the faces of people within its field of view using a camera and transmitting them to the server, means including a microphone for capturing and transmitting audio to the server, means equipped with a display for displaying the generated information, server means for analyzing face data and audio data and returning the recognized information, emotion engine means for analyzing audio data and face image data and determining the user's emotional state, and generation AI server means for retrieving newly stored text data, past conversation data, and emotional information from a database and generating summary text and emotional information. This makes it possible to provide memory support information and information based on emotional state.

[1602] A "camera" is a device that captures images and acquires that digital data.

[1603] A "server" is a computer system that operates on a network and provides data processing and storage services.

[1604] A "microphone" is a device that captures sound and acquires its digital data.

[1605] A "display" is a device that displays visual information.

[1606] "Facial recognition" is a technology that analyzes captured facial images to identify specific individuals.

[1607] "Speech recognition" is a technology that analyzes captured audio and converts it into text data.

[1608] An "emotion engine" is a technology that analyzes voice data and facial image data to determine the user's emotional state.

[1609] A "database" is a system for systematically storing and managing data.

[1610] A "generating AI server" is a server equipped with artificial intelligence that generates summarized text and sentiment information using stored data.

[1611] "Memory support information" refers to information that supports a user's memory by referencing specific records or histories.

[1612] A "summary text" is a document that concisely summarizes multiple pieces of information.

[1613] This invention relates to a system combining facial recognition means, speech recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. The system consists of smart glasses terminals, a facial recognition server, a speech recognition server, a generation AI server, an emotion engine, and a database as its main components.

[1614] Performing facial recognition

[1615] Hardware and software to be used

[1616] Smart Glasses Device

[1617] Face recognition server

[1618] Camera (high resolution)

[1619] OpenCV (face recognition library)

[1620] operation

[1621] The smart glasses device captures the faces of people within the user's field of view via its camera and sends them to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database, and identifies specific individuals. The results are sent back to the smart glasses device, and the person's name and related information are displayed on the screen.

[1622] Executing speech recognition

[1623] Hardware and software to be used

[1624] Smart Glasses Device

[1625] Speech recognition server

[1626] microphone

[1627] Google Speech-to-Text API

[1628] operation

[1629] The microphone on the smart glasses device captures the conversation audio in real time and sends it to a speech recognition server. The speech recognition server converts the audio data into text and stores it in a database.

[1630] Execution of the Emotion Engine

[1631] Hardware and software to be used

[1632] Emotional Engine

[1633] Emotion analysis algorithms (voice tone analysis, facial expression analysis)

[1634] operation

[1635] The emotion engine analyzes voice data and facial image data to determine the user's emotional state. The determined emotional state is then stored in a database as emotion information.

[1636] Execution of Generative AI

[1637] Hardware and software to be used

[1638] AI generation server

[1639] GPT-4 (Natural Language Processing Model)

[1640] operation

[1641] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes them, and generates summarized text and sentiment information. This generated information is sent to the smart glasses terminal and displayed on the screen.

[1642] Specific example

[1643] For example, if a dementia patient is wearing smart glasses, when a family member enters the room, the smart glasses' camera captures the family member's face and sends it to a facial recognition server. The facial recognition server compares the family member's face image with a database and sends the result, "This person is family member AA," back to the smart glasses terminal. The smart glasses then display, "This person is family member AA. We talked about BB during the last visit."

[1644] Additionally, once a conversation begins, the smart glasses' microphone captures the audio and sends it to a speech recognition server. The speech recognition server converts the conversation into text and stores it in a database. The emotion engine determines the user's emotional state from the audio and facial expression data, generates emotion information such as "The user is currently excited," and stores it in the database. The generating AI server analyzes the new conversation data, past conversation data, and emotion information to generate a summary such as "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," and sends it to the smart glasses for display.

[1645] Example of a prompt

[1646] "Last time we talked about BB, but the new topic we discussed this time is CC. Users are currently very excited about it."

[1647] This system enables smooth communication in service industries dealing with dementia patients and large customer bases, thereby improving the user's quality of life. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, leading to more appropriate responses.

[1648] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1649] Step 1: Face Capture

[1650] terminal

[1651] The smart glasses' camera captures the faces of people within the user's field of view in real time. The input data is images of people's faces within the field of view, and a high-resolution camera is used to obtain clear images. The output is the captured face image.

[1652] Step 2: Send your face image

[1653] Terminal → Server (Facial Recognition)

[1654] The captured facial image is sent to a facial recognition server. The facial image data is compressed and transferred to the facial recognition server in real time using a secure communication protocol (e.g., HTTPS). The input data is the captured facial image, and the output is the facial image data sent to the facial recognition server.

[1655] Step 3: Analysis of facial data

[1656] Server (facial recognition)

[1657] The face recognition server analyzes the received face image. The software used is a face recognition library such as OpenCV, and the input data is the transmitted face image. This analysis extracts facial feature points. The output is face feature data.

[1658] Step 4: Match with database

[1659] Server (facial recognition)

[1660] The analyzed facial data is compared with a database. A specific matching algorithm is used to identify individuals whose facial features match those in the existing database. The input data is facial feature data, and the output is information about the identified individuals.

[1661] Step 5: Submit the identification result

[1662] Server (facial recognition) → Terminal

[1663] The server sends the name of the identified person and past conversation details back to the smart glasses device. The server encrypts and transmits this information. The input data is the identified person's information, and the output is the identification information sent to the device.

[1664] Step 6: Displaying Identification Information

[1665] terminal

[1666] The smart glasses display the name and related information of the recognized person. The input data is identification information, and the output is the information displayed on the screen.

[1667] Step 7: Capture audio

[1668] User

[1669] When a user starts a conversation, the smart glasses' microphone captures the audio in real time. The input data is the conversation audio, and the output is the captured audio data.

[1670] Step 8: Sending audio data

[1671] Terminal → Server (Speech Recognition)

[1672] The captured audio data is sent to the speech recognition server. Before transmission, the data is converted to an appropriate format (e.g., WAV format). The input data is the audio data, and the output is the data sent to the speech recognition server.

[1673] Step 9: Analyzing the audio data

[1674] Server (Speech Recognition)

[1675] A speech recognition server analyzes the audio data and converts it to text. The input data is audio, and the software used is the Google Speech-to-Text API. The output is text data.

[1676] Step 10: Saving the text data

[1677] Server (Speech Recognition)

[1678] The data is converted to text and saved to a database. The input data is the converted text data, which is saved with a timestamp. The output is the text data saved in the database.

[1679] Step 11: Analysis of emotional state

[1680] Emotional Engine

[1681] This system analyzes voice and facial image data to determine the user's emotional state. Based on an emotion analysis algorithm, the input data consists of voice and facial image data. The output is the determined emotion information.

[1682] Step 12: Preserving emotional information

[1683] Emotional Engine

[1684] The determined emotional state is stored in a database as emotional information. The input data is emotional information, and the output is the emotional information stored in the database.

[1685] Step 13: Summarizing the conversation

[1686] AI generation server

[1687] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them. The technology used is GPT-4. The input data consists of text data, past conversation data, and sentiment information, and the output is summarized text and sentiment information.

[1688] Step 14: Sending generated information

[1689] AI generation server → Terminal

[1690] The generated summary text and sentiment information are sent to the smart glasses device. Encryption is used to maintain security. The input data consists of the summary text and sentiment information; the output data is the data sent to the smart glasses device.

[1691] Step 15: Display summary information

[1692] terminal

[1693] The smart glasses display a summarized conversation and sentiment information. The input data is the summarized text and sentiment information, and the output is the information displayed on the screen.

[1694] (Application Example 2)

[1695] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1696] Traditional in-store customer service systems only used facial and voice recognition, resulting in insufficient personalized service. Furthermore, it was difficult to grasp customers' emotional states in real time, hindering the ability to take prompt and appropriate action in situations requiring special attention.

[1697] In Application Example 2, the identification processing performed by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for analyzing emotional states, means for generating summaries based on past conversation content and emotional information, means for capturing the faces of people within the field of view using a camera and transmitting them to the server, and means for capturing conversations using a microphone and transmitting them to the recognition server. This enables the provision of information based on the customer's emotional state and past conversation history, allowing for the rapid and appropriate provision of personalized services.

[1698] "Means of performing facial recognition" refers to technology that analyzes and identifies a person's face image captured by a camera.

[1699] "Means of performing speech recognition" refers to technology for converting audio data captured by a microphone into text format and then analyzing it.

[1700] "Means for displaying generated information" refers to display and interface technologies for visually providing analyzed and processed data to the user.

[1701] "Means for analyzing emotional states" refers to technologies that use voice data and facial image data to determine the emotional state of a subject (such as joy, anger, sadness, etc.).

[1702] "Means for generating summaries based on past conversation content and sentiment information" refers to a generative AI model that analyzes past conversation content and sentiment information in a database to generate important information and summaries.

[1703] A "system that provides personalized service information" is a set of technologies that provide information and services to individual customers based on their individual needs and past history.

[1704] To carry out this invention, a system comprising the following main elements is required. The system comprises the following components.

[1705] System Configuration Details

[1706] 1. Smart glasses device

[1707] 2. Face recognition server

[1708] 3. Speech recognition server

[1709] 4. Generation AI Server

[1710] 5. Emotional Engine

[1711] 6. Database

[1712] Hardware and software

[1713] Hardware: Smart glasses (e.g., Microsoft HoloLens), Servers (high-performance cloud servers)

[1714] Software: Face recognition software (e.g., Amazon Rekognition), speech recognition software (e.g., Google Speech-to-Text), sentiment analysis software (e.g., Microsoft Azure's Text Analytics for sentiment analysis), generative AI models (e.g., OpenAI's GPT-4)

[1715] System program processing details

[1716] Face recognition processing

[1717] The smart glasses device captures the faces of people within the user's field of view using its camera. The captured facial images are sent to a facial recognition server in real time or in batch processing. This facial recognition server performs image analysis and identifies specific individuals by comparing them with existing data in its database. The person's name and related information are sent back to the smart glasses device and displayed to the user.

[1718] Speech recognition processing

[1719] The smart glasses' microphone captures conversations between the user and customers in real time and sends the audio data to a speech recognition server. This server converts the audio data into text and stores the text data in a database.

[1720] Emotional Engine Processing

[1721] The emotion engine analyzes voice and facial image data to determine the user's emotional state. The determined emotional state is stored in a database and used by the generating AI server.

[1722] Processing of generated AI models

[1723] The AI ​​server analyzes newly stored text data, past conversation data, and sentiment information to generate conversation summaries and key information. The generated information is sent to the smart glasses terminal and provided to the user.

[1724] Specific example

[1725] Examples of customer service in physical stores

[1726] When a user wears smart glasses and captures the face of a customer entering the store, a facial recognition server identifies the customer's name and past visit history. For example, information such as, "This customer is Mr. / Ms. Tanaka. During their last visit, they showed interest in allergy-free products," is displayed. When the customer and the user begin a conversation, the conversation is recorded, and after it ends, a summary such as, "Mr. / Ms. Tanaka is interested in new organic products," is generated. Furthermore, an emotion engine analyzes the customer's emotional state in real time and provides information such as, "The customer is excited," enabling the user to respond appropriately.

[1727] Example of a prompt

[1728] Examples of prompts for a generative AI model are as follows:

[1729] prompt:

[1730] Past conversation data: {conversation data}

[1731] Latest conversation: {New conversation}

[1732] Past emotional information: {Past emotional information}

[1733] New emotional state: {New emotional state}

[1734] output:

[1735] 1. Summary of the latest conversation

[1736] 2. Customer concerns and proposed solutions

[1737] Through this format, personalized services can be provided in physical stores, significantly improving the quality of customer service.

[1738] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1739] Step 1:

[1740] The smart glasses' camera captures the faces of people within the user's field of view. The captured face images are sent from the smart glasses device to a face recognition server. At this point, the captured image serves as the input, and the transmission process is initiated.

[1741] Step 2:

[1742] The facial recognition server analyzes the received facial image and compares it with existing data in the database to identify a specific person. The input is a facial image, and the output is the name of the identified person and related information. The process involves facial image analysis and database matching.

[1743] Step 3:

[1744] The facial recognition server sends the name and related information of the identified person back to the smart glasses terminal. The output information includes the identified person's name and past conversation content. The user then receives the information displayed on the screen.

[1745] Step 4:

[1746] The smart glasses' microphone captures the conversation between the user and the customer in real time. The captured audio data is sent from the smart glasses device to the speech recognition server. At this point, the captured audio data is the input, and the transmission process is performed.

[1747] Step 5:

[1748] The speech recognition server analyzes the received audio data and converts it into text format. The input is audio data, and the output is text data. Audio analysis and text conversion processes are performed.

[1749] Step 6:

[1750] The text data generated by the speech recognition server is saved to the database. The text data is used as input for the database registration process.

[1751] Step 7:

[1752] The emotion engine analyzes voice data and facial image data in the database to determine the user's emotional state. The input is voice data and facial image data, and the output is the emotional state. Emotion analysis processing is then performed.

[1753] Step 8:

[1754] The emotion engine determines the emotional state, generates emotion information, and stores it in the database. The emotional state is the input, and the process of generating emotion information and registering it in the database is performed.

[1755] Step 9:

[1756] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes both, and generates a summary. The input consists of text data, past conversation data, and sentiment information, while the output is summarized text information. Data analysis and summary generation processes are performed.

[1757] Step 10:

[1758] The generation AI server sends the generated summary text and sentiment information to the smart glasses terminal. The output consists of the summary text and sentiment information, and the process of sending them to the smart glasses is performed.

[1759] Step 11:

[1760] The smart glasses display shows a summarized conversation and emotional information. Based on this, the user can provide personalized service information to the customer. The displayed information is the output.

[1761] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1762] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1763] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1764] [Fourth Embodiment]

[1765] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1766] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1767] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1768] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1769] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1770] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1771] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1772] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1773] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1774] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1775] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1776] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1777] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1778] This invention relates to a system that provides memory support information by combining facial recognition means, voice recognition means, and means for displaying generated information. A specific embodiment of this system, including its program and processing, is described below.

[1779] System Configuration

[1780] This system consists of the following main elements:

[1781] 1. Smart glasses device

[1782] 2. Face recognition server

[1783] 3. Speech recognition server

[1784] 4. Generation AI Server

[1785] 5. Database

[1786] Program processing

[1787] Performing facial recognition

[1788] 1. Terminal

[1789] The smart glasses' camera captures the faces of people within the user's field of view.

[1790] 2. Terminal → Server (Facial Recognition)

[1791] The captured facial image is sent to a facial recognition server.

[1792] 3. Server (facial recognition)

[1793] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[1794] 4. Server (facial recognition) → Terminal

[1795] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[1796] 5. Terminal

[1797] The smart glasses display the name of the recognized person.

[1798] Executing speech recognition

[1799] 1. Terminal

[1800] The smart glasses' microphone continuously captures the audio of the conversation.

[1801] 2. Terminal → Server (Speech Recognition)

[1802] The captured audio data is sent to the speech recognition server.

[1803] 3. Server (Speech Recognition)

[1804] Convert audio data to text.

[1805] 4. Server (Speech Recognition)

[1806] Save text data to a database.

[1807] Execution of Generative AI

[1808] 1. Generation AI Server

[1809] The generation AI server analyzes and merges past conversation data with newly acquired conversation data.

[1810] 2. Generation AI server → Terminal

[1811] The generated summary text is sent to the smart glasses device.

[1812] 3. Terminal

[1813] The smart glasses display a summarized version of the conversation.

[1814] Specific example

[1815] Scenes involving dementia patients and their caregivers

[1816] 1. User (Patient)

[1817] A dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face.

[1818] 2. Terminal → Server (Facial Recognition)

[1819] The captured facial image is sent to the facial recognition server.

[1820] 3. Server (facial recognition)

[1821] The facial recognition server matches the helper's facial image against the database and returns the result, "This person is helper AA."

[1822] 4. Server (facial recognition) → Terminal

[1823] The helper's name and past conversation history are sent to the smart glasses.

[1824] 5. Terminal

[1825] The smart glasses display the message, "This is caregiver AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[1826] 6. Terminal → Server (Speech Recognition)

[1827] The captured audio data is sent to the speech recognition server.

[1828] 7. Server (Speech Recognition)

[1829] Convert the conversation content into text and save it to a database.

[1830] 8. Generation AI Server

[1831] The generating AI server analyzes new conversation data and past conversation data to generate a summary such as, "Last time we talked about BB, but the new topic mentioned this time is CC."

[1832] 9. Generation AI Server → Terminal

[1833] The generated summary is sent to the smart glasses.

[1834] 10. Terminal

[1835] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC," making it easier for the user to understand the flow of the conversation.

[1836] In this way, the system of the present invention can facilitate communication with dementia patients and in service industries dealing with many customers, thereby improving the quality of life for users.

[1837] The following describes the processing flow.

[1838] Step 1:

[1839] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[1840] Step 2:

[1841] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[1842] Step 3:

[1843] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[1844] Step 4:

[1845] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[1846] Step 5:

[1847] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[1848] Step 6:

[1849] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[1850] Step 7:

[1851] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[1852] Step 8:

[1853] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[1854] Step 9:

[1855] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[1856] Step 10:

[1857] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[1858] Step 11:

[1859] The server (speech recognition) saves the transcribed conversation to a database.

[1860] Step 12:

[1861] The generating AI server retrieves newly saved text data and past conversation data from the database and analyzes both.

[1862] Step 13:

[1863] The generation AI server merges past and new conversation data to generate a summary. This summary includes key points from the current conversation and references to past information.

[1864] Step 14:

[1865] The generated summary text is sent from the generation AI server to the terminal.

[1866] Step 15:

[1867] The device displays a summary it has received on the smart glasses' screen. This allows the user to understand, "In our last conversation, we talked about AA. This time, we're talking about CC."

[1868] This series of processes enables smooth communication in service industries dealing with dementia patients and large numbers of customers by utilizing facial recognition, voice recognition, and generative AI.

[1869] (Example 1)

[1870] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1871] Conventional memory support systems have a drawback: they lack features to help users remember people and conversation content. In particular, there was a need for effective support in situations where memory assistance is required, such as for dementia patients and service industries that involve interaction with many customers. Furthermore, existing systems sometimes suffered from low accuracy in facial and voice recognition, resulting in inaccurate memory support information.

[1872] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1873] In this invention, the server includes means for the terminal to capture a person's face, means for the terminal to transmit the captured face image to a face recognition server, means for the face recognition server to analyze the face image and compare it with a database, means for the face recognition server to return identified person information to the terminal, means for the terminal to display the identified person information, means for the terminal to capture the audio of a conversation, means for the terminal to transmit the captured audio data to a speech recognition server, means for the speech recognition server to convert the audio data into text, means for the speech recognition server to store the text data in a database, means for the generating AI server to analyze and merge past conversation data and new conversation data, means for the generating AI server to transmit the generated summary text to the terminal, and means for the terminal to display the summarized conversation content. As a result, the user can instantly identify people and check past conversation content, enabling the conversation to proceed smoothly.

[1874] A "terminal" is a device that a user wears and operates, and in this invention, this specifically refers to smart glasses, etc.

[1875] A "face recognition server" is a server that analyzes facial images transmitted from a terminal, compares them with a database, and identifies the corresponding person.

[1876] A "speech recognition server" is a server that converts speech data sent from a terminal into text.

[1877] A "generative AI server" is a server that analyzes and merges past conversation data with new conversation data to generate summarized text.

[1878] A "database" is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[1879] A "camera" is a device built into a terminal that captures the faces of people within the user's field of view.

[1880] A "microphone" is an audio input device built into a terminal that captures the sound of conversations.

[1881] A "face image" is image data of a person's face captured by the device's camera.

[1882] "Audio data" refers to audio data of a conversation captured by the device's microphone.

[1883] "Text data" refers to character data converted from speech data by a speech recognition server.

[1884] A "summary text" is a summary that integrates past conversation content and new conversation content, generated by a generation AI server.

[1885] This invention relates to a system for providing memory support information to a user. This system combines facial recognition means, voice recognition means, and means for displaying generated information to provide effective memory support to the user.

[1886] System Configuration

[1887] This system consists of the following main elements:

[1888] 1. Terminal

[1889] This includes wearable devices such as smart glasses. These devices are equipped with cameras and microphones that capture the faces of people and the audio of conversations within the user's field of view.

[1890] 2. Face recognition server

[1891] The face recognition server analyzes face images sent from the terminal and matches them against a database to identify the corresponding person. It uses face recognition libraries such as OpenCV or AWS Rekognition.

[1892] 3. Speech recognition server

[1893] The speech recognition server converts the audio data sent from the terminal into text. It uses speech recognition services such as Google Cloud Speech-to-Text or IBM Watson Speech to Text.

[1894] 4. Generation AI Server

[1895] The generative AI server analyzes and merges past conversation data with newly acquired conversation data to generate summarized text. Examples of generative AI models include GPT-3 and BERT.

[1896] 5. Database

[1897] This is a data storage system for storing and managing various types of data related to the present invention, such as facial image data and text data.

[1898] Program processing

[1899] In this system, when the user wears smart glasses, a series of processes are performed, from facial recognition of a person to speech recognition of conversations, and then the generation and display of summarized text by a generative AI.

[1900] Facial recognition means

[1901] The device's camera captures the faces of people within the user's field of view and sends the facial images to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database to identify the corresponding individuals, and sends the results back to the device. The device then displays the returned person information on its screen.

[1902] Speech recognition means

[1903] The device's microphone captures the conversation audio and sends the audio data to a speech recognition server. The speech recognition server converts the audio data into text and stores the resulting text data in a database. This makes searching and analysis easier later on.

[1904] Generation AI means

[1905] The AI ​​server analyzes and merges previously stored conversation data with newly acquired conversation data to generate a summary text. The generated summary text is sent to the device and displayed on the device's screen. This allows the user to smoothly understand the content of past and current conversations.

[1906] Specific example

[1907] Consider a scenario where a dementia patient is wearing smart glasses. When a caregiver enters the room, the smart glasses' camera captures the caregiver's face and sends the information to a server. The server sends back the relevant information, and the device displays, "This is caregiver AA. We talked about BB during the last visit." When a conversation begins, the smart glasses' microphone captures the audio and sends the audio data to the server. The server converts the audio to text and stores it in a database. Later, a generative AI server analyzes this data, generates a summary text, and displays it on the device.

[1908] An example of a prompt message would be: "Generate a program that analyzes facial images captured by smart glasses and displays information about the recognized person. Also, include a function to convert spoken conversation into text and summarize it."

[1909] Thus, by providing memory support information, the present invention can facilitate communication in service industries that interact with dementia patients and many customers, thereby improving the quality of life for users.

[1910] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1911] Step 1:

[1912] terminal

[1913] The smart glasses' camera captures the faces of people within the user's field of view in high resolution. The input is a person's face, which is saved as a still image using the camera. The output is the captured face image. This face image data is used in the next step.

[1914] Step 2:

[1915] Terminal → Server (Facial Recognition)

[1916] The terminal captures a face image and sends it to a face recognition server using a secure communication protocol (e.g., HTTPS). The input is the face image data obtained in step 1. The output is the face image data sent to the face recognition server.

[1917] Step 3:

[1918] Server (facial recognition)

[1919] The face recognition server analyzes the received face image and extracts feature points. Specifically, the relevant system (e.g., OpenCV or AWS Rekognition) receives face image data as input and outputs the coordinates of feature points and other identification information. Next, these feature points are compared with existing face data stored in the database to identify the corresponding person. As a result of the comparison, person information is output.

[1920] Step 4:

[1921] Server (facial recognition) → Terminal

[1922] The facial recognition server sends back relevant information to the terminal, such as the name of the identified person and past conversation content. The input is the identified person information data, and the output is the person information data sent to the terminal.

[1923] Step 5:

[1924] terminal

[1925] The device displays the person's information, which has been returned, on the smart glasses' display. The input is person information data received from the server, and the output is display information that the user can visually confirm (e.g., "This person is helper AA. We talked about BB during the last visit").

[1926] Step 6:

[1927] terminal

[1928] The smart glasses' microphone continuously captures the conversation audio. The input is the conversational audio, which is saved as audio data using the microphone. The output is the captured audio data, which is used in the next step.

[1929] Step 7:

[1930] Terminal → Server (Speech Recognition)

[1931] The terminal transmits the captured audio data to the speech recognition server in real time. The input is the audio data obtained in step 6, which is transmitted via a secure communication protocol. The output is the audio data sent to the speech recognition server.

[1932] Step 8:

[1933] Server (Speech Recognition)

[1934] The speech recognition server converts the audio data into text. Specifically, it uses Google Cloud Speech-to-Text or IBM Watson Speech to Text to receive audio data as input and generate text data as output.

[1935] Step 9:

[1936] Server (Speech Recognition)

[1937] The converted text data is saved to a database. The input is the generated text data, and the output is the text data saved in the database.

[1938] Step 10:

[1939] AI generation server

[1940] The generative AI server analyzes and fuses past conversation data with newly acquired conversation data. The input is stored text data, which is analyzed using generative AI models such as GPT-3 and BERT to generate a summarized text. The output is the generated summarized text.

[1941] Step 11:

[1942] AI generation server → Terminal

[1943] The AI ​​server generates a summary text and sends it to the terminal. The input is the summary text, which is sent via a secure communication protocol. The output is the summary text that was sent to the terminal.

[1944] Step 12:

[1945] terminal

[1946] The device displays the summarized text on the smart glasses' display. The input is the summarized text received from the generating AI server, and the output is summarized information that the user can visually confirm (e.g., "Last time we talked about BB, but the new topic mentioned this time is CC"). This display allows the user to smoothly understand past and current conversations.

[1947] (Application Example 1)

[1948] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1949] In traditional brick-and-mortar stores, it was difficult for employees to instantly recognize customers' faces and access their past interactions and purchase history. This could lead to communication breakdowns and decreased customer satisfaction. Furthermore, a lack of information to appropriately suggest new products was also a problem.

[1950] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1951] In this invention, the server includes means for facial recognition, means for voice recognition, means for displaying generated information, means for providing memory support information, means for customer recognition, means for providing a summary of conversation content, and means for suggesting recommended products. This enables the recognition of the customer's face, instant display of past conversations and purchase history, and facilitates effective communication and appropriate product suggestions.

[1952] "Face recognition means" refers to a technology that uses a camera to capture the faces of people within its field of view and transmits them to a server.

[1953] "Speech recognition means" refers to a technology that captures conversations using a microphone and transmits them to a recognition server.

[1954] "Means for displaying generated information" refers to technologies that display information obtained from facial recognition or voice recognition on a display.

[1955] "Methods for providing memory support information" refers to technologies that provide memory support to users based on information such as past conversation data and purchase history.

[1956] "Customer recognition means" refers to technology that captures a customer's face and identifies and displays their past interactions and purchase history.

[1957] A "means for providing summaries of conversation content" is a technology that analyzes past conversation data and new conversation data and displays it as a summary.

[1958] "Recommended product suggestions" refers to techniques that suggest the most suitable products based on a customer's past purchase history and conversations.

[1959] The system for carrying out this invention consists of the following main elements.

[1960] System Configuration

[1961] 1. Smart Glasses

[1962] 2. Face recognition server

[1963] 3. Speech recognition server

[1964] 4. Generation AI Server

[1965] 5. Database

[1966] Details of the embodiment

[1967] Hardware configuration

[1968] 1. Smart Glasses

[1969] It has a built-in camera and microphone, and captures the faces of people within the user's field of view.

[1970] The system displays information generated through the display to the user.

[1971] 2. Server cluster

[1972] Face recognition server: Analyzes captured facial images and matches them against a database to identify individuals.

[1973] Speech recognition server: Converts captured audio data into text.

[1974] Generation AI Server: Analyzes and merges past and new conversation data to generate summarized information.

[1975] Database: Stores customer facial images, past conversation data, purchase history, etc.

[1976] System program

[1977] 1. Customer awareness

[1978] The smart glasses' camera captures the faces of people within the user's field of vision.

[1979] The facial image is sent to a facial recognition server and compared against a database.

[1980] The relevant person's information (name and past purchase history) is sent to the smart glasses and displayed on the screen.

[1981] 2. Speech Recognition

[1982] The microphone in the smart glasses captures the audio of the conversation.

[1983] The audio data is sent to a speech recognition server and converted into text.

[1984] The converted text data is stored in the database.

[1985] 3. Providing a summary of the conversation.

[1986] The generation AI server analyzes past and new conversation data to generate a summary text.

[1987] The generated summary text is sent to the smart glasses and displayed on the screen.

[1988] 4. Suggestions for recommended products

[1989] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[1990] Information about the suggested products is displayed on the smart glasses.

[1991] Specific example

[1992] 1. Customer service scenarios

[1993] Employees wear smart glasses to capture the faces of customers who come into the store.

[1994] A facial image is sent to a facial recognition server, and the server returns information such as, "This person is customer A. They purchased a vacuum cleaner last time."

[1995] The smart glasses display the message, "This person is Customer A. They purchased a vacuum cleaner last time."

[1996] The conversation audio is captured and sent to a speech recognition server.

[1997] The audio data is converted to text and stored in a database.

[1998] The AI ​​server analyzes the old and new conversation data and generates a summary such as, "Last time we talked about vacuum cleaners. The new topic is washing machines."

[1999] The generated summary is displayed on smart glasses.

[2000] Examples of prompts to input into a generative AI model

[2001] Customer data: {"Name": "Customer A", "Previously purchased items": ["Vacuum cleaner"], "Previous conversation content": ["About the vacuum cleaner's functions"]}

[2002] New conversation topic: "I'd like to ask for advice about washing machines."

[2003] Output: "Last time we talked about vacuum cleaners, but the new topic mentioned this time is washing machines."

[2004] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[2005] Step 1:

[2006] The smart glasses' camera captures the faces of people within the user's field of vision.

[2007] Input: Camera video (face image)

[2008] Output: Captured facial image

[2009] Specific operation: The smart glasses detect the faces of people within the user's field of vision and capture those faces with the camera.

[2010] Step 2:

[2011] The captured facial image is sent to a facial recognition server.

[2012] Input: Captured facial image

[2013] Output: Notification of successful image transmission to face recognition server

[2014] Specific operation: The smart glasses use wireless communication to send the captured facial image to a facial recognition server.

[2015] Step 3:

[2016] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[2017] Input: Face image

[2018] Output: Information about the identified person (name, past purchase history)

[2019] Specific operation: The face recognition server analyzes the face image and compares it with the person's record stored in the database.

[2020] Step 4:

[2021] The information of the person in question is sent to the smart glasses.

[2022] Input: Information of an identified person

[2023] Output: Instructions for displaying information on smart glasses

[2024] Specific operation: The facial recognition server sends information about the identified person (name, past purchase history) to the smart glasses.

[2025] Step 5:

[2026] The smart glasses display information about the recognized person.

[2027] Input: Information of an identified person

[2028] Output: Person information displayed on the screen

[2029] Specific operation: The smart glasses display the person information they receive on the screen, allowing the user to verify it.

[2030] Step 6:

[2031] The microphone in the smart glasses captures the audio of the conversation.

[2032] Input: Conversation audio

[2033] Output: Captured audio data

[2034] Specific operation: The smart glasses use the microphone to record the audio of the conversation.

[2035] Step 7:

[2036] The captured audio data is sent to the speech recognition server.

[2037] Input: Captured audio data

[2038] Output: Notification of successful transmission of voice data to the speech recognition server

[2039] Specific operation: The smart glasses use wireless communication to send the captured audio data to the speech recognition server.

[2040] Step 8:

[2041] The speech recognition server converts the speech data into text and stores it in a database.

[2042] Input: Audio data

[2043] Output: Text data

[2044] Specific operation: The speech recognition server converts speech data into text using speech recognition technology and saves the converted text data to a database.

[2045] Step 9:

[2046] The generation AI server analyzes past and new conversation data to generate a summary text.

[2047] Input: Past conversation data, new conversation data

[2048] Output: Summary text

[2049] Specific operation: The generative AI server uses a generative AI model to analyze past and recent conversation data, extract key points, and generate a summarized text.

[2050] Step 10:

[2051] The generated summary text is sent to smart glasses.

[2052] Input: Summary text

[2053] Output: Instructions to display summarized text on smart glasses

[2054] Specific operation: The generation AI server sends the generated summary text to the smart glasses, and the smart glasses display it on the screen.

[2055] Step 11:

[2056] The smart glasses display a summarized version of the conversation.

[2057] Input: Summary text

[2058] Output: Summary information displayed on the screen

[2059] Specific action: The smart glasses display a summary text on the screen for the user to review.

[2060] Step 12:

[2061] The AI ​​server analyzes the customer's past purchase history and conversation content to suggest the most suitable products.

[2062] Input: Past purchase history, conversation content

[2063] Output: Proposed product information

[2064] Specific operation: The generation AI server uses a generation AI model based on purchase history and conversation content to suggest the most suitable products.

[2065] Step 13:

[2066] Information about the suggested products is displayed on smart glasses.

[2067] Input: Proposed product information

[2068] Output: Suggested product information displayed on the screen

[2069] Specific operation: The smart glasses display suggested product information on the screen for the user to review.

[2070] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2071] This invention relates to a system combining facial recognition means, voice recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. A specific embodiment of this system, including its program and processing, is described below.

[2072] System Configuration

[2073] This system consists of the following main elements:

[2074] 1. Smart glasses device

[2075] 2. Face recognition server

[2076] 3. Speech recognition server

[2077] 4. Generation AI Server

[2078] 5. Emotional Engine

[2079] 6. Database

[2080] Program processing

[2081] Performing facial recognition

[2082] 1. Terminal

[2083] The smart glasses' camera captures the faces of people within the user's field of view.

[2084] 2. Terminal → Server (Facial Recognition)

[2085] The captured facial image is sent to a facial recognition server.

[2086] 3. Server (facial recognition)

[2087] A facial recognition server analyzes facial images and compares them to a database to identify the corresponding individuals.

[2088] 4. Server (facial recognition) → Terminal

[2089] The name of the identified person and related information (such as past conversation content) are sent back to the smart glasses device.

[2090] 5. Terminal

[2091] The smart glasses display the name of the recognized person.

[2092] Executing speech recognition

[2093] 1. Terminal

[2094] The smart glasses' microphone captures the conversation audio in real time and temporarily stores that audio data.

[2095] 2. Terminal → Server (Speech Recognition)

[2096] The captured audio data is sent to the speech recognition server.

[2097] 3. Server (Speech Recognition)

[2098] Analyzes audio data and converts it to text.

[2099] 4. Server (Speech Recognition)

[2100] Save text data to a database.

[2101] Execution of the Emotion Engine

[2102] 1. Emotional Engine

[2103] The system analyzes voice data and facial image data to determine the user's emotional state (e.g., joy, anger, sadness, etc.).

[2104] 2. Emotional Engine

[2105] The determined emotional state is generated as emotional information and stored in a database.

[2106] Execution of Generative AI

[2107] 1. Generation AI Server

[2108] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes both.

[2109] 2. Generation AI server → Terminal

[2110] The generated summary text and sentiment information are sent to the smart glasses device.

[2111] 3. Terminal

[2112] The smart glasses display a summary of the conversation and emotional information.

[2113] Specific example

[2114] Scenes involving dementia patients and their families

[2115] 1. User (Patient)

[2116] A dementia patient is wearing smart glasses. When a family member enters the room, the smart glasses' camera captures the family member's face.

[2117] 2. Terminal → Server (Facial Recognition)

[2118] The captured facial image is sent to the facial recognition server.

[2119] 3. Server (facial recognition)

[2120] The facial recognition server matches the family's facial images against a database and returns the result, "This person is family member AA."

[2121] 4. Server (facial recognition) → Terminal

[2122] Family members' names and past conversation history are sent to the smart glasses.

[2123] 5. Terminal

[2124] The smart glasses display the message, "This is family member AA. We talked about BB during your last visit." The smart glasses' microphone continues to capture the current conversation.

[2125] 6. Terminal → Server (Speech Recognition)

[2126] The captured audio data is sent to the speech recognition server.

[2127] 7. Server (Speech Recognition)

[2128] Convert the conversation content into text and save it to a database.

[2129] 8. Emotional Engine

[2130] The system analyzes captured audio and facial expression data to determine the user's emotional state. For example, it might generate emotional information such as, "The user is currently excited."

[2131] 9. Emotion Engine → Database

[2132] Emotional information is stored in a database.

[2133] 10. Generation AI Server

[2134] The generating AI server analyzes new conversation data, past conversation data, and sentiment information to generate summaries such as, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited."

[2135] 11. Generation AI Server → Terminal

[2136] The generated summary is sent to the smart glasses.

[2137] 12. Terminal

[2138] The smart glasses display a message saying, "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," making it easier for the user to understand the flow of the conversation.

[2139] In this way, the system of the present invention can enable smooth communication in service industries dealing with dementia patients and many customers, thereby improving the quality of life for users. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, enabling more appropriate responses.

[2140] The following describes the processing flow.

[2141] Step 1:

[2142] The user puts on the smart glasses and starts the system. The smart glasses activate the camera and microphone.

[2143] Step 2:

[2144] The device (smart glasses) captures the faces of people within the user's field of view using its camera. The captured face images are temporarily stored on the device.

[2145] Step 3:

[2146] The device captures a facial image and sends it to a facial recognition server. This transmission utilizes a network connection.

[2147] Step 4:

[2148] The server (for facial recognition) analyzes the received facial image and compares it with facial data in the database. A facial recognition algorithm is then applied to identify the person.

[2149] Step 5:

[2150] The server (using facial recognition) retrieves the name and identification information of the person who matches the criteria. This information also includes past conversation data.

[2151] Step 6:

[2152] The server (using facial recognition) sends the name and identification information of the person it identifies back to the terminal.

[2153] Step 7:

[2154] The device displays the name and identification information of the person it receives on the smart glasses' display. For example, it might say, "This person is AA."

[2155] Step 8:

[2156] The microphone on the device (smart glasses) captures the audio of the conversation in real time and temporarily stores that audio data.

[2157] Step 9:

[2158] The device sends the captured audio data to the speech recognition server. This is also done over the network.

[2159] Step 10:

[2160] The server (speech recognition) analyzes the received audio data and applies a speech recognition algorithm to convert it into text.

[2161] Step 11:

[2162] The server (speech recognition) saves the transcribed conversation to a database.

[2163] Step 12:

[2164] The emotion engine retrieves voice data and facial expression data from the database. The emotion engine then begins analyzing the voice and facial expressions.

[2165] Step 13:

[2166] The emotion engine determines the user's emotional state based on the analysis results. For example, it identifies one of the following emotional states: joy, anger, sadness, or pleasure.

[2167] Step 14:

[2168] The emotion engine generates emotion information based on the determined emotional state and stores that information in a database.

[2169] Step 15:

[2170] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them.

[2171] Step 16:

[2172] The generation AI server fuses past conversation data, new conversation data, and sentiment information to generate a summary. This summary includes key points from the current conversation, references to past information, and the user's emotional state.

[2173] Step 17:

[2174] The generated summary text and sentiment information are sent from the generating AI server to the terminal.

[2175] Step 18:

[2176] The device displays the summary and sentiment information it receives on the smart glasses' display. This allows the user to understand, "In our last conversation we talked about AA. This time we're talking about CC, and the user is currently excited."

[2177] This series of processes utilizes facial recognition, voice recognition, generative AI, and an emotion engine to enable smooth communication in service industries dealing with dementia patients and large customer bases.

[2178] (Example 2)

[2179] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2180] Conventional systems provided limited memory support information and lacked information based on the user's emotional state. This made it difficult to improve the user's quality of life and facilitate smooth communication. Furthermore, it was challenging to provide appropriate support for dementia patients and service industries dealing with large customer bases.

[2181] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2182] In this invention, the server includes means for capturing the faces of people within its field of view using a camera and transmitting them to the server, means including a microphone for capturing and transmitting audio to the server, means equipped with a display for displaying the generated information, server means for analyzing face data and audio data and returning the recognized information, emotion engine means for analyzing audio data and face image data and determining the user's emotional state, and generation AI server means for retrieving newly stored text data, past conversation data, and emotional information from a database and generating summary text and emotional information. This makes it possible to provide memory support information and information based on emotional state.

[2183] A "camera" is a device that captures images and acquires that digital data.

[2184] A "server" is a computer system that operates on a network and provides data processing and storage services.

[2185] A "microphone" is a device that captures sound and acquires its digital data.

[2186] A "display" is a device that displays visual information.

[2187] "Facial recognition" is a technology that analyzes captured facial images to identify specific individuals.

[2188] "Speech recognition" is a technology that analyzes captured audio and converts it into text data.

[2189] An "emotion engine" is a technology that analyzes voice data and facial image data to determine the user's emotional state.

[2190] A "database" is a system for systematically storing and managing data.

[2191] A "generating AI server" is a server equipped with artificial intelligence that generates summarized text and sentiment information using stored data.

[2192] "Memory support information" refers to information that supports a user's memory by referencing specific records or histories.

[2193] A "summary text" is a document that concisely summarizes multiple pieces of information.

[2194] This invention relates to a system combining facial recognition means, speech recognition means, means for displaying generated information, and an emotion engine, which provides memory support information and information based on the user's emotional state. The system consists of smart glasses terminals, a facial recognition server, a speech recognition server, a generation AI server, an emotion engine, and a database as its main components.

[2195] Performing facial recognition

[2196] Hardware and software to be used

[2197] Smart Glasses Device

[2198] Face recognition server

[2199] Camera (high resolution)

[2200] OpenCV (face recognition library)

[2201] operation

[2202] The smart glasses device captures the faces of people within the user's field of view via its camera and sends them to a facial recognition server. The facial recognition server analyzes the facial images, compares them to a database, and identifies specific individuals. The results are sent back to the smart glasses device, and the person's name and related information are displayed on the screen.

[2203] Executing speech recognition

[2204] Hardware and software to be used

[2205] Smart Glasses Device

[2206] Speech recognition server

[2207] microphone

[2208] Google Speech-to-Text API

[2209] operation

[2210] The microphone on the smart glasses device captures the conversation audio in real time and sends it to a speech recognition server. The speech recognition server converts the audio data into text and stores it in a database.

[2211] Execution of the Emotion Engine

[2212] Hardware and software to be used

[2213] Emotional Engine

[2214] Emotion analysis algorithms (voice tone analysis, facial expression analysis)

[2215] operation

[2216] The emotion engine analyzes voice data and facial image data to determine the user's emotional state. The determined emotional state is then stored in a database as emotion information.

[2217] Execution of Generative AI

[2218] Hardware and software to be used

[2219] AI generation server

[2220] GPT-4 (Natural Language Processing Model)

[2221] operation

[2222] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes them, and generates summarized text and sentiment information. This generated information is sent to the smart glasses terminal and displayed on the screen.

[2223] Specific example

[2224] For example, if a dementia patient is wearing smart glasses, when a family member enters the room, the smart glasses' camera captures the family member's face and sends it to a facial recognition server. The facial recognition server compares the family member's face image with a database and sends the result, "This person is family member AA," back to the smart glasses terminal. The smart glasses then display, "This person is family member AA. We talked about BB during the last visit."

[2225] Additionally, once a conversation begins, the smart glasses' microphone captures the audio and sends it to a speech recognition server. The speech recognition server converts the conversation into text and stores it in a database. The emotion engine determines the user's emotional state from the audio and facial expression data, generates emotion information such as "The user is currently excited," and stores it in the database. The generating AI server analyzes the new conversation data, past conversation data, and emotion information to generate a summary such as "Last time we talked about BB, but the new topic mentioned this time is CC, and the user is excited," and sends it to the smart glasses for display.

[2226] Example of a prompt

[2227] "Last time we talked about BB, but the new topic we discussed this time is CC. Users are currently very excited about it."

[2228] This system enables smooth communication in service industries dealing with dementia patients and large customer bases, thereby improving the user's quality of life. Furthermore, by combining it with an emotion engine, it becomes possible to provide information based on the user's emotions, leading to more appropriate responses.

[2229] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2230] Step 1: Face Capture

[2231] terminal

[2232] The smart glasses' camera captures the faces of people within the user's field of view in real time. The input data is images of people's faces within the field of view, and a high-resolution camera is used to obtain clear images. The output is the captured face image.

[2233] Step 2: Send your face image

[2234] Terminal → Server (Facial Recognition)

[2235] The captured facial image is sent to a facial recognition server. The facial image data is compressed and transferred to the facial recognition server in real time using a secure communication protocol (e.g., HTTPS). The input data is the captured facial image, and the output is the facial image data sent to the facial recognition server.

[2236] Step 3: Analysis of facial data

[2237] Server (facial recognition)

[2238] The face recognition server analyzes the received face image. The software used is a face recognition library such as OpenCV, and the input data is the transmitted face image. This analysis extracts facial feature points. The output is face feature data.

[2239] Step 4: Match with database

[2240] Server (facial recognition)

[2241] The analyzed facial data is compared with a database. A specific matching algorithm is used to identify individuals whose facial features match those in the existing database. The input data is facial feature data, and the output is information about the identified individuals.

[2242] Step 5: Submit the identification result

[2243] Server (facial recognition) → Terminal

[2244] The server sends the name of the identified person and past conversation details back to the smart glasses device. The server encrypts and transmits this information. The input data is the identified person's information, and the output is the identification information sent to the device.

[2245] Step 6: Displaying Identification Information

[2246] terminal

[2247] The smart glasses display the name and related information of the recognized person. The input data is identification information, and the output is the information displayed on the screen.

[2248] Step 7: Capture audio

[2249] User

[2250] When a user starts a conversation, the smart glasses' microphone captures the audio in real time. The input data is the conversation audio, and the output is the captured audio data.

[2251] Step 8: Sending audio data

[2252] Terminal → Server (Speech Recognition)

[2253] The captured audio data is sent to the speech recognition server. Before transmission, the data is converted to an appropriate format (e.g., WAV format). The input data is the audio data, and the output is the data sent to the speech recognition server.

[2254] Step 9: Analyzing the audio data

[2255] Server (Speech Recognition)

[2256] A speech recognition server analyzes the audio data and converts it to text. The input data is audio, and the software used is the Google Speech-to-Text API. The output is text data.

[2257] Step 10: Saving the text data

[2258] Server (Speech Recognition)

[2259] The data is converted to text and saved to a database. The input data is the converted text data, which is saved with a timestamp. The output is the text data saved in the database.

[2260] Step 11: Analysis of emotional state

[2261] Emotional Engine

[2262] This system analyzes voice and facial image data to determine the user's emotional state. Based on an emotion analysis algorithm, the input data consists of voice and facial image data. The output is the determined emotion information.

[2263] Step 12: Preserving emotional information

[2264] Emotional Engine

[2265] The determined emotional state is stored in a database as emotional information. The input data is emotional information, and the output is the emotional information stored in the database.

[2266] Step 13: Summarizing the conversation

[2267] AI generation server

[2268] The generating AI server retrieves newly saved text data, past conversation data, and sentiment information from the database and analyzes them. The technology used is GPT-4. The input data consists of text data, past conversation data, and sentiment information, and the output is summarized text and sentiment information.

[2269] Step 14: Sending generated information

[2270] AI generation server → Terminal

[2271] The generated summary text and sentiment information are sent to the smart glasses device. Encryption is used to maintain security. The input data consists of the summary text and sentiment information; the output data is the data sent to the smart glasses device.

[2272] Step 15: Display summary information

[2273] terminal

[2274] The smart glasses display a summarized conversation and sentiment information. The input data is the summarized text and sentiment information, and the output is the information displayed on the screen.

[2275] (Application Example 2)

[2276] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2277] Traditional in-store customer service systems only used facial and voice recognition, resulting in insufficient personalized service. Furthermore, it was difficult to grasp customers' emotional states in real time, hindering the ability to take prompt and appropriate action in situations requiring special attention.

[2278] In Application Example 2, the identification processing performed by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for analyzing emotional states, means for generating summaries based on past conversation content and emotional information, means for capturing the faces of people within the field of view using a camera and transmitting them to the server, and means for capturing conversations using a microphone and transmitting them to the recognition server. This enables the provision of information based on the customer's emotional state and past conversation history, allowing for the rapid and appropriate provision of personalized services.

[2279] "Means of performing facial recognition" refers to technology that analyzes and identifies a person's face image captured by a camera.

[2280] "Means of performing speech recognition" refers to technology for converting audio data captured by a microphone into text format and then analyzing it.

[2281] "Means for displaying generated information" refers to display and interface technologies for visually providing analyzed and processed data to the user.

[2282] "Means for analyzing emotional states" refers to technologies that use voice data and facial image data to determine the emotional state of a subject (such as joy, anger, sadness, etc.).

[2283] "Means for generating summaries based on past conversation content and sentiment information" refers to a generative AI model that analyzes past conversation content and sentiment information in a database to generate important information and summaries.

[2284] A "system that provides personalized service information" is a set of technologies that provide information and services to individual customers based on their individual needs and past history.

[2285] To carry out this invention, a system comprising the following main elements is required. The system comprises the following components.

[2286] System Configuration Details

[2287] 1. Smart glasses device

[2288] 2. Face recognition server

[2289] 3. Speech recognition server

[2290] 4. Generation AI Server

[2291] 5. Emotional Engine

[2292] 6. Database

[2293] Hardware and software

[2294] Hardware: Smart glasses (e.g., Microsoft HoloLens), Servers (high-performance cloud servers)

[2295] Software: Face recognition software (e.g., Amazon Rekognition), speech recognition software (e.g., Google Speech-to-Text), sentiment analysis software (e.g., Microsoft Azure's Text Analytics for sentiment analysis), generative AI models (e.g., OpenAI's GPT-4)

[2296] System program processing details

[2297] Face recognition processing

[2298] The smart glasses device captures the faces of people within the user's field of view using its camera. The captured facial images are sent to a facial recognition server in real time or in batch processing. This facial recognition server performs image analysis and identifies specific individuals by comparing them with existing data in its database. The person's name and related information are sent back to the smart glasses device and displayed to the user.

[2299] Speech recognition processing

[2300] The smart glasses' microphone captures conversations between the user and customers in real time and sends the audio data to a speech recognition server. This server converts the audio data into text and stores the text data in a database.

[2301] Emotional Engine Processing

[2302] The emotion engine analyzes voice and facial image data to determine the user's emotional state. The determined emotional state is stored in a database and used by the generating AI server.

[2303] Processing of generated AI models

[2304] The AI ​​server analyzes newly stored text data, past conversation data, and sentiment information to generate conversation summaries and key information. The generated information is sent to the smart glasses terminal and provided to the user.

[2305] Specific example

[2306] Examples of customer service in physical stores

[2307] When a user wears smart glasses and captures the face of a customer entering the store, a facial recognition server identifies the customer's name and past visit history. For example, information such as, "This customer is Mr. / Ms. Tanaka. During their last visit, they showed interest in allergy-free products," is displayed. When the customer and the user begin a conversation, the conversation is recorded, and after it ends, a summary such as, "Mr. / Ms. Tanaka is interested in new organic products," is generated. Furthermore, an emotion engine analyzes the customer's emotional state in real time and provides information such as, "The customer is excited," enabling the user to respond appropriately.

[2308] Example of a prompt

[2309] Examples of prompts for a generative AI model are as follows:

[2310] prompt:

[2311] Past conversation data: {conversation data}

[2312] Latest conversation: {New conversation}

[2313] Past emotional information: {Past emotional information}

[2314] New emotional state: {New emotional state}

[2315] output:

[2316] 1. Summary of the latest conversation

[2317] 2. Customer concerns and proposed solutions

[2318] Through this format, personalized services can be provided in physical stores, significantly improving the quality of customer service.

[2319] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2320] Step 1:

[2321] The smart glasses' camera captures the faces of people within the user's field of view. The captured face images are sent from the smart glasses device to a face recognition server. At this point, the captured image serves as the input, and the transmission process is initiated.

[2322] Step 2:

[2323] The facial recognition server analyzes the received facial image and compares it with existing data in the database to identify a specific person. The input is a facial image, and the output is the name of the identified person and related information. The process involves facial image analysis and database matching.

[2324] Step 3:

[2325] The facial recognition server sends the name and related information of the identified person back to the smart glasses terminal. The output information includes the identified person's name and past conversation content. The user then receives the information displayed on the screen.

[2326] Step 4:

[2327] The smart glasses' microphone captures the conversation between the user and the customer in real time. The captured audio data is sent from the smart glasses device to the speech recognition server. At this point, the captured audio data is the input, and the transmission process is performed.

[2328] Step 5:

[2329] The speech recognition server analyzes the received audio data and converts it into text format. The input is audio data, and the output is text data. Audio analysis and text conversion processes are performed.

[2330] Step 6:

[2331] The text data generated by the speech recognition server is saved to the database. The text data is used as input for the database registration process.

[2332] Step 7:

[2333] The emotion engine analyzes voice data and facial image data in the database to determine the user's emotional state. The input is voice data and facial image data, and the output is the emotional state. Emotion analysis processing is then performed.

[2334] Step 8:

[2335] The emotion engine determines the emotional state, generates emotion information, and stores it in the database. The emotional state is the input, and the process of generating emotion information and registering it in the database is performed.

[2336] Step 9:

[2337] The generation AI server retrieves newly saved text data, past conversation data, and sentiment information from the database, analyzes both, and generates a summary. The input consists of text data, past conversation data, and sentiment information, while the output is summarized text information. Data analysis and summary generation processes are performed.

[2338] Step 10:

[2339] The generation AI server sends the generated summary text and sentiment information to the smart glasses terminal. The output consists of the summary text and sentiment information, and the process of sending them to the smart glasses is performed.

[2340] Step 11:

[2341] The smart glasses display shows a summarized conversation and emotional information. Based on this, the user can provide personalized service information to the customer. The displayed information is the output.

[2342] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2343] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2344] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[2345] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2346] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[2347] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[2348] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[2349] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[2350] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[2351] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[2352] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[2353] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[2354] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[2355] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2356] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[2357] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[2358] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[2359] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[2360] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[2361] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[2362] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[2363] The following is further disclosed regarding the embodiments described above.

[2364] (Claim 1)

[2365] Methods for performing facial recognition,

[2366] A means of performing speech recognition,

[2367] A means of displaying the generated information,

[2368] A system that provides memory support information by coordinating the above methods.

[2369] (Claim 2)

[2370] The means for performing facial recognition includes means for capturing the faces of people within the field of view using a camera and transmitting them to a server.

[2371] The system according to claim 1.

[2372] (Claim 3)

[2373] The means for performing speech recognition includes means for capturing a conversation using a microphone and transmitting it to a recognition server.

[2374] The system according to claim 1.

[2375] "Example 1"

[2376] (Claim 1)

[2377] The means by which the device captures a person's face,

[2378] A means for sending a facial image captured by the terminal to a facial recognition server,

[2379] A means by which a face recognition server analyzes face images and compares them with a database,

[2380] A means of sending identified person information back from a facial recognition server to a terminal,

[2381] A means of displaying the information of a person identified by the terminal,

[2382] The device has a means of capturing the audio of the conversation,

[2383] A means for sending audio data captured by the terminal to a speech recognition server,

[2384] A means by which a speech recognition server converts speech data into text,

[2385] A means for the speech recognition server to save text data to a database,

[2386] The generation AI server analyzes and merges past conversation data with new conversation data,

[2387] A means of sending the generated summary text from the generation AI server to the terminal,

[2388] A means for the device to display the summarized conversation content,

[2389] A system that provides memory support information by coordinating the above methods.

[2390] (Claim 2)

[2391] The system according to claim 1, comprising means for capturing the faces of people within the field of view using a camera and transmitting them to a face recognition server.

[2392] (Claim 3)

[2393] The system according to claim 1, comprising means for capturing a conversation using a microphone and transmitting it to a speech recognition server.

[2394] "Application Example 1"

[2395] (Claim 1)

[2396] Methods for performing facial recognition,

[2397] A means of performing speech recognition,

[2398] A means of displaying the generated information,

[2399] Means of providing memory support information,

[2400] Customer recognition methods,

[2401] Means of providing a summary of the conversation content,

[2402] Methods for suggesting recommended products,

[2403] A system that provides information by coordinating the above methods.

[2404] (Claim 2)

[2405] The means for performing facial recognition includes means for capturing the faces of people within the field of view using a camera and transmitting them to a server.

[2406] The system according to claim 1.

[2407] (Claim 3)

[2408] The means for performing speech recognition includes means for capturing a conversation using a microphone and transmitting it to a recognition server.

[2409] The system according to claim 1.

[2410] "Example 2 of combining an emotion engine"

[2411] (Claim 1)

[2412] A means of capturing the faces of people within the field of view using a camera and sending them to a server,

[2413] A means including a microphone that captures audio and sends it to a server,

[2414] A means equipped with a display for displaying generated information,

[2415] A server means that analyzes facial data and voice data and returns the recognized information,

[2416] An emotion engine means that analyzes voice data and facial image data to determine the user's emotional state,

[2417] A generative AI server means that retrieves newly saved text data, past conversation data, and sentiment information from a database, and generates summarized text and sentiment information.

[2418] A system that combines the above methods to provide memory support information and information based on emotional state.

[2419] (Claim 2)

[2420] The system according to claim 1, comprising means for capturing the faces of people within its field of view using a camera and transmitting them to a server.

[2421] (Claim 3)

[2422] The system according to claim 1, comprising means including a microphone for capturing audio and transmitting it to a server.

[2423] "Application example 2 when combining with an emotional engine"

[2424] (Claim 1)

[2425] Methods for performing facial recognition,

[2426] A means of performing speech recognition,

[2427] A means of displaying the generated information,

[2428] A means of analyzing emotional states,

[2429] A means of generating a summary based on past conversation content and emotional information,

[2430] A system that provides personalized service information by coordinating the above methods.

[2431] (Claim 2)

[2432] This includes means of capturing the faces of people within the field of view using a camera and transmitting them to a server.

[2433] The system according to claim 1.

[2434] (Claim 3)

[2435] This includes means for capturing conversations using a microphone and sending them to a recognition server.

[2436] The system according to claim 1. [Explanation of Symbols]

[2437] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Methods for performing facial recognition, A means of performing speech recognition, A means of displaying the generated information, A system that provides memory support information by coordinating the above methods.

2. The means for performing facial recognition includes means for capturing the faces of people within the field of view using a camera and transmitting them to a server. The system according to claim 1.

3. The means for performing speech recognition includes means for capturing a conversation using a microphone and transmitting it to a recognition server. The system according to claim 1.

Images