System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing visual data; means for capturing audio data; means for analyzing the captured visual data and audio data; means for generating support information in real-time based on the analysis; and means for providing the generated support information to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Early symptoms of dementia include memory impairment, aphasia, and executive dysfunction, and as these symptoms progress, they cause significant difficulties in daily life. However, there is currently a lack of effective technological support to address these symptoms. The present invention aims to provide technical support for the daily lives of patients with early dementia and reduce the burden on patients and their families and caregivers. [Means for solving the problem]

[0005] The present invention provides a system including a means for capturing visual data, a means for capturing audio data, a means for analyzing the captured visual data and audio data, a means for generating support information in real time based on the analysis results, and a means for providing the generated support information to a user. This system can identify problems or difficulties faced by a user in real time based on the visual and audio data, and provide appropriate support information. This can support the daily lives of patients with early-stage dementia, improve the user's quality of life, and reduce the burden on caregivers.

[0006] "Visual data" refers to video information that appears in the user's field of vision and is captured by an image capture device such as a camera.

[0007] "Audio data" refers to audio information of the user and the surroundings, which is captured by an audio capture device such as a microphone.

[0008] "Means for analyzing" refers to algorithms and devices that process captured visual and audio data to understand and interpret its content.

[0009] "Means for generating support information in real time" refers to a function for instantly generating appropriate advice and support information for the user based on the analysis results.

[0010] "Means for providing to the user" refers to devices and algorithms for communicating the generated support information to the user in audio or other form. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0015] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0019] [First embodiment]

[0020] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0021] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0024] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0027] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0031] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0032] The present invention is a system for supporting the daily lives of patients with early-stage dementia, capturing visual and audio data, analyzing the data, and providing appropriate support information in real time. The system and its embodiments are described in detail below.

[0033] Overall system configuration

[0034] This system consists of a glasses-type device (terminal) worn by the user and a server that receives and analyzes the data. The glasses-type device has a built-in camera and microphone that captures the user's visual and audio data. The captured data is sent to the server in real time, where it is analyzed and appropriate support information is generated. The generated support information is then provided to the user via the glasses-type device.

[0035] Visual and audio data capture

[0036] User

[0037] When a user wears the eyeglasses, the camera in the glasses continuously captures images (visual data) within the user's field of vision, and the built-in microphone records the user's and their surrounding sounds (audio data).

[0038] Sending data

[0039] Terminal

[0040] The captured visual and audio data is temporarily stored in a buffer on the device, then compressed, encrypted, and sent to a server. This communication occurs in real time, minimizing the delay between when the data is collected and when it reaches the server.

[0041] Receiving and analyzing data

[0042] server

[0043] The server receives visual and audio data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[0044] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[0045] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[0046] Generate support information

[0047] server

[0048] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[0049] Providing support information

[0050] Terminal

[0051] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[0052] Specific examples

[0053] Example 1: Person recognition

[0054] User: "Who are you?"

[0055] The device's microphone captures this audio and sends the data to a server.

[0056] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0057] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[0058] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0059] Example 2: Directional indication

[0060] If the user forgets where the door is in the room,

[0061] The device's camera captures the view and sends it to the server.

[0062] The server analyzes the visual data and does not detect the presence of a "door."

[0063] As support information, the message "The door is on the right" is generated and sent to the terminal.

[0064] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0065] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user puts on the eyeglasses-type device, which enables the device to capture the user's visual and audio data.

[0069] Step 2:

[0070] The device's camera continuously captures the image of the user's field of vision (visual data), and the microphone records the user's and surrounding sounds (audio data).

[0071] Step 3:

[0072] The device temporarily stores the captured visual and audio data in a buffer memory, where it is compressed and encrypted in real time.

[0073] Step 4:

[0074] The device transmits compressed and encrypted visual and audio data to the server, and this communication occurs in real time to minimize latency.

[0075] Step 5:

[0076] The server receives the visual and audio data sent from the terminal, and decompresses and decodes the respective data.

[0077] Step 6:

[0078] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[0079] Step 7:

[0080] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[0081] Step 8:

[0082] The server analyzes visual and audio data to identify the problem the user is facing and generates real-time support information, such as the name of a person if the user does not recognize them and their relationship to the person.

[0083] Step 9:

[0084] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[0085] Step 10:

[0086] The server transmits the generated voice data to the terminal.

[0087] Step 11:

[0088] The device plays back the received audio data, providing support information to the user through audio and giving necessary advice in real time. For example, if the user gets lost, audio instructions will be given on the direction they should go.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Patients with early-stage dementia may experience problems in their daily lives, such as difficulty recognizing objects and people, or a lack of understanding of intentions. This not only makes it difficult for patients to make appropriate decisions and reduces their quality of life, but also places a burden on caregivers and family members. There is a need for a support system that can solve these problems and enable patients to live their daily lives more smoothly.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for acquiring visual information, means for acquiring acoustic information, means for temporarily storing the acquired visual information and acoustic information and compressing and encrypting it, means for transmitting the compressed and encrypted information in real time, means for analyzing the received visual information and acoustic information, means for generating support information in real time based on the analysis results, and means for converting the generated support information into audio and providing it to the user. This makes it possible to provide real-time support for the difficulties faced by early-stage dementia patients in their daily lives and improve the quality of life of the patients.

[0094] "Visual information" refers to image data captured in the user's field of vision by an image capture device such as a camera in a glasses-type device.

[0095] "Acoustic information" refers to audio data emitted from the user and the environment, which is acquired by an audio acquisition device such as a microphone of the eyeglass-type device.

[0096] "Temporary storage" refers to a process of temporarily storing acquired visual and acoustic information in memory or storage.

[0097] "Compression" is data processing performed to reduce the data volume of visual and audio information.

[0098] "Encryption" is the process of protecting data with encryption technology in order to securely transmit visual and audio information.

[0099] "Transmission" is the process of sending compressed and encrypted visual and audio information from the terminal to the server over the network.

[0100] "Reception" refers to the process in which the server receives visual and audio information sent from the terminal.

[0101] "Analysis" refers to analyzing received visual information using image recognition algorithms, and converting acoustic information into text data using speech recognition algorithms and analyzing it through natural language processing.

[0102] "Support information" is information that includes advice and instructions for the user based on the analysis results, and is information that is generated in real time for problem solving.

[0103] "Conversion to voice" refers to a process of converting the generated assistance information into voice data using voice synthesis technology so that it can be provided to the user audibly.

[0104] "Providing to the user" refers to providing the converted audio data in a form that the user can hear through the speaker of the eyeglass-type device, etc.

[0105] The present invention is a system for supporting the daily lives of patients with early-stage dementia. This system captures visual and acoustic information, analyzes the acquired data, and provides support information in real time. Detailed embodiments of the present invention will be described below.

[0106] Hardware Configuration

[0107] This system consists of a glasses-type device (hereafter referred to as the terminal) worn by the user and a server that receives and analyzes data. The terminal has a built-in camera and microphone, which are responsible for capturing visual and acoustic information. The server is a general-purpose computer equipped with a high-performance CPU and GPU.

[0108] Software Configuration

[0109] The device is installed with software for capturing visual and acoustic information: driver software for acquiring images from the camera and audio processing software for acquiring audio from the microphone.

[0110] The server has the following software modules installed:

[0111] Image recognition algorithms: e.g., YOLOv5 and OpenCV

[0112] Speech recognition algorithms, such as Google Cloud Speech-to-Text or Microsoft Azure Speech Service

[0113] Natural Language Processing (NLP) modules: e.g., BERT models

[0114] Speech synthesis technology: e.g., Google Text-to-Speech or Amazon Polly

[0115] Operating procedures and data processing

[0116] When a user wears the glasses, the device's camera captures the user's visual information in real time, and the microphone records audio information. The device temporarily stores this data, compresses and encrypts it, and then transmits it to a server.

[0117] The server receives the visual and audio information sent from the device, first decompresses and decodes the data, then uses an image recognition algorithm to analyze the visual information and identify objects and people in the user's field of view, and simultaneously uses a speech recognition algorithm to convert the audio information into text data and understand the user's speech and intent through natural language processing.

[0118] Based on the analysis results, the server generates real-time support information. For example, if the user does not recognize a person, it will provide the person's name and relationship to the person. If the user gets lost, it will provide advice on the direction to go. This support information is then compressed and encrypted again before being sent to the device.

[0119] The device converts the received assistance information into voice data using voice synthesis technology and plays it back through a built-in speaker, allowing users to obtain the necessary information in real time by voice.

[0120] Specific examples

[0121] Example 1: Person recognition

[0122] User: Ask "Who are you?"

[0123] Device: The microphone captures this audio and sends the data to the server.

[0124] Server: Analyzes the voice data and understands the user's question.

[0125] Server: Analyzes visual data to identify the person being questioned.

[0126] Server: Generate supporting information: "This is [Name], one of your doctors."

[0127] Server: Sends this information to the device.

[0128] Terminal: Support information is converted into voice using speech synthesis technology and provided to the user.

[0129] Example 2: Directional indication

[0130] If the user forgets the location of the door in the room, visual data is captured and sent to the server.

[0131] The server does not analyze the visual data to detect the location of the door.

[0132] The server generates the supporting information "The door is on the right."

[0133] The server sends this information to the terminal.

[0134] The terminal uses voice synthesis technology to convert support information into voice and provide it to the user.

[0135] Example prompts to input to the generative AI model

[0136] Example prompt 1: "When a user asks who they are, analyze the speech data to determine signs of aphasia and generate supporting information."

[0137] Prompt example 2: "When the user forgets the location of the door through visual data, and the server analyzes and cannot determine the location of the door from the visual data."

[0138] As described above, this system is capable of analyzing visual and acoustic information in real time and providing appropriate support information in the form of voice to support the daily lives of patients with early-stage dementia.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] The user puts on the eyeglasses-type device. This activates the device's camera and microphone, and the device begins capturing visual and acoustic information. The user's wearing of the device is required as input, and the device's sensors are activated as output.

[0142] Step 2:

[0143] The device uses a camera to capture the user's visual information in real time and a microphone to record audio information. The input is the user's visual and audio environment, and the output is the captured visual and audio information.

[0144] Step 3:

[0145] The terminal temporarily stores the captured visual and audio information. The temporarily stored data is temporarily stored in a buffer. The captured visual and audio information is required as input, and the temporarily stored data is obtained as output.

[0146] Step 4:

[0147] The terminal compresses and encrypts the temporarily stored visual and audio information. The input is the temporarily stored data. The output is the compressed and encrypted data. Specifically, data compression uses a compression algorithm such as JPEG or MP3, and encryption uses an encryption algorithm such as AES.

[0148] Step 5:

[0149] The terminal sends compressed and encrypted data to the server. This transmission is done in real time with minimal latency. The input is the compressed and encrypted data. The output is the transmitted data.

[0150] Step 6:

[0151] The server receives the data sent from the terminal. After receiving it, it decompresses and decrypts the data. The input is the sent data, and the output is the decompressed and decrypted data. Specifically, it uses a decompression algorithm such as gzip or bzip2 for decompression, and the AES decryption algorithm for decryption.

[0152] Step 7:

[0153] The server analyzes the decompressed and decoded visual information using an image recognition algorithm. For example, it uses YOLOv5 or OpenCV to identify objects and people in the user's field of view. Visual information is required as input, and the analysis results are obtained as output. Specifically, the image recognition algorithm analyzes the visual information and recognizes objects and people.

[0154] Step 8:

[0155] The server converts the decompressed and decoded audio information into text data using a speech recognition algorithm and understands the user's speech and intent through natural language processing (NLP). The input is audio information, and the output is analyzed text data. Specifically, the speech recognition algorithm analyzes the audio information and converts it into text data, and then the NLP module analyzes the text data.

[0156] Step 9:

[0157] The server generates support information in real time based on the results of visual and audio data analysis. For example, if the user cannot recognize a person, it will convey the person's name and relationship to them. Also, if the user gets lost, it will instruct the user on the direction to go. The analysis results are required as input, and the generated support information is obtained as output.

[0158] Step 10:

[0159] The server compresses and encrypts the generated support information again and sends it to the terminal. The generated support information is required as input, and the compressed and encrypted support information is obtained as output.

[0160] Step 11:

[0161] The terminal decompresses and decodes the assistance information received from the server. As input, it takes compressed and encrypted assistance information, and as output, it gets the decompressed and decoded assistance information.

[0162] Step 12:

[0163] The device converts the decompressed and decoded assistance information into audio data using speech synthesis technology and provides it to the user through a built-in speaker. The decoded assistance information is required as input, and audio data is obtained as output. Specifically, the device uses speech synthesis technology to convert the assistance information into natural audio and provides it to the user audibly.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] Patients with early-stage dementia often face difficulties in their daily lives due to being unable to recognize objects and people and losing their sense of direction. This also increases the likelihood of them getting into dangerous situations, increasing their safety and the burden on their families and caregivers. To solve these problems, a system is needed that provides appropriate support in real time in the various situations patients face, and detects danger and issues warnings.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes a means for analyzing visual data and audio data in real time, a means for monitoring the user's surroundings based on the analysis results and detecting danger, and a means for generating warnings and advice when danger is detected. This makes it possible to provide real-time support for early-stage dementia patients to live their daily lives safely and appropriately avoid danger.

[0169] A "means for capturing visual data" is a device or method for acquiring images that appear in the user's field of vision using a device such as a camera.

[0170] "Means for capturing audio data" refers to a device or method for capturing sounds from the user and their surroundings using a device such as a microphone.

[0171] "Means for analyzing captured visual and audio data" refers to a device or method that uses a specific algorithm to understand the content of the acquired visual and audio data and generate analytical results.

[0172] The "means for generating support information in real time based on the analysis results" refers to a device or method for instantly generating support information required by the user based on the analysis results of the visual data and audio data.

[0173] The "means for providing the generated support information to the user" refers to a device or method for transmitting the generated support information to the user by audio or visual means.

[0174] The "means for monitoring the situation around the user based on the analysis results and detecting danger" refers to a device or method for monitoring the environment and behavior around the user based on the results of data analysis and identifying dangerous situations.

[0175] A "means for generating a warning or advice upon detection of a danger" is a device or method that provides an appropriate warning or advice to a user when a dangerous situation is identified.

[0176] This invention is a support system to help patients with early-stage dementia live their daily lives more safely. The system begins operation when the user puts on the smart glasses. The smart glasses have a built-in camera and microphone that capture visual and audio data. The captured data is temporarily stored in the device, compressed and encrypted, and then transmitted to a server in real time.

[0177] The server analyzes the received visual data using image recognition algorithms. Specifically, it uses image processing libraries such as OpenCV to identify objects and people in the visual data and determine what the user is looking at and the situation they are facing. It also uses speech recognition algorithms to convert audio data into text data and analyzes the user's intent using natural language processing techniques (e.g., NLTK, spaCy).

[0178] Based on the analysis results, the server generates support information in real time. Specifically, if the user does not recognize a person, it will tell them the person's name and relationship, or if the user gets lost, it will give them directions. The analysis results also monitor the user's surroundings, and if a dangerous situation is identified, it will generate appropriate warnings or advice (e.g., "The door is open" or "Did you forget to take your medicine?").

[0179] The generated support information is then provided to the user via the smart glasses. Using speech synthesis technology (e.g., gTTS), the text data is converted into speech and played back through the glasses' speakers, allowing the user to receive the necessary support in real time.

[0180] For example:

[0181] Example 1: Detecting an open door

[0182] If the camera does not detect the presence of a door even though the user frequently uses the word "door" in conversation, the server will generate a warning saying "Door is open" and notify the user audibly through the smart glasses.

[0183] Example 2: Medication reminders

[0184] The server analyzes the user's voice data, and if the word "medicine" is not detected within a certain period of time, it generates a reminder "Have you taken your medicine?" and notifies the user via voice through the smart glasses.

[0185] Example prompts for generative AI models

[0186] "Please use visual and audio data to identify dangerous situations that early-stage dementia patients face in their daily lives and generate appropriate warning information. The visual data should be a byte array in JPEG format, and the audio data should be in WAV format with a sampling rate of 8 kHz. Please output in the following format:

[0187] {

[0188] "status": "safe" or "danger",

[0189] "warning_text": "Danger warning message"

[0190] }"

[0191] This system allows people with early-stage dementia to receive the support they need in real time in their daily lives, significantly improving safety and reducing the burden on their families and caregivers.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] A user puts on smart glasses. The glasses are equipped with a camera and microphone, and begin capturing the user's visual and audio data. The input is the image in the user's field of vision and the surrounding audio, and the output is the captured visual and audio data.

[0195] Step 2:

[0196] The device temporarily stores the captured visual and audio data in a buffer. The stored data is compressed and encrypted. The input is the captured visual and audio data, and the output is the compressed and encrypted visual and audio data.

[0197] Step 3:

[0198] The terminal transmits compressed and encrypted visual and audio data to the server in real time. The input is the compressed and encrypted visual and audio data, and the output is the data transmission to the server.

[0199] Step 4:

[0200] The server analyzes the received visual data using an image recognition algorithm (e.g., OpenCV). The image recognition algorithm identifies objects and people in the visual data and determines what is in the user's field of view. The input is the visual data sent to the server, and the output is the analyzed visual information.

[0201] Step 5:

[0202] The server converts the received voice data into text data using a voice recognition algorithm and analyzes the user's intent using natural language processing technology (e.g., NLTK, spaCy). The input is the voice data sent to the server, and the output is the text data and analyzed intent information.

[0203] Step 6:

[0204] The server generates support information in real time based on the analysis results. The generated support information includes solutions to problems the user is facing and warnings. For example, it can be direction guidance if the user gets lost or person recognition information. The input is the analyzed visual information and intention information, and the output is support information and warning messages.

[0205] Step 7:

[0206] The server transmits the generated support information to the terminal again. The input is the generated support information, and the output is data transmission to the terminal.

[0207] Step 8:

[0208] The device provides the user with the support information received from the server. Using speech synthesis technology (e.g., gTTS), the support information is converted into speech and played back through the smart glasses' speaker. The input is the support information sent from the server, and the output is a voice notification to the user.

[0209] Step 9:

[0210] The user receives a voice notification and takes appropriate action, such as closing the door or taking medicine after receiving the warning. The input is the voice notification, and the output is the user's action.

[0211] In this way, all processing steps work together to provide real-time support to patients with early-stage dementia who are faced with problems and dangers in their daily lives, enabling them to maintain a safe life.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] The present invention incorporates an emotion engine into a system that captures and analyzes visual and audio data to support the daily lives of people with early-stage dementia, thereby recognizing the user's emotional state and providing personalized responses. The system and its embodiments are described in detail below.

[0214] Overall system configuration

[0215] This system consists of an eyeglass-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine. The eyeglass-type device has a built-in camera and microphone that captures the user's visual, audio, and emotional data. The captured data is sent in real time to the server, which analyzes it and generates appropriate support information. The generated support information is then provided to the user again via the eyeglass-type device.

[0216] Capture visual, audio, and emotional data

[0217] User

[0218] When a user wears the glasses-type device, the device's camera captures the image in the user's field of vision (visual data) and the user's facial expressions. The built-in microphone also records the user's and their surroundings' sounds (audio data). Emotional data is acquired by analyzing voice tone, facial expressions, and body movements.

[0219] Sending data

[0220] Terminal

[0221] The captured visual, audio, and emotional data is temporarily stored in a buffer on the device, then compressed, encrypted, and transmitted to a server in real time. This real-time communication minimizes the delay between the time the data is collected and the time it reaches the server.

[0222] Receiving and analyzing data

[0223] server

[0224] The server receives visual, audio, and emotional data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[0225] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[0226] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[0227] Emotion data analysis is performed by an emotion engine, which assesses the user's emotional state based on voice tone, facial expressions, and body movements.

[0228] Generate support information

[0229] server

[0230] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[0231] Additionally, the emotion engine's analysis can provide support based on the user's emotional state: for example, if the user is feeling anxious, it will provide instructions in a calmer, more reassuring voice.

[0232] Providing support information

[0233] Terminal

[0234] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[0235] Specific examples

[0236] Example 1: Person recognition

[0237] User: "Who are you?"

[0238] The device's microphone captures this audio and sends the data to a server.

[0239] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0240] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[0241] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0242] Example 2: Directional indication

[0243] If the user forgets where the door is in the room,

[0244] The device's camera captures the view and sends it to the server.

[0245] The server analyzes the visual data and does not detect the presence of a "door."

[0246] As support information, the message "The door is on the right" is generated and sent to the terminal.

[0247] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0248] Example 3: Emotion Recognition

[0249] If the user looks anxious,

[0250] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[0251] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[0252] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[0253] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0254] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[0255] The processing flow will be explained below.

[0256] Step 1:

[0257] The user puts on the eyeglasses, which allows the device to capture the user's visual, audio, and emotional data.

[0258] Step 2:

[0259] The device's camera continuously captures the image (visual data) of the user's field of vision, while the microphone records the user's and surrounding sounds (audio data). At the same time, the device captures the user's facial expressions, tone of voice, and body movements and records them as emotional data.

[0260] Step 3:

[0261] The device temporarily stores the captured visual, audio, and emotional data in a buffer memory, which is then compressed, encrypted, and sent to the server in real time.

[0262] Step 4:

[0263] The server receives the visual data, audio data, and emotion data sent from the terminal, and decompresses and decodes each data.

[0264] Step 5:

[0265] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[0266] Step 6:

[0267] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[0268] Step 7:

[0269] The server analyzes the emotion data and uses an emotion engine to assess the user's emotional state based on voice tone, facial expressions, and body movements.

[0270] Step 8:

[0271] Based on the analysis results, the server identifies the problem the user is facing and generates support information in real time. For example, if the user does not recognize a person, it will tell them the person's name and relationship to them. Also, if the user gets lost, it will give them directions to go.

[0272] Step 9:

[0273] The server uses the analysis results of the emotion engine to generate support information according to the user's emotional state. For example, if the user is feeling anxious, it will choose calm and reassuring words.

[0274] Step 10:

[0275] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[0276] Step 11:

[0277] The server transmits the generated voice data to the terminal.

[0278] Step 12:

[0279] The device plays back the received audio data, providing the user with support information and necessary advice in real time. For example, if the user gets lost, the device will provide voice instructions on the direction to go. If the user feels anxious, the device will offer reassuring words.

[0280] For example, if a user cannot recognize a person or gets lost, the server generates appropriate support information and provides it to the user via voice via the device. The emotion engine also evaluates the user's emotional state, and if the user is feeling anxious, for example, it provides reassuring words in a calm tone. This series of processes makes it possible to support the user's daily life in real time.

[0281] Example 2

[0282] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] Until now, there has been no system that can effectively support the daily lives of patients with early-stage dementia by collecting and analyzing comprehensive data, including not only visual and audio data but also emotional data, in real time. As a result, real-time support including the user's emotional state has not been provided, and difficulties in daily life have not been adequately resolved.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0285] In this invention, the server includes means for analyzing visual data using an image recognition algorithm, means for analyzing audio data using a voice recognition algorithm, and means for analyzing emotional data using an emotion engine, thereby enabling comprehensive analysis of visual, audio, and emotional data and providing real-time support information based on the user's emotional state.

[0286] "Visual data" refers to image and video data acquired by a device that captures visual information, such as a camera.

[0287] "Audio data" refers to data that includes sounds and voices acquired by an audio input device such as a microphone.

[0288] "Emotional data" is data used to analyze a user's emotional state by capturing vocal tones, facial expressions, body movements, etc.

[0289] An "image recognition algorithm" is an algorithm that automatically detects and recognizes specific objects, people, and scenes from images and videos.

[0290] A "voice recognition algorithm" is an algorithm for analyzing voice data and converting the voice into text data.

[0291] The "emotion engine" is software that evaluates and analyzes the user's emotional state based on captured data.

[0292] "Support information" refers to information and instructions generated based on the results of analyzing the user's visual data, audio data, and emotional data.

[0293] "Real-time" refers to the extremely short time between data collection and the generation of analysis results, meaning that processing is immediate.

[0294] This invention is a system that captures visual, audio, and emotional data and analyzes them in real time to support the daily lives of patients with early-stage dementia. This system consists of a glasses-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine.

[0295] Overall system configuration

[0296] User

[0297] The user wears a pair of glasses equipped with a camera and microphone that captures the user's visual, audio, and emotional data.

[0298] Terminal

[0299] The captured visual, audio, and emotion data is temporarily stored in a buffer on the device. This data is then compressed and encrypted in real time and sent to the server. Specifically, image data is converted to JPEG format, audio data is converted to AAC format, etc., and the data is encrypted using the SSL / TLS communication protocol before being sent to the server.

[0300] Sending data

[0301] server

[0302] The server receives the visual, audio, and emotion data sent from the device. The received data is immediately analyzed. Specifically, the following process is performed:

[0303] Visual data analysis: Using image recognition algorithms (e.g., OpenCV) on the server, objects and people are identified in the video.

[0304] Analyzing speech data: Using speech recognition algorithms (e.g., Google Cloud Speech-to-Text API) to convert speech into text data, and then applying natural language processing (NLP) techniques to understand the content.

[0305] Emotional data analysis: Using an emotion engine (e.g., Affectiva) to assess emotional state from vocal tone, facial expressions, and body movements.

[0306] Generate support information

[0307] server

[0308] Based on the analysis results, the server generates the support information the user needs in real time. For example, if a user looks at a friend's face and asks, "Who are you?", this voice is captured and analyzed by the server, and an appropriate answer (e.g., "This is [Name]. I'm one of your friends.") is generated. If the user gets lost, the situation is analyzed from visual data and directional instructions such as "The door is on the right" are generated. If the user looks anxious, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated based on the analysis of emotional data.

[0309] Providing support information

[0310] Terminal

[0311] The generated support information is then sent back to the device and provided to the user via voice. Using speech synthesis technology (e.g., Amazon Polly), the support information is converted into natural-sounding speech and presented to the user through the device's speaker.

[0312] Specific examples of embodiments

[0313] Example 1: Person recognition

[0314] User: "Who are you?"

[0315] The device's microphone captures this audio and sends it to the server.

[0316] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0317] Support information such as "This is [name], one of your friends" is generated and sent to the device.

[0318] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0319] Example 2: Directional indication

[0320] If the user forgets where the door is in the room,

[0321] The device's camera captures the view and sends it to the server.

[0322] The server does not analyze the visual data to detect the presence of a "door."

[0323] The support information "The door is on the right" is generated and sent to the terminal.

[0324] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0325] Example 3: Emotion Recognition

[0326] If the user looks anxious,

[0327] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[0328] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[0329] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[0330] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0331] The above is a specific embodiment of the present invention. This system enables real-time support for early-stage dementia patients in dealing with the various difficulties they face in their daily lives, and is expected to improve the user's quality of life and reduce the burden on family and caregivers.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1: Data Capture

[0334] User

[0335] The user wears the eyeglass-type device.

[0336] Input: The environment in which the user generates visual and audio data.

[0337] Output: Visual, audio and emotional data captured by the device.

[0338] How it works: The camera captures the user's field of view in real time, and the microphone records the user's speech and the surrounding environmental sounds. It also captures the tone of voice and facial expressions as emotional data.

[0339] Step 2: Temporarily save data

[0340] Terminal

[0341] The captured visual data, audio data, and emotion data are temporarily stored in the device's built-in buffer.

[0342] Input: Visual, audio, and emotional data captured by the device.

[0343] Output: The data stored in the buffer.

[0344] Specific operation: Temporarily stores captured data in memory and checks whether the data is in a suitable format for further processing.

[0345] Step 3: Compress and Encrypt the Data

[0346] Terminal

[0347] The stored data is compressed and encrypted and prepared for transmission to the server.

[0348] Input: Visual, audio and emotion data in a buffer.

[0349] Output: Compressed and encrypted data.

[0350] Specific operations: Image data is compressed to JPEG format, audio data is converted to AAC format, and data is encrypted using the SSL / TLS protocol.

[0351] Step 4: Sending data

[0352] Terminal

[0353] The compressed and encrypted data is sent to the server in real time.

[0354] Input: Compressed and encrypted data.

[0355] Output: The data received by the server.

[0356] Specific operation: Send data to the server through the terminal's communication module and confirm the success of the transmission.

[0357] Step 5: Receiving the data

[0358] server

[0359] The server receives the data sent from the terminal.

[0360] Input: Compressed and encrypted data sent from the terminal.

[0361] Output: Raw data stored in the server.

[0362] Specific operation: The server decrypts and unpacks the received data and prepares it for analysis.

[0363] Step 6: Analyze the visual data

[0364] server

[0365] The server analyzes the received visual data using image recognition algorithms.

[0366] Input: Decoded and unpacked visual data.

[0367] Output: Analysis results (e.g., identification of specific objects or people).

[0368] Specific operation: Using the OpenCV library, it identifies objects and people from video and identifies what the user is looking at.

[0369] Step 7: Analyze the audio data

[0370] server

[0371] The server analyzes the received voice data using a voice recognition algorithm.

[0372] Input: Decoded and decompressed audio data.

[0373] Output: Analysis results (e.g., converting audio into text data).

[0374] Specific operation: Uses Google Cloud Speech-to-Text API to convert speech into text data and uses NLP technology to understand the content.

[0375] Step 8: Analyze the sentiment data

[0376] server

[0377] The emotion data received by the server is analyzed using an emotion engine.

[0378] Input: Decoded and unpacked emotion data.

[0379] Output: Analysis results (e.g., assessment of the user's emotional state).

[0380] Specific behavior: Using the Affectiva emotion engine, we assess the user's emotional state from their vocal tone, facial expressions, and body movements.

[0381] Step 9: Generate support information

[0382] server

[0383] Support information is generated in real time based on the analysis results.

[0384] Input: Analysis results of visual, audio and emotion data.

[0385] Output: The generated supporting information.

[0386] Specific operation: Generates the solutions and advice the user needs in text format and creates support information tailored to the user's situation.

[0387] Step 10: Provide support information

[0388] Terminal

[0389] The generated support information is sent again to the terminal and provided to the user by voice.

[0390] Input: Generated supporting information.

[0391] Output: Natural-sounding voice delivered through speech synthesis.

[0392] Specific operation: Using speech synthesis technology such as Amazon Polly, the generated text data is converted into speech and provided to the user through the device's speaker.

[0393] (Application example 2)

[0394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] To ensure the efficiency and safety of workers in factories, it is necessary to provide appropriate work instructions and safety warnings in real time. However, conventional methods have made it difficult to provide appropriate support that takes into account the emotional state, fatigue, and stress of workers. In addition, since rapid situational assessment is required on-site, a real-time analysis system with minimal delay is required.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0397] In this invention, the server includes means for capturing visual data, means for capturing audio data, means for analyzing the captured visual data and audio data, means for generating support information in real time based on the analysis results, means for providing the generated support information to the user, means for capturing and analyzing emotional data, and means for generating and providing work instructions, safety warnings, and information for fatigue and stress management based on the analysis results. This enables appropriate support that takes into account the emotional state, fatigue, and stress of the worker, and rapid situation assessment.

[0398] "Visual data" is video data that appears in the user's field of vision and is information captured by a camera.

[0399] "Audio data" refers to data that indicates audio information of the user and their surroundings and is recorded by a microphone.

[0400] "Analysis means" refers to the systems and algorithms that process the captured visual and audio data and generate appropriate information based on the results.

[0401] "Support information" is information such as specific instructions and advice generated by the analysis means to support the user's daily life and work.

[0402] "Emotion data" is information about the user's emotional state obtained by analyzing voice tone, facial expressions, body movements, and the like.

[0403] "Work instructions" are information that guides the user on the specific work content and procedures that should be performed.

[0404] "Safety warnings" are information that alert users when they are faced with a dangerous situation.

[0405] "Fatigue and stress management" is information that detects fatigue and stress from the user's facial expressions, voice, etc., and suggests measures to take or take a break.

[0406] The present invention is a system that uses a head-mounted display (hereinafter referred to as a smart helmet) worn by a worker, and aims to support work efficiently and safely by capturing and analyzing visual data, audio data, and emotional data. The specific configuration and operation of this system are described below.

[0407] Overall system configuration

[0408] The system consists of a smart helmet worn by the user, a server that receives and analyzes data, and a means of providing the user with information generated based on the analysis results.The smart helmet has a built-in camera and microphone that captures the user's visual and audio data, as well as emotional data.

[0409] Capture visual, audio, and emotional data

[0410] When a user wears a smart helmet, the built-in camera captures visual data (images seen by the user) and facial expressions, and the built-in microphone records audio data of the user and their surroundings. This data is captured in real time and sent to a server for analysis.

[0411] Sending data

[0412] The smart helmet temporarily stores captured visual, audio, and emotional data in a buffer, which is then compressed, encrypted, and sent to a server in real time. This low-latency communication allows for real-time analysis.

[0413] Receiving and analyzing data

[0414] The server receives and analyzes data sent from the smart helmet in real time. It uses an image recognition algorithm to analyze visual data and recognize specific objects and people. It uses a voice recognition algorithm to analyze audio data, converting it into text data and then analyzing the user's intent using natural language processing (NLP). It uses an emotion engine to analyze emotional data, which allows it to evaluate the worker's emotional state, as well as their fatigue and stress levels.

[0415] Generate support information

[0416] Based on the analysis results, the server generates supporting information in real time, such as:

[0417] Work Instructions: Understand what you're currently doing and provide specific instructions on what to do next.

[0418] Safety warning: When a dangerous situation is detected, it provides warning information and instructions on how to avoid it.

[0419] Fatigue and stress management: Detects worker fatigue and stress and suggests breaks.

[0420] This support information is generated in real time and sent back to the smart helmet, where it is provided to the user audibly using voice synthesis technology.

[0421] Specific examples

[0422] Example 1: Work instruction support

[0423] If a worker forgets a particular step and looks confused, the smart helmet captures video and audio and sends them to the server, which analyzes the procedure and generates instructions, such as "Next, pull the left lever."

[0424] Example prompt: "If I forget this step, tell me what to do next."

[0425] Example 2: Safety warning

[0426] When a worker approaches a dangerous area, the camera in the smart helmet captures the surrounding image and sends it to the server. The server analyzes the image and, if it detects a dangerous area, generates an audio warning saying, "Please retreat. This is dangerous."

[0427] Example prompt: "If a worker enters a dangerous area, give instructions to evacuate immediately."

[0428] Example 3: Fatigue and stress management

[0429] If a worker shows signs of fatigue, the smart helmet captures facial expressions and voice data and sends it to a server. The server analyzes the data, and if it detects fatigue, it suggests, "It's time for a break. Let's take a short rest."

[0430] Sample prompt: "If a worker is fatigued, how can you encourage them to take a break?"

[0431] In this way, the system of the present invention can provide real-time support to workers in dealing with various difficulties they may face, thereby improving worker efficiency and ensuring safety.

[0432] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0433] Step 1:

[0434] The smart helmet captures visual and audio data.

[0435] Specifically, the smart helmet acquires visual information from the user using a built-in camera, and audio information from a built-in microphone.

[0436] Input: User's visual and audio information

[0437] Output: Captured visual and audio data

[0438] Step 2:

[0439] The visual and audio data captured by the smart helmet is temporarily stored in a buffer.

[0440] This prepares the data for analysis in real time.

[0441] Input: Captured visual and audio data

[0442] Output: Data stored in the buffer

[0443] Step 3:

[0444] The smart helmet compresses and encrypts the buffered data and sends it to the server.

[0445] Since data is transmitted in real time, data size is reduced and high security is maintained.

[0446] Input: Data stored in the buffer

[0447] Output: Compressed and encrypted data

[0448] Step 4:

[0449] The server receives the data transmitted from the smart helmet.

[0450] The received data is decoded for analysis.

[0451] Input: Compressed and encrypted data

[0452] Output: Decoded visual, audio, and emotion data

[0453] Step 5:

[0454] The server analyzes the visual data and recognizes specific objects and people.

[0455] Uses image recognition algorithms (e.g., OpenCV or TensorFlow)

[0456] Input: Decoded visual data

[0457] Output: Analysis results (information on recognized objects and people)

[0458] Step 6:

[0459] The server analyzes the voice data and converts it into text data.

[0460] Uses a speech recognition algorithm (e.g., Google Speech-to-Text API)

[0461] Input: Decoded audio data

[0462] Output: Converted text data

[0463] Step 7:

[0464] The server analyzes the user's intent based on the text data.

[0465] Uses natural language processing (NLP) techniques (e.g. spaCy and BERT)

[0466] Input: Converted text data

[0467] Output: Analysis results (user intent information)

[0468] Step 8:

[0469] The server analyzes the emotional data and evaluates the user's emotional state.

[0470] Use an emotion engine (e.g., Emotion API)

[0471] Input: Decoded emotion data

[0472] Output: Analysis result (user's emotional state)

[0473] Step 9:

[0474] A server generates work instructions, safety warnings, and supporting information for fatigue and stress management.

[0475] Use a generative AI model (e.g. GPT-3)

[0476] Input: Analysis results (object / person information, user intent, emotional state)

[0477] Output: Supporting information (work instructions, warnings, break suggestions)

[0478] Step 10:

[0479] The server transmits the generated support information to the smart helmet.

[0480] Input: Support Information

[0481] Output: Support information sent to the smart helmet

[0482] Step 11:

[0483] The smart helmet provides support information to the user via voice.

[0484] Uses voice synthesis technology (e.g., Amazon Polly)

[0485] Input: Support Information

[0486] Output: Supporting information provided by voice

[0487] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0488] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0489] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0490] [Second embodiment]

[0491] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0492] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0493] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0494] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0495] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0496] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0497] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0498] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0499] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0500] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0501] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0502] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0503] The present invention is a system for supporting the daily lives of patients with early-stage dementia, capturing visual and audio data, analyzing the data, and providing appropriate support information in real time. The system and its embodiments are described in detail below.

[0504] Overall system configuration

[0505] This system consists of a glasses-type device (terminal) worn by the user and a server that receives and analyzes the data. The glasses-type device has a built-in camera and microphone that captures the user's visual and audio data. The captured data is sent to the server in real time, where it is analyzed and appropriate support information is generated. The generated support information is then provided to the user via the glasses-type device.

[0506] Visual and audio data capture

[0507] User

[0508] When a user wears the eyeglasses, the camera in the glasses continuously captures images (visual data) within the user's field of vision, and the built-in microphone records the user's and their surrounding sounds (audio data).

[0509] Sending data

[0510] Terminal

[0511] The captured visual and audio data is temporarily stored in a buffer on the device, then compressed, encrypted, and sent to a server. This communication occurs in real time, minimizing the delay between when the data is collected and when it reaches the server.

[0512] Receiving and analyzing data

[0513] server

[0514] The server receives visual and audio data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[0515] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[0516] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[0517] Generate support information

[0518] server

[0519] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[0520] Providing support information

[0521] Terminal

[0522] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[0523] Specific examples

[0524] Example 1: Person recognition

[0525] User: "Who are you?"

[0526] The device's microphone captures this audio and sends the data to a server.

[0527] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0528] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[0529] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0530] Example 2: Directional indication

[0531] If the user forgets where the door is in the room,

[0532] The device's camera captures the view and sends it to the server.

[0533] The server analyzes the visual data and does not detect the presence of a "door."

[0534] As support information, the message "The door is on the right" is generated and sent to the terminal.

[0535] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0536] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[0537] The processing flow will be explained below.

[0538] Step 1:

[0539] The user puts on the eyeglasses-type device, which enables the device to capture the user's visual and audio data.

[0540] Step 2:

[0541] The device's camera continuously captures the image of the user's field of vision (visual data), and the microphone records the user's and surrounding sounds (audio data).

[0542] Step 3:

[0543] The device temporarily stores the captured visual and audio data in a buffer memory, where it is compressed and encrypted in real time.

[0544] Step 4:

[0545] The device transmits compressed and encrypted visual and audio data to the server, and this communication occurs in real time to minimize latency.

[0546] Step 5:

[0547] The server receives the visual and audio data sent from the terminal, and decompresses and decodes the respective data.

[0548] Step 6:

[0549] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[0550] Step 7:

[0551] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[0552] Step 8:

[0553] The server analyzes visual and audio data to identify the problem the user is facing and generates real-time support information, such as the name of a person if the user does not recognize them and their relationship to the person.

[0554] Step 9:

[0555] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[0556] Step 10:

[0557] The server transmits the generated voice data to the terminal.

[0558] Step 11:

[0559] The device plays back the received audio data, providing support information to the user through audio and giving necessary advice in real time. For example, if the user gets lost, audio instructions will be given on the direction they should go.

[0560] Example 1

[0561] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0562] Patients with early-stage dementia may experience problems in their daily lives, such as difficulty recognizing objects and people, or a lack of understanding of intentions. This not only makes it difficult for patients to make appropriate decisions and reduces their quality of life, but also places a burden on caregivers and family members. There is a need for a support system that can solve these problems and enable patients to live their daily lives more smoothly.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0564] In this invention, the server includes means for acquiring visual information, means for acquiring acoustic information, means for temporarily storing the acquired visual information and acoustic information and compressing and encrypting it, means for transmitting the compressed and encrypted information in real time, means for analyzing the received visual information and acoustic information, means for generating support information in real time based on the analysis results, and means for converting the generated support information into audio and providing it to the user. This makes it possible to provide real-time support for the difficulties faced by early-stage dementia patients in their daily lives and improve the quality of life of the patients.

[0565] "Visual information" refers to image data captured in the user's field of vision by an image capture device such as a camera in a glasses-type device.

[0566] "Acoustic information" refers to audio data emitted from the user and the environment, which is acquired by an audio acquisition device such as a microphone of the eyeglass-type device.

[0567] "Temporary storage" refers to a process of temporarily storing acquired visual and acoustic information in memory or storage.

[0568] "Compression" is data processing performed to reduce the data volume of visual and audio information.

[0569] "Encryption" is the process of protecting data with encryption technology in order to securely transmit visual and audio information.

[0570] "Transmission" is the process of sending compressed and encrypted visual and audio information from the terminal to the server over the network.

[0571] "Reception" refers to the process in which the server receives visual and audio information sent from the terminal.

[0572] "Analysis" refers to analyzing received visual information using image recognition algorithms, and converting acoustic information into text data using speech recognition algorithms and analyzing it through natural language processing.

[0573] "Support information" is information that includes advice and instructions for the user based on the analysis results, and is information that is generated in real time for problem solving.

[0574] "Conversion to voice" refers to a process of converting the generated assistance information into voice data using voice synthesis technology so that it can be provided to the user audibly.

[0575] "Providing to the user" refers to providing the converted audio data in a form that the user can hear through the speaker of the eyeglass-type device, etc.

[0576] The present invention is a system for supporting the daily lives of patients with early-stage dementia. This system captures visual and acoustic information, analyzes the acquired data, and provides support information in real time. Detailed embodiments of the present invention will be described below.

[0577] Hardware Configuration

[0578] This system consists of a glasses-type device (hereafter referred to as the terminal) worn by the user and a server that receives and analyzes data. The terminal has a built-in camera and microphone, which are responsible for capturing visual and acoustic information. The server is a general-purpose computer equipped with a high-performance CPU and GPU.

[0579] Software Configuration

[0580] The device is installed with software for capturing visual and acoustic information: driver software for acquiring images from the camera and audio processing software for acquiring audio from the microphone.

[0581] The server has the following software modules installed:

[0582] Image recognition algorithms: e.g., YOLOv5 and OpenCV

[0583] Speech recognition algorithms, such as Google Cloud Speech-to-Text or Microsoft Azure Speech Service

[0584] Natural Language Processing (NLP) modules: e.g., BERT models

[0585] Speech synthesis technology: e.g., Google Text-to-Speech or Amazon Polly

[0586] Operating procedures and data processing

[0587] When a user wears the glasses, the device's camera captures the user's visual information in real time, and the microphone records audio information. The device temporarily stores this data, compresses and encrypts it, and then transmits it to a server.

[0588] The server receives the visual and audio information sent from the device, first decompresses and decodes the data, then uses an image recognition algorithm to analyze the visual information and identify objects and people in the user's field of view, and simultaneously uses a speech recognition algorithm to convert the audio information into text data and understand the user's speech and intent through natural language processing.

[0589] Based on the analysis results, the server generates real-time support information. For example, if the user does not recognize a person, it will provide the person's name and relationship to the person. If the user gets lost, it will provide advice on the direction to go. This support information is then compressed and encrypted again before being sent to the device.

[0590] The device converts the received assistance information into voice data using voice synthesis technology and plays it back through a built-in speaker, allowing users to obtain the necessary information in real time by voice.

[0591] Specific examples

[0592] Example 1: Person recognition

[0593] User: Ask "Who are you?"

[0594] Device: The microphone captures this audio and sends the data to the server.

[0595] Server: Analyzes the voice data and understands the user's question.

[0596] Server: Analyzes visual data to identify the person being questioned.

[0597] Server: Generate supporting information: "This is [Name], one of your doctors."

[0598] Server: Sends this information to the device.

[0599] Terminal: Support information is converted into voice using speech synthesis technology and provided to the user.

[0600] Example 2: Directional indication

[0601] If the user forgets the location of the door in the room, visual data is captured and sent to the server.

[0602] The server does not analyze the visual data to detect the location of the door.

[0603] The server generates the supporting information "The door is on the right."

[0604] The server sends this information to the terminal.

[0605] The terminal uses voice synthesis technology to convert support information into voice and provide it to the user.

[0606] Example prompts to input to the generative AI model

[0607] Example prompt 1: "When a user asks who they are, analyze the speech data to determine signs of aphasia and generate supporting information."

[0608] Prompt example 2: "When the user forgets the location of the door through visual data, and the server analyzes and cannot determine the location of the door from the visual data."

[0609] As described above, this system is capable of analyzing visual and acoustic information in real time and providing appropriate support information in the form of voice to support the daily lives of patients with early-stage dementia.

[0610] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0611] Step 1:

[0612] The user puts on the eyeglasses-type device. This activates the device's camera and microphone, and the device begins capturing visual and acoustic information. The user's wearing of the device is required as input, and the device's sensors are activated as output.

[0613] Step 2:

[0614] The device uses a camera to capture the user's visual information in real time and a microphone to record audio information. The input is the user's visual and audio environment, and the output is the captured visual and audio information.

[0615] Step 3:

[0616] The terminal temporarily stores the captured visual and audio information. The temporarily stored data is temporarily stored in a buffer. The captured visual and audio information is required as input, and the temporarily stored data is obtained as output.

[0617] Step 4:

[0618] The terminal compresses and encrypts the temporarily stored visual and audio information. The input is the temporarily stored data. The output is the compressed and encrypted data. Specifically, data compression uses a compression algorithm such as JPEG or MP3, and encryption uses an encryption algorithm such as AES.

[0619] Step 5:

[0620] The terminal sends compressed and encrypted data to the server. This transmission is done in real time with minimal latency. The input is the compressed and encrypted data. The output is the transmitted data.

[0621] Step 6:

[0622] The server receives the data sent from the terminal. After receiving it, it decompresses and decrypts the data. The input is the sent data, and the output is the decompressed and decrypted data. Specifically, it uses a decompression algorithm such as gzip or bzip2 for decompression, and the AES decryption algorithm for decryption.

[0623] Step 7:

[0624] The server analyzes the decompressed and decoded visual information using an image recognition algorithm. For example, it uses YOLOv5 or OpenCV to identify objects and people in the user's field of view. Visual information is required as input, and the analysis results are obtained as output. Specifically, the image recognition algorithm analyzes the visual information and recognizes objects and people.

[0625] Step 8:

[0626] The server converts the decompressed and decoded audio information into text data using a speech recognition algorithm and understands the user's speech and intent through natural language processing (NLP). The input is audio information, and the output is analyzed text data. Specifically, the speech recognition algorithm analyzes the audio information and converts it into text data, and then the NLP module analyzes the text data.

[0627] Step 9:

[0628] The server generates support information in real time based on the results of visual and audio data analysis. For example, if the user cannot recognize a person, it will convey the person's name and relationship to them. Also, if the user gets lost, it will instruct the user on the direction to go. The analysis results are required as input, and the generated support information is obtained as output.

[0629] Step 10:

[0630] The server compresses and encrypts the generated support information again and sends it to the terminal. The generated support information is required as input, and the compressed and encrypted support information is obtained as output.

[0631] Step 11:

[0632] The terminal decompresses and decodes the assistance information received from the server. As input, it takes compressed and encrypted assistance information, and as output, it gets the decompressed and decoded assistance information.

[0633] Step 12:

[0634] The device converts the decompressed and decoded assistance information into audio data using speech synthesis technology and provides it to the user through a built-in speaker. The decoded assistance information is required as input, and audio data is obtained as output. Specifically, the device uses speech synthesis technology to convert the assistance information into natural audio and provides it to the user audibly.

[0635] (Application example 1)

[0636] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] Patients with early-stage dementia often face difficulties in their daily lives due to being unable to recognize objects and people and losing their sense of direction. This also increases the likelihood of them getting into dangerous situations, increasing their safety and the burden on their families and caregivers. To solve these problems, a system is needed that provides appropriate support in real time in the various situations patients face, and detects danger and issues warnings.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0639] In this invention, the server includes a means for analyzing visual data and audio data in real time, a means for monitoring the user's surroundings based on the analysis results and detecting danger, and a means for generating warnings and advice when danger is detected. This makes it possible to provide real-time support for early-stage dementia patients to live their daily lives safely and appropriately avoid danger.

[0640] A "means for capturing visual data" is a device or method for acquiring images that appear in the user's field of vision using a device such as a camera.

[0641] "Means for capturing audio data" refers to a device or method for capturing sounds from the user and their surroundings using a device such as a microphone.

[0642] "Means for analyzing captured visual and audio data" refers to a device or method that uses a specific algorithm to understand the content of the acquired visual and audio data and generate analytical results.

[0643] The "means for generating support information in real time based on the analysis results" refers to a device or method for instantly generating support information required by the user based on the analysis results of the visual data and audio data.

[0644] The "means for providing the generated support information to the user" refers to a device or method for transmitting the generated support information to the user by audio or visual means.

[0645] The "means for monitoring the situation around the user based on the analysis results and detecting danger" refers to a device or method for monitoring the environment and behavior around the user based on the results of data analysis and identifying dangerous situations.

[0646] A "means for generating a warning or advice upon detection of a danger" is a device or method that provides an appropriate warning or advice to a user when a dangerous situation is identified.

[0647] This invention is a support system to help patients with early-stage dementia live their daily lives more safely. The system begins operation when the user puts on the smart glasses. The smart glasses have a built-in camera and microphone that capture visual and audio data. The captured data is temporarily stored in the device, compressed and encrypted, and then transmitted to a server in real time.

[0648] The server analyzes the received visual data using image recognition algorithms. Specifically, it uses image processing libraries such as OpenCV to identify objects and people in the visual data and determine what the user is looking at and the situation they are facing. It also uses speech recognition algorithms to convert audio data into text data and analyzes the user's intent using natural language processing techniques (e.g., NLTK, spaCy).

[0649] Based on the analysis results, the server generates support information in real time. Specifically, if the user does not recognize a person, it will tell them the person's name and relationship, or if the user gets lost, it will give them directions. The analysis results also monitor the user's surroundings, and if a dangerous situation is identified, it will generate appropriate warnings or advice (e.g., "The door is open" or "Did you forget to take your medicine?").

[0650] The generated support information is then provided to the user via the smart glasses. Using speech synthesis technology (e.g., gTTS), the text data is converted into speech and played back through the glasses' speakers, allowing the user to receive the necessary support in real time.

[0651] For example:

[0652] Example 1: Detecting an open door

[0653] If the camera does not detect the presence of a door even though the user frequently uses the word "door" in conversation, the server will generate a warning saying "Door is open" and notify the user audibly through the smart glasses.

[0654] Example 2: Medication reminders

[0655] The server analyzes the user's voice data, and if the word "medicine" is not detected within a certain period of time, it generates a reminder "Have you taken your medicine?" and notifies the user via voice through the smart glasses.

[0656] Example prompts for generative AI models

[0657] "Please use visual and audio data to identify dangerous situations that early-stage dementia patients face in their daily lives and generate appropriate warning information. The visual data should be a byte array in JPEG format, and the audio data should be in WAV format with a sampling rate of 8 kHz. Please output in the following format:

[0658] {

[0659] "status": "safe" or "danger",

[0660] "warning_text": "Danger warning message"

[0661] }"

[0662] This system allows people with early-stage dementia to receive the support they need in real time in their daily lives, significantly improving safety and reducing the burden on their families and caregivers.

[0663] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0664] Step 1:

[0665] A user puts on smart glasses. The glasses are equipped with a camera and microphone, and begin capturing the user's visual and audio data. The input is the image in the user's field of vision and the surrounding audio, and the output is the captured visual and audio data.

[0666] Step 2:

[0667] The device temporarily stores the captured visual and audio data in a buffer. The stored data is compressed and encrypted. The input is the captured visual and audio data, and the output is the compressed and encrypted visual and audio data.

[0668] Step 3:

[0669] The terminal transmits compressed and encrypted visual and audio data to the server in real time. The input is the compressed and encrypted visual and audio data, and the output is the data transmission to the server.

[0670] Step 4:

[0671] The server analyzes the received visual data using an image recognition algorithm (e.g., OpenCV). The image recognition algorithm identifies objects and people in the visual data and determines what is in the user's field of view. The input is the visual data sent to the server, and the output is the analyzed visual information.

[0672] Step 5:

[0673] The server converts the received voice data into text data using a voice recognition algorithm and analyzes the user's intent using natural language processing technology (e.g., NLTK, spaCy). The input is the voice data sent to the server, and the output is the text data and analyzed intent information.

[0674] Step 6:

[0675] The server generates support information in real time based on the analysis results. The generated support information includes solutions to problems the user is facing and warnings. For example, it can be direction guidance if the user gets lost or person recognition information. The input is the analyzed visual information and intention information, and the output is support information and warning messages.

[0676] Step 7:

[0677] The server transmits the generated support information to the terminal again. The input is the generated support information, and the output is data transmission to the terminal.

[0678] Step 8:

[0679] The device provides the user with the support information received from the server. Using speech synthesis technology (e.g., gTTS), the support information is converted into speech and played back through the smart glasses' speaker. The input is the support information sent from the server, and the output is a voice notification to the user.

[0680] Step 9:

[0681] The user receives a voice notification and takes appropriate action, such as closing the door or taking medicine after receiving the warning. The input is the voice notification, and the output is the user's action.

[0682] In this way, all processing steps work together to provide real-time support to patients with early-stage dementia who are faced with problems and dangers in their daily lives, enabling them to maintain a safe life.

[0683] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0684] The present invention incorporates an emotion engine into a system that captures and analyzes visual and audio data to support the daily lives of people with early-stage dementia, thereby recognizing the user's emotional state and providing personalized responses. The system and its embodiments are described in detail below.

[0685] Overall system configuration

[0686] This system consists of an eyeglass-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine. The eyeglass-type device has a built-in camera and microphone that captures the user's visual, audio, and emotional data. The captured data is sent in real time to the server, which analyzes it and generates appropriate support information. The generated support information is then provided to the user again via the eyeglass-type device.

[0687] Capture visual, audio, and emotional data

[0688] User

[0689] When a user wears the glasses-type device, the device's camera captures the image in the user's field of vision (visual data) and the user's facial expressions. The built-in microphone also records the user's and their surroundings' sounds (audio data). Emotional data is acquired by analyzing voice tone, facial expressions, and body movements.

[0690] Sending data

[0691] Terminal

[0692] The captured visual, audio, and emotional data is temporarily stored in a buffer on the device, then compressed, encrypted, and transmitted to a server in real time. This real-time communication minimizes the delay between the time the data is collected and the time it reaches the server.

[0693] Receiving and analyzing data

[0694] server

[0695] The server receives visual, audio, and emotional data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[0696] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[0697] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[0698] Emotion data analysis is performed by an emotion engine, which assesses the user's emotional state based on voice tone, facial expressions, and body movements.

[0699] Generate support information

[0700] server

[0701] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[0702] Additionally, the emotion engine's analysis can provide support based on the user's emotional state: for example, if the user is feeling anxious, it will provide instructions in a calmer, more reassuring voice.

[0703] Providing support information

[0704] Terminal

[0705] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[0706] Specific examples

[0707] Example 1: Person recognition

[0708] User: "Who are you?"

[0709] The device's microphone captures this audio and sends the data to a server.

[0710] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0711] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[0712] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0713] Example 2: Directional indication

[0714] If the user forgets where the door is in the room,

[0715] The device's camera captures the view and sends it to the server.

[0716] The server analyzes the visual data and does not detect the presence of a "door."

[0717] As support information, the message "The door is on the right" is generated and sent to the terminal.

[0718] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0719] Example 3: Emotion Recognition

[0720] If the user looks anxious,

[0721] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[0722] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[0723] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[0724] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0725] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[0726] The processing flow will be explained below.

[0727] Step 1:

[0728] The user puts on the eyeglasses, which allows the device to capture the user's visual, audio, and emotional data.

[0729] Step 2:

[0730] The device's camera continuously captures the image (visual data) of the user's field of vision, while the microphone records the user's and surrounding sounds (audio data). At the same time, the device captures the user's facial expressions, tone of voice, and body movements and records them as emotional data.

[0731] Step 3:

[0732] The device temporarily stores the captured visual, audio, and emotional data in a buffer memory, which is then compressed, encrypted, and sent to the server in real time.

[0733] Step 4:

[0734] The server receives the visual data, audio data, and emotion data sent from the terminal, and decompresses and decodes each data.

[0735] Step 5:

[0736] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[0737] Step 6:

[0738] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[0739] Step 7:

[0740] The server analyzes the emotion data and uses an emotion engine to assess the user's emotional state based on voice tone, facial expressions, and body movements.

[0741] Step 8:

[0742] Based on the analysis results, the server identifies the problem the user is facing and generates support information in real time. For example, if the user does not recognize a person, it will tell them the person's name and relationship to them. Also, if the user gets lost, it will give them directions to go.

[0743] Step 9:

[0744] The server uses the analysis results of the emotion engine to generate support information according to the user's emotional state. For example, if the user is feeling anxious, it will choose calm and reassuring words.

[0745] Step 10:

[0746] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[0747] Step 11:

[0748] The server transmits the generated voice data to the terminal.

[0749] Step 12:

[0750] The device plays back the received audio data, providing the user with support information and necessary advice in real time. For example, if the user gets lost, the device will provide voice instructions on the direction to go. If the user feels anxious, the device will offer reassuring words.

[0751] For example, if a user cannot recognize a person or gets lost, the server generates appropriate support information and provides it to the user via voice via the device. The emotion engine also evaluates the user's emotional state, and if the user is feeling anxious, for example, it provides reassuring words in a calm tone. This series of processes makes it possible to support the user's daily life in real time.

[0752] Example 2

[0753] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] Until now, there has been no system that can effectively support the daily lives of patients with early-stage dementia by collecting and analyzing comprehensive data, including not only visual and audio data but also emotional data, in real time. As a result, real-time support including the user's emotional state has not been provided, and difficulties in daily life have not been adequately resolved.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0756] In this invention, the server includes means for analyzing visual data using an image recognition algorithm, means for analyzing audio data using a voice recognition algorithm, and means for analyzing emotional data using an emotion engine, thereby enabling comprehensive analysis of visual, audio, and emotional data and providing real-time support information based on the user's emotional state.

[0757] "Visual data" refers to image and video data acquired by a device that captures visual information, such as a camera.

[0758] "Audio data" refers to data that includes sounds and voices acquired by an audio input device such as a microphone.

[0759] "Emotional data" is data used to analyze a user's emotional state by capturing vocal tones, facial expressions, body movements, etc.

[0760] An "image recognition algorithm" is an algorithm that automatically detects and recognizes specific objects, people, and scenes from images and videos.

[0761] A "voice recognition algorithm" is an algorithm for analyzing voice data and converting the voice into text data.

[0762] The "emotion engine" is software that evaluates and analyzes the user's emotional state based on captured data.

[0763] "Support information" refers to information and instructions generated based on the results of analyzing the user's visual data, audio data, and emotional data.

[0764] "Real-time" refers to the extremely short time between data collection and the generation of analysis results, meaning that processing is immediate.

[0765] This invention is a system that captures visual, audio, and emotional data and analyzes them in real time to support the daily lives of patients with early-stage dementia. This system consists of a glasses-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine.

[0766] Overall system configuration

[0767] User

[0768] The user wears a pair of glasses equipped with a camera and microphone that captures the user's visual, audio, and emotional data.

[0769] Terminal

[0770] The captured visual, audio, and emotion data is temporarily stored in a buffer on the device. This data is then compressed and encrypted in real time and sent to the server. Specifically, image data is converted to JPEG format, audio data is converted to AAC format, etc., and the data is encrypted using the SSL / TLS communication protocol before being sent to the server.

[0771] Sending data

[0772] server

[0773] The server receives the visual, audio, and emotion data sent from the device. The received data is immediately analyzed. Specifically, the following process is performed:

[0774] Visual data analysis: Using image recognition algorithms (e.g., OpenCV) on the server, objects and people are identified in the video.

[0775] Analyzing speech data: Using speech recognition algorithms (e.g., Google Cloud Speech-to-Text API) to convert speech into text data, and then applying natural language processing (NLP) techniques to understand the content.

[0776] Emotional data analysis: Using an emotion engine (e.g., Affectiva) to assess emotional state from vocal tone, facial expressions, and body movements.

[0777] Generate support information

[0778] server

[0779] Based on the analysis results, the server generates the support information the user needs in real time. For example, if a user looks at a friend's face and asks, "Who are you?", this voice is captured and analyzed by the server, and an appropriate answer (e.g., "This is [Name]. I'm one of your friends.") is generated. If the user gets lost, the situation is analyzed from visual data and directional instructions such as "The door is on the right" are generated. If the user looks anxious, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated based on the analysis of emotional data.

[0780] Providing support information

[0781] Terminal

[0782] The generated support information is then sent back to the device and provided to the user via voice. Using speech synthesis technology (e.g., Amazon Polly), the support information is converted into natural-sounding speech and presented to the user through the device's speaker.

[0783] Specific examples of embodiments

[0784] Example 1: Person recognition

[0785] User: "Who are you?"

[0786] The device's microphone captures this audio and sends it to the server.

[0787] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0788] Support information such as "This is [name], one of your friends" is generated and sent to the device.

[0789] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0790] Example 2: Directional indication

[0791] If the user forgets where the door is in the room,

[0792] The device's camera captures the view and sends it to the server.

[0793] The server does not analyze the visual data to detect the presence of a "door."

[0794] The support information "The door is on the right" is generated and sent to the terminal.

[0795] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0796] Example 3: Emotion Recognition

[0797] If the user looks anxious,

[0798] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[0799] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[0800] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[0801] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[0802] The above is a specific embodiment of the present invention. This system enables real-time support for early-stage dementia patients in dealing with the various difficulties they face in their daily lives, and is expected to improve the user's quality of life and reduce the burden on family and caregivers.

[0803] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0804] Step 1: Data Capture

[0805] User

[0806] The user wears the eyeglass-type device.

[0807] Input: The environment in which the user generates visual and audio data.

[0808] Output: Visual, audio and emotional data captured by the device.

[0809] How it works: The camera captures the user's field of view in real time, and the microphone records the user's speech and the surrounding environmental sounds. It also captures the tone of voice and facial expressions as emotional data.

[0810] Step 2: Temporarily save data

[0811] Terminal

[0812] The captured visual data, audio data, and emotion data are temporarily stored in the device's built-in buffer.

[0813] Input: Visual, audio, and emotional data captured by the device.

[0814] Output: The data stored in the buffer.

[0815] Specific operation: Temporarily stores captured data in memory and checks whether the data is in a suitable format for further processing.

[0816] Step 3: Compress and Encrypt the Data

[0817] Terminal

[0818] The stored data is compressed and encrypted and prepared for transmission to the server.

[0819] Input: Visual, audio and emotion data in a buffer.

[0820] Output: Compressed and encrypted data.

[0821] Specific operations: Image data is compressed to JPEG format, audio data is converted to AAC format, and data is encrypted using the SSL / TLS protocol.

[0822] Step 4: Sending data

[0823] Terminal

[0824] The compressed and encrypted data is sent to the server in real time.

[0825] Input: Compressed and encrypted data.

[0826] Output: The data received by the server.

[0827] Specific operation: Send data to the server through the terminal's communication module and confirm the success of the transmission.

[0828] Step 5: Receiving the data

[0829] server

[0830] The server receives the data sent from the terminal.

[0831] Input: Compressed and encrypted data sent from the terminal.

[0832] Output: Raw data stored in the server.

[0833] Specific operation: The server decrypts and unpacks the received data and prepares it for analysis.

[0834] Step 6: Analyze the visual data

[0835] server

[0836] The server analyzes the received visual data using image recognition algorithms.

[0837] Input: Decoded and unpacked visual data.

[0838] Output: Analysis results (e.g., identification of specific objects or people).

[0839] Specific operation: Using the OpenCV library, it identifies objects and people from video and identifies what the user is looking at.

[0840] Step 7: Analyze the audio data

[0841] server

[0842] The server analyzes the received voice data using a voice recognition algorithm.

[0843] Input: Decoded and decompressed audio data.

[0844] Output: Analysis results (e.g., converting audio into text data).

[0845] Specific operation: Uses Google Cloud Speech-to-Text API to convert speech into text data and uses NLP technology to understand the content.

[0846] Step 8: Analyze the sentiment data

[0847] server

[0848] The emotion data received by the server is analyzed using an emotion engine.

[0849] Input: Decoded and unpacked emotion data.

[0850] Output: Analysis results (e.g., assessment of the user's emotional state).

[0851] Specific behavior: Using the Affectiva emotion engine, we assess the user's emotional state from their vocal tone, facial expressions, and body movements.

[0852] Step 9: Generate support information

[0853] server

[0854] Support information is generated in real time based on the analysis results.

[0855] Input: Analysis results of visual, audio and emotion data.

[0856] Output: The generated supporting information.

[0857] Specific operation: Generates the solutions and advice the user needs in text format and creates support information tailored to the user's situation.

[0858] Step 10: Provide support information

[0859] Terminal

[0860] The generated support information is sent again to the terminal and provided to the user by voice.

[0861] Input: Generated supporting information.

[0862] Output: Natural-sounding voice delivered through speech synthesis.

[0863] Specific operation: Using speech synthesis technology such as Amazon Polly, the generated text data is converted into speech and provided to the user through the device's speaker.

[0864] (Application example 2)

[0865] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0866] To ensure the efficiency and safety of workers in factories, it is necessary to provide appropriate work instructions and safety warnings in real time. However, conventional methods have made it difficult to provide appropriate support that takes into account the emotional state, fatigue, and stress of workers. In addition, since rapid situational assessment is required on-site, a real-time analysis system with minimal delay is required.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0868] In this invention, the server includes means for capturing visual data, means for capturing audio data, means for analyzing the captured visual data and audio data, means for generating support information in real time based on the analysis results, means for providing the generated support information to the user, means for capturing and analyzing emotional data, and means for generating and providing work instructions, safety warnings, and information for fatigue and stress management based on the analysis results. This enables appropriate support that takes into account the emotional state, fatigue, and stress of the worker, and rapid situation assessment.

[0869] "Visual data" is video data that appears in the user's field of vision and is information captured by a camera.

[0870] "Audio data" refers to data that indicates audio information of the user and their surroundings and is recorded by a microphone.

[0871] "Analysis means" refers to the systems and algorithms that process the captured visual and audio data and generate appropriate information based on the results.

[0872] "Support information" is information such as specific instructions and advice generated by the analysis means to support the user's daily life and work.

[0873] "Emotion data" is information about the user's emotional state obtained by analyzing voice tone, facial expressions, body movements, and the like.

[0874] "Work instructions" are information that guides the user on the specific work content and procedures that should be performed.

[0875] "Safety warnings" are information that alert users when they are faced with a dangerous situation.

[0876] "Fatigue and stress management" is information that detects fatigue and stress from the user's facial expressions, voice, etc., and suggests measures to take or take a break.

[0877] The present invention is a system that uses a head-mounted display (hereinafter referred to as a smart helmet) worn by a worker, and aims to support work efficiently and safely by capturing and analyzing visual data, audio data, and emotional data. The specific configuration and operation of this system are described below.

[0878] Overall system configuration

[0879] The system consists of a smart helmet worn by the user, a server that receives and analyzes data, and a means of providing the user with information generated based on the analysis results.The smart helmet has a built-in camera and microphone that captures the user's visual and audio data, as well as emotional data.

[0880] Capture visual, audio, and emotional data

[0881] When a user wears a smart helmet, the built-in camera captures visual data (images seen by the user) and facial expressions, and the built-in microphone records audio data of the user and their surroundings. This data is captured in real time and sent to a server for analysis.

[0882] Sending data

[0883] The smart helmet temporarily stores captured visual, audio, and emotional data in a buffer, which is then compressed, encrypted, and sent to a server in real time. This low-latency communication allows for real-time analysis.

[0884] Receiving and analyzing data

[0885] The server receives and analyzes data sent from the smart helmet in real time. It uses an image recognition algorithm to analyze visual data and recognize specific objects and people. It uses a voice recognition algorithm to analyze audio data, converting it into text data and then analyzing the user's intent using natural language processing (NLP). It uses an emotion engine to analyze emotional data, which allows it to evaluate the worker's emotional state, as well as their fatigue and stress levels.

[0886] Generate support information

[0887] Based on the analysis results, the server generates supporting information in real time, such as:

[0888] Work Instructions: Understand what you're currently doing and provide specific instructions on what to do next.

[0889] Safety warning: When a dangerous situation is detected, it provides warning information and instructions on how to avoid it.

[0890] Fatigue and stress management: Detects worker fatigue and stress and suggests breaks.

[0891] This support information is generated in real time and sent back to the smart helmet, where it is provided to the user audibly using voice synthesis technology.

[0892] Specific examples

[0893] Example 1: Work instruction support

[0894] If a worker forgets a particular step and looks confused, the smart helmet captures video and audio and sends them to the server, which analyzes the procedure and generates instructions, such as "Next, pull the left lever."

[0895] Example prompt: "If I forget this step, tell me what to do next."

[0896] Example 2: Safety warning

[0897] When a worker approaches a dangerous area, the camera in the smart helmet captures the surrounding image and sends it to the server. The server analyzes the image and, if it detects a dangerous area, generates an audio warning saying, "Please retreat. This is dangerous."

[0898] Example prompt: "If a worker enters a dangerous area, give instructions to evacuate immediately."

[0899] Example 3: Fatigue and stress management

[0900] If a worker shows signs of fatigue, the smart helmet captures facial expressions and voice data and sends it to a server. The server analyzes the data, and if it detects fatigue, it suggests, "It's time for a break. Let's take a short rest."

[0901] Sample prompt: "If a worker is fatigued, how can you encourage them to take a break?"

[0902] In this way, the system of the present invention can provide real-time support to workers in dealing with various difficulties they may face, thereby improving worker efficiency and ensuring safety.

[0903] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0904] Step 1:

[0905] The smart helmet captures visual and audio data.

[0906] Specifically, the smart helmet acquires visual information from the user using a built-in camera, and audio information from a built-in microphone.

[0907] Input: User's visual and audio information

[0908] Output: Captured visual and audio data

[0909] Step 2:

[0910] The visual and audio data captured by the smart helmet is temporarily stored in a buffer.

[0911] This prepares the data for analysis in real time.

[0912] Input: Captured visual and audio data

[0913] Output: Data stored in the buffer

[0914] Step 3:

[0915] The smart helmet compresses and encrypts the buffered data and sends it to the server.

[0916] Since data is transmitted in real time, data size is reduced and high security is maintained.

[0917] Input: Data stored in the buffer

[0918] Output: Compressed and encrypted data

[0919] Step 4:

[0920] The server receives the data transmitted from the smart helmet.

[0921] The received data is decoded for analysis.

[0922] Input: Compressed and encrypted data

[0923] Output: Decoded visual, audio, and emotion data

[0924] Step 5:

[0925] The server analyzes the visual data and recognizes specific objects and people.

[0926] Uses image recognition algorithms (e.g., OpenCV or TensorFlow)

[0927] Input: Decoded visual data

[0928] Output: Analysis results (information on recognized objects and people)

[0929] Step 6:

[0930] The server analyzes the voice data and converts it into text data.

[0931] Uses a speech recognition algorithm (e.g., Google Speech-to-Text API)

[0932] Input: Decoded audio data

[0933] Output: Converted text data

[0934] Step 7:

[0935] The server analyzes the user's intent based on the text data.

[0936] Uses natural language processing (NLP) techniques (e.g. spaCy and BERT)

[0937] Input: Converted text data

[0938] Output: Analysis results (user intent information)

[0939] Step 8:

[0940] The server analyzes the emotional data and evaluates the user's emotional state.

[0941] Use an emotion engine (e.g., Emotion API)

[0942] Input: Decoded emotion data

[0943] Output: Analysis result (user's emotional state)

[0944] Step 9:

[0945] A server generates work instructions, safety warnings, and supporting information for fatigue and stress management.

[0946] Use a generative AI model (e.g. GPT-3)

[0947] Input: Analysis results (object / person information, user intent, emotional state)

[0948] Output: Supporting information (work instructions, warnings, break suggestions)

[0949] Step 10:

[0950] The server transmits the generated support information to the smart helmet.

[0951] Input: Support Information

[0952] Output: Support information sent to the smart helmet

[0953] Step 11:

[0954] The smart helmet provides support information to the user via voice.

[0955] Uses voice synthesis technology (e.g., Amazon Polly)

[0956] Input: Support Information

[0957] Output: Supporting information provided by voice

[0958] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0959] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0960] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0961] [Third embodiment]

[0962] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0963] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0964] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0965] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0966] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0967] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0968] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0969] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0970] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0971] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0972] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0973] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0974] The present invention is a system for supporting the daily lives of patients with early-stage dementia, capturing visual and audio data, analyzing the data, and providing appropriate support information in real time. The system and its embodiments are described in detail below.

[0975] Overall system configuration

[0976] This system consists of a glasses-type device (terminal) worn by the user and a server that receives and analyzes the data. The glasses-type device has a built-in camera and microphone that captures the user's visual and audio data. The captured data is sent to the server in real time, where it is analyzed and appropriate support information is generated. The generated support information is then provided to the user via the glasses-type device.

[0977] Visual and audio data capture

[0978] User

[0979] When a user wears the eyeglasses, the camera in the glasses continuously captures images (visual data) within the user's field of vision, and the built-in microphone records the user's and their surrounding sounds (audio data).

[0980] Sending data

[0981] Terminal

[0982] The captured visual and audio data is temporarily stored in a buffer on the device, then compressed, encrypted, and sent to a server. This communication occurs in real time, minimizing the delay between when the data is collected and when it reaches the server.

[0983] Receiving and analyzing data

[0984] server

[0985] The server receives visual and audio data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[0986] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[0987] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[0988] Generate support information

[0989] server

[0990] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[0991] Providing support information

[0992] Terminal

[0993] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[0994] Specific examples

[0995] Example 1: Person recognition

[0996] User: "Who are you?"

[0997] The device's microphone captures this audio and sends the data to a server.

[0998] The server analyzes the voice data and determines whether it is a sign of aphasia.

[0999] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[1000] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1001] Example 2: Directional indication

[1002] If the user forgets where the door is in the room,

[1003] The device's camera captures the view and sends it to the server.

[1004] The server analyzes the visual data and does not detect the presence of a "door."

[1005] As support information, the message "The door is on the right" is generated and sent to the terminal.

[1006] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1007] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[1008] The processing flow will be explained below.

[1009] Step 1:

[1010] The user puts on the eyeglasses-type device, which enables the device to capture the user's visual and audio data.

[1011] Step 2:

[1012] The device's camera continuously captures the image of the user's field of vision (visual data), and the microphone records the user's and surrounding sounds (audio data).

[1013] Step 3:

[1014] The device temporarily stores the captured visual and audio data in a buffer memory, where it is compressed and encrypted in real time.

[1015] Step 4:

[1016] The device transmits compressed and encrypted visual and audio data to the server, and this communication occurs in real time to minimize latency.

[1017] Step 5:

[1018] The server receives the visual and audio data sent from the terminal, and decompresses and decodes the respective data.

[1019] Step 6:

[1020] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[1021] Step 7:

[1022] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[1023] Step 8:

[1024] The server analyzes visual and audio data to identify the problem the user is facing and generates real-time support information, such as the name of a person if the user does not recognize them and their relationship to the person.

[1025] Step 9:

[1026] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[1027] Step 10:

[1028] The server transmits the generated voice data to the terminal.

[1029] Step 11:

[1030] The device plays back the received audio data, providing support information to the user through audio and giving necessary advice in real time. For example, if the user gets lost, audio instructions will be given on the direction they should go.

[1031] Example 1

[1032] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Patients with early-stage dementia may experience problems in their daily lives, such as difficulty recognizing objects and people, or a lack of understanding of intentions. This not only makes it difficult for patients to make appropriate decisions and reduces their quality of life, but also places a burden on caregivers and family members. There is a need for a support system that can solve these problems and enable patients to live their daily lives more smoothly.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1035] In this invention, the server includes means for acquiring visual information, means for acquiring acoustic information, means for temporarily storing the acquired visual information and acoustic information and compressing and encrypting it, means for transmitting the compressed and encrypted information in real time, means for analyzing the received visual information and acoustic information, means for generating support information in real time based on the analysis results, and means for converting the generated support information into audio and providing it to the user. This makes it possible to provide real-time support for the difficulties faced by early-stage dementia patients in their daily lives and improve the quality of life of the patients.

[1036] "Visual information" refers to image data captured in the user's field of vision by an image capture device such as a camera in a glasses-type device.

[1037] "Acoustic information" refers to audio data emitted from the user and the environment, which is acquired by an audio acquisition device such as a microphone of the eyeglass-type device.

[1038] "Temporary storage" refers to a process of temporarily storing acquired visual and acoustic information in memory or storage.

[1039] "Compression" is data processing performed to reduce the data volume of visual and audio information.

[1040] "Encryption" is the process of protecting data with encryption technology in order to securely transmit visual and audio information.

[1041] "Transmission" is the process of sending compressed and encrypted visual and audio information from the terminal to the server over the network.

[1042] "Reception" refers to the process in which the server receives visual and audio information sent from the terminal.

[1043] "Analysis" refers to analyzing received visual information using image recognition algorithms, and converting acoustic information into text data using speech recognition algorithms and analyzing it through natural language processing.

[1044] "Support information" is information that includes advice and instructions for the user based on the analysis results, and is information that is generated in real time for problem solving.

[1045] "Conversion to voice" refers to a process of converting the generated assistance information into voice data using voice synthesis technology so that it can be provided to the user audibly.

[1046] "Providing to the user" refers to providing the converted audio data in a form that the user can hear through the speaker of the eyeglass-type device, etc.

[1047] The present invention is a system for supporting the daily lives of patients with early-stage dementia. This system captures visual and acoustic information, analyzes the acquired data, and provides support information in real time. Detailed embodiments of the present invention will be described below.

[1048] Hardware Configuration

[1049] This system consists of a glasses-type device (hereafter referred to as the terminal) worn by the user and a server that receives and analyzes data. The terminal has a built-in camera and microphone, which are responsible for capturing visual and acoustic information. The server is a general-purpose computer equipped with a high-performance CPU and GPU.

[1050] Software Configuration

[1051] The device is installed with software for capturing visual and acoustic information: driver software for acquiring images from the camera and audio processing software for acquiring audio from the microphone.

[1052] The server has the following software modules installed:

[1053] Image recognition algorithms: e.g., YOLOv5 and OpenCV

[1054] Speech recognition algorithms, such as Google Cloud Speech-to-Text or Microsoft Azure Speech Service

[1055] Natural Language Processing (NLP) modules: e.g., BERT models

[1056] Speech synthesis technology: e.g., Google Text-to-Speech or Amazon Polly

[1057] Operating procedures and data processing

[1058] When a user wears the glasses, the device's camera captures the user's visual information in real time, and the microphone records audio information. The device temporarily stores this data, compresses and encrypts it, and then transmits it to a server.

[1059] The server receives the visual and audio information sent from the device, first decompresses and decodes the data, then uses an image recognition algorithm to analyze the visual information and identify objects and people in the user's field of view, and simultaneously uses a speech recognition algorithm to convert the audio information into text data and understand the user's speech and intent through natural language processing.

[1060] Based on the analysis results, the server generates real-time support information. For example, if the user does not recognize a person, it will provide the person's name and relationship to the person. If the user gets lost, it will provide advice on the direction to go. This support information is then compressed and encrypted again before being sent to the device.

[1061] The device converts the received assistance information into voice data using voice synthesis technology and plays it back through a built-in speaker, allowing users to obtain the necessary information in real time by voice.

[1062] Specific examples

[1063] Example 1: Person recognition

[1064] User: Ask "Who are you?"

[1065] Device: The microphone captures this audio and sends the data to the server.

[1066] Server: Analyzes the voice data and understands the user's question.

[1067] Server: Analyzes visual data to identify the person being questioned.

[1068] Server: Generate supporting information: "This is [Name], one of your doctors."

[1069] Server: Sends this information to the device.

[1070] Terminal: Support information is converted into voice using speech synthesis technology and provided to the user.

[1071] Example 2: Directional indication

[1072] If the user forgets the location of the door in the room, visual data is captured and sent to the server.

[1073] The server does not analyze the visual data to detect the location of the door.

[1074] The server generates the supporting information "The door is on the right."

[1075] The server sends this information to the terminal.

[1076] The terminal uses voice synthesis technology to convert support information into voice and provide it to the user.

[1077] Example prompts to input to the generative AI model

[1078] Example prompt 1: "When a user asks who they are, analyze the speech data to determine signs of aphasia and generate supporting information."

[1079] Prompt example 2: "When the user forgets the location of the door through visual data, and the server analyzes and cannot determine the location of the door from the visual data."

[1080] As described above, this system is capable of analyzing visual and acoustic information in real time and providing appropriate support information in the form of voice to support the daily lives of patients with early-stage dementia.

[1081] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1082] Step 1:

[1083] The user puts on the eyeglasses-type device. This activates the device's camera and microphone, and the device begins capturing visual and acoustic information. The user's wearing of the device is required as input, and the device's sensors are activated as output.

[1084] Step 2:

[1085] The device uses a camera to capture the user's visual information in real time and a microphone to record audio information. The input is the user's visual and audio environment, and the output is the captured visual and audio information.

[1086] Step 3:

[1087] The terminal temporarily stores the captured visual and audio information. The temporarily stored data is temporarily stored in a buffer. The captured visual and audio information is required as input, and the temporarily stored data is obtained as output.

[1088] Step 4:

[1089] The terminal compresses and encrypts the temporarily stored visual and audio information. The input is the temporarily stored data. The output is the compressed and encrypted data. Specifically, data compression uses a compression algorithm such as JPEG or MP3, and encryption uses an encryption algorithm such as AES.

[1090] Step 5:

[1091] The terminal sends compressed and encrypted data to the server. This transmission is done in real time with minimal latency. The input is the compressed and encrypted data. The output is the transmitted data.

[1092] Step 6:

[1093] The server receives the data sent from the terminal. After receiving it, it decompresses and decrypts the data. The input is the sent data, and the output is the decompressed and decrypted data. Specifically, it uses a decompression algorithm such as gzip or bzip2 for decompression, and the AES decryption algorithm for decryption.

[1094] Step 7:

[1095] The server analyzes the decompressed and decoded visual information using an image recognition algorithm. For example, it uses YOLOv5 or OpenCV to identify objects and people in the user's field of view. Visual information is required as input, and the analysis results are obtained as output. Specifically, the image recognition algorithm analyzes the visual information and recognizes objects and people.

[1096] Step 8:

[1097] The server converts the decompressed and decoded audio information into text data using a speech recognition algorithm and understands the user's speech and intent through natural language processing (NLP). The input is audio information, and the output is analyzed text data. Specifically, the speech recognition algorithm analyzes the audio information and converts it into text data, and then the NLP module analyzes the text data.

[1098] Step 9:

[1099] The server generates support information in real time based on the results of visual and audio data analysis. For example, if the user cannot recognize a person, it will convey the person's name and relationship to them. Also, if the user gets lost, it will instruct the user on the direction to go. The analysis results are required as input, and the generated support information is obtained as output.

[1100] Step 10:

[1101] The server compresses and encrypts the generated support information again and sends it to the terminal. The generated support information is required as input, and the compressed and encrypted support information is obtained as output.

[1102] Step 11:

[1103] The terminal decompresses and decodes the assistance information received from the server. As input, it takes compressed and encrypted assistance information, and as output, it gets the decompressed and decoded assistance information.

[1104] Step 12:

[1105] The device converts the decompressed and decoded assistance information into audio data using speech synthesis technology and provides it to the user through a built-in speaker. The decoded assistance information is required as input, and audio data is obtained as output. Specifically, the device uses speech synthesis technology to convert the assistance information into natural audio and provides it to the user audibly.

[1106] (Application example 1)

[1107] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Patients with early-stage dementia often face difficulties in their daily lives due to being unable to recognize objects and people and losing their sense of direction. This also increases the likelihood of them getting into dangerous situations, increasing their safety and the burden on their families and caregivers. To solve these problems, a system is needed that provides appropriate support in real time in the various situations patients face, and detects danger and issues warnings.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1110] In this invention, the server includes a means for analyzing visual data and audio data in real time, a means for monitoring the user's surroundings based on the analysis results and detecting danger, and a means for generating warnings and advice when danger is detected. This makes it possible to provide real-time support for early-stage dementia patients to live their daily lives safely and appropriately avoid danger.

[1111] A "means for capturing visual data" is a device or method for acquiring images that appear in the user's field of vision using a device such as a camera.

[1112] "Means for capturing audio data" refers to a device or method for capturing sounds from the user and their surroundings using a device such as a microphone.

[1113] "Means for analyzing captured visual and audio data" refers to a device or method that uses a specific algorithm to understand the content of the acquired visual and audio data and generate analytical results.

[1114] The "means for generating support information in real time based on the analysis results" refers to a device or method for instantly generating support information required by the user based on the analysis results of the visual data and audio data.

[1115] The "means for providing the generated support information to the user" refers to a device or method for transmitting the generated support information to the user by audio or visual means.

[1116] The "means for monitoring the situation around the user based on the analysis results and detecting danger" refers to a device or method for monitoring the environment and behavior around the user based on the results of data analysis and identifying dangerous situations.

[1117] A "means for generating a warning or advice upon detection of a danger" is a device or method that provides an appropriate warning or advice to a user when a dangerous situation is identified.

[1118] This invention is a support system to help patients with early-stage dementia live their daily lives more safely. The system begins operation when the user puts on the smart glasses. The smart glasses have a built-in camera and microphone that capture visual and audio data. The captured data is temporarily stored in the device, compressed and encrypted, and then transmitted to a server in real time.

[1119] The server analyzes the received visual data using image recognition algorithms. Specifically, it uses image processing libraries such as OpenCV to identify objects and people in the visual data and determine what the user is looking at and the situation they are facing. It also uses speech recognition algorithms to convert audio data into text data and analyzes the user's intent using natural language processing techniques (e.g., NLTK, spaCy).

[1120] Based on the analysis results, the server generates support information in real time. Specifically, if the user does not recognize a person, it will tell them the person's name and relationship, or if the user gets lost, it will give them directions. The analysis results also monitor the user's surroundings, and if a dangerous situation is identified, it will generate appropriate warnings or advice (e.g., "The door is open" or "Did you forget to take your medicine?").

[1121] The generated support information is then provided to the user via the smart glasses. Using speech synthesis technology (e.g., gTTS), the text data is converted into speech and played back through the glasses' speakers, allowing the user to receive the necessary support in real time.

[1122] For example:

[1123] Example 1: Detecting an open door

[1124] If the camera does not detect the presence of a door even though the user frequently uses the word "door" in conversation, the server will generate a warning saying "Door is open" and notify the user audibly through the smart glasses.

[1125] Example 2: Medication reminders

[1126] The server analyzes the user's voice data, and if the word "medicine" is not detected within a certain period of time, it generates a reminder "Have you taken your medicine?" and notifies the user via voice through the smart glasses.

[1127] Example prompts for generative AI models

[1128] "Please use visual and audio data to identify dangerous situations that early-stage dementia patients face in their daily lives and generate appropriate warning information. The visual data should be a byte array in JPEG format, and the audio data should be in WAV format with a sampling rate of 8 kHz. Please output in the following format:

[1129] {

[1130] "status": "safe" or "danger",

[1131] "warning_text": "Danger warning message"

[1132] }"

[1133] This system allows people with early-stage dementia to receive the support they need in real time in their daily lives, significantly improving safety and reducing the burden on their families and caregivers.

[1134] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1135] Step 1:

[1136] A user puts on smart glasses. The glasses are equipped with a camera and microphone, and begin capturing the user's visual and audio data. The input is the image in the user's field of vision and the surrounding audio, and the output is the captured visual and audio data.

[1137] Step 2:

[1138] The device temporarily stores the captured visual and audio data in a buffer. The stored data is compressed and encrypted. The input is the captured visual and audio data, and the output is the compressed and encrypted visual and audio data.

[1139] Step 3:

[1140] The terminal transmits compressed and encrypted visual and audio data to the server in real time. The input is the compressed and encrypted visual and audio data, and the output is the data transmission to the server.

[1141] Step 4:

[1142] The server analyzes the received visual data using an image recognition algorithm (e.g., OpenCV). The image recognition algorithm identifies objects and people in the visual data and determines what is in the user's field of view. The input is the visual data sent to the server, and the output is the analyzed visual information.

[1143] Step 5:

[1144] The server converts the received voice data into text data using a voice recognition algorithm and analyzes the user's intent using natural language processing technology (e.g., NLTK, spaCy). The input is the voice data sent to the server, and the output is the text data and analyzed intent information.

[1145] Step 6:

[1146] The server generates support information in real time based on the analysis results. The generated support information includes solutions to problems the user is facing and warnings. For example, it can be direction guidance if the user gets lost or person recognition information. The input is the analyzed visual information and intention information, and the output is support information and warning messages.

[1147] Step 7:

[1148] The server transmits the generated support information to the terminal again. The input is the generated support information, and the output is data transmission to the terminal.

[1149] Step 8:

[1150] The device provides the user with the support information received from the server. Using speech synthesis technology (e.g., gTTS), the support information is converted into speech and played back through the smart glasses' speaker. The input is the support information sent from the server, and the output is a voice notification to the user.

[1151] Step 9:

[1152] The user receives a voice notification and takes appropriate action, such as closing the door or taking medicine after receiving the warning. The input is the voice notification, and the output is the user's action.

[1153] In this way, all processing steps work together to provide real-time support to patients with early-stage dementia who are faced with problems and dangers in their daily lives, enabling them to maintain a safe life.

[1154] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1155] The present invention incorporates an emotion engine into a system that captures and analyzes visual and audio data to support the daily lives of people with early-stage dementia, thereby recognizing the user's emotional state and providing personalized responses. The system and its embodiments are described in detail below.

[1156] Overall system configuration

[1157] This system consists of an eyeglass-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine. The eyeglass-type device has a built-in camera and microphone that captures the user's visual, audio, and emotional data. The captured data is sent in real time to the server, which analyzes it and generates appropriate support information. The generated support information is then provided to the user again via the eyeglass-type device.

[1158] Capture visual, audio, and emotional data

[1159] User

[1160] When a user wears the glasses-type device, the device's camera captures the image in the user's field of vision (visual data) and the user's facial expressions. The built-in microphone also records the user's and their surroundings' sounds (audio data). Emotional data is acquired by analyzing voice tone, facial expressions, and body movements.

[1161] Sending data

[1162] Terminal

[1163] The captured visual, audio, and emotional data is temporarily stored in a buffer on the device, then compressed, encrypted, and transmitted to a server in real time. This real-time communication minimizes the delay between the time the data is collected and the time it reaches the server.

[1164] Receiving and analyzing data

[1165] server

[1166] The server receives visual, audio, and emotional data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[1167] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[1168] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[1169] Emotion data analysis is performed by an emotion engine, which assesses the user's emotional state based on voice tone, facial expressions, and body movements.

[1170] Generate support information

[1171] server

[1172] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[1173] Additionally, the emotion engine's analysis can provide support based on the user's emotional state: for example, if the user is feeling anxious, it will provide instructions in a calmer, more reassuring voice.

[1174] Providing support information

[1175] Terminal

[1176] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[1177] Specific examples

[1178] Example 1: Person recognition

[1179] User: "Who are you?"

[1180] The device's microphone captures this audio and sends the data to a server.

[1181] The server analyzes the voice data and determines whether it is a sign of aphasia.

[1182] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[1183] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1184] Example 2: Directional indication

[1185] If the user forgets where the door is in the room,

[1186] The device's camera captures the view and sends it to the server.

[1187] The server analyzes the visual data and does not detect the presence of a "door."

[1188] As support information, the message "The door is on the right" is generated and sent to the terminal.

[1189] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1190] Example 3: Emotion Recognition

[1191] If the user looks anxious,

[1192] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[1193] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[1194] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[1195] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1196] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[1197] The processing flow will be explained below.

[1198] Step 1:

[1199] The user puts on the eyeglasses, which allows the device to capture the user's visual, audio, and emotional data.

[1200] Step 2:

[1201] The device's camera continuously captures the image (visual data) of the user's field of vision, while the microphone records the user's and surrounding sounds (audio data). At the same time, the device captures the user's facial expressions, tone of voice, and body movements and records them as emotional data.

[1202] Step 3:

[1203] The device temporarily stores the captured visual, audio, and emotional data in a buffer memory, which is then compressed, encrypted, and sent to the server in real time.

[1204] Step 4:

[1205] The server receives the visual data, audio data, and emotion data sent from the terminal, and decompresses and decodes each data.

[1206] Step 5:

[1207] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[1208] Step 6:

[1209] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[1210] Step 7:

[1211] The server analyzes the emotion data and uses an emotion engine to assess the user's emotional state based on voice tone, facial expressions, and body movements.

[1212] Step 8:

[1213] Based on the analysis results, the server identifies the problem the user is facing and generates support information in real time. For example, if the user does not recognize a person, it will tell them the person's name and relationship to them. Also, if the user gets lost, it will give them directions to go.

[1214] Step 9:

[1215] The server uses the analysis results of the emotion engine to generate support information according to the user's emotional state. For example, if the user is feeling anxious, it will choose calm and reassuring words.

[1216] Step 10:

[1217] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[1218] Step 11:

[1219] The server transmits the generated voice data to the terminal.

[1220] Step 12:

[1221] The device plays back the received audio data, providing the user with support information and necessary advice in real time. For example, if the user gets lost, the device will provide voice instructions on the direction to go. If the user feels anxious, the device will offer reassuring words.

[1222] For example, if a user cannot recognize a person or gets lost, the server generates appropriate support information and provides it to the user via voice via the device. The emotion engine also evaluates the user's emotional state, and if the user is feeling anxious, for example, it provides reassuring words in a calm tone. This series of processes makes it possible to support the user's daily life in real time.

[1223] Example 2

[1224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] Until now, there has been no system that can effectively support the daily lives of patients with early-stage dementia by collecting and analyzing comprehensive data, including not only visual and audio data but also emotional data, in real time. As a result, real-time support including the user's emotional state has not been provided, and difficulties in daily life have not been adequately resolved.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes means for analyzing visual data using an image recognition algorithm, means for analyzing audio data using a voice recognition algorithm, and means for analyzing emotional data using an emotion engine, thereby enabling comprehensive analysis of visual, audio, and emotional data and providing real-time support information based on the user's emotional state.

[1228] "Visual data" refers to image and video data acquired by a device that captures visual information, such as a camera.

[1229] "Audio data" refers to data that includes sounds and voices acquired by an audio input device such as a microphone.

[1230] "Emotional data" is data used to analyze a user's emotional state by capturing vocal tones, facial expressions, body movements, etc.

[1231] An "image recognition algorithm" is an algorithm that automatically detects and recognizes specific objects, people, and scenes from images and videos.

[1232] A "voice recognition algorithm" is an algorithm for analyzing voice data and converting the voice into text data.

[1233] The "emotion engine" is software that evaluates and analyzes the user's emotional state based on captured data.

[1234] "Support information" refers to information and instructions generated based on the results of analyzing the user's visual data, audio data, and emotional data.

[1235] "Real-time" refers to the extremely short time between data collection and the generation of analysis results, meaning that processing is immediate.

[1236] This invention is a system that captures visual, audio, and emotional data and analyzes them in real time to support the daily lives of patients with early-stage dementia. This system consists of a glasses-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine.

[1237] Overall system configuration

[1238] User

[1239] The user wears a pair of glasses equipped with a camera and microphone that captures the user's visual, audio, and emotional data.

[1240] Terminal

[1241] The captured visual, audio, and emotion data is temporarily stored in a buffer on the device. This data is then compressed and encrypted in real time and sent to the server. Specifically, image data is converted to JPEG format, audio data is converted to AAC format, etc., and the data is encrypted using the SSL / TLS communication protocol before being sent to the server.

[1242] Sending data

[1243] server

[1244] The server receives the visual, audio, and emotion data sent from the device. The received data is immediately analyzed. Specifically, the following process is performed:

[1245] Visual data analysis: Using image recognition algorithms (e.g., OpenCV) on the server, objects and people are identified in the video.

[1246] Analyzing speech data: Using speech recognition algorithms (e.g., Google Cloud Speech-to-Text API) to convert speech into text data, and then applying natural language processing (NLP) techniques to understand the content.

[1247] Emotional data analysis: Using an emotion engine (e.g., Affectiva) to assess emotional state from vocal tone, facial expressions, and body movements.

[1248] Generate support information

[1249] server

[1250] Based on the analysis results, the server generates the support information the user needs in real time. For example, if a user looks at a friend's face and asks, "Who are you?", this voice is captured and analyzed by the server, and an appropriate answer (e.g., "This is [Name]. I'm one of your friends.") is generated. If the user gets lost, the situation is analyzed from visual data and directional instructions such as "The door is on the right" are generated. If the user looks anxious, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated based on the analysis of emotional data.

[1251] Providing support information

[1252] Terminal

[1253] The generated support information is then sent back to the device and provided to the user via voice. Using speech synthesis technology (e.g., Amazon Polly), the support information is converted into natural-sounding speech and presented to the user through the device's speaker.

[1254] Specific examples of embodiments

[1255] Example 1: Person recognition

[1256] User: "Who are you?"

[1257] The device's microphone captures this audio and sends it to the server.

[1258] The server analyzes the voice data and determines whether it is a sign of aphasia.

[1259] Support information such as "This is [name], one of your friends" is generated and sent to the device.

[1260] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1261] Example 2: Directional indication

[1262] If the user forgets where the door is in the room,

[1263] The device's camera captures the view and sends it to the server.

[1264] The server does not analyze the visual data to detect the presence of a "door."

[1265] The support information "The door is on the right" is generated and sent to the terminal.

[1266] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1267] Example 3: Emotion Recognition

[1268] If the user looks anxious,

[1269] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[1270] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[1271] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[1272] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1273] The above is a specific embodiment of the present invention. This system enables real-time support for early-stage dementia patients in dealing with the various difficulties they face in their daily lives, and is expected to improve the user's quality of life and reduce the burden on family and caregivers.

[1274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1275] Step 1: Data Capture

[1276] User

[1277] The user wears the eyeglass-type device.

[1278] Input: The environment in which the user generates visual and audio data.

[1279] Output: Visual, audio and emotional data captured by the device.

[1280] How it works: The camera captures the user's field of view in real time, and the microphone records the user's speech and the surrounding environmental sounds. It also captures the tone of voice and facial expressions as emotional data.

[1281] Step 2: Temporarily save data

[1282] Terminal

[1283] The captured visual data, audio data, and emotion data are temporarily stored in the device's built-in buffer.

[1284] Input: Visual, audio, and emotional data captured by the device.

[1285] Output: The data stored in the buffer.

[1286] Specific operation: Temporarily stores captured data in memory and checks whether the data is in a suitable format for further processing.

[1287] Step 3: Compress and Encrypt the Data

[1288] Terminal

[1289] The stored data is compressed and encrypted and prepared for transmission to the server.

[1290] Input: Visual, audio and emotion data in a buffer.

[1291] Output: Compressed and encrypted data.

[1292] Specific operations: Image data is compressed to JPEG format, audio data is converted to AAC format, and data is encrypted using the SSL / TLS protocol.

[1293] Step 4: Sending data

[1294] Terminal

[1295] The compressed and encrypted data is sent to the server in real time.

[1296] Input: Compressed and encrypted data.

[1297] Output: The data received by the server.

[1298] Specific operation: Send data to the server through the terminal's communication module and confirm the success of the transmission.

[1299] Step 5: Receiving the data

[1300] server

[1301] The server receives the data sent from the terminal.

[1302] Input: Compressed and encrypted data sent from the terminal.

[1303] Output: Raw data stored in the server.

[1304] Specific operation: The server decrypts and unpacks the received data and prepares it for analysis.

[1305] Step 6: Analyze the visual data

[1306] server

[1307] The server analyzes the received visual data using image recognition algorithms.

[1308] Input: Decoded and unpacked visual data.

[1309] Output: Analysis results (e.g., identification of specific objects or people).

[1310] Specific operation: Using the OpenCV library, it identifies objects and people from video and identifies what the user is looking at.

[1311] Step 7: Analyze the audio data

[1312] server

[1313] The server analyzes the received voice data using a voice recognition algorithm.

[1314] Input: Decoded and decompressed audio data.

[1315] Output: Analysis results (e.g., converting audio into text data).

[1316] Specific operation: Uses Google Cloud Speech-to-Text API to convert speech into text data and uses NLP technology to understand the content.

[1317] Step 8: Analyze the sentiment data

[1318] server

[1319] The emotion data received by the server is analyzed using an emotion engine.

[1320] Input: Decoded and unpacked emotion data.

[1321] Output: Analysis results (e.g., assessment of the user's emotional state).

[1322] Specific behavior: Using the Affectiva emotion engine, we assess the user's emotional state from their vocal tone, facial expressions, and body movements.

[1323] Step 9: Generate support information

[1324] server

[1325] Support information is generated in real time based on the analysis results.

[1326] Input: Analysis results of visual, audio and emotion data.

[1327] Output: The generated supporting information.

[1328] Specific operation: Generates the solutions and advice the user needs in text format and creates support information tailored to the user's situation.

[1329] Step 10: Provide support information

[1330] Terminal

[1331] The generated support information is sent again to the terminal and provided to the user by voice.

[1332] Input: Generated supporting information.

[1333] Output: Natural-sounding voice delivered through speech synthesis.

[1334] Specific operation: Using speech synthesis technology such as Amazon Polly, the generated text data is converted into speech and provided to the user through the device's speaker.

[1335] (Application example 2)

[1336] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1337] To ensure the efficiency and safety of workers in factories, it is necessary to provide appropriate work instructions and safety warnings in real time. However, conventional methods have made it difficult to provide appropriate support that takes into account the emotional state, fatigue, and stress of workers. In addition, since rapid situational assessment is required on-site, a real-time analysis system with minimal delay is required.

[1338] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1339] In this invention, the server includes means for capturing visual data, means for capturing audio data, means for analyzing the captured visual data and audio data, means for generating support information in real time based on the analysis results, means for providing the generated support information to the user, means for capturing and analyzing emotional data, and means for generating and providing work instructions, safety warnings, and information for fatigue and stress management based on the analysis results. This enables appropriate support that takes into account the emotional state, fatigue, and stress of the worker, and rapid situation assessment.

[1340] "Visual data" is video data that appears in the user's field of vision and is information captured by a camera.

[1341] "Audio data" refers to data that indicates audio information of the user and their surroundings and is recorded by a microphone.

[1342] "Analysis means" refers to the systems and algorithms that process the captured visual and audio data and generate appropriate information based on the results.

[1343] "Support information" is information such as specific instructions and advice generated by the analysis means to support the user's daily life and work.

[1344] "Emotion data" is information about the user's emotional state obtained by analyzing voice tone, facial expressions, body movements, and the like.

[1345] "Work instructions" are information that guides the user on the specific work content and procedures that should be performed.

[1346] "Safety warnings" are information that alert users when they are faced with a dangerous situation.

[1347] "Fatigue and stress management" is information that detects fatigue and stress from the user's facial expressions, voice, etc., and suggests measures to take or take a break.

[1348] The present invention is a system that uses a head-mounted display (hereinafter referred to as a smart helmet) worn by a worker, and aims to support work efficiently and safely by capturing and analyzing visual data, audio data, and emotional data. The specific configuration and operation of this system are described below.

[1349] Overall system configuration

[1350] The system consists of a smart helmet worn by the user, a server that receives and analyzes data, and a means of providing the user with information generated based on the analysis results.The smart helmet has a built-in camera and microphone that captures the user's visual and audio data, as well as emotional data.

[1351] Capture visual, audio, and emotional data

[1352] When a user wears a smart helmet, the built-in camera captures visual data (images seen by the user) and facial expressions, and the built-in microphone records audio data of the user and their surroundings. This data is captured in real time and sent to a server for analysis.

[1353] Sending data

[1354] The smart helmet temporarily stores captured visual, audio, and emotional data in a buffer, which is then compressed, encrypted, and sent to a server in real time. This low-latency communication allows for real-time analysis.

[1355] Receiving and analyzing data

[1356] The server receives and analyzes data sent from the smart helmet in real time. It uses an image recognition algorithm to analyze visual data and recognize specific objects and people. It uses a voice recognition algorithm to analyze audio data, converting it into text data and then analyzing the user's intent using natural language processing (NLP). It uses an emotion engine to analyze emotional data, which allows it to evaluate the worker's emotional state, as well as their fatigue and stress levels.

[1357] Generate support information

[1358] Based on the analysis results, the server generates supporting information in real time, such as:

[1359] Work Instructions: Understand what you're currently doing and provide specific instructions on what to do next.

[1360] Safety warning: When a dangerous situation is detected, it provides warning information and instructions on how to avoid it.

[1361] Fatigue and stress management: Detects worker fatigue and stress and suggests breaks.

[1362] This support information is generated in real time and sent back to the smart helmet, where it is provided to the user audibly using voice synthesis technology.

[1363] Specific examples

[1364] Example 1: Work instruction support

[1365] If a worker forgets a particular step and looks confused, the smart helmet captures video and audio and sends them to the server, which analyzes the procedure and generates instructions, such as "Next, pull the left lever."

[1366] Example prompt: "If I forget this step, tell me what to do next."

[1367] Example 2: Safety warning

[1368] When a worker approaches a dangerous area, the camera in the smart helmet captures the surrounding image and sends it to the server. The server analyzes the image and, if it detects a dangerous area, generates an audio warning saying, "Please retreat. This is dangerous."

[1369] Example prompt: "If a worker enters a dangerous area, give instructions to evacuate immediately."

[1370] Example 3: Fatigue and stress management

[1371] If a worker shows signs of fatigue, the smart helmet captures facial expressions and voice data and sends it to a server. The server analyzes the data, and if it detects fatigue, it suggests, "It's time for a break. Let's take a short rest."

[1372] Sample prompt: "If a worker is fatigued, how can you encourage them to take a break?"

[1373] In this way, the system of the present invention can provide real-time support to workers in dealing with various difficulties they may face, thereby improving worker efficiency and ensuring safety.

[1374] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1375] Step 1:

[1376] The smart helmet captures visual and audio data.

[1377] Specifically, the smart helmet acquires visual information from the user using a built-in camera, and audio information from a built-in microphone.

[1378] Input: User's visual and audio information

[1379] Output: Captured visual and audio data

[1380] Step 2:

[1381] The visual and audio data captured by the smart helmet is temporarily stored in a buffer.

[1382] This prepares the data for analysis in real time.

[1383] Input: Captured visual and audio data

[1384] Output: Data stored in the buffer

[1385] Step 3:

[1386] The smart helmet compresses and encrypts the buffered data and sends it to the server.

[1387] Since data is transmitted in real time, data size is reduced and high security is maintained.

[1388] Input: Data stored in the buffer

[1389] Output: Compressed and encrypted data

[1390] Step 4:

[1391] The server receives the data transmitted from the smart helmet.

[1392] The received data is decoded for analysis.

[1393] Input: Compressed and encrypted data

[1394] Output: Decoded visual, audio, and emotion data

[1395] Step 5:

[1396] The server analyzes the visual data and recognizes specific objects and people.

[1397] Uses image recognition algorithms (e.g., OpenCV or TensorFlow)

[1398] Input: Decoded visual data

[1399] Output: Analysis results (information on recognized objects and people)

[1400] Step 6:

[1401] The server analyzes the voice data and converts it into text data.

[1402] Uses a speech recognition algorithm (e.g., Google Speech-to-Text API)

[1403] Input: Decoded audio data

[1404] Output: Converted text data

[1405] Step 7:

[1406] The server analyzes the user's intent based on the text data.

[1407] Uses natural language processing (NLP) techniques (e.g. spaCy and BERT)

[1408] Input: Converted text data

[1409] Output: Analysis results (user intent information)

[1410] Step 8:

[1411] The server analyzes the emotional data and evaluates the user's emotional state.

[1412] Use an emotion engine (e.g., Emotion API)

[1413] Input: Decoded emotion data

[1414] Output: Analysis result (user's emotional state)

[1415] Step 9:

[1416] A server generates work instructions, safety warnings, and supporting information for fatigue and stress management.

[1417] Use a generative AI model (e.g. GPT-3)

[1418] Input: Analysis results (object / person information, user intent, emotional state)

[1419] Output: Supporting information (work instructions, warnings, break suggestions)

[1420] Step 10:

[1421] The server transmits the generated support information to the smart helmet.

[1422] Input: Support Information

[1423] Output: Support information sent to the smart helmet

[1424] Step 11:

[1425] The smart helmet provides support information to the user via voice.

[1426] Uses voice synthesis technology (e.g., Amazon Polly)

[1427] Input: Support Information

[1428] Output: Supporting information provided by voice

[1429] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1430] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1431] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1432] [Fourth embodiment]

[1433] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1434] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1435] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1436] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1437] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1439] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1440] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1441] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1442] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1443] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1444] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1445] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1446] The present invention is a system for supporting the daily lives of patients with early-stage dementia, capturing visual and audio data, analyzing the data, and providing appropriate support information in real time. The system and its embodiments are described in detail below.

[1447] Overall system configuration

[1448] This system consists of a glasses-type device (terminal) worn by the user and a server that receives and analyzes the data. The glasses-type device has a built-in camera and microphone that captures the user's visual and audio data. The captured data is sent to the server in real time, where it is analyzed and appropriate support information is generated. The generated support information is then provided to the user via the glasses-type device.

[1449] Visual and audio data capture

[1450] User

[1451] When a user wears the eyeglasses, the camera in the glasses continuously captures images (visual data) within the user's field of vision, and the built-in microphone records the user's and their surrounding sounds (audio data).

[1452] Sending data

[1453] Terminal

[1454] The captured visual and audio data is temporarily stored in a buffer on the device, then compressed, encrypted, and sent to a server. This communication occurs in real time, minimizing the delay between when the data is collected and when it reaches the server.

[1455] Receiving and analyzing data

[1456] server

[1457] The server receives visual and audio data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[1458] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[1459] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[1460] Generate support information

[1461] server

[1462] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[1463] Providing support information

[1464] Terminal

[1465] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[1466] Specific examples

[1467] Example 1: Person recognition

[1468] User: "Who are you?"

[1469] The device's microphone captures this audio and sends the data to a server.

[1470] The server analyzes the voice data and determines whether it is a sign of aphasia.

[1471] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[1472] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1473] Example 2: Directional indication

[1474] If the user forgets where the door is in the room,

[1475] The device's camera captures the view and sends it to the server.

[1476] The server analyzes the visual data and does not detect the presence of a "door."

[1477] As support information, the message "The door is on the right" is generated and sent to the terminal.

[1478] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1479] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[1480] The processing flow will be explained below.

[1481] Step 1:

[1482] The user puts on the eyeglasses-type device, which enables the device to capture the user's visual and audio data.

[1483] Step 2:

[1484] The device's camera continuously captures the image of the user's field of vision (visual data), and the microphone records the user's and surrounding sounds (audio data).

[1485] Step 3:

[1486] The device temporarily stores the captured visual and audio data in a buffer memory, where it is compressed and encrypted in real time.

[1487] Step 4:

[1488] The device transmits compressed and encrypted visual and audio data to the server, and this communication occurs in real time to minimize latency.

[1489] Step 5:

[1490] The server receives the visual and audio data sent from the terminal, and decompresses and decodes the respective data.

[1491] Step 6:

[1492] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[1493] Step 7:

[1494] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[1495] Step 8:

[1496] The server analyzes visual and audio data to identify the problem the user is facing and generates real-time support information, such as the name of a person if the user does not recognize them and their relationship to the person.

[1497] Step 9:

[1498] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[1499] Step 10:

[1500] The server transmits the generated voice data to the terminal.

[1501] Step 11:

[1502] The device plays back the received audio data, providing support information to the user through audio and giving necessary advice in real time. For example, if the user gets lost, audio instructions will be given on the direction they should go.

[1503] Example 1

[1504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1505] Patients with early-stage dementia may experience problems in their daily lives, such as difficulty recognizing objects and people, or a lack of understanding of intentions. This not only makes it difficult for patients to make appropriate decisions and reduces their quality of life, but also places a burden on caregivers and family members. There is a need for a support system that can solve these problems and enable patients to live their daily lives more smoothly.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1507] In this invention, the server includes means for acquiring visual information, means for acquiring acoustic information, means for temporarily storing the acquired visual information and acoustic information and compressing and encrypting it, means for transmitting the compressed and encrypted information in real time, means for analyzing the received visual information and acoustic information, means for generating support information in real time based on the analysis results, and means for converting the generated support information into audio and providing it to the user. This makes it possible to provide real-time support for the difficulties faced by early-stage dementia patients in their daily lives and improve the quality of life of the patients.

[1508] "Visual information" refers to image data captured in the user's field of vision by an image capture device such as a camera in a glasses-type device.

[1509] "Acoustic information" refers to audio data emitted from the user and the environment, which is acquired by an audio acquisition device such as a microphone of the eyeglass-type device.

[1510] "Temporary storage" refers to a process of temporarily storing acquired visual and acoustic information in memory or storage.

[1511] "Compression" is data processing performed to reduce the data volume of visual and audio information.

[1512] "Encryption" is the process of protecting data with encryption technology in order to securely transmit visual and audio information.

[1513] "Transmission" is the process of sending compressed and encrypted visual and audio information from the terminal to the server over the network.

[1514] "Reception" refers to the process in which the server receives visual and audio information sent from the terminal.

[1515] "Analysis" refers to analyzing received visual information using image recognition algorithms, and converting acoustic information into text data using speech recognition algorithms and analyzing it through natural language processing.

[1516] "Support information" is information that includes advice and instructions for the user based on the analysis results, and is information that is generated in real time for problem solving.

[1517] "Conversion to voice" refers to a process of converting the generated assistance information into voice data using voice synthesis technology so that it can be provided to the user audibly.

[1518] "Providing to the user" refers to providing the converted audio data in a form that the user can hear through the speaker of the eyeglass-type device, etc.

[1519] The present invention is a system for supporting the daily lives of patients with early-stage dementia. This system captures visual and acoustic information, analyzes the acquired data, and provides support information in real time. Detailed embodiments of the present invention will be described below.

[1520] Hardware Configuration

[1521] This system consists of a glasses-type device (hereafter referred to as the terminal) worn by the user and a server that receives and analyzes data. The terminal has a built-in camera and microphone, which are responsible for capturing visual and acoustic information. The server is a general-purpose computer equipped with a high-performance CPU and GPU.

[1522] Software Configuration

[1523] The device is installed with software for capturing visual and acoustic information: driver software for acquiring images from the camera and audio processing software for acquiring audio from the microphone.

[1524] The server has the following software modules installed:

[1525] Image recognition algorithms: e.g., YOLOv5 and OpenCV

[1526] Speech recognition algorithms, such as Google Cloud Speech-to-Text or Microsoft Azure Speech Service

[1527] Natural Language Processing (NLP) modules: e.g., BERT models

[1528] Speech synthesis technology: e.g., Google Text-to-Speech or Amazon Polly

[1529] Operating procedures and data processing

[1530] When a user wears the glasses, the device's camera captures the user's visual information in real time, and the microphone records audio information. The device temporarily stores this data, compresses and encrypts it, and then transmits it to a server.

[1531] The server receives the visual and audio information sent from the device, first decompresses and decodes the data, then uses an image recognition algorithm to analyze the visual information and identify objects and people in the user's field of view, and simultaneously uses a speech recognition algorithm to convert the audio information into text data and understand the user's speech and intent through natural language processing.

[1532] Based on the analysis results, the server generates real-time support information. For example, if the user does not recognize a person, it will provide the person's name and relationship to the person. If the user gets lost, it will provide advice on the direction to go. This support information is then compressed and encrypted again before being sent to the device.

[1533] The device converts the received assistance information into voice data using voice synthesis technology and plays it back through a built-in speaker, allowing users to obtain the necessary information in real time by voice.

[1534] Specific examples

[1535] Example 1: Person recognition

[1536] User: Ask "Who are you?"

[1537] Device: The microphone captures this audio and sends the data to the server.

[1538] Server: Analyzes the voice data and understands the user's question.

[1539] Server: Analyzes visual data to identify the person being questioned.

[1540] Server: Generate supporting information: "This is [Name], one of your doctors."

[1541] Server: Sends this information to the device.

[1542] Terminal: Support information is converted into voice using speech synthesis technology and provided to the user.

[1543] Example 2: Directional indication

[1544] If the user forgets the location of the door in the room, visual data is captured and sent to the server.

[1545] The server does not analyze the visual data to detect the location of the door.

[1546] The server generates the supporting information "The door is on the right."

[1547] The server sends this information to the terminal.

[1548] The terminal uses voice synthesis technology to convert support information into voice and provide it to the user.

[1549] Example prompts to input to the generative AI model

[1550] Example prompt 1: "When a user asks who they are, analyze the speech data to determine signs of aphasia and generate supporting information."

[1551] Prompt example 2: "When the user forgets the location of the door through visual data, and the server analyzes and cannot determine the location of the door from the visual data."

[1552] As described above, this system is capable of analyzing visual and acoustic information in real time and providing appropriate support information in the form of voice to support the daily lives of patients with early-stage dementia.

[1553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1554] Step 1:

[1555] The user puts on the eyeglasses-type device. This activates the device's camera and microphone, and the device begins capturing visual and acoustic information. The user's wearing of the device is required as input, and the device's sensors are activated as output.

[1556] Step 2:

[1557] The device uses a camera to capture the user's visual information in real time and a microphone to record audio information. The input is the user's visual and audio environment, and the output is the captured visual and audio information.

[1558] Step 3:

[1559] The terminal temporarily stores the captured visual and audio information. The temporarily stored data is temporarily stored in a buffer. The captured visual and audio information is required as input, and the temporarily stored data is obtained as output.

[1560] Step 4:

[1561] The terminal compresses and encrypts the temporarily stored visual and audio information. The input is the temporarily stored data. The output is the compressed and encrypted data. Specifically, data compression uses a compression algorithm such as JPEG or MP3, and encryption uses an encryption algorithm such as AES.

[1562] Step 5:

[1563] The terminal sends compressed and encrypted data to the server. This transmission is done in real time with minimal latency. The input is the compressed and encrypted data. The output is the transmitted data.

[1564] Step 6:

[1565] The server receives the data sent from the terminal. After receiving it, it decompresses and decrypts the data. The input is the sent data, and the output is the decompressed and decrypted data. Specifically, it uses a decompression algorithm such as gzip or bzip2 for decompression, and the AES decryption algorithm for decryption.

[1566] Step 7:

[1567] The server analyzes the decompressed and decoded visual information using an image recognition algorithm. For example, it uses YOLOv5 or OpenCV to identify objects and people in the user's field of view. Visual information is required as input, and the analysis results are obtained as output. Specifically, the image recognition algorithm analyzes the visual information and recognizes objects and people.

[1568] Step 8:

[1569] The server converts the decompressed and decoded audio information into text data using a speech recognition algorithm and understands the user's speech and intent through natural language processing (NLP). The input is audio information, and the output is analyzed text data. Specifically, the speech recognition algorithm analyzes the audio information and converts it into text data, and then the NLP module analyzes the text data.

[1570] Step 9:

[1571] The server generates support information in real time based on the results of visual and audio data analysis. For example, if the user cannot recognize a person, it will convey the person's name and relationship to them. Also, if the user gets lost, it will instruct the user on the direction to go. The analysis results are required as input, and the generated support information is obtained as output.

[1572] Step 10:

[1573] The server compresses and encrypts the generated support information again and sends it to the terminal. The generated support information is required as input, and the compressed and encrypted support information is obtained as output.

[1574] Step 11:

[1575] The terminal decompresses and decodes the assistance information received from the server. As input, it takes compressed and encrypted assistance information, and as output, it gets the decompressed and decoded assistance information.

[1576] Step 12:

[1577] The device converts the decompressed and decoded assistance information into audio data using speech synthesis technology and provides it to the user through a built-in speaker. The decoded assistance information is required as input, and audio data is obtained as output. Specifically, the device uses speech synthesis technology to convert the assistance information into natural audio and provides it to the user audibly.

[1578] (Application example 1)

[1579] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1580] Patients with early-stage dementia often face difficulties in their daily lives due to being unable to recognize objects and people and losing their sense of direction. This also increases the likelihood of them getting into dangerous situations, increasing their safety and the burden on their families and caregivers. To solve these problems, a system is needed that provides appropriate support in real time in the various situations patients face, and detects danger and issues warnings.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1582] In this invention, the server includes a means for analyzing visual data and audio data in real time, a means for monitoring the user's surroundings based on the analysis results and detecting danger, and a means for generating warnings and advice when danger is detected. This makes it possible to provide real-time support for early-stage dementia patients to live their daily lives safely and appropriately avoid danger.

[1583] A "means for capturing visual data" is a device or method for acquiring images that appear in the user's field of vision using a device such as a camera.

[1584] "Means for capturing audio data" refers to a device or method for capturing sounds from the user and their surroundings using a device such as a microphone.

[1585] "Means for analyzing captured visual and audio data" refers to a device or method that uses a specific algorithm to understand the content of the acquired visual and audio data and generate analytical results.

[1586] The "means for generating support information in real time based on the analysis results" refers to a device or method for instantly generating support information required by the user based on the analysis results of the visual data and audio data.

[1587] The "means for providing the generated support information to the user" refers to a device or method for transmitting the generated support information to the user by audio or visual means.

[1588] The "means for monitoring the situation around the user based on the analysis results and detecting danger" refers to a device or method for monitoring the environment and behavior around the user based on the results of data analysis and identifying dangerous situations.

[1589] A "means for generating a warning or advice upon detection of a danger" is a device or method that provides an appropriate warning or advice to a user when a dangerous situation is identified.

[1590] This invention is a support system to help patients with early-stage dementia live their daily lives more safely. The system begins operation when the user puts on the smart glasses. The smart glasses have a built-in camera and microphone that capture visual and audio data. The captured data is temporarily stored in the device, compressed and encrypted, and then transmitted to a server in real time.

[1591] The server analyzes the received visual data using image recognition algorithms. Specifically, it uses image processing libraries such as OpenCV to identify objects and people in the visual data and determine what the user is looking at and the situation they are facing. It also uses speech recognition algorithms to convert audio data into text data and analyzes the user's intent using natural language processing techniques (e.g., NLTK, spaCy).

[1592] Based on the analysis results, the server generates support information in real time. Specifically, if the user does not recognize a person, it will tell them the person's name and relationship, or if the user gets lost, it will give them directions. The analysis results also monitor the user's surroundings, and if a dangerous situation is identified, it will generate appropriate warnings or advice (e.g., "The door is open" or "Did you forget to take your medicine?").

[1593] The generated support information is then provided to the user via the smart glasses. Using speech synthesis technology (e.g., gTTS), the text data is converted into speech and played back through the glasses' speakers, allowing the user to receive the necessary support in real time.

[1594] For example:

[1595] Example 1: Detecting an open door

[1596] If the camera does not detect the presence of a door even though the user frequently uses the word "door" in conversation, the server will generate a warning saying "Door is open" and notify the user audibly through the smart glasses.

[1597] Example 2: Medication reminders

[1598] The server analyzes the user's voice data, and if the word "medicine" is not detected within a certain period of time, it generates a reminder "Have you taken your medicine?" and notifies the user via voice through the smart glasses.

[1599] Example prompts for generative AI models

[1600] "Please use visual and audio data to identify dangerous situations that early-stage dementia patients face in their daily lives and generate appropriate warning information. The visual data should be a byte array in JPEG format, and the audio data should be in WAV format with a sampling rate of 8 kHz. Please output in the following format:

[1601] {

[1602] "status": "safe" or "danger",

[1603] "warning_text": "Danger warning message"

[1604] }"

[1605] This system allows people with early-stage dementia to receive the support they need in real time in their daily lives, significantly improving safety and reducing the burden on their families and caregivers.

[1606] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1607] Step 1:

[1608] A user puts on smart glasses. The glasses are equipped with a camera and microphone, and begin capturing the user's visual and audio data. The input is the image in the user's field of vision and the surrounding audio, and the output is the captured visual and audio data.

[1609] Step 2:

[1610] The device temporarily stores the captured visual and audio data in a buffer. The stored data is compressed and encrypted. The input is the captured visual and audio data, and the output is the compressed and encrypted visual and audio data.

[1611] Step 3:

[1612] The terminal transmits compressed and encrypted visual and audio data to the server in real time. The input is the compressed and encrypted visual and audio data, and the output is the data transmission to the server.

[1613] Step 4:

[1614] The server analyzes the received visual data using an image recognition algorithm (e.g., OpenCV). The image recognition algorithm identifies objects and people in the visual data and determines what is in the user's field of view. The input is the visual data sent to the server, and the output is the analyzed visual information.

[1615] Step 5:

[1616] The server converts the received voice data into text data using a voice recognition algorithm and analyzes the user's intent using natural language processing technology (e.g., NLTK, spaCy). The input is the voice data sent to the server, and the output is the text data and analyzed intent information.

[1617] Step 6:

[1618] The server generates support information in real time based on the analysis results. The generated support information includes solutions to problems the user is facing and warnings. For example, it can be direction guidance if the user gets lost or person recognition information. The input is the analyzed visual information and intention information, and the output is support information and warning messages.

[1619] Step 7:

[1620] The server transmits the generated support information to the terminal again. The input is the generated support information, and the output is data transmission to the terminal.

[1621] Step 8:

[1622] The device provides the user with the support information received from the server. Using speech synthesis technology (e.g., gTTS), the support information is converted into speech and played back through the smart glasses' speaker. The input is the support information sent from the server, and the output is a voice notification to the user.

[1623] Step 9:

[1624] The user receives a voice notification and takes appropriate action, such as closing the door or taking medicine after receiving the warning. The input is the voice notification, and the output is the user's action.

[1625] In this way, all processing steps work together to provide real-time support to patients with early-stage dementia who are faced with problems and dangers in their daily lives, enabling them to maintain a safe life.

[1626] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1627] The present invention incorporates an emotion engine into a system that captures and analyzes visual and audio data to support the daily lives of people with early-stage dementia, thereby recognizing the user's emotional state and providing personalized responses. The system and its embodiments are described in detail below.

[1628] Overall system configuration

[1629] This system consists of an eyeglass-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine. The eyeglass-type device has a built-in camera and microphone that captures the user's visual, audio, and emotional data. The captured data is sent in real time to the server, which analyzes it and generates appropriate support information. The generated support information is then provided to the user again via the eyeglass-type device.

[1630] Capture visual, audio, and emotional data

[1631] User

[1632] When a user wears the glasses-type device, the device's camera captures the image in the user's field of vision (visual data) and the user's facial expressions. The built-in microphone also records the user's and their surroundings' sounds (audio data). Emotional data is acquired by analyzing voice tone, facial expressions, and body movements.

[1633] Sending data

[1634] Terminal

[1635] The captured visual, audio, and emotional data is temporarily stored in a buffer on the device, then compressed, encrypted, and transmitted to a server in real time. This real-time communication minimizes the delay between the time the data is collected and the time it reaches the server.

[1636] Receiving and analyzing data

[1637] server

[1638] The server receives visual, audio, and emotional data sent from the device, which is then instantly analyzed to identify any problems or difficulties the user may be experiencing as early symptoms of dementia.

[1639] Visual data is analyzed using image recognition algorithms, for example to identify objects or people in the field of view and determine what the user is looking at.

[1640] Voice data is analyzed by converting it into text data using a voice recognition algorithm, and then natural language processing is used to understand what the user is saying and their intent.

[1641] Emotion data analysis is performed by an emotion engine, which assesses the user's emotional state based on voice tone, facial expressions, and body movements.

[1642] Generate support information

[1643] server

[1644] Based on the analysis results on the server, solutions and advice for the problems the user is facing are generated in real time. For example, if the user does not recognize a person, the system will tell them the person's name and relationship to them. Also, if the user gets lost, the system will show them the direction they should go.

[1645] Additionally, the emotion engine's analysis can provide support based on the user's emotional state: for example, if the user is feeling anxious, it will provide instructions in a calmer, more reassuring voice.

[1646] Providing support information

[1647] Terminal

[1648] The generated support information is then sent back to the device and provided to the user via voice. Natural-sounding voice is generated from the text data using speech synthesis technology and played back through the glasses-type device's speaker.

[1649] Specific examples

[1650] Example 1: Person recognition

[1651] User: "Who are you?"

[1652] The device's microphone captures this audio and sends the data to a server.

[1653] The server analyzes the voice data and determines whether it is a sign of aphasia.

[1654] As support information, the following is generated and sent to the terminal: "This is [Name], one of your doctors."

[1655] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1656] Example 2: Directional indication

[1657] If the user forgets where the door is in the room,

[1658] The device's camera captures the view and sends it to the server.

[1659] The server analyzes the visual data and does not detect the presence of a "door."

[1660] As support information, the message "The door is on the right" is generated and sent to the terminal.

[1661] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1662] Example 3: Emotion Recognition

[1663] If the user looks anxious,

[1664] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[1665] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[1666] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[1667] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1668] Through this processing, the system of the present invention can provide real-time support to patients with early-stage dementia who face various difficulties in their daily lives, thereby improving the quality of life of the user and reducing the burden on their family and caregivers.

[1669] The processing flow will be explained below.

[1670] Step 1:

[1671] The user puts on the eyeglasses, which allows the device to capture the user's visual, audio, and emotional data.

[1672] Step 2:

[1673] The device's camera continuously captures the image (visual data) of the user's field of vision, while the microphone records the user's and surrounding sounds (audio data). At the same time, the device captures the user's facial expressions, tone of voice, and body movements and records them as emotional data.

[1674] Step 3:

[1675] The device temporarily stores the captured visual, audio, and emotional data in a buffer memory, which is then compressed, encrypted, and sent to the server in real time.

[1676] Step 4:

[1677] The server receives the visual data, audio data, and emotion data sent from the terminal, and decompresses and decodes each data.

[1678] Step 5:

[1679] The server analyzes the visual data, using image recognition algorithms to identify objects and people in the video and determine what the user is looking at.

[1680] Step 6:

[1681] The server analyzes the voice data, converts it into text using speech recognition algorithms, and uses natural language processing to understand what the user is saying and what they mean.

[1682] Step 7:

[1683] The server analyzes the emotion data and uses an emotion engine to assess the user's emotional state based on voice tone, facial expressions, and body movements.

[1684] Step 8:

[1685] Based on the analysis results, the server identifies the problem the user is facing and generates support information in real time. For example, if the user does not recognize a person, it will tell them the person's name and relationship to them. Also, if the user gets lost, it will give them directions to go.

[1686] Step 9:

[1687] The server uses the analysis results of the emotion engine to generate support information according to the user's emotional state. For example, if the user is feeling anxious, it will choose calm and reassuring words.

[1688] Step 10:

[1689] The server converts the generated support information into voice data, using voice synthesis technology to generate natural-sounding voice from the text data.

[1690] Step 11:

[1691] The server transmits the generated voice data to the terminal.

[1692] Step 12:

[1693] The device plays back the received audio data, providing the user with support information and necessary advice in real time. For example, if the user gets lost, the device will provide voice instructions on the direction to go. If the user feels anxious, the device will offer reassuring words.

[1694] For example, if a user cannot recognize a person or gets lost, the server generates appropriate support information and provides it to the user via voice via the device. The emotion engine also evaluates the user's emotional state, and if the user is feeling anxious, for example, it provides reassuring words in a calm tone. This series of processes makes it possible to support the user's daily life in real time.

[1695] Example 2

[1696] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1697] Until now, there has been no system that can effectively support the daily lives of patients with early-stage dementia by collecting and analyzing comprehensive data, including not only visual and audio data but also emotional data, in real time. As a result, real-time support including the user's emotional state has not been provided, and difficulties in daily life have not been adequately resolved.

[1698] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1699] In this invention, the server includes means for analyzing visual data using an image recognition algorithm, means for analyzing audio data using a voice recognition algorithm, and means for analyzing emotional data using an emotion engine, thereby enabling comprehensive analysis of visual, audio, and emotional data and providing real-time support information based on the user's emotional state.

[1700] "Visual data" refers to image and video data acquired by a device that captures visual information, such as a camera.

[1701] "Audio data" refers to data that includes sounds and voices acquired by an audio input device such as a microphone.

[1702] "Emotional data" is data used to analyze a user's emotional state by capturing vocal tones, facial expressions, body movements, etc.

[1703] An "image recognition algorithm" is an algorithm that automatically detects and recognizes specific objects, people, and scenes from images and videos.

[1704] A "voice recognition algorithm" is an algorithm for analyzing voice data and converting the voice into text data.

[1705] The "emotion engine" is software that evaluates and analyzes the user's emotional state based on captured data.

[1706] "Support information" refers to information and instructions generated based on the results of analyzing the user's visual data, audio data, and emotional data.

[1707] "Real-time" refers to the extremely short time between data collection and the generation of analysis results, meaning that processing is immediate.

[1708] This invention is a system that captures visual, audio, and emotional data and analyzes them in real time to support the daily lives of patients with early-stage dementia. This system consists of a glasses-type device (terminal) worn by the user, a server that receives and analyzes the data, and an emotion engine.

[1709] Overall system configuration

[1710] User

[1711] The user wears a pair of glasses equipped with a camera and microphone that captures the user's visual, audio, and emotional data.

[1712] Terminal

[1713] The captured visual, audio, and emotion data is temporarily stored in a buffer on the device. This data is then compressed and encrypted in real time and sent to the server. Specifically, image data is converted to JPEG format, audio data is converted to AAC format, etc., and the data is encrypted using the SSL / TLS communication protocol before being sent to the server.

[1714] Sending data

[1715] server

[1716] The server receives the visual, audio, and emotion data sent from the device. The received data is immediately analyzed. Specifically, the following process is performed:

[1717] Visual data analysis: Using image recognition algorithms (e.g., OpenCV) on the server, objects and people are identified in the video.

[1718] Analyzing speech data: Using speech recognition algorithms (e.g., Google Cloud Speech-to-Text API) to convert speech into text data, and then applying natural language processing (NLP) techniques to understand the content.

[1719] Emotional data analysis: Using an emotion engine (e.g., Affectiva) to assess emotional state from vocal tone, facial expressions, and body movements.

[1720] Generate support information

[1721] server

[1722] Based on the analysis results, the server generates the support information the user needs in real time. For example, if a user looks at a friend's face and asks, "Who are you?", this voice is captured and analyzed by the server, and an appropriate answer (e.g., "This is [Name]. I'm one of your friends.") is generated. If the user gets lost, the situation is analyzed from visual data and directional instructions such as "The door is on the right" are generated. If the user looks anxious, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated based on the analysis of emotional data.

[1723] Providing support information

[1724] Terminal

[1725] The generated support information is then sent back to the device and provided to the user via voice. Using speech synthesis technology (e.g., Amazon Polly), the support information is converted into natural-sounding speech and presented to the user through the device's speaker.

[1726] Specific examples of embodiments

[1727] Example 1: Person recognition

[1728] User: "Who are you?"

[1729] The device's microphone captures this audio and sends it to the server.

[1730] The server analyzes the voice data and determines whether it is a sign of aphasia.

[1731] Support information such as "This is [name], one of your friends" is generated and sent to the device.

[1732] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1733] Example 2: Directional indication

[1734] If the user forgets where the door is in the room,

[1735] The device's camera captures the view and sends it to the server.

[1736] The server does not analyze the visual data to detect the presence of a "door."

[1737] The support information "The door is on the right" is generated and sent to the terminal.

[1738] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1739] Example 3: Emotion Recognition

[1740] If the user looks anxious,

[1741] The device's camera captures the user's facial expressions and sends them to the server along with the audio data.

[1742] The server's emotion engine performs analysis and determines that the user is feeling anxious.

[1743] As support information, a reassuring message such as "Are you OK? Is there anything I can help you with?" is generated and sent to the device.

[1744] The terminal uses speech synthesis technology to convert this text into speech and provide it to the user.

[1745] The above is a specific embodiment of the present invention. This system enables real-time support for early-stage dementia patients in dealing with the various difficulties they face in their daily lives, and is expected to improve the user's quality of life and reduce the burden on family and caregivers.

[1746] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1747] Step 1: Data Capture

[1748] User

[1749] The user wears the eyeglass-type device.

[1750] Input: The environment in which the user generates visual and audio data.

[1751] Output: Visual, audio and emotional data captured by the device.

[1752] How it works: The camera captures the user's field of view in real time, and the microphone records the user's speech and the surrounding environmental sounds. It also captures the tone of voice and facial expressions as emotional data.

[1753] Step 2: Temporarily save data

[1754] Terminal

[1755] The captured visual data, audio data, and emotion data are temporarily stored in the device's built-in buffer.

[1756] Input: Visual, audio, and emotional data captured by the device.

[1757] Output: The data stored in the buffer.

[1758] Specific operation: Temporarily stores captured data in memory and checks whether the data is in a suitable format for further processing.

[1759] Step 3: Compress and Encrypt the Data

[1760] Terminal

[1761] The stored data is compressed and encrypted and prepared for transmission to the server.

[1762] Input: Visual, audio and emotion data in a buffer.

[1763] Output: Compressed and encrypted data.

[1764] Specific operations: Image data is compressed to JPEG format, audio data is converted to AAC format, and data is encrypted using the SSL / TLS protocol.

[1765] Step 4: Sending data

[1766] Terminal

[1767] The compressed and encrypted data is sent to the server in real time.

[1768] Input: Compressed and encrypted data.

[1769] Output: The data received by the server.

[1770] Specific operation: Send data to the server through the terminal's communication module and confirm the success of the transmission.

[1771] Step 5: Receiving the data

[1772] server

[1773] The server receives the data sent from the terminal.

[1774] Input: Compressed and encrypted data sent from the terminal.

[1775] Output: Raw data stored in the server.

[1776] Specific operation: The server decrypts and unpacks the received data and prepares it for analysis.

[1777] Step 6: Analyze the visual data

[1778] server

[1779] The server analyzes the received visual data using image recognition algorithms.

[1780] Input: Decoded and unpacked visual data.

[1781] Output: Analysis results (e.g., identification of specific objects or people).

[1782] Specific operation: Using the OpenCV library, it identifies objects and people from video and identifies what the user is looking at.

[1783] Step 7: Analyze the audio data

[1784] server

[1785] The server analyzes the received voice data using a voice recognition algorithm.

[1786] Input: Decoded and decompressed audio data.

[1787] Output: Analysis results (e.g., converting audio into text data).

[1788] Specific operation: Uses Google Cloud Speech-to-Text API to convert speech into text data and uses NLP technology to understand the content.

[1789] Step 8: Analyze the sentiment data

[1790] server

[1791] The emotion data received by the server is analyzed using an emotion engine.

[1792] Input: Decoded and unpacked emotion data.

[1793] Output: Analysis results (e.g., assessment of the user's emotional state).

[1794] Specific behavior: Using the Affectiva emotion engine, we assess the user's emotional state from their vocal tone, facial expressions, and body movements.

[1795] Step 9: Generate support information

[1796] server

[1797] Support information is generated in real time based on the analysis results.

[1798] Input: Analysis results of visual, audio and emotion data.

[1799] Output: The generated supporting information.

[1800] Specific operation: Generates the solutions and advice the user needs in text format and creates support information tailored to the user's situation.

[1801] Step 10: Provide support information

[1802] Terminal

[1803] The generated support information is sent again to the terminal and provided to the user by voice.

[1804] Input: Generated supporting information.

[1805] Output: Natural-sounding voice delivered through speech synthesis.

[1806] Specific operation: Using speech synthesis technology such as Amazon Polly, the generated text data is converted into speech and provided to the user through the device's speaker.

[1807] (Application example 2)

[1808] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1809] To ensure the efficiency and safety of workers in factories, it is necessary to provide appropriate work instructions and safety warnings in real time. However, conventional methods have made it difficult to provide appropriate support that takes into account the emotional state, fatigue, and stress of workers. In addition, since rapid situational assessment is required on-site, a real-time analysis system with minimal delay is required.

[1810] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1811] In this invention, the server includes means for capturing visual data, means for capturing audio data, means for analyzing the captured visual data and audio data, means for generating support information in real time based on the analysis results, means for providing the generated support information to the user, means for capturing and analyzing emotional data, and means for generating and providing work instructions, safety warnings, and information for fatigue and stress management based on the analysis results. This enables appropriate support that takes into account the emotional state, fatigue, and stress of the worker, and rapid situation assessment.

[1812] "Visual data" is video data that appears in the user's field of vision and is information captured by a camera.

[1813] "Audio data" refers to data that indicates audio information of the user and their surroundings and is recorded by a microphone.

[1814] "Analysis means" refers to the systems and algorithms that process the captured visual and audio data and generate appropriate information based on the results.

[1815] "Support information" is information such as specific instructions and advice generated by the analysis means to support the user's daily life and work.

[1816] "Emotion data" is information about the user's emotional state obtained by analyzing voice tone, facial expressions, body movements, and the like.

[1817] "Work instructions" are information that guides the user on the specific work content and procedures that should be performed.

[1818] "Safety warnings" are information that alert users when they are faced with a dangerous situation.

[1819] "Fatigue and stress management" is information that detects fatigue and stress from the user's facial expressions, voice, etc., and suggests measures to take or take a break.

[1820] The present invention is a system that uses a head-mounted display (hereinafter referred to as a smart helmet) worn by a worker, and aims to support work efficiently and safely by capturing and analyzing visual data, audio data, and emotional data. The specific configuration and operation of this system are described below.

[1821] Overall system configuration

[1822] The system consists of a smart helmet worn by the user, a server that receives and analyzes data, and a means of providing the user with information generated based on the analysis results.The smart helmet has a built-in camera and microphone that captures the user's visual and audio data, as well as emotional data.

[1823] Capture visual, audio, and emotional data

[1824] When a user wears a smart helmet, the built-in camera captures visual data (images seen by the user) and facial expressions, and the built-in microphone records audio data of the user and their surroundings. This data is captured in real time and sent to a server for analysis.

[1825] Sending data

[1826] The smart helmet temporarily stores captured visual, audio, and emotional data in a buffer, which is then compressed, encrypted, and sent to a server in real time. This low-latency communication allows for real-time analysis.

[1827] Receiving and analyzing data

[1828] The server receives and analyzes data sent from the smart helmet in real time. It uses an image recognition algorithm to analyze visual data and recognize specific objects and people. It uses a voice recognition algorithm to analyze audio data, converting it into text data and then analyzing the user's intent using natural language processing (NLP). It uses an emotion engine to analyze emotional data, which allows it to evaluate the worker's emotional state, as well as their fatigue and stress levels.

[1829] Generate support information

[1830] Based on the analysis results, the server generates supporting information in real time, such as:

[1831] Work Instructions: Understand what you're currently doing and provide specific instructions on what to do next.

[1832] Safety warning: When a dangerous situation is detected, it provides warning information and instructions on how to avoid it.

[1833] Fatigue and stress management: Detects worker fatigue and stress and suggests breaks.

[1834] This support information is generated in real time and sent back to the smart helmet, where it is provided to the user audibly using voice synthesis technology.

[1835] Specific examples

[1836] Example 1: Work instruction support

[1837] If a worker forgets a particular step and looks confused, the smart helmet captures video and audio and sends them to the server, which analyzes the procedure and generates instructions, such as "Next, pull the left lever."

[1838] Example prompt: "If I forget this step, tell me what to do next."

[1839] Example 2: Safety warning

[1840] When a worker approaches a dangerous area, the camera in the smart helmet captures the surrounding image and sends it to the server. The server analyzes the image and, if it detects a dangerous area, generates an audio warning saying, "Please retreat. This is dangerous."

[1841] Example prompt: "If a worker enters a dangerous area, give instructions to evacuate immediately."

[1842] Example 3: Fatigue and stress management

[1843] If a worker shows signs of fatigue, the smart helmet captures facial expressions and voice data and sends it to a server. The server analyzes the data, and if it detects fatigue, it suggests, "It's time for a break. Let's take a short rest."

[1844] Sample prompt: "If a worker is fatigued, how can you encourage them to take a break?"

[1845] In this way, the system of the present invention can provide real-time support to workers in dealing with various difficulties they may face, thereby improving worker efficiency and ensuring safety.

[1846] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1847] Step 1:

[1848] The smart helmet captures visual and audio data.

[1849] Specifically, the smart helmet acquires visual information from the user using a built-in camera, and audio information from a built-in microphone.

[1850] Input: User's visual and audio information

[1851] Output: Captured visual and audio data

[1852] Step 2:

[1853] The visual and audio data captured by the smart helmet is temporarily stored in a buffer.

[1854] This prepares the data for analysis in real time.

[1855] Input: Captured visual and audio data

[1856] Output: Data stored in the buffer

[1857] Step 3:

[1858] The smart helmet compresses and encrypts the buffered data and sends it to the server.

[1859] Since data is transmitted in real time, data size is reduced and high security is maintained.

[1860] Input: Data stored in the buffer

[1861] Output: Compressed and encrypted data

[1862] Step 4:

[1863] The server receives the data transmitted from the smart helmet.

[1864] The received data is decoded for analysis.

[1865] Input: Compressed and encrypted data

[1866] Output: Decoded visual, audio, and emotion data

[1867] Step 5:

[1868] The server analyzes the visual data and recognizes specific objects and people.

[1869] Uses image recognition algorithms (e.g., OpenCV or TensorFlow)

[1870] Input: Decoded visual data

[1871] Output: Analysis results (information on recognized objects and people)

[1872] Step 6:

[1873] The server analyzes the voice data and converts it into text data.

[1874] Uses a speech recognition algorithm (e.g., Google Speech-to-Text API)

[1875] Input: Decoded audio data

[1876] Output: Converted text data

[1877] Step 7:

[1878] The server analyzes the user's intent based on the text data.

[1879] Uses natural language processing (NLP) techniques (e.g. spaCy and BERT)

[1880] Input: Converted text data

[1881] Output: Analysis results (user intent information)

[1882] Step 8:

[1883] The server analyzes the emotional data and evaluates the user's emotional state.

[1884] Use an emotion engine (e.g., Emotion API)

[1885] Input: Decoded emotion data

[1886] Output: Analysis result (user's emotional state)

[1887] Step 9:

[1888] A server generates work instructions, safety warnings, and supporting information for fatigue and stress management.

[1889] Use a generative AI model (e.g. GPT-3)

[1890] Input: Analysis results (object / person information, user intent, emotional state)

[1891] Output: Supporting information (work instructions, warnings, break suggestions)

[1892] Step 10:

[1893] The server transmits the generated support information to the smart helmet.

[1894] Input: Support Information

[1895] Output: Support information sent to the smart helmet

[1896] Step 11:

[1897] The smart helmet provides support information to the user via voice.

[1898] Uses voice synthesis technology (e.g., Amazon Polly)

[1899] Input: Support Information

[1900] Output: Supporting information provided by voice

[1901] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1902] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1903] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1904] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1905] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1906] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1907] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1908] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1909] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1910] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1911] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1912] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1913] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1914] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1915] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1916] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1917] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1918] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1919] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1920] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1921] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1922] The following is further disclosed regarding the above embodiment.

[1923] (Claim 1)

[1924] a means for capturing visual data;

[1925] means for capturing audio data;

[1926] means for analyzing the captured visual and audio data;

[1927] a means for generating support information in real time based on the analysis results;

[1928] means for providing the generated support information to a user;

[1929] A system including:

[1930] (Claim 2)

[1931] 10. The system of claim 1, wherein the analyzing means uses an object detection algorithm to analyze the visual data and recognize specific objects.

[1932] (Claim 3)

[1933] 2. The system according to claim 1, wherein the analyzing means converts voice data into text data using a voice recognition algorithm, and analyzes the user's intention based on the converted text data.

[1934] "Example 1"

[1935] (Claim 1)

[1936] a means for acquiring visual information;

[1937] a means for acquiring acoustic information;

[1938] means for temporarily storing, compressing and encrypting the acquired visual and acoustic information;

[1939] means for transmitting the compressed and encrypted information in real time;

[1940] means for analyzing the received visual and acoustic information;

[1941] a means for generating support information in real time based on the analysis results;

[1942] means for converting the generated assistance information into speech and providing it to a user;

[1943] A system including:

[1944] (Claim 2)

[1945] 10. The system of claim 1, wherein the analyzing means analyzes the visual information using an image recognition algorithm to recognize specific objects.

[1946] (Claim 3)

[1947] 2. The system according to claim 1, wherein the analyzing means converts acoustic information into text information using a speech recognition algorithm, and analyzes the user's intention based on the converted text information.

[1948] "Application Example 1"

[1949] (Claim 1)

[1950] a means for capturing visual data;

[1951] means for capturing audio data;

[1952] means for analyzing the captured visual and audio data;

[1953] a means for generating support information in real time based on the analysis results;

[1954] means for providing the generated support information to a user;

[1955] a means for monitoring the situation around the user based on the analysis result and detecting danger;

[1956] means for generating warnings and advice upon detection of a hazard;

[1957] a means for providing generated warnings and advice to a user;

[1958] A system including:

[1959] (Claim 2)

[1960] 10. The system of claim 1, wherein the analyzing means uses an object detection algorithm to analyze the visual data and recognize specific objects.

[1961] (Claim 3)

[1962] 2. The system according to claim 1, wherein the analyzing means converts voice data into text data using a voice recognition algorithm, and analyzes the user's intention based on the converted text data.

[1963] "Example 2: Combining Emotion Engines"

[1964] (Claim 1)

[1965] a means for capturing visual data;

[1966] means for capturing audio data;

[1967] means for capturing emotional data based on an emotional state;

[1968] means for storing the captured visual, audio and emotional data;

[1969] means for transmitting the stored visual, audio and emotional data in real time;

[1970] means for analyzing the received visual data using an image recognition algorithm;

[1971] means for analyzing the received voice data using a voice recognition algorithm;

[1972] means for analyzing the received emotion data using an emotion engine;

[1973] a means for generating support information in real time based on the analysis results;

[1974] means for providing the generated support information to a user;

[1975] A system including:

[1976] (Claim 2)

[1977] 10. The system of claim 1, wherein the analyzing means uses an image recognition algorithm to analyze the visual data and recognize specific objects.

[1978] (Claim 3)

[1979] 2. The system according to claim 1, wherein the analyzing means converts voice data into text data using a voice recognition algorithm, and analyzes the user's intention based on the converted text data.

[1980] (Claim 4)

[1981] 2. The system of claim 1, wherein the analyzing means uses an emotion engine to analyze the emotion data and assess the user's emotional state.

[1982] "Application example 2 when combining emotion engines"

[1983] (Claim 1)

[1984] a means for capturing visual data;

[1985] means for capturing audio data;

[1986] means for analyzing the captured visual and audio data;

[1987] a means for generating support information in real time based on the analysis results;

[1988] means for providing the generated support information to a user;

[1989] a means for capturing and analyzing emotion data;

[1990] means for generating and providing work instructions, safety warnings, and information for fatigue and stress management based on the analysis results;

[1991] A system including:

[1992] (Claim 2)

[1993] 10. The system of claim 1, wherein the analyzing means uses an object detection algorithm to analyze the visual data and recognize specific objects.

[1994] (Claim 3)

[1995] 2. The system according to claim 1, wherein the analyzing means converts voice data into text data using a voice recognition algorithm, and analyzes the user's intention based on the converted text data. [Explanation of symbols]

[1996] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing visual data; means for capturing audio data; means for analyzing the captured visual and audio data; a means for generating support information in real time based on the analysis results; means for providing the generated support information to a user; A system including:

2. 2. The system of claim 1, wherein said analyzing means uses an object detection algorithm to analyze the visual data and recognize specific objects.

3. 2. The system according to claim 1, wherein said analyzing means converts voice data into text data using a voice recognition algorithm, and analyzes the user's intention based on the converted text data.

Images