System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting vital data, a walking pace, and a rhythm of a user in real time; means for analyzing a camera image and detecting unevenness of a road or an obstacle; and means for providing the user with an alert or guidance by voice based on the data collected by the means and an analysis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As our society ages, it is necessary to provide an environment where seniors who are concerned about walking can go out with peace of mind and support their independent lifestyles. However, the risk of falling while walking is increasing, and an increasing number of seniors are hesitant to go out. As a result, lack of exercise and social isolation are further contributing to the deterioration of seniors' health. Conventional walking aids and services lack the ability to analyze users' real-time data and provide immediate feedback, making them inadequate for preventing falls and supporting seniors' walking. [Means for solving the problem]

[0005] The present invention is a system equipped with a means for collecting a user's vital data, walking pace, and rhythm in real time, and a means for analyzing camera images to detect road irregularities and obstacles. It also includes a means for providing the user with audio warnings and guidance based on this data and the analysis results. Furthermore, based on the analysis results of the vital data and walking pace and rhythm, it provides the user with music and rhythms to help pace themselves appropriately, suggests taking a break or slowing down, and recommends exercises to prevent falls. This reduces the risk of seniors falling and creates an environment where they can go out with peace of mind.

[0006] "Vital data" refers to biometric information that indicates a user's health condition, such as heart rate, blood pressure, and body temperature.

[0007] "Walking pace" refers to the speed and rhythm of the stride when the user walks.

[0008] "Real-time" refers to processing and analyzing data immediately at the moment it is generated.

[0009] "Camera image" refers to visual information obtained from a device worn by a user.

[0010] "Road unevenness" refers to a state in which the surface of the road on which the user walks is not flat, but has differences in elevation or steps.

[0011] An "obstacle" refers to an object that exists on the user's walking path and obstructs the user's walking.

[0012] "Warning" refers to the act of informing the user of danger by voice or other means and urging caution.

[0013] "Guidance" refers to audio and visual content that guides and instructs users to walk safely.

[0014] "Pacemaking" refers to providing music or rhythm to keep the user's walking pace constant.

[0015] "Exercise" refers to an exercise program undertaken to maintain or improve physical strength or muscle strength. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a walking assistance system for seniors, designed to allow users to go out with peace of mind. The system collects the user's vital signs and walking pace in real time, and analyzes camera footage to determine the surrounding situation. Furthermore, by providing appropriate audio warnings and guidance, it reduces the risk of falls and supports the user's independent lifestyle.

[0038] Main components

[0039] 1. Terminal

[0040] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0041] The terminal is responsible for transmitting this data to the server.

[0042] 2. Server

[0043] It receives data sent from the terminal and analyzes it in real time.

[0044] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0045] 3. Users

[0046] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0047] Explanation of program processing

[0048] Environmental sensor data collection

[0049] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time, as well as camera footage, and transmits this data to a server.

[0050] Real-time data analysis

[0051] The server receives the data sent from the device and analyzes it in real time. Specifically, it monitors the user's heart rate and fluctuations in walking pace to determine if there are any abnormalities. At the same time, it analyzes the camera footage to detect unevenness and obstacles in the road.

[0052] Pacing and Guidance

[0053] Based on the analysis results, the server generates music and rhythms to stabilize the user's walking pace. This is provided to the user via the device, and audio warnings and guidance are also provided. For example, if there is a step ahead of the user, the server will say, "There is a step ahead. Please be careful."

[0054] Personalized Coaching

[0055] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice to "take a short break." It will also recommend exercises to prevent falls and provide audio instructions on how to do them.

[0056] Emergency response

[0057] If the device detects that the user has fallen, it will assess the situation based on data from the acceleration sensor and issue an emergency alert. At this time, the device will sound a loud alarm to alert those around it and simultaneously send information about the location of the fall and video to the server. The server then automatically sends a message containing the location and video of the fall to pre-registered family members and emergency contacts to ensure the user's safety.

[0058] Specific examples

[0059] For example, imagine a scenario where a senior user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking pace, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0060] In this way, the system provides multifunctional support to help users go out safely and reduce the risk of falling.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The device collects the user's vital data, walking pace, and rhythm in real time, including biometric information such as heart rate, blood pressure, and body temperature, and also collects camera footage of the user while walking.

[0064] Step 2:

[0065] The device sends the collected data (vital data, walking pace, camera footage, etc.) to a server. The data is sent to the server in real time using a secure communication protocol.

[0066] Step 3:

[0067] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles.

[0068] Step 4:

[0069] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, the server may communicate voice guidance such as "There is a step ahead. Be careful" to the user via the device.

[0070] Step 5:

[0071] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It will also recommend exercises to prevent falls and provide audio instructions.

[0072] Step 6:

[0073] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0074] Step 7:

[0075] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0076] Step 8:

[0077] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0078] Step 9:

[0079] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as a phone call or notifying medical facilities.

[0080] This series of processes allows the user to enjoy going out with peace of mind and effectively reduces the risk of falling.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] To provide an environment where elderly people can go out safely, a system that monitors their vital signs and walking pace in real time and provides appropriate guidance and alerts is needed. However, existing technologies have difficulty effectively collecting and analyzing this data and providing appropriate feedback. Another issue is that emergency response is insufficient.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for collecting the user's biometric information, walking speed, and walking pattern in real time, means for analyzing video and detecting road undulations and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting the user's sudden movements or falls, immediately issuing an alert, and transmitting fall position information and video to the server, and means for the server to send an emergency alert message. This provides an environment where elderly people can go out with peace of mind, reduces the risk of falls, and enables rapid response in emergencies.

[0086] A "user" is a person who uses the system and whose vital data, walking speed, walking pattern, etc. are monitored.

[0087] "Biometric information" is data indicating the user's health condition, such as heart rate, blood pressure, and body temperature.

[0088] "Walking speed" refers to the speed at which a user moves, and is an index of the distance traveled within a certain period of time.

[0089] The "walking pattern" refers to the rhythm and repetition of movements when the user walks, and is an index for detecting movements that differ from normal movements.

[0090] "Means for collecting in real time" refers to a device or method for instantly collecting a user's biometric information, walking speed, and walking pattern and transmitting them to the system.

[0091] "Means for analyzing images" refers to technology that processes image data acquired by cameras and other devices to detect road irregularities and obstacles.

[0092] "Means for providing audio warnings and instructions" refers to a device or method that uses audio to alert a user to safety concerns or encourage appropriate behavior.

[0093] "Means for detecting sudden movements and falls" refers to sensors and algorithms that immediately detect abnormalities in the user's movements or falls and notify the system.

[0094] "Means for issuing an alert" refers to a device or method for notifying the user and surrounding people of an abnormality by voice or visual means.

[0095] The "means for transmitting fall position information and video" refers to a device or method for transmitting position information and video of the scene of a fall to a server when a user falls.

[0096] "Means for sending an emergency alert message" refers to a device or method that sends information to pre-set contacts in the event of an emergency such as a fall, encouraging a prompt response.

[0097] This invention is a walking support system that provides an environment where elderly people can go out with peace of mind. Specifically, this system collects and analyzes the user's biometric information, walking speed, and walking pattern in real time, and can provide appropriate audio guidance and emergency response. This system mainly consists of three components: a terminal, a server, and a user.

[0098] Terminal

[0099] The terminal is a wearable device worn by the user. This terminal collects biometric information such as heart rate, blood pressure, and body temperature, as well as walking speed and walking pattern in real time. It also collects images of the surrounding area using a built-in camera and transmits this data to a server via Bluetooth or Wi-Fi.

[0100] The terminal used is, for example, a wearable device from Garmin, which has the ability to measure heart rate and blood pressure data every second and record video at 30 frames per second.

[0101] server

[0102] The server receives data sent from the device and analyzes it in real time. Specifically, it uses a Python script to monitor fluctuations in heart rate and walking speed and check for any abnormalities. This uses data analysis libraries such as numpy and pandas. It also uses the OpenCV library to analyze camera footage and detect road undulations and obstacles.

[0103] Based on the results of the analysis, Text-to-Speech (TTS) software is used to generate voice guidance such as "Please slow down a bit" and provide this to the user via the device.

[0104] Emergency response

[0105] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, it immediately emits a loud alert and simultaneously sends the location of the fall and camera footage to a server. The server then uses services such as Twilio and AWS SNS to automatically send an emergency alert message to pre-defined contacts to ensure safety.

[0106] Examples of specific examples and prompts

[0107] For example, imagine a scenario where an elderly user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking speed, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0108] Based on a concrete example of what this system achieves, the following is an example of a prompt sentence to input to the generative AI model:

[0109] Imagine situations where senior citizen A is at high risk of falling while walking, and generate real-time warnings. For example, if there is a step ahead, the voice will instruct, "There is a step ahead. Be careful." And if the heart rate rises, the voice will guide, "Take a short break."

[0110] In this way, the present invention provides a system that allows users to go out with peace of mind, reduces the risk of falling, and enables quick response in emergencies.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1: Collecting Environmental Sensor Data

[0113] The device collects the user's biometric information (heart rate, blood pressure, body temperature, etc.), walking speed, and walking patterns in real time. Specifically, the device measures heart rate and blood pressure data every second and records video at 30 frames per second using the built-in camera. This data is sent to a server via Bluetooth or Wi-Fi.

[0114] Input: User's biometric information, walking speed, camera footage

[0115] Output: Real-time data sent to the server

[0116] Step 2: Real-time analysis of data

[0117] The server receives the data sent from the device and analyzes it in real time. It uses a Python script to monitor fluctuations in heart rate and walking speed to check for any abnormalities. It also uses the OpenCV library to analyze camera footage and identify road undulations and obstacles.

[0118] Input: Real-time data sent from the device (biometric information, walking speed, camera footage)

[0119] Output: Anomaly detection results, road undulations and obstacles identification results

[0120] Step 3: Generate audio guidance

[0121] Based on the analysis results, the server generates voice guidance and musical rhythms to stabilize the user's walking pace. Using Text-to-Speech (TTS) software, it generates voice guidance such as "Please slow down a bit" and provides this to the user via their device.

[0122] Input: Analysis results (user's walking pace, road undulations and obstacle identification results)

[0123] Output: Voice guidance and musical rhythm

[0124] Step 4: Personalized Coaching

[0125] Based on the analysis of the user's biometric information and walking speed, the server will suggest rest periods and provide audio guidance on exercises to prevent falls. The server will analyze the user's past data using a machine learning algorithm to provide optimal advice. For example, it may suggest, "Your heart rate has increased, so please take a short break."

[0126] Input: User vital data and historical data

[0127] Output: Audio suggestions for rest and exercise

[0128] Step 5: Emergency response

[0129] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, the device immediately emits a loud alert and simultaneously transmits the location of the fall and camera footage to a server. The server then receives this information and automatically sends an emergency alert message to pre-defined contacts using services such as Twilio or AWS SNS.

[0130] Input: Acceleration sensor data, fall location information, camera footage

[0131] Output: Loud alert, emergency alert message

[0132] (Application example 1)

[0133] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] Conventional automated driving systems do not adequately address the need to ensure the safety of elderly drivers. In particular, they lack features such as real-time monitoring of the driver's biometric information and driving pace, emergency detection, automatic stopping, and notification to emergency contacts. As a result, elderly people find it difficult to use automated vehicles with confidence, and they remain anxious about traveling alone. The present invention aims to enable elderly people to use automated vehicles with confidence and significantly improve driving safety.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0136] In this invention, the server includes means for collecting the user's biometric information, driving pace, and rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting an emergency and automatically stopping the vehicle, means for automatically notifying emergency contacts, means for adjusting the vehicle's driving pace based on the driver's vital data, and means for understanding the situation around the vehicle based on video analysis. This allows even elderly people to use self-driving vehicles with peace of mind, and significantly improves safety while driving.

[0137] "User" refers to a driver who uses the system or an individual for whom the system is provided.

[0138] "Biometric information" refers to data that shows the driver's health condition in real time, such as heart rate, blood pressure, and body temperature.

[0139] "Driving pace" refers to fluctuations in the speed and rhythm of a driver's vehicle operation.

[0140] "Rhythm" refers to the regularity or periodicity of a driver's driving behavior.

[0141] "Camera footage" refers to video data captured by a camera used to monitor the inside and outside of a vehicle.

[0142] "Analysis" refers to the process of processing collected data to extract useful information.

[0143] "Road irregularities and obstacles" refers to physical undulations and objects that obstruct movement in front of the driver.

[0144] "Warning" refers to the act of notifying drivers of danger.

[0145] "Guidance" refers to instructions to encourage drivers to drive safely and behave appropriately.

[0146] An "emergency situation" refers to a situation that threatens the safety of the driver or vehicle.

[0147] "Automatic stop" means that the system stops the vehicle's operation without driver intervention.

[0148] "Emergency contacts" refer to the people you should contact first in the event of an emergency.

[0149] "Notification" refers to the act of a system communicating specific information to a designated recipient.

[0150] "Driving pace adjustment" refers to automatically changing the vehicle's speed and operation based on the driver's biometric information.

[0151] "Surrounding conditions" refers to all physical conditions and events inside and outside the vehicle and in the driving environment.

[0152] "Means" refers to devices or methods used to achieve a particular purpose.

[0153] The present invention provides an "automated driving vehicle support system for seniors" that enables elderly people to use automated driving vehicles safely and with peace of mind. Specific embodiments will be described below.

[0154] Hardware Configuration

[0155] 1. Wearable devices: fitness bands and smartwatches worn by seniors that collect real-time biometric information such as heart rate, blood pressure, and body temperature.

[0156] 2. Cameras inside and outside the vehicle: These cameras are used to monitor the inside and outside of the vehicle and collect video data.

[0157] 3. Smartphone or tablet: A device for data processing.

[0158] 4. Server: A central system for analyzing and processing data.

[0159] Software Configuration

[0160] 1. Biometric monitoring system: Software for collecting and analyzing driver vital data, such as Apple HealthKit or Google Fit API.

[0161] 2. Video analysis system: Using OpenCV, the system analyzes camera images from inside and outside the vehicle to detect road irregularities and obstacles.

[0162] 3. Voice Guidance System: Uses Google Text-to-Speech API and Amazon Polly to provide voice guidance, reminders, and pacing.

[0163] 4. Emergency Response System: Uses the Twilio API to automatically notify emergency contacts in the event of an emergency.

[0164] Processing flow

[0165] 1. Data Collection:

[0166] The wearable device collects the user's biometric information in real time and transmits it to a server.

[0167] Cameras inside and outside the vehicle collect video data in real time, which is also sent to a server.

[0168] 2. Data Analysis:

[0169] The server analyzes the collected vital data and video data to monitor the driver's physical condition and the driving environment.

[0170] 3. Real-time support:

[0171] It adjusts the driving pace and provides voice guidance to encourage the driver to take appropriate driving actions.

[0172] In the event of an emergency, the system will automatically shut down and notify emergency contacts.

[0173] Specific examples

[0174] For example, consider a scenario in which elderly person B goes out in a self-driving vehicle. B is wearing a wearable device, and an in-car camera monitors footage while driving, collecting and analyzing fluctuations in his or her physical condition. If B's ​​heart rate rises significantly while driving, a voice prompt will automatically advise him or her, saying, "Try not to push yourself, and take a break." Furthermore, if the external camera detects a danger ahead, a voice prompt will warn him or her, saying, "There is an obstacle ahead. Be careful."

[0175] Prompt Sentence Examples

[0176] "We are considering a system that will enable elderly people to safely use self-driving vehicles, by collecting biometric information in real time and analyzing driving behavior. Specifically, the system will monitor heart rate and driving pace, and issue audio warnings if it detects danger based on video analysis. What kind of functions and software should we use?"

[0177] Based on the above description, the present invention provides specific technical means for improving the safety of elderly people when using self-driving vehicles.

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] Data collection:

[0181] The terminal (wearable device) collects the user's biometric information (heart rate, blood pressure, body temperature, etc.) in real time and sends it to a server. In addition, cameras inside and outside the vehicle collect video data in real time and send it to the server.

[0182] Input: User vital data, camera footage from inside and outside the vehicle.

[0183] Output: Biometric information and video data sent to the server.

[0184] Step 2:

[0185] Data Analysis:

[0186] The server analyzes the received biometric information and video data to monitor the driver's physical condition and the driving environment, including analyzing fluctuations in biometric information and analyzing the road environment using image processing technology.

[0187] Input: Biometric information and video data sent to the server.

[0188] Output: Analyzed user's physical condition and the situation around the vehicle.

[0189] Step 3:

[0190] Real-time guidance provided:

[0191] The server provides voice guidance to the driver based on the analysis results. For example, if the driver's heart rate rises, a message is sent urging them to take a break. If there is an obstacle ahead, a voice message will be sent saying, "There is an obstacle ahead. Be careful."

[0192] Input: Analyzed physical condition and vehicle surroundings.

[0193] Output: Voice guidance (break suggestions and obstacle notifications).

[0194] Step 4:

[0195] Pace your driving:

[0196] The server adjusts the vehicle's driving pace based on the user's vital data, taking measures such as slowing down if the user's heart rate is high.

[0197] Input: User's biometric information.

[0198] Output: Adjusted driving pace instructions.

[0199] Step 5:

[0200] Emergency automatic stop:

[0201] If the server detects an emergency (for example, a sudden change in the driver's health), it issues an instruction to automatically stop the vehicle.

[0202] Input: Detection of sudden changes in vital signs.

[0203] Output: Auto-stop command for vehicle.

[0204] Step 6:

[0205] Emergency notification:

[0206] The server automatically notifies the user's emergency contacts when an emergency shutdown occurs, including details about the user's location and current status.

[0207] Input: Detection of sudden changes in vital signs, location information.

[0208] Output: Notification message to emergency contacts.

[0209] Through these steps, the system will operate to enable elderly people to safely use self-driving vehicles.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] This invention combines an emotion engine with a walking support system for seniors to recognize the user's emotional state in real time and provide pacemaking, warnings, and guidance based on this information, allowing users to enjoy going out with greater peace of mind and reducing the risk of falling.

[0212] Main components

[0213] 1. Terminal

[0214] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0215] The device is also equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state.

[0216] The terminal is responsible for transmitting this data to the server.

[0217] 2. Server

[0218] It receives data sent from the terminal and analyzes it in real time.

[0219] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0220] Based on the recognition results of the emotion engine, advice and warnings are given taking into account the emotional state.

[0221] 3. Users

[0222] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0223] Explanation of program processing

[0224] Environmental sensor data collection

[0225] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. In addition, the device analyzes the camera footage and the user's facial expressions and voice to recognize their emotional state. This data is then sent to a server.

[0226] Real-time data analysis

[0227] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles. It also analyzes the results of the emotion engine to understand the user's emotional state.

[0228] Pacing and Guidance

[0229] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, it will say, "There is a step ahead. Please be careful." In addition, based on the emotional state obtained from the emotion engine, if the user is feeling anxious, it will provide guidance in a soft voice, saying, "Please walk in a relaxed manner."

[0230] Personalized Coaching

[0231] The server suggests to the user to take a break or slow down based on the analysis of the user's vital data and walking pace. For example, if the user's heart rate rises, the server will prompt the user by voice, "Take a short break." It also provides advice based on the user's emotional state, such as "Try taking a deep breath to refresh yourself." Furthermore, the server will recommend exercises to prevent falls and provide audio instructions on how to do them.

[0232] Emergency response

[0233] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0234] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0235] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0236] Specific examples

[0237] For example, imagine a scenario in which a senior user, Person B, goes out for a walk. Person B is wearing a wearable device, which collects his / her heart rate, walking pace, camera footage, and emotional state in real time from facial expressions and voice. The server receives this data, and if it determines that Person B is walking too fast, it suggests that he / she slow down. This suggestion is made through voice, saying, "Slow down a bit." If the server also analyzes the camera footage and detects that there is a step ahead of Person B, it warns him / her with a voice saying, "There is a step ahead. Please be careful." On the other hand, if it recognizes that Person B is anxious from his / her facial expression, it provides guidance in a soft voice saying, "Please walk relaxedly."

[0238] In this way, this system provides multifunctional support that takes into account the user's emotional state, creating an environment in which users can go out with greater peace of mind.

[0239] The processing flow will be explained below.

[0240] Step 1:

[0241] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. The device also collects camera footage and the user's facial expressions and voice, and analyzes these with an emotion engine to recognize the user's emotional state.

[0242] Step 2:

[0243] The device sends the collected data (vital data, walking pace, rhythm, camera footage, and emotional data) to a server. The data is sent securely and in real time using a communication protocol.

[0244] Step 3:

[0245] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. At the same time, it analyzes the camera footage to detect road irregularities and obstacles, and recognizes the user's emotional state based on the results of the emotion engine.

[0246] Step 4:

[0247] The server will pace itself, taking into account the user's emotional state. For example, if the user feels anxious, it will provide relaxing music and rhythms and give a voice guidance saying, "Walk in a relaxed manner." Under normal circumstances, it will provide appropriate music and rhythms based on the analysis results.

[0248] Step 5:

[0249] The server will then warn the user based on the results of the camera image analysis. For example, if there is a step ahead, the server will say, "There is a step ahead. Please be careful." If the user's emotional state is unstable, the server will give additional advice such as, "Walk slowly."

[0250] Step 6:

[0251] The server analyzes the user's vital data and walking pace and suggests that the user take a break or slow down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It also takes into account the user's emotional state and provides advice on how to change their mood (e.g., "Try taking a deep breath").

[0252] Step 7:

[0253] The device constantly monitors the user's fall using data from the accelerometer and gyroscope, and if it detects any signs of a possible fall, it will sound a loud noise to alert those around it.

[0254] Step 8:

[0255] When the device detects a fall, it immediately sends the location of the fall and camera footage to the server, allowing the server to accurately grasp the situation.

[0256] Step 9:

[0257] Based on the information received by the server, a message containing location information and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing for prompt assistance.

[0258] Step 10:

[0259] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as notifying medical facilities or making additional contact.

[0260] This series of processes allows users to receive personalized support that takes into account their emotional state, allowing them to enjoy going out with peace of mind. It also effectively reduces the risk of falling.

[0261] Example 2

[0262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0263] To enable elderly people to go out and walk safely and with peace of mind, it is important to monitor not only vital signs and walking status but also emotional status in real time and provide appropriate guidance and warnings. However, current walking support systems do not adequately recognize emotional status or respond dynamically based on it. A system that can quickly respond to the risk of falls is also needed.

[0264] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0265] In this invention, the server includes means for collecting the user's biometric data, walking speed, and movement rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, and means for detecting the user's falls. This makes it possible to comprehensively monitor the elderly person's emotional and physical condition and provide appropriate guidance and warnings in real time.

[0266] "Biometric data" refers to data that indicates a person's physical condition, such as heart rate, blood pressure, and body temperature.

[0267] "Walking speed" refers to the speed at which the user walks, and is data indicating the walking pace.

[0268] "Movement rhythm" refers to the periodic pattern of the user's movement while walking, and is data that indicates the tempo and rhythm of walking.

[0269] "Camera images" refer to visual data acquired by a video recording device, and are data for visually capturing the surrounding situation.

[0270] "Emotional state" refers to the user's state of mind or feelings, including emotions such as joy, sadness, and anxiety.

[0271] "Warning" refers to warning or calling for caution to the user, and is a way to avoid danger.

[0272] "Guidance" means providing appropriate instructions and directions to the user.

[0273] The "means for detecting falls" is part of the system for monitoring a user's falls in real time and detecting them immediately when they occur.

[0274] "Means of collecting data in real time" refers to a method for obtaining data immediately without delay and providing it to the system.

[0275] "Means of analysis" are methods for analyzing collected data and converting it into meaningful information.

[0276] This invention is a system for supporting elderly people to go out and walk safely and with peace of mind. This system has the function of collecting the user's biometric data, walking speed, and movement rhythm in real time, and providing appropriate guidance and warnings based on that data.

[0277] The system consists of the following main components: The terminal is a wearable device worn by the user that collects heart rate, walking speed, rhythm, and camera images in real time. Furthermore, the terminal is equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state. This data is sequentially sent to the server.

[0278] The server receives data sent from the device and analyzes it in real time. Specific analysis includes algorithms for checking fluctuations in the user's heart rate and walking speed, image recognition technology for detecting road irregularities and obstacles from camera images, and an integrated analysis system that comprehensively evaluates the emotion engine's analysis results. Based on the analysis results, the server generates voice guidance and musical rhythms and provides them to the user via the device at the appropriate time.

[0279] In addition, for emergency response, the device uses an accelerometer and gyroscope to monitor the user's falls in real time. If it detects a movement that could indicate a fall, it will sound a loud alert and send the location of the fall and camera images taken at the time to a server. The server also has the function of automatically sending notifications to emergency contacts based on the information it receives.

[0280] For example, when an elderly user, Person A, goes out for a walk, Person A is wearing a wearable device. This device instantly collects Person A's heart rate, walking speed, camera images, facial expressions, and voice, and sends them to the server in real time. The server analyzes this data, and if it determines that Person A is walking too fast, it provides voice guidance such as "Slow down a bit." If the server detects a step ahead from the camera image, it warns, "There is a step ahead. Be careful." Furthermore, if it detects that Person A has an anxious expression, it sends guidance that takes emotions into consideration, such as "Walk relaxedly."

[0281] Below is an example of a prompt sentence.

[0282] "Generate a program that collects the user's biometric data and analyzes their walking speed."

[0283] "How can I detect a fall using an accelerometer and gyroscope?"

[0284] "Explain how you can use an emotion engine to analyze emotional states and provide dynamic guidance."

[0285] In this way, the system can provide multifunctional walking assistance that supports the user's safety and security.

[0286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0287] Step 1:

[0288] The device collects the user's biometric data, walking speed, and movement rhythm in real time.

[0289] Input: Heart rate, blood pressure, body temperature, walking speed, and rhythm data obtained from the user's body.

[0290] Processing: These data are stored and recorded using sensors built into the device. Biometric data is acquired every second, and rhythm is quantified by analyzing walking tempo.

[0291] Output: Collected biometric data, walking speed, and rhythm data are sent to the server.

[0292] Step 2:

[0293] The device captures camera images and analyzes the user's facial expressions and voice.

[0294] Input: Video data input to the device's camera, user's voice data.

[0295] Processing: Using video recognition technology and voice recognition algorithms, the device analyzes the user's emotional state from their facial expressions and voice. The device's emotion engine identifies emotions such as smile, anxiety, and anger.

[0296] Output: Send the parsed emotional state data to the server.

[0297] Step 3:

[0298] The server receives the biometric data, walking speed, rhythm data, and emotional state data sent from the terminal and analyzes them in real time.

[0299] Input: Biometric data, walking speed, rhythm data, and emotional state data obtained from the device.

[0300] Processing: Analyzes data fluctuations over time to determine whether they are within the normal range. For example, if your heart rate rises sharply, it is determined that you are under exercise stress. This data is then comprehensively analyzed using an AI model.

[0301] Output: The analysis results are sent to the voice guidance generation unit.

[0302] Step 4:

[0303] Based on the analysis results, the server generates pacemaking and guidance for the user.

[0304] Input: The analyzed data and its evaluation results.

[0305] Processing: Taking into account the user's situation, appropriate music and voice guidance are created. For example, if it is determined that the walking speed is too fast, voice guidance such as "Please slow down a little" is created. If there is a step ahead, a warning such as "There is a step ahead. Be careful." If an anxious emotional state is detected from the emotion engine, guidance is generated in a soft voice such as "Please walk in a relaxed manner."

[0306] Output: Send the generated voice guidance to the device.

[0307] Step 5:

[0308] The terminal provides the user with the voice guidance received from the server.

[0309] Input: Voice guidance data sent from the server.

[0310] Processing: Play back audio guidance to the user through the device's built-in speaker.

[0311] Output: The user receives the audio prompt.

[0312] Step 6:

[0313] The device detects when the user falls and takes emergency action.

[0314] Input: Motion data from the accelerometer and gyroscope.

[0315] Processing: The system constantly monitors the user's movements and posture, and applies an algorithm to detect potential falls. If detected, it issues a loud alert and sends the user's location and camera images to a server.

[0316] Output: The location information of the fall and the camera image are sent to the server, and an alert is issued to those in the vicinity.

[0317] Step 7:

[0318] The server receives the fall information and sends a notification to emergency contacts.

[0319] Input: Fall location information and camera images sent from the device.

[0320] Processing: The received fall information is analyzed and a message is automatically sent to pre-registered emergency contacts. The message includes the location and video of the fall.

[0321] Output: Send a notification to emergency contacts with information about the fall.

[0322] (Application example 2)

[0323] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] In current food delivery services, staff are often at risk of accidents and mistakes due to excessive stress, fatigue, and an unstable walking pace. A system is needed that can monitor the mental and physical state of employees in real time and provide appropriate guidance.

[0325] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's vital signs, walking pace, and rhythm in real time, means for analyzing camera images and detecting road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for recognizing the emotional state in real time and providing pacemaking and warnings based thereon, and means for detecting falls and obtaining and notifying location information. This makes it possible to monitor the stress and fatigue felt by staff during work in real time and encourage breaks and give warnings at appropriate times.

[0326] "Vital data" refers to data that indicates a user's vital signs, such as heart rate, blood pressure, and body temperature, in real time.

[0327] "Walking pace" is data on the speed and rhythm when the user walks.

[0328] "Camera footage" refers to real-time visual information acquired by the camera of a wearable device.

[0329] An "emotional state" is a state related to a user's feelings such as stress, anxiety, happiness, etc.

[0330] "Pacemaking" refers to adjusting the pace in real time so that the user can continue walking safely and efficiently.

[0331] "Attention" refers to alerting the user by audio or visual information.

[0332] "Guidance" refers to providing users with appropriate information and instructions to encourage safe and efficient behavior.

[0333] "Fall detection" means recognizing in real time that a user has fallen.

[0334] "Location information" is data indicating the current location of the user.

[0335] "Notification" means sending important information to the user and people around them.

[0336] This invention provides a wearable assistance system that helps food delivery staff perform their work more safely and efficiently. The wearable device worn by the delivery staff collects vital signs, walking pace and rhythm in real time, and analyzes camera footage. The device also has an emotion engine that can recognize the user's emotional state in real time.

[0337] The data collected by the wearable device is sent to a server, which analyzes the data in real time. In particular, the server analyzes vital signs, walking pace and rhythm, as well as emotional state recognition. This allows the server to provide appropriate pacing, alerts, and guidance to the user.

[0338] The hardware used includes wearable devices (e.g., smart glasses and head-mounted displays) equipped with heart rate sensors and cameras, and the software used includes HeartRateMonitor for collecting vital data, EmotionRecognizer for analyzing facial expressions, and VoiceAssistant for providing voice guidance.

[0339] Furthermore, the server also has the means to respond to cases such as when the user falls or other emergencies. For example, if the server detects a fall, it can immediately obtain the user's location information and notify pre-registered emergency contacts, which allows for a rapid response.

[0340] For example, if a delivery staff member feels stressed while walking, EmotionRecognizer will recognize this and the server will provide guidance to relax. Also, if the heart rate rises, the server can provide specific voice guidance such as "Your heart rate is high. Please take a short break."

[0341] Example prompt sentence:

[0342] "Develop a system that recognizes the emotional state of food delivery staff in real time and provides pacemaking, warnings, and guidance based on that. Emotional states include stress, anxiety, and fatigue. Heart rate data will also be used to prompt staff to take a break if fatigue is detected. Hardware to be used includes a wearable device, heart rate sensor, and camera, and software libraries include OpenCV and a voice assistant library."

[0343] In this way, the present invention is a system that provides an environment in which food delivery staff can work with peace of mind.

[0344] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0345] Step 1:

[0346] The user wears a wearable device, which collects the user's vital data (e.g., heart rate), walking pace, rhythm, and camera footage in real time. The input here is raw data obtained from the sensors, and the output is the collection of these data.

[0347] Step 2:

[0348] The device sends the collected data to the server. At this point, the input is the vital signs, walking pace, rhythm, and camera footage collected by the device, and the output is this data sent to the server.

[0349] Step 3:

[0350] The server analyzes the received data. First, it analyzes the vital data and determines whether the heart rate and blood pressure are within normal ranges. The input is the vital data received by the server, and the output is the analysis result regarding the user's health condition.

[0351] Step 4:

[0352] The server analyzes the walking pace and rhythm and determines whether the user is walking safely and efficiently. The input is the walking pace and rhythm data, and the output is the analysis result.

[0353] Step 5:

[0354] The server analyzes the camera images and detects road irregularities and obstacles. The input is the camera images, and the output is the analysis results on the road conditions.

[0355] Step 6:

[0356] The device's emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The input is real-time facial and voice data from the wearable device, and the output is the analysis result of the user's emotional state.

[0357] Step 7:

[0358] The server determines appropriate guidance for the user based on vital data, walking pace, rhythm, analysis of camera footage, and recognition of emotional state. For example, if the heart rate is high or there is an obstacle on the road, the server generates appropriate voice guidance. The input is the results of various analyses, and the output is the guidance content provided to the user.

[0359] Step 8:

[0360] The terminal conveys the guidance content provided by the server to the user as voice. Voice guidance includes content such as "Please slow down a little" or "There is a step ahead. Please be careful." The input is the guidance content generated by the server, and the output is voice guidance.

[0361] Step 9:

[0362] The device constantly monitors the data from the accelerometer and gyroscope to detect falls. The input is real-time data from these sensors, and the output is the fall detection result.

[0363] Step 10:

[0364] When the device detects a fall, it immediately sends location information and camera footage to the server. The input is the fall detection result, location information, and camera footage, and the output is the transmission of this information to the server.

[0365] Step 11:

[0366] The server sends a notification to the emergency contact based on the received fall information. The input here is the fall information and the emergency contact information, and the output is a notification message.

[0367] This series of processes enables food delivery staff to carry out their duties safely and efficiently, and enables quick response in emergencies.

[0368] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0370] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0371] [Second embodiment]

[0372] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0373] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0375] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0379] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0380] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0381] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0382] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0383] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0384] This invention is a walking assistance system for seniors, designed to allow users to go out with peace of mind. The system collects the user's vital signs and walking pace in real time, and analyzes camera footage to determine the surrounding situation. Furthermore, by providing appropriate audio warnings and guidance, it reduces the risk of falls and supports the user's independent lifestyle.

[0385] Main components

[0386] 1. Terminal

[0387] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0388] The terminal is responsible for transmitting this data to the server.

[0389] 2. Server

[0390] It receives data sent from the terminal and analyzes it in real time.

[0391] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0392] 3. Users

[0393] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0394] Explanation of program processing

[0395] Environmental sensor data collection

[0396] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time, as well as camera footage, and transmits this data to a server.

[0397] Real-time data analysis

[0398] The server receives the data sent from the device and analyzes it in real time. Specifically, it monitors the user's heart rate and fluctuations in walking pace to determine if there are any abnormalities. At the same time, it analyzes the camera footage to detect unevenness and obstacles in the road.

[0399] Pacing and Guidance

[0400] Based on the analysis results, the server generates music and rhythms to stabilize the user's walking pace. This is provided to the user via the device, and audio warnings and guidance are also provided. For example, if there is a step ahead of the user, the server will say, "There is a step ahead. Please be careful."

[0401] Personalized Coaching

[0402] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice to "take a short break." It will also recommend exercises to prevent falls and provide audio instructions on how to do them.

[0403] Emergency response

[0404] If the device detects that the user has fallen, it will assess the situation based on data from the acceleration sensor and issue an emergency alert. At this time, the device will sound a loud alarm to alert those around it and simultaneously send information about the location of the fall and video to the server. The server then automatically sends a message containing the location and video of the fall to pre-registered family members and emergency contacts to ensure the user's safety.

[0405] Specific examples

[0406] For example, imagine a scenario where a senior user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking pace, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0407] In this way, the system provides multifunctional support to help users go out safely and reduce the risk of falling.

[0408] The processing flow will be explained below.

[0409] Step 1:

[0410] The device collects the user's vital data, walking pace, and rhythm in real time, including biometric information such as heart rate, blood pressure, and body temperature, and also collects camera footage of the user while walking.

[0411] Step 2:

[0412] The device sends the collected data (vital data, walking pace, camera footage, etc.) to a server. The data is sent to the server in real time using a secure communication protocol.

[0413] Step 3:

[0414] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles.

[0415] Step 4:

[0416] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, the server may communicate voice guidance such as "There is a step ahead. Be careful" to the user via the device.

[0417] Step 5:

[0418] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It will also recommend exercises to prevent falls and provide audio instructions.

[0419] Step 6:

[0420] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0421] Step 7:

[0422] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0423] Step 8:

[0424] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0425] Step 9:

[0426] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as a phone call or notifying medical facilities.

[0427] This series of processes allows the user to enjoy going out with peace of mind and effectively reduces the risk of falling.

[0428] Example 1

[0429] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0430] To provide an environment where elderly people can go out safely, a system that monitors their vital signs and walking pace in real time and provides appropriate guidance and alerts is needed. However, existing technologies have difficulty effectively collecting and analyzing this data and providing appropriate feedback. Another issue is that emergency response is insufficient.

[0431] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0432] In this invention, the server includes means for collecting the user's biometric information, walking speed, and walking pattern in real time, means for analyzing video and detecting road undulations and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting the user's sudden movements or falls, immediately issuing an alert, and transmitting fall position information and video to the server, and means for the server to send an emergency alert message. This provides an environment where elderly people can go out with peace of mind, reduces the risk of falls, and enables rapid response in emergencies.

[0433] A "user" is a person who uses the system and whose vital data, walking speed, walking pattern, etc. are monitored.

[0434] "Biometric information" is data indicating the user's health condition, such as heart rate, blood pressure, and body temperature.

[0435] "Walking speed" refers to the speed at which a user moves, and is an index of the distance traveled within a certain period of time.

[0436] The "walking pattern" refers to the rhythm and repetition of movements when the user walks, and is an index for detecting movements that differ from normal movements.

[0437] "Means for collecting in real time" refers to a device or method for instantly collecting a user's biometric information, walking speed, and walking pattern and transmitting them to the system.

[0438] "Means for analyzing images" refers to technology that processes image data acquired by cameras and other devices to detect road irregularities and obstacles.

[0439] "Means for providing audio warnings and instructions" refers to a device or method that uses audio to alert a user to safety concerns or encourage appropriate behavior.

[0440] "Means for detecting sudden movements and falls" refers to sensors and algorithms that immediately detect abnormalities in the user's movements or falls and notify the system.

[0441] "Means for issuing an alert" refers to a device or method for notifying the user and surrounding people of an abnormality by voice or visual means.

[0442] The "means for transmitting fall position information and video" refers to a device or method for transmitting position information and video of the scene of a fall to a server when a user falls.

[0443] "Means for sending an emergency alert message" refers to a device or method that sends information to pre-set contacts in the event of an emergency such as a fall, encouraging a prompt response.

[0444] This invention is a walking support system that provides an environment where elderly people can go out with peace of mind. Specifically, this system collects and analyzes the user's biometric information, walking speed, and walking pattern in real time, and can provide appropriate audio guidance and emergency response. This system mainly consists of three components: a terminal, a server, and a user.

[0445] Terminal

[0446] The terminal is a wearable device worn by the user. This terminal collects biometric information such as heart rate, blood pressure, and body temperature, as well as walking speed and walking pattern in real time. It also collects images of the surrounding area using a built-in camera and transmits this data to a server via Bluetooth or Wi-Fi.

[0447] The terminal used is, for example, a wearable device from Garmin, which has the ability to measure heart rate and blood pressure data every second and record video at 30 frames per second.

[0448] server

[0449] The server receives data sent from the device and analyzes it in real time. Specifically, it uses a Python script to monitor fluctuations in heart rate and walking speed and check for any abnormalities. This uses data analysis libraries such as numpy and pandas. It also uses the OpenCV library to analyze camera footage and detect road undulations and obstacles.

[0450] Based on the results of the analysis, Text-to-Speech (TTS) software is used to generate voice guidance such as "Please slow down a bit" and provide this to the user via the device.

[0451] Emergency response

[0452] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, it immediately emits a loud alert and simultaneously sends the location of the fall and camera footage to a server. The server then uses services such as Twilio and AWS SNS to automatically send an emergency alert message to pre-defined contacts to ensure safety.

[0453] Examples of specific examples and prompts

[0454] For example, imagine a scenario where an elderly user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking speed, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0455] Based on a concrete example of what this system achieves, the following is an example of a prompt sentence to input to the generative AI model:

[0456] Imagine situations where senior citizen A is at high risk of falling while walking, and generate real-time warnings. For example, if there is a step ahead, the voice will instruct, "There is a step ahead. Be careful." And if the heart rate rises, the voice will guide, "Take a short break."

[0457] In this way, the present invention provides a system that allows users to go out with peace of mind, reduces the risk of falling, and enables quick response in emergencies.

[0458] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0459] Step 1: Collecting Environmental Sensor Data

[0460] The device collects the user's biometric information (heart rate, blood pressure, body temperature, etc.), walking speed, and walking patterns in real time. Specifically, the device measures heart rate and blood pressure data every second and records video at 30 frames per second using the built-in camera. This data is sent to a server via Bluetooth or Wi-Fi.

[0461] Input: User's biometric information, walking speed, camera footage

[0462] Output: Real-time data sent to the server

[0463] Step 2: Real-time analysis of data

[0464] The server receives the data sent from the device and analyzes it in real time. It uses a Python script to monitor fluctuations in heart rate and walking speed to check for any abnormalities. It also uses the OpenCV library to analyze camera footage and identify road undulations and obstacles.

[0465] Input: Real-time data sent from the device (biometric information, walking speed, camera footage)

[0466] Output: Anomaly detection results, road undulations and obstacles identification results

[0467] Step 3: Generate audio guidance

[0468] Based on the analysis results, the server generates voice guidance and musical rhythms to stabilize the user's walking pace. Using Text-to-Speech (TTS) software, it generates voice guidance such as "Please slow down a bit" and provides this to the user via their device.

[0469] Input: Analysis results (user's walking pace, road undulations and obstacle identification results)

[0470] Output: Voice guidance and musical rhythm

[0471] Step 4: Personalized Coaching

[0472] Based on the analysis of the user's biometric information and walking speed, the server will suggest rest periods and provide audio guidance on exercises to prevent falls. The server will analyze the user's past data using a machine learning algorithm to provide optimal advice. For example, it may suggest, "Your heart rate has increased, so please take a short break."

[0473] Input: User vital data and historical data

[0474] Output: Audio suggestions for rest and exercise

[0475] Step 5: Emergency response

[0476] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, the device immediately emits a loud alert and simultaneously transmits the location of the fall and camera footage to a server. The server then receives this information and automatically sends an emergency alert message to pre-defined contacts using services such as Twilio or AWS SNS.

[0477] Input: Acceleration sensor data, fall location information, camera footage

[0478] Output: Loud alert, emergency alert message

[0479] (Application example 1)

[0480] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0481] Conventional automated driving systems do not adequately address the need to ensure the safety of elderly drivers. In particular, they lack features such as real-time monitoring of the driver's biometric information and driving pace, emergency detection, automatic stopping, and notification to emergency contacts. As a result, elderly people find it difficult to use automated vehicles with confidence, and they remain anxious about traveling alone. The present invention aims to enable elderly people to use automated vehicles with confidence and significantly improve driving safety.

[0482] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0483] In this invention, the server includes means for collecting the user's biometric information, driving pace, and rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting an emergency and automatically stopping the vehicle, means for automatically notifying emergency contacts, means for adjusting the vehicle's driving pace based on the driver's vital data, and means for understanding the situation around the vehicle based on video analysis. This allows even elderly people to use self-driving vehicles with peace of mind, and significantly improves safety while driving.

[0484] "User" refers to a driver who uses the system or an individual for whom the system is provided.

[0485] "Biometric information" refers to data that shows the driver's health condition in real time, such as heart rate, blood pressure, and body temperature.

[0486] "Driving pace" refers to fluctuations in the speed and rhythm of a driver's vehicle operation.

[0487] "Rhythm" refers to the regularity or periodicity of a driver's driving behavior.

[0488] "Camera footage" refers to video data captured by a camera used to monitor the inside and outside of a vehicle.

[0489] "Analysis" refers to the process of processing collected data to extract useful information.

[0490] "Road irregularities and obstacles" refers to physical undulations and objects that obstruct movement in front of the driver.

[0491] "Warning" refers to the act of notifying drivers of danger.

[0492] "Guidance" refers to instructions to encourage drivers to drive safely and behave appropriately.

[0493] An "emergency situation" refers to a situation that threatens the safety of the driver or vehicle.

[0494] "Automatic stop" means that the system stops the vehicle's operation without driver intervention.

[0495] "Emergency contacts" refer to the people you should contact first in the event of an emergency.

[0496] "Notification" refers to the act of a system communicating specific information to a designated recipient.

[0497] "Driving pace adjustment" refers to automatically changing the vehicle's speed and operation based on the driver's biometric information.

[0498] "Surrounding conditions" refers to all physical conditions and events inside and outside the vehicle and in the driving environment.

[0499] "Means" refers to devices or methods used to achieve a particular purpose.

[0500] The present invention provides an "automated driving vehicle support system for seniors" that enables elderly people to use automated driving vehicles safely and with peace of mind. Specific embodiments will be described below.

[0501] Hardware Configuration

[0502] 1. Wearable devices: fitness bands and smartwatches worn by seniors that collect real-time biometric information such as heart rate, blood pressure, and body temperature.

[0503] 2. Cameras inside and outside the vehicle: These cameras are used to monitor the inside and outside of the vehicle and collect video data.

[0504] 3. Smartphone or tablet: A device for data processing.

[0505] 4. Server: A central system for analyzing and processing data.

[0506] Software Configuration

[0507] 1. Biometric monitoring system: Software for collecting and analyzing driver vital data, such as Apple HealthKit or Google Fit API.

[0508] 2. Video analysis system: Using OpenCV, the system analyzes camera images from inside and outside the vehicle to detect road irregularities and obstacles.

[0509] 3. Voice Guidance System: Uses Google Text-to-Speech API and Amazon Polly to provide voice guidance, reminders, and pacing.

[0510] 4. Emergency Response System: Uses the Twilio API to automatically notify emergency contacts in the event of an emergency.

[0511] Processing flow

[0512] 1. Data Collection:

[0513] The wearable device collects the user's biometric information in real time and transmits it to a server.

[0514] Cameras inside and outside the vehicle collect video data in real time, which is also sent to a server.

[0515] 2. Data Analysis:

[0516] The server analyzes the collected vital data and video data to monitor the driver's physical condition and the driving environment.

[0517] 3. Real-time support:

[0518] It adjusts the driving pace and provides voice guidance to encourage the driver to take appropriate driving actions.

[0519] In the event of an emergency, the system will automatically shut down and notify emergency contacts.

[0520] Specific examples

[0521] For example, consider a scenario in which elderly person B goes out in a self-driving vehicle. B is wearing a wearable device, and an in-car camera monitors footage while driving, collecting and analyzing fluctuations in his or her physical condition. If B's ​​heart rate rises significantly while driving, a voice prompt will automatically advise him or her, saying, "Try not to push yourself, and take a break." Furthermore, if the external camera detects a danger ahead, a voice prompt will warn him or her, saying, "There is an obstacle ahead. Be careful."

[0522] Prompt Sentence Examples

[0523] "We are considering a system that will enable elderly people to safely use self-driving vehicles, by collecting biometric information in real time and analyzing driving behavior. Specifically, the system will monitor heart rate and driving pace, and issue audio warnings if it detects danger based on video analysis. What kind of functions and software should we use?"

[0524] Based on the above description, the present invention provides specific technical means for improving the safety of elderly people when using self-driving vehicles.

[0525] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0526] Step 1:

[0527] Data collection:

[0528] The terminal (wearable device) collects the user's biometric information (heart rate, blood pressure, body temperature, etc.) in real time and sends it to a server. In addition, cameras inside and outside the vehicle collect video data in real time and send it to the server.

[0529] Input: User vital data, camera footage from inside and outside the vehicle.

[0530] Output: Biometric information and video data sent to the server.

[0531] Step 2:

[0532] Data Analysis:

[0533] The server analyzes the received biometric information and video data to monitor the driver's physical condition and the driving environment, including analyzing fluctuations in biometric information and analyzing the road environment using image processing technology.

[0534] Input: Biometric information and video data sent to the server.

[0535] Output: Analyzed user's physical condition and the situation around the vehicle.

[0536] Step 3:

[0537] Real-time guidance provided:

[0538] The server provides voice guidance to the driver based on the analysis results. For example, if the driver's heart rate rises, a message is sent urging them to take a break. If there is an obstacle ahead, a voice message will be sent saying, "There is an obstacle ahead. Be careful."

[0539] Input: Analyzed physical condition and vehicle surroundings.

[0540] Output: Voice guidance (break suggestions and obstacle notifications).

[0541] Step 4:

[0542] Pace your driving:

[0543] The server adjusts the vehicle's driving pace based on the user's vital data, taking measures such as slowing down if the user's heart rate is high.

[0544] Input: User's biometric information.

[0545] Output: Adjusted driving pace instructions.

[0546] Step 5:

[0547] Emergency automatic stop:

[0548] If the server detects an emergency (for example, a sudden change in the driver's health), it issues an instruction to automatically stop the vehicle.

[0549] Input: Detection of sudden changes in vital signs.

[0550] Output: Auto-stop command for vehicle.

[0551] Step 6:

[0552] Emergency notification:

[0553] The server automatically notifies the user's emergency contacts when an emergency shutdown occurs, including details about the user's location and current status.

[0554] Input: Detection of sudden changes in vital signs, location information.

[0555] Output: Notification message to emergency contacts.

[0556] Through these steps, the system will operate to enable elderly people to safely use self-driving vehicles.

[0557] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0558] This invention combines an emotion engine with a walking support system for seniors to recognize the user's emotional state in real time and provide pacemaking, warnings, and guidance based on this information, allowing users to enjoy going out with greater peace of mind and reducing the risk of falling.

[0559] Main components

[0560] 1. Terminal

[0561] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0562] The device is also equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state.

[0563] The terminal is responsible for transmitting this data to the server.

[0564] 2. Server

[0565] It receives data sent from the terminal and analyzes it in real time.

[0566] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0567] Based on the recognition results of the emotion engine, advice and warnings are given taking into account the emotional state.

[0568] 3. Users

[0569] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0570] Explanation of program processing

[0571] Environmental sensor data collection

[0572] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. In addition, the device analyzes the camera footage and the user's facial expressions and voice to recognize their emotional state. This data is then sent to a server.

[0573] Real-time data analysis

[0574] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles. It also analyzes the results of the emotion engine to understand the user's emotional state.

[0575] Pacing and Guidance

[0576] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, it will say, "There is a step ahead. Please be careful." In addition, based on the emotional state obtained from the emotion engine, if the user is feeling anxious, it will provide guidance in a soft voice, saying, "Please walk in a relaxed manner."

[0577] Personalized Coaching

[0578] The server suggests to the user to take a break or slow down based on the analysis of the user's vital data and walking pace. For example, if the user's heart rate rises, the server will prompt the user by voice, "Take a short break." It also provides advice based on the user's emotional state, such as "Try taking a deep breath to refresh yourself." Furthermore, the server will recommend exercises to prevent falls and provide audio instructions on how to do them.

[0579] Emergency response

[0580] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0581] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0582] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0583] Specific examples

[0584] For example, imagine a scenario in which a senior user, Person B, goes out for a walk. Person B is wearing a wearable device, which collects his / her heart rate, walking pace, camera footage, and emotional state in real time from facial expressions and voice. The server receives this data, and if it determines that Person B is walking too fast, it suggests that he / she slow down. This suggestion is made through voice, saying, "Slow down a bit." If the server also analyzes the camera footage and detects that there is a step ahead of Person B, it warns him / her with a voice saying, "There is a step ahead. Please be careful." On the other hand, if it recognizes that Person B is anxious from his / her facial expression, it provides guidance in a soft voice saying, "Please walk relaxedly."

[0585] In this way, this system provides multifunctional support that takes into account the user's emotional state, creating an environment in which users can go out with greater peace of mind.

[0586] The processing flow will be explained below.

[0587] Step 1:

[0588] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. The device also collects camera footage and the user's facial expressions and voice, and analyzes these with an emotion engine to recognize the user's emotional state.

[0589] Step 2:

[0590] The device sends the collected data (vital data, walking pace, rhythm, camera footage, and emotional data) to a server. The data is sent securely and in real time using a communication protocol.

[0591] Step 3:

[0592] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. At the same time, it analyzes the camera footage to detect road irregularities and obstacles, and recognizes the user's emotional state based on the results of the emotion engine.

[0593] Step 4:

[0594] The server will pace itself, taking into account the user's emotional state. For example, if the user feels anxious, it will provide relaxing music and rhythms and give a voice guidance saying, "Walk in a relaxed manner." Under normal circumstances, it will provide appropriate music and rhythms based on the analysis results.

[0595] Step 5:

[0596] The server will then warn the user based on the results of the camera image analysis. For example, if there is a step ahead, the server will say, "There is a step ahead. Please be careful." If the user's emotional state is unstable, the server will give additional advice such as, "Walk slowly."

[0597] Step 6:

[0598] The server analyzes the user's vital data and walking pace and suggests that the user take a break or slow down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It also takes into account the user's emotional state and provides advice on how to change their mood (e.g., "Try taking a deep breath").

[0599] Step 7:

[0600] The device constantly monitors the user's fall using data from the accelerometer and gyroscope, and if it detects any signs of a possible fall, it will sound a loud noise to alert those around it.

[0601] Step 8:

[0602] When the device detects a fall, it immediately sends the location of the fall and camera footage to the server, allowing the server to accurately grasp the situation.

[0603] Step 9:

[0604] Based on the information received by the server, a message containing location information and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing for prompt assistance.

[0605] Step 10:

[0606] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as notifying medical facilities or making additional contact.

[0607] This series of processes allows users to receive personalized support that takes into account their emotional state, allowing them to enjoy going out with peace of mind. It also effectively reduces the risk of falling.

[0608] Example 2

[0609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] To enable elderly people to go out and walk safely and with peace of mind, it is important to monitor not only vital signs and walking status but also emotional status in real time and provide appropriate guidance and warnings. However, current walking support systems do not adequately recognize emotional status or respond dynamically based on it. A system that can quickly respond to the risk of falls is also needed.

[0611] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0612] In this invention, the server includes means for collecting the user's biometric data, walking speed, and movement rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, and means for detecting the user's falls. This makes it possible to comprehensively monitor the elderly person's emotional and physical condition and provide appropriate guidance and warnings in real time.

[0613] "Biometric data" refers to data that indicates a person's physical condition, such as heart rate, blood pressure, and body temperature.

[0614] "Walking speed" refers to the speed at which the user walks, and is data indicating the walking pace.

[0615] "Movement rhythm" refers to the periodic pattern of the user's movement while walking, and is data that indicates the tempo and rhythm of walking.

[0616] "Camera images" refer to visual data acquired by a video recording device, and are data for visually capturing the surrounding situation.

[0617] "Emotional state" refers to the user's state of mind or feelings, including emotions such as joy, sadness, and anxiety.

[0618] "Warning" refers to warning or calling for caution to the user, and is a way to avoid danger.

[0619] "Guidance" means providing appropriate instructions and directions to the user.

[0620] The "means for detecting falls" is part of the system for monitoring a user's falls in real time and detecting them immediately when they occur.

[0621] "Means of collecting data in real time" refers to a method for obtaining data immediately without delay and providing it to the system.

[0622] "Means of analysis" are methods for analyzing collected data and converting it into meaningful information.

[0623] This invention is a system for supporting elderly people to go out and walk safely and with peace of mind. This system has the function of collecting the user's biometric data, walking speed, and movement rhythm in real time, and providing appropriate guidance and warnings based on that data.

[0624] The system consists of the following main components: The terminal is a wearable device worn by the user that collects heart rate, walking speed, rhythm, and camera images in real time. Furthermore, the terminal is equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state. This data is sequentially sent to the server.

[0625] The server receives data sent from the device and analyzes it in real time. Specific analysis includes algorithms for checking fluctuations in the user's heart rate and walking speed, image recognition technology for detecting road irregularities and obstacles from camera images, and an integrated analysis system that comprehensively evaluates the emotion engine's analysis results. Based on the analysis results, the server generates voice guidance and musical rhythms and provides them to the user via the device at the appropriate time.

[0626] In addition, for emergency response, the device uses an accelerometer and gyroscope to monitor the user's falls in real time. If it detects a movement that could indicate a fall, it will sound a loud alert and send the location of the fall and camera images taken at the time to a server. The server also has the function of automatically sending notifications to emergency contacts based on the information it receives.

[0627] For example, when an elderly user, Person A, goes out for a walk, Person A is wearing a wearable device. This device instantly collects Person A's heart rate, walking speed, camera images, facial expressions, and voice, and sends them to the server in real time. The server analyzes this data, and if it determines that Person A is walking too fast, it provides voice guidance such as "Slow down a bit." If the server detects a step ahead from the camera image, it warns, "There is a step ahead. Be careful." Furthermore, if it detects that Person A has an anxious expression, it sends guidance that takes emotions into consideration, such as "Walk relaxedly."

[0628] Below is an example of a prompt sentence.

[0629] "Generate a program that collects the user's biometric data and analyzes their walking speed."

[0630] "How can I detect a fall using an accelerometer and gyroscope?"

[0631] "Explain how you can use an emotion engine to analyze emotional states and provide dynamic guidance."

[0632] In this way, the system can provide multifunctional walking assistance that supports the user's safety and security.

[0633] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0634] Step 1:

[0635] The device collects the user's biometric data, walking speed, and movement rhythm in real time.

[0636] Input: Heart rate, blood pressure, body temperature, walking speed, and rhythm data obtained from the user's body.

[0637] Processing: These data are stored and recorded using sensors built into the device. Biometric data is acquired every second, and rhythm is quantified by analyzing walking tempo.

[0638] Output: Collected biometric data, walking speed, and rhythm data are sent to the server.

[0639] Step 2:

[0640] The device captures camera images and analyzes the user's facial expressions and voice.

[0641] Input: Video data input to the device's camera, user's voice data.

[0642] Processing: Using video recognition technology and voice recognition algorithms, the device analyzes the user's emotional state from their facial expressions and voice. The device's emotion engine identifies emotions such as smile, anxiety, and anger.

[0643] Output: Send the parsed emotional state data to the server.

[0644] Step 3:

[0645] The server receives the biometric data, walking speed, rhythm data, and emotional state data sent from the terminal and analyzes them in real time.

[0646] Input: Biometric data, walking speed, rhythm data, and emotional state data obtained from the device.

[0647] Processing: Analyzes data fluctuations over time to determine whether they are within the normal range. For example, if your heart rate rises sharply, it is determined that you are under exercise stress. This data is then comprehensively analyzed using an AI model.

[0648] Output: The analysis results are sent to the voice guidance generation unit.

[0649] Step 4:

[0650] Based on the analysis results, the server generates pacemaking and guidance for the user.

[0651] Input: The analyzed data and its evaluation results.

[0652] Processing: Taking into account the user's situation, appropriate music and voice guidance are created. For example, if it is determined that the walking speed is too fast, voice guidance such as "Please slow down a little" is created. If there is a step ahead, a warning such as "There is a step ahead. Be careful." If an anxious emotional state is detected from the emotion engine, guidance is generated in a soft voice such as "Please walk in a relaxed manner."

[0653] Output: Send the generated voice guidance to the device.

[0654] Step 5:

[0655] The terminal provides the user with the voice guidance received from the server.

[0656] Input: Voice guidance data sent from the server.

[0657] Processing: Play back audio guidance to the user through the device's built-in speaker.

[0658] Output: The user receives the audio prompt.

[0659] Step 6:

[0660] The device detects when the user falls and takes emergency action.

[0661] Input: Motion data from the accelerometer and gyroscope.

[0662] Processing: The system constantly monitors the user's movements and posture, and applies an algorithm to detect potential falls. If detected, it issues a loud alert and sends the user's location and camera images to a server.

[0663] Output: The location information of the fall and the camera image are sent to the server, and an alert is issued to those in the vicinity.

[0664] Step 7:

[0665] The server receives the fall information and sends a notification to emergency contacts.

[0666] Input: Fall location information and camera images sent from the device.

[0667] Processing: The received fall information is analyzed and a message is automatically sent to pre-registered emergency contacts. The message includes the location and video of the fall.

[0668] Output: Send a notification to emergency contacts with information about the fall.

[0669] (Application example 2)

[0670] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In current food delivery services, staff are often at risk of accidents and mistakes due to excessive stress, fatigue, and an unstable walking pace. A system is needed that can monitor the mental and physical state of employees in real time and provide appropriate guidance.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's vital signs, walking pace, and rhythm in real time, means for analyzing camera images and detecting road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for recognizing the emotional state in real time and providing pacemaking and warnings based thereon, and means for detecting falls and obtaining and notifying location information. This makes it possible to monitor the stress and fatigue felt by staff during work in real time and encourage breaks and give warnings at appropriate times.

[0673] "Vital data" refers to data that indicates a user's vital signs, such as heart rate, blood pressure, and body temperature, in real time.

[0674] "Walking pace" is data on the speed and rhythm when the user walks.

[0675] "Camera footage" refers to real-time visual information acquired by the camera of a wearable device.

[0676] An "emotional state" is a state related to a user's feelings such as stress, anxiety, happiness, etc.

[0677] "Pacemaking" refers to adjusting the pace in real time so that the user can continue walking safely and efficiently.

[0678] "Attention" refers to alerting the user by audio or visual information.

[0679] "Guidance" refers to providing users with appropriate information and instructions to encourage safe and efficient behavior.

[0680] "Fall detection" means recognizing in real time that a user has fallen.

[0681] "Location information" is data indicating the current location of the user.

[0682] "Notification" means sending important information to the user and people around them.

[0683] This invention provides a wearable assistance system that helps food delivery staff perform their work more safely and efficiently. The wearable device worn by the delivery staff collects vital signs, walking pace and rhythm in real time, and analyzes camera footage. The device also has an emotion engine that can recognize the user's emotional state in real time.

[0684] The data collected by the wearable device is sent to a server, which analyzes the data in real time. In particular, the server analyzes vital signs, walking pace and rhythm, as well as emotional state recognition. This allows the server to provide appropriate pacing, alerts, and guidance to the user.

[0685] The hardware used includes wearable devices (e.g., smart glasses and head-mounted displays) equipped with heart rate sensors and cameras, and the software used includes HeartRateMonitor for collecting vital data, EmotionRecognizer for analyzing facial expressions, and VoiceAssistant for providing voice guidance.

[0686] Furthermore, the server also has the means to respond to cases such as when the user falls or other emergencies. For example, if the server detects a fall, it can immediately obtain the user's location information and notify pre-registered emergency contacts, which allows for a rapid response.

[0687] For example, if a delivery staff member feels stressed while walking, EmotionRecognizer will recognize this and the server will provide guidance to relax. Also, if the heart rate rises, the server can provide specific voice guidance such as "Your heart rate is high. Please take a short break."

[0688] Example prompt sentence:

[0689] "Develop a system that recognizes the emotional state of food delivery staff in real time and provides pacemaking, warnings, and guidance based on that. Emotional states include stress, anxiety, and fatigue. Heart rate data will also be used to prompt staff to take a break if fatigue is detected. Hardware to be used includes a wearable device, heart rate sensor, and camera, and software libraries include OpenCV and a voice assistant library."

[0690] In this way, the present invention is a system that provides an environment in which food delivery staff can work with peace of mind.

[0691] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0692] Step 1:

[0693] The user wears a wearable device, which collects the user's vital data (e.g., heart rate), walking pace, rhythm, and camera footage in real time. The input here is raw data obtained from the sensors, and the output is the collection of these data.

[0694] Step 2:

[0695] The device sends the collected data to the server. At this point, the input is the vital signs, walking pace, rhythm, and camera footage collected by the device, and the output is this data sent to the server.

[0696] Step 3:

[0697] The server analyzes the received data. First, it analyzes the vital data and determines whether the heart rate and blood pressure are within normal ranges. The input is the vital data received by the server, and the output is the analysis result regarding the user's health condition.

[0698] Step 4:

[0699] The server analyzes the walking pace and rhythm and determines whether the user is walking safely and efficiently. The input is the walking pace and rhythm data, and the output is the analysis result.

[0700] Step 5:

[0701] The server analyzes the camera images and detects road irregularities and obstacles. The input is the camera images, and the output is the analysis results on the road conditions.

[0702] Step 6:

[0703] The device's emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The input is real-time facial and voice data from the wearable device, and the output is the analysis result of the user's emotional state.

[0704] Step 7:

[0705] The server determines appropriate guidance for the user based on vital data, walking pace, rhythm, analysis of camera footage, and recognition of emotional state. For example, if the heart rate is high or there is an obstacle on the road, the server generates appropriate voice guidance. The input is the results of various analyses, and the output is the guidance content provided to the user.

[0706] Step 8:

[0707] The terminal conveys the guidance content provided by the server to the user as voice. Voice guidance includes content such as "Please slow down a little" or "There is a step ahead. Please be careful." The input is the guidance content generated by the server, and the output is voice guidance.

[0708] Step 9:

[0709] The device constantly monitors the data from the accelerometer and gyroscope to detect falls. The input is real-time data from these sensors, and the output is the fall detection result.

[0710] Step 10:

[0711] When the device detects a fall, it immediately sends location information and camera footage to the server. The input is the fall detection result, location information, and camera footage, and the output is the transmission of this information to the server.

[0712] Step 11:

[0713] The server sends a notification to the emergency contact based on the received fall information. The input here is the fall information and the emergency contact information, and the output is a notification message.

[0714] This series of processes enables food delivery staff to carry out their duties safely and efficiently, and enables quick response in emergencies.

[0715] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0717] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0718] [Third embodiment]

[0719] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0720] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0721] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0722] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0723] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0724] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0725] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0726] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0727] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0728] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0729] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0730] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0731] This invention is a walking assistance system for seniors, designed to allow users to go out with peace of mind. The system collects the user's vital signs and walking pace in real time, and analyzes camera footage to determine the surrounding situation. Furthermore, by providing appropriate audio warnings and guidance, it reduces the risk of falls and supports the user's independent lifestyle.

[0732] Main components

[0733] 1. Terminal

[0734] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0735] The terminal is responsible for transmitting this data to the server.

[0736] 2. Server

[0737] It receives data sent from the terminal and analyzes it in real time.

[0738] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0739] 3. Users

[0740] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0741] Explanation of program processing

[0742] Environmental sensor data collection

[0743] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time, as well as camera footage, and transmits this data to a server.

[0744] Real-time data analysis

[0745] The server receives the data sent from the device and analyzes it in real time. Specifically, it monitors the user's heart rate and fluctuations in walking pace to determine if there are any abnormalities. At the same time, it analyzes the camera footage to detect unevenness and obstacles in the road.

[0746] Pacing and Guidance

[0747] Based on the analysis results, the server generates music and rhythms to stabilize the user's walking pace. This is provided to the user via the device, and audio warnings and guidance are also provided. For example, if there is a step ahead of the user, the server will say, "There is a step ahead. Please be careful."

[0748] Personalized Coaching

[0749] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice to "take a short break." It will also recommend exercises to prevent falls and provide audio instructions on how to do them.

[0750] Emergency response

[0751] If the device detects that the user has fallen, it will assess the situation based on data from the acceleration sensor and issue an emergency alert. At this time, the device will sound a loud alarm to alert those around it and simultaneously send information about the location of the fall and video to the server. The server then automatically sends a message containing the location and video of the fall to pre-registered family members and emergency contacts to ensure the user's safety.

[0752] Specific examples

[0753] For example, imagine a scenario where a senior user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking pace, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0754] In this way, the system provides multifunctional support to help users go out safely and reduce the risk of falling.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] The device collects the user's vital data, walking pace, and rhythm in real time, including biometric information such as heart rate, blood pressure, and body temperature, and also collects camera footage of the user while walking.

[0758] Step 2:

[0759] The device sends the collected data (vital data, walking pace, camera footage, etc.) to a server. The data is sent to the server in real time using a secure communication protocol.

[0760] Step 3:

[0761] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles.

[0762] Step 4:

[0763] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, the server may communicate voice guidance such as "There is a step ahead. Be careful" to the user via the device.

[0764] Step 5:

[0765] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It will also recommend exercises to prevent falls and provide audio instructions.

[0766] Step 6:

[0767] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0768] Step 7:

[0769] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0770] Step 8:

[0771] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0772] Step 9:

[0773] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as a phone call or notifying medical facilities.

[0774] This series of processes allows the user to enjoy going out with peace of mind and effectively reduces the risk of falling.

[0775] Example 1

[0776] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0777] To provide an environment where elderly people can go out safely, a system that monitors their vital signs and walking pace in real time and provides appropriate guidance and alerts is needed. However, existing technologies have difficulty effectively collecting and analyzing this data and providing appropriate feedback. Another issue is that emergency response is insufficient.

[0778] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0779] In this invention, the server includes means for collecting the user's biometric information, walking speed, and walking pattern in real time, means for analyzing video and detecting road undulations and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting the user's sudden movements or falls, immediately issuing an alert, and transmitting fall position information and video to the server, and means for the server to send an emergency alert message. This provides an environment where elderly people can go out with peace of mind, reduces the risk of falls, and enables rapid response in emergencies.

[0780] A "user" is a person who uses the system and whose vital data, walking speed, walking pattern, etc. are monitored.

[0781] "Biometric information" is data indicating the user's health condition, such as heart rate, blood pressure, and body temperature.

[0782] "Walking speed" refers to the speed at which a user moves, and is an index of the distance traveled within a certain period of time.

[0783] The "walking pattern" refers to the rhythm and repetition of movements when the user walks, and is an index for detecting movements that differ from normal movements.

[0784] "Means for collecting in real time" refers to a device or method for instantly collecting a user's biometric information, walking speed, and walking pattern and transmitting them to the system.

[0785] "Means for analyzing images" refers to technology that processes image data acquired by cameras and other devices to detect road irregularities and obstacles.

[0786] "Means for providing audio warnings and instructions" refers to a device or method that uses audio to alert a user to safety concerns or encourage appropriate behavior.

[0787] "Means for detecting sudden movements and falls" refers to sensors and algorithms that immediately detect abnormalities in the user's movements or falls and notify the system.

[0788] "Means for issuing an alert" refers to a device or method for notifying the user and surrounding people of an abnormality by voice or visual means.

[0789] The "means for transmitting fall position information and video" refers to a device or method for transmitting position information and video of the scene of a fall to a server when a user falls.

[0790] "Means for sending an emergency alert message" refers to a device or method that sends information to pre-set contacts in the event of an emergency such as a fall, encouraging a prompt response.

[0791] This invention is a walking support system that provides an environment where elderly people can go out with peace of mind. Specifically, this system collects and analyzes the user's biometric information, walking speed, and walking pattern in real time, and can provide appropriate audio guidance and emergency response. This system mainly consists of three components: a terminal, a server, and a user.

[0792] Terminal

[0793] The terminal is a wearable device worn by the user. This terminal collects biometric information such as heart rate, blood pressure, and body temperature, as well as walking speed and walking pattern in real time. It also collects images of the surrounding area using a built-in camera and transmits this data to a server via Bluetooth or Wi-Fi.

[0794] The terminal used is, for example, a wearable device from Garmin, which has the ability to measure heart rate and blood pressure data every second and record video at 30 frames per second.

[0795] server

[0796] The server receives data sent from the device and analyzes it in real time. Specifically, it uses a Python script to monitor fluctuations in heart rate and walking speed and check for any abnormalities. This uses data analysis libraries such as numpy and pandas. It also uses the OpenCV library to analyze camera footage and detect road undulations and obstacles.

[0797] Based on the results of the analysis, Text-to-Speech (TTS) software is used to generate voice guidance such as "Please slow down a bit" and provide this to the user via the device.

[0798] Emergency response

[0799] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, it immediately emits a loud alert and simultaneously sends the location of the fall and camera footage to a server. The server then uses services such as Twilio and AWS SNS to automatically send an emergency alert message to pre-defined contacts to ensure safety.

[0800] Examples of specific examples and prompts

[0801] For example, imagine a scenario where an elderly user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking speed, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[0802] Based on a concrete example of what this system achieves, the following is an example of a prompt sentence to input to the generative AI model:

[0803] Imagine situations where senior citizen A is at high risk of falling while walking, and generate real-time warnings. For example, if there is a step ahead, the voice will instruct, "There is a step ahead. Be careful." And if the heart rate rises, the voice will guide, "Take a short break."

[0804] In this way, the present invention provides a system that allows users to go out with peace of mind, reduces the risk of falling, and enables quick response in emergencies.

[0805] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0806] Step 1: Collecting Environmental Sensor Data

[0807] The device collects the user's biometric information (heart rate, blood pressure, body temperature, etc.), walking speed, and walking patterns in real time. Specifically, the device measures heart rate and blood pressure data every second and records video at 30 frames per second using the built-in camera. This data is sent to a server via Bluetooth or Wi-Fi.

[0808] Input: User's biometric information, walking speed, camera footage

[0809] Output: Real-time data sent to the server

[0810] Step 2: Real-time analysis of data

[0811] The server receives the data sent from the device and analyzes it in real time. It uses a Python script to monitor fluctuations in heart rate and walking speed to check for any abnormalities. It also uses the OpenCV library to analyze camera footage and identify road undulations and obstacles.

[0812] Input: Real-time data sent from the device (biometric information, walking speed, camera footage)

[0813] Output: Anomaly detection results, road undulations and obstacles identification results

[0814] Step 3: Generate audio guidance

[0815] Based on the analysis results, the server generates voice guidance and musical rhythms to stabilize the user's walking pace. Using Text-to-Speech (TTS) software, it generates voice guidance such as "Please slow down a bit" and provides this to the user via their device.

[0816] Input: Analysis results (user's walking pace, road undulations and obstacle identification results)

[0817] Output: Voice guidance and musical rhythm

[0818] Step 4: Personalized Coaching

[0819] Based on the analysis of the user's biometric information and walking speed, the server will suggest rest periods and provide audio guidance on exercises to prevent falls. The server will analyze the user's past data using a machine learning algorithm to provide optimal advice. For example, it may suggest, "Your heart rate has increased, so please take a short break."

[0820] Input: User vital data and historical data

[0821] Output: Audio suggestions for rest and exercise

[0822] Step 5: Emergency response

[0823] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, the device immediately emits a loud alert and simultaneously transmits the location of the fall and camera footage to a server. The server then receives this information and automatically sends an emergency alert message to pre-defined contacts using services such as Twilio or AWS SNS.

[0824] Input: Acceleration sensor data, fall location information, camera footage

[0825] Output: Loud alert, emergency alert message

[0826] (Application example 1)

[0827] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0828] Conventional automated driving systems do not adequately address the need to ensure the safety of elderly drivers. In particular, they lack features such as real-time monitoring of the driver's biometric information and driving pace, emergency detection, automatic stopping, and notification to emergency contacts. As a result, elderly people find it difficult to use automated vehicles with confidence, and they remain anxious about traveling alone. The present invention aims to enable elderly people to use automated vehicles with confidence and significantly improve driving safety.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0830] In this invention, the server includes means for collecting the user's biometric information, driving pace, and rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting an emergency and automatically stopping the vehicle, means for automatically notifying emergency contacts, means for adjusting the vehicle's driving pace based on the driver's vital data, and means for understanding the situation around the vehicle based on video analysis. This allows even elderly people to use self-driving vehicles with peace of mind, and significantly improves safety while driving.

[0831] "User" refers to a driver who uses the system or an individual for whom the system is provided.

[0832] "Biometric information" refers to data that shows the driver's health condition in real time, such as heart rate, blood pressure, and body temperature.

[0833] "Driving pace" refers to fluctuations in the speed and rhythm of a driver's vehicle operation.

[0834] "Rhythm" refers to the regularity or periodicity of a driver's driving behavior.

[0835] "Camera footage" refers to video data captured by a camera used to monitor the inside and outside of a vehicle.

[0836] "Analysis" refers to the process of processing collected data to extract useful information.

[0837] "Road irregularities and obstacles" refers to physical undulations and objects that obstruct movement in front of the driver.

[0838] "Warning" refers to the act of notifying drivers of danger.

[0839] "Guidance" refers to instructions to encourage drivers to drive safely and behave appropriately.

[0840] An "emergency situation" refers to a situation that threatens the safety of the driver or vehicle.

[0841] "Automatic stop" means that the system stops the vehicle's operation without driver intervention.

[0842] "Emergency contacts" refer to the people you should contact first in the event of an emergency.

[0843] "Notification" refers to the act of a system communicating specific information to a designated recipient.

[0844] "Driving pace adjustment" refers to automatically changing the vehicle's speed and operation based on the driver's biometric information.

[0845] "Surrounding conditions" refers to all physical conditions and events inside and outside the vehicle and in the driving environment.

[0846] "Means" refers to devices or methods used to achieve a particular purpose.

[0847] The present invention provides an "automated driving vehicle support system for seniors" that enables elderly people to use automated driving vehicles safely and with peace of mind. Specific embodiments will be described below.

[0848] Hardware Configuration

[0849] 1. Wearable devices: fitness bands and smartwatches worn by seniors that collect real-time biometric information such as heart rate, blood pressure, and body temperature.

[0850] 2. Cameras inside and outside the vehicle: These cameras are used to monitor the inside and outside of the vehicle and collect video data.

[0851] 3. Smartphone or tablet: A device for data processing.

[0852] 4. Server: A central system for analyzing and processing data.

[0853] Software Configuration

[0854] 1. Biometric monitoring system: Software for collecting and analyzing driver vital data, such as Apple HealthKit or Google Fit API.

[0855] 2. Video analysis system: Using OpenCV, the system analyzes camera images from inside and outside the vehicle to detect road irregularities and obstacles.

[0856] 3. Voice Guidance System: Uses Google Text-to-Speech API and Amazon Polly to provide voice guidance, reminders, and pacing.

[0857] 4. Emergency Response System: Uses the Twilio API to automatically notify emergency contacts in the event of an emergency.

[0858] Processing flow

[0859] 1. Data Collection:

[0860] The wearable device collects the user's biometric information in real time and transmits it to a server.

[0861] Cameras inside and outside the vehicle collect video data in real time, which is also sent to a server.

[0862] 2. Data Analysis:

[0863] The server analyzes the collected vital data and video data to monitor the driver's physical condition and the driving environment.

[0864] 3. Real-time support:

[0865] It adjusts the driving pace and provides voice guidance to encourage the driver to take appropriate driving actions.

[0866] In the event of an emergency, the system will automatically shut down and notify emergency contacts.

[0867] Specific examples

[0868] For example, consider a scenario in which elderly person B goes out in a self-driving vehicle. B is wearing a wearable device, and an in-car camera monitors footage while driving, collecting and analyzing fluctuations in his or her physical condition. If B's ​​heart rate rises significantly while driving, a voice prompt will automatically advise him or her, saying, "Try not to push yourself, and take a break." Furthermore, if the external camera detects a danger ahead, a voice prompt will warn him or her, saying, "There is an obstacle ahead. Be careful."

[0869] Prompt Sentence Examples

[0870] "We are considering a system that will enable elderly people to safely use self-driving vehicles, by collecting biometric information in real time and analyzing driving behavior. Specifically, the system will monitor heart rate and driving pace, and issue audio warnings if it detects danger based on video analysis. What kind of functions and software should we use?"

[0871] Based on the above description, the present invention provides specific technical means for improving the safety of elderly people when using self-driving vehicles.

[0872] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0873] Step 1:

[0874] Data collection:

[0875] The terminal (wearable device) collects the user's biometric information (heart rate, blood pressure, body temperature, etc.) in real time and sends it to a server. In addition, cameras inside and outside the vehicle collect video data in real time and send it to the server.

[0876] Input: User vital data, camera footage from inside and outside the vehicle.

[0877] Output: Biometric information and video data sent to the server.

[0878] Step 2:

[0879] Data Analysis:

[0880] The server analyzes the received biometric information and video data to monitor the driver's physical condition and the driving environment, including analyzing fluctuations in biometric information and analyzing the road environment using image processing technology.

[0881] Input: Biometric information and video data sent to the server.

[0882] Output: Analyzed user's physical condition and the situation around the vehicle.

[0883] Step 3:

[0884] Real-time guidance provided:

[0885] The server provides voice guidance to the driver based on the analysis results. For example, if the driver's heart rate rises, a message is sent urging them to take a break. If there is an obstacle ahead, a voice message will be sent saying, "There is an obstacle ahead. Be careful."

[0886] Input: Analyzed physical condition and vehicle surroundings.

[0887] Output: Voice guidance (break suggestions and obstacle notifications).

[0888] Step 4:

[0889] Pace your driving:

[0890] The server adjusts the vehicle's driving pace based on the user's vital data, taking measures such as slowing down if the user's heart rate is high.

[0891] Input: User's biometric information.

[0892] Output: Adjusted driving pace instructions.

[0893] Step 5:

[0894] Emergency automatic stop:

[0895] If the server detects an emergency (for example, a sudden change in the driver's health), it issues an instruction to automatically stop the vehicle.

[0896] Input: Detection of sudden changes in vital signs.

[0897] Output: Auto-stop command for vehicle.

[0898] Step 6:

[0899] Emergency notification:

[0900] The server automatically notifies the user's emergency contacts when an emergency shutdown occurs, including details about the user's location and current status.

[0901] Input: Detection of sudden changes in vital signs, location information.

[0902] Output: Notification message to emergency contacts.

[0903] Through these steps, the system will operate to enable elderly people to safely use self-driving vehicles.

[0904] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0905] This invention combines an emotion engine with a walking support system for seniors to recognize the user's emotional state in real time and provide pacemaking, warnings, and guidance based on this information, allowing users to enjoy going out with greater peace of mind and reducing the risk of falling.

[0906] Main components

[0907] 1. Terminal

[0908] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[0909] The device is also equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state.

[0910] The terminal is responsible for transmitting this data to the server.

[0911] 2. Server

[0912] It receives data sent from the terminal and analyzes it in real time.

[0913] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[0914] Based on the recognition results of the emotion engine, advice and warnings are given taking into account the emotional state.

[0915] 3. Users

[0916] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[0917] Explanation of program processing

[0918] Environmental sensor data collection

[0919] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. In addition, the device analyzes the camera footage and the user's facial expressions and voice to recognize their emotional state. This data is then sent to a server.

[0920] Real-time data analysis

[0921] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles. It also analyzes the results of the emotion engine to understand the user's emotional state.

[0922] Pacing and Guidance

[0923] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, it will say, "There is a step ahead. Please be careful." In addition, based on the emotional state obtained from the emotion engine, if the user is feeling anxious, it will provide guidance in a soft voice, saying, "Please walk in a relaxed manner."

[0924] Personalized Coaching

[0925] The server suggests to the user to take a break or slow down based on the analysis of the user's vital data and walking pace. For example, if the user's heart rate rises, the server will prompt the user by voice, "Take a short break." It also provides advice based on the user's emotional state, such as "Try taking a deep breath to refresh yourself." Furthermore, the server will recommend exercises to prevent falls and provide audio instructions on how to do them.

[0926] Emergency response

[0927] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[0928] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[0929] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[0930] Specific examples

[0931] For example, imagine a scenario in which a senior user, Person B, goes out for a walk. Person B is wearing a wearable device, which collects his / her heart rate, walking pace, camera footage, and emotional state in real time from facial expressions and voice. The server receives this data, and if it determines that Person B is walking too fast, it suggests that he / she slow down. This suggestion is made through voice, saying, "Slow down a bit." If the server also analyzes the camera footage and detects that there is a step ahead of Person B, it warns him / her with a voice saying, "There is a step ahead. Please be careful." On the other hand, if it recognizes that Person B is anxious from his / her facial expression, it provides guidance in a soft voice saying, "Please walk relaxedly."

[0932] In this way, this system provides multifunctional support that takes into account the user's emotional state, creating an environment in which users can go out with greater peace of mind.

[0933] The processing flow will be explained below.

[0934] Step 1:

[0935] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. The device also collects camera footage and the user's facial expressions and voice, and analyzes these with an emotion engine to recognize the user's emotional state.

[0936] Step 2:

[0937] The device sends the collected data (vital data, walking pace, rhythm, camera footage, and emotional data) to a server. The data is sent securely and in real time using a communication protocol.

[0938] Step 3:

[0939] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. At the same time, it analyzes the camera footage to detect road irregularities and obstacles, and recognizes the user's emotional state based on the results of the emotion engine.

[0940] Step 4:

[0941] The server will pace itself, taking into account the user's emotional state. For example, if the user feels anxious, it will provide relaxing music and rhythms and give a voice guidance saying, "Walk in a relaxed manner." Under normal circumstances, it will provide appropriate music and rhythms based on the analysis results.

[0942] Step 5:

[0943] The server will then warn the user based on the results of the camera image analysis. For example, if there is a step ahead, the server will say, "There is a step ahead. Please be careful." If the user's emotional state is unstable, the server will give additional advice such as, "Walk slowly."

[0944] Step 6:

[0945] The server analyzes the user's vital data and walking pace and suggests that the user take a break or slow down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It also takes into account the user's emotional state and provides advice on how to change their mood (e.g., "Try taking a deep breath").

[0946] Step 7:

[0947] The device constantly monitors the user's fall using data from the accelerometer and gyroscope, and if it detects any signs of a possible fall, it will sound a loud noise to alert those around it.

[0948] Step 8:

[0949] When the device detects a fall, it immediately sends the location of the fall and camera footage to the server, allowing the server to accurately grasp the situation.

[0950] Step 9:

[0951] Based on the information received by the server, a message containing location information and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing for prompt assistance.

[0952] Step 10:

[0953] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as notifying medical facilities or making additional contact.

[0954] This series of processes allows users to receive personalized support that takes into account their emotional state, allowing them to enjoy going out with peace of mind. It also effectively reduces the risk of falling.

[0955] Example 2

[0956] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0957] To enable elderly people to go out and walk safely and with peace of mind, it is important to monitor not only vital signs and walking status but also emotional status in real time and provide appropriate guidance and warnings. However, current walking support systems do not adequately recognize emotional status or respond dynamically based on it. A system that can quickly respond to the risk of falls is also needed.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0959] In this invention, the server includes means for collecting the user's biometric data, walking speed, and movement rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, and means for detecting the user's falls. This makes it possible to comprehensively monitor the elderly person's emotional and physical condition and provide appropriate guidance and warnings in real time.

[0960] "Biometric data" refers to data that indicates a person's physical condition, such as heart rate, blood pressure, and body temperature.

[0961] "Walking speed" refers to the speed at which the user walks, and is data indicating the walking pace.

[0962] "Movement rhythm" refers to the periodic pattern of the user's movement while walking, and is data that indicates the tempo and rhythm of walking.

[0963] "Camera images" refer to visual data acquired by a video recording device, and are data for visually capturing the surrounding situation.

[0964] "Emotional state" refers to the user's state of mind or feelings, including emotions such as joy, sadness, and anxiety.

[0965] "Warning" refers to warning or calling for caution to the user, and is a way to avoid danger.

[0966] "Guidance" means providing appropriate instructions and directions to the user.

[0967] The "means for detecting falls" is part of the system for monitoring a user's falls in real time and detecting them immediately when they occur.

[0968] "Means of collecting data in real time" refers to a method for obtaining data immediately without delay and providing it to the system.

[0969] "Means of analysis" are methods for analyzing collected data and converting it into meaningful information.

[0970] This invention is a system for supporting elderly people to go out and walk safely and with peace of mind. This system has the function of collecting the user's biometric data, walking speed, and movement rhythm in real time, and providing appropriate guidance and warnings based on that data.

[0971] The system consists of the following main components: The terminal is a wearable device worn by the user that collects heart rate, walking speed, rhythm, and camera images in real time. Furthermore, the terminal is equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state. This data is sequentially sent to the server.

[0972] The server receives data sent from the device and analyzes it in real time. Specific analysis includes algorithms for checking fluctuations in the user's heart rate and walking speed, image recognition technology for detecting road irregularities and obstacles from camera images, and an integrated analysis system that comprehensively evaluates the emotion engine's analysis results. Based on the analysis results, the server generates voice guidance and musical rhythms and provides them to the user via the device at the appropriate time.

[0973] In addition, for emergency response, the device uses an accelerometer and gyroscope to monitor the user's falls in real time. If it detects a movement that could indicate a fall, it will sound a loud alert and send the location of the fall and camera images taken at the time to a server. The server also has the function of automatically sending notifications to emergency contacts based on the information it receives.

[0974] For example, when an elderly user, Person A, goes out for a walk, Person A is wearing a wearable device. This device instantly collects Person A's heart rate, walking speed, camera images, facial expressions, and voice, and sends them to the server in real time. The server analyzes this data, and if it determines that Person A is walking too fast, it provides voice guidance such as "Slow down a bit." If the server detects a step ahead from the camera image, it warns, "There is a step ahead. Be careful." Furthermore, if it detects that Person A has an anxious expression, it sends guidance that takes emotions into consideration, such as "Walk relaxedly."

[0975] Below is an example of a prompt sentence.

[0976] "Generate a program that collects the user's biometric data and analyzes their walking speed."

[0977] "How can I detect a fall using an accelerometer and gyroscope?"

[0978] "Explain how you can use an emotion engine to analyze emotional states and provide dynamic guidance."

[0979] In this way, the system can provide multifunctional walking assistance that supports the user's safety and security.

[0980] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] The device collects the user's biometric data, walking speed, and movement rhythm in real time.

[0983] Input: Heart rate, blood pressure, body temperature, walking speed, and rhythm data obtained from the user's body.

[0984] Processing: These data are stored and recorded using sensors built into the device. Biometric data is acquired every second, and rhythm is quantified by analyzing walking tempo.

[0985] Output: Collected biometric data, walking speed, and rhythm data are sent to the server.

[0986] Step 2:

[0987] The device captures camera images and analyzes the user's facial expressions and voice.

[0988] Input: Video data input to the device's camera, user's voice data.

[0989] Processing: Using video recognition technology and voice recognition algorithms, the device analyzes the user's emotional state from their facial expressions and voice. The device's emotion engine identifies emotions such as smile, anxiety, and anger.

[0990] Output: Send the parsed emotional state data to the server.

[0991] Step 3:

[0992] The server receives the biometric data, walking speed, rhythm data, and emotional state data sent from the terminal and analyzes them in real time.

[0993] Input: Biometric data, walking speed, rhythm data, and emotional state data obtained from the device.

[0994] Processing: Analyzes data fluctuations over time to determine whether they are within the normal range. For example, if your heart rate rises sharply, it is determined that you are under exercise stress. This data is then comprehensively analyzed using an AI model.

[0995] Output: The analysis results are sent to the voice guidance generation unit.

[0996] Step 4:

[0997] Based on the analysis results, the server generates pacemaking and guidance for the user.

[0998] Input: The analyzed data and its evaluation results.

[0999] Processing: Taking into account the user's situation, appropriate music and voice guidance are created. For example, if it is determined that the walking speed is too fast, voice guidance such as "Please slow down a little" is created. If there is a step ahead, a warning such as "There is a step ahead. Be careful." If an anxious emotional state is detected from the emotion engine, guidance is generated in a soft voice such as "Please walk in a relaxed manner."

[1000] Output: Send the generated voice guidance to the device.

[1001] Step 5:

[1002] The terminal provides the user with the voice guidance received from the server.

[1003] Input: Voice guidance data sent from the server.

[1004] Processing: Play back audio guidance to the user through the device's built-in speaker.

[1005] Output: The user receives the audio prompt.

[1006] Step 6:

[1007] The device detects when the user falls and takes emergency action.

[1008] Input: Motion data from the accelerometer and gyroscope.

[1009] Processing: The system constantly monitors the user's movements and posture, and applies an algorithm to detect potential falls. If detected, it issues a loud alert and sends the user's location and camera images to a server.

[1010] Output: The location information of the fall and the camera image are sent to the server, and an alert is issued to those in the vicinity.

[1011] Step 7:

[1012] The server receives the fall information and sends a notification to emergency contacts.

[1013] Input: Fall location information and camera images sent from the device.

[1014] Processing: The received fall information is analyzed and a message is automatically sent to pre-registered emergency contacts. The message includes the location and video of the fall.

[1015] Output: Send a notification to emergency contacts with information about the fall.

[1016] (Application example 2)

[1017] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1018] In current food delivery services, staff are often at risk of accidents and mistakes due to excessive stress, fatigue, and an unstable walking pace. A system is needed that can monitor the mental and physical state of employees in real time and provide appropriate guidance.

[1019] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's vital signs, walking pace, and rhythm in real time, means for analyzing camera images and detecting road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for recognizing the emotional state in real time and providing pacemaking and warnings based thereon, and means for detecting falls and obtaining and notifying location information. This makes it possible to monitor the stress and fatigue felt by staff during work in real time and encourage breaks and give warnings at appropriate times.

[1020] "Vital data" refers to data that indicates a user's vital signs, such as heart rate, blood pressure, and body temperature, in real time.

[1021] "Walking pace" is data on the speed and rhythm when the user walks.

[1022] "Camera footage" refers to real-time visual information acquired by the camera of a wearable device.

[1023] An "emotional state" is a state related to a user's feelings such as stress, anxiety, happiness, etc.

[1024] "Pacemaking" refers to adjusting the pace in real time so that the user can continue walking safely and efficiently.

[1025] "Attention" refers to alerting the user by audio or visual information.

[1026] "Guidance" refers to providing users with appropriate information and instructions to encourage safe and efficient behavior.

[1027] "Fall detection" means recognizing in real time that a user has fallen.

[1028] "Location information" is data indicating the current location of the user.

[1029] "Notification" means sending important information to the user and people around them.

[1030] This invention provides a wearable assistance system that helps food delivery staff perform their work more safely and efficiently. The wearable device worn by the delivery staff collects vital signs, walking pace and rhythm in real time, and analyzes camera footage. The device also has an emotion engine that can recognize the user's emotional state in real time.

[1031] The data collected by the wearable device is sent to a server, which analyzes the data in real time. In particular, the server analyzes vital signs, walking pace and rhythm, as well as emotional state recognition. This allows the server to provide appropriate pacing, alerts, and guidance to the user.

[1032] The hardware used includes wearable devices (e.g., smart glasses and head-mounted displays) equipped with heart rate sensors and cameras, and the software used includes HeartRateMonitor for collecting vital data, EmotionRecognizer for analyzing facial expressions, and VoiceAssistant for providing voice guidance.

[1033] Furthermore, the server also has the means to respond to cases such as when the user falls or other emergencies. For example, if the server detects a fall, it can immediately obtain the user's location information and notify pre-registered emergency contacts, which allows for a rapid response.

[1034] For example, if a delivery staff member feels stressed while walking, EmotionRecognizer will recognize this and the server will provide guidance to relax. Also, if the heart rate rises, the server can provide specific voice guidance such as "Your heart rate is high. Please take a short break."

[1035] Example prompt sentence:

[1036] "Develop a system that recognizes the emotional state of food delivery staff in real time and provides pacemaking, warnings, and guidance based on that. Emotional states include stress, anxiety, and fatigue. Heart rate data will also be used to prompt staff to take a break if fatigue is detected. Hardware to be used includes a wearable device, heart rate sensor, and camera, and software libraries include OpenCV and a voice assistant library."

[1037] In this way, the present invention is a system that provides an environment in which food delivery staff can work with peace of mind.

[1038] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1039] Step 1:

[1040] The user wears a wearable device, which collects the user's vital data (e.g., heart rate), walking pace, rhythm, and camera footage in real time. The input here is raw data obtained from the sensors, and the output is the collection of these data.

[1041] Step 2:

[1042] The device sends the collected data to the server. At this point, the input is the vital signs, walking pace, rhythm, and camera footage collected by the device, and the output is this data sent to the server.

[1043] Step 3:

[1044] The server analyzes the received data. First, it analyzes the vital data and determines whether the heart rate and blood pressure are within normal ranges. The input is the vital data received by the server, and the output is the analysis result regarding the user's health condition.

[1045] Step 4:

[1046] The server analyzes the walking pace and rhythm and determines whether the user is walking safely and efficiently. The input is the walking pace and rhythm data, and the output is the analysis result.

[1047] Step 5:

[1048] The server analyzes the camera images and detects road irregularities and obstacles. The input is the camera images, and the output is the analysis results on the road conditions.

[1049] Step 6:

[1050] The device's emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The input is real-time facial and voice data from the wearable device, and the output is the analysis result of the user's emotional state.

[1051] Step 7:

[1052] The server determines appropriate guidance for the user based on vital data, walking pace, rhythm, analysis of camera footage, and recognition of emotional state. For example, if the heart rate is high or there is an obstacle on the road, the server generates appropriate voice guidance. The input is the results of various analyses, and the output is the guidance content provided to the user.

[1053] Step 8:

[1054] The terminal conveys the guidance content provided by the server to the user as voice. Voice guidance includes content such as "Please slow down a little" or "There is a step ahead. Please be careful." The input is the guidance content generated by the server, and the output is voice guidance.

[1055] Step 9:

[1056] The device constantly monitors the data from the accelerometer and gyroscope to detect falls. The input is real-time data from these sensors, and the output is the fall detection result.

[1057] Step 10:

[1058] When the device detects a fall, it immediately sends location information and camera footage to the server. The input is the fall detection result, location information, and camera footage, and the output is the transmission of this information to the server.

[1059] Step 11:

[1060] The server sends a notification to the emergency contact based on the received fall information. The input here is the fall information and the emergency contact information, and the output is a notification message.

[1061] This series of processes enables food delivery staff to carry out their duties safely and efficiently, and enables quick response in emergencies.

[1062] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1063] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1064] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1065] [Fourth embodiment]

[1066] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1067] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1068] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1069] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1070] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1071] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1072] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1073] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1074] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1075] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1076] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1077] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1078] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1079] This invention is a walking assistance system for seniors, designed to allow users to go out with peace of mind. The system collects the user's vital signs and walking pace in real time, and analyzes camera footage to determine the surrounding situation. Furthermore, by providing appropriate audio warnings and guidance, it reduces the risk of falls and supports the user's independent lifestyle.

[1080] Main components

[1081] 1. Terminal

[1082] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[1083] The terminal is responsible for transmitting this data to the server.

[1084] 2. Server

[1085] It receives data sent from the terminal and analyzes it in real time.

[1086] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[1087] 3. Users

[1088] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[1089] Explanation of program processing

[1090] Environmental sensor data collection

[1091] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time, as well as camera footage, and transmits this data to a server.

[1092] Real-time data analysis

[1093] The server receives the data sent from the device and analyzes it in real time. Specifically, it monitors the user's heart rate and fluctuations in walking pace to determine if there are any abnormalities. At the same time, it analyzes the camera footage to detect unevenness and obstacles in the road.

[1094] Pacing and Guidance

[1095] Based on the analysis results, the server generates music and rhythms to stabilize the user's walking pace. This is provided to the user via the device, and audio warnings and guidance are also provided. For example, if there is a step ahead of the user, the server will say, "There is a step ahead. Please be careful."

[1096] Personalized Coaching

[1097] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice to "take a short break." It will also recommend exercises to prevent falls and provide audio instructions on how to do them.

[1098] Emergency response

[1099] If the device detects that the user has fallen, it will assess the situation based on data from the acceleration sensor and issue an emergency alert. At this time, the device will sound a loud alarm to alert those around it and simultaneously send information about the location of the fall and video to the server. The server then automatically sends a message containing the location and video of the fall to pre-registered family members and emergency contacts to ensure the user's safety.

[1100] Specific examples

[1101] For example, imagine a scenario where a senior user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking pace, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[1102] In this way, the system provides multifunctional support to help users go out safely and reduce the risk of falling.

[1103] The processing flow will be explained below.

[1104] Step 1:

[1105] The device collects the user's vital data, walking pace, and rhythm in real time, including biometric information such as heart rate, blood pressure, and body temperature, and also collects camera footage of the user while walking.

[1106] Step 2:

[1107] The device sends the collected data (vital data, walking pace, camera footage, etc.) to a server. The data is sent to the server in real time using a secure communication protocol.

[1108] Step 3:

[1109] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles.

[1110] Step 4:

[1111] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, the server may communicate voice guidance such as "There is a step ahead. Be careful" to the user via the device.

[1112] Step 5:

[1113] The server analyzes the user's vital data and walking pace and suggests taking a break or slowing down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It will also recommend exercises to prevent falls and provide audio instructions.

[1114] Step 6:

[1115] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[1116] Step 7:

[1117] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[1118] Step 8:

[1119] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[1120] Step 9:

[1121] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as a phone call or notifying medical facilities.

[1122] This series of processes allows the user to enjoy going out with peace of mind and effectively reduces the risk of falling.

[1123] Example 1

[1124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1125] To provide an environment where elderly people can go out safely, a system that monitors their vital signs and walking pace in real time and provides appropriate guidance and alerts is needed. However, existing technologies have difficulty effectively collecting and analyzing this data and providing appropriate feedback. Another issue is that emergency response is insufficient.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1127] In this invention, the server includes means for collecting the user's biometric information, walking speed, and walking pattern in real time, means for analyzing video and detecting road undulations and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting the user's sudden movements or falls, immediately issuing an alert, and transmitting fall position information and video to the server, and means for the server to send an emergency alert message. This provides an environment where elderly people can go out with peace of mind, reduces the risk of falls, and enables rapid response in emergencies.

[1128] A "user" is a person who uses the system and whose vital data, walking speed, walking pattern, etc. are monitored.

[1129] "Biometric information" is data indicating the user's health condition, such as heart rate, blood pressure, and body temperature.

[1130] "Walking speed" refers to the speed at which a user moves, and is an index of the distance traveled within a certain period of time.

[1131] The "walking pattern" refers to the rhythm and repetition of movements when the user walks, and is an index for detecting movements that differ from normal movements.

[1132] "Means for collecting in real time" refers to a device or method for instantly collecting a user's biometric information, walking speed, and walking pattern and transmitting them to the system.

[1133] "Means for analyzing images" refers to technology that processes image data acquired by cameras and other devices to detect road irregularities and obstacles.

[1134] "Means for providing audio warnings and instructions" refers to a device or method that uses audio to alert a user to safety concerns or encourage appropriate behavior.

[1135] "Means for detecting sudden movements and falls" refers to sensors and algorithms that immediately detect abnormalities in the user's movements or falls and notify the system.

[1136] "Means for issuing an alert" refers to a device or method for notifying the user and surrounding people of an abnormality by voice or visual means.

[1137] The "means for transmitting fall position information and video" refers to a device or method for transmitting position information and video of the scene of a fall to a server when a user falls.

[1138] "Means for sending an emergency alert message" refers to a device or method that sends information to pre-set contacts in the event of an emergency such as a fall, encouraging a prompt response.

[1139] This invention is a walking support system that provides an environment where elderly people can go out with peace of mind. Specifically, this system collects and analyzes the user's biometric information, walking speed, and walking pattern in real time, and can provide appropriate audio guidance and emergency response. This system mainly consists of three components: a terminal, a server, and a user.

[1140] Terminal

[1141] The terminal is a wearable device worn by the user. This terminal collects biometric information such as heart rate, blood pressure, and body temperature, as well as walking speed and walking pattern in real time. It also collects images of the surrounding area using a built-in camera and transmits this data to a server via Bluetooth or Wi-Fi.

[1142] The terminal used is, for example, a wearable device from Garmin, which has the ability to measure heart rate and blood pressure data every second and record video at 30 frames per second.

[1143] server

[1144] The server receives data sent from the device and analyzes it in real time. Specifically, it uses a Python script to monitor fluctuations in heart rate and walking speed and check for any abnormalities. This uses data analysis libraries such as numpy and pandas. It also uses the OpenCV library to analyze camera footage and detect road undulations and obstacles.

[1145] Based on the results of the analysis, Text-to-Speech (TTS) software is used to generate voice guidance such as "Please slow down a bit" and provide this to the user via the device.

[1146] Emergency response

[1147] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, it immediately emits a loud alert and simultaneously sends the location of the fall and camera footage to a server. The server then uses services such as Twilio and AWS SNS to automatically send an emergency alert message to pre-defined contacts to ensure safety.

[1148] Examples of specific examples and prompts

[1149] For example, imagine a scenario where an elderly user, Person A, goes out for a walk. Person A is wearing a wearable device, which collects his heart rate, walking speed, and camera footage in real time. The server receives this data, and if it determines that Person A's walking pace is too fast, it suggests that he slow down. This suggestion is made by voice, saying, "Let's slow down a bit." Furthermore, if the server analyzes the camera footage and detects that there is a step ahead of Person A, it warns him by voice, saying, "There is a step ahead. Be careful."

[1150] Based on a concrete example of what this system achieves, the following is an example of a prompt sentence to input to the generative AI model:

[1151] Imagine situations where senior citizen A is at high risk of falling while walking, and generate real-time warnings. For example, if there is a step ahead, the voice will instruct, "There is a step ahead. Be careful." And if the heart rate rises, the voice will guide, "Take a short break."

[1152] In this way, the present invention provides a system that allows users to go out with peace of mind, reduces the risk of falling, and enables quick response in emergencies.

[1153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1154] Step 1: Collecting Environmental Sensor Data

[1155] The device collects the user's biometric information (heart rate, blood pressure, body temperature, etc.), walking speed, and walking patterns in real time. Specifically, the device measures heart rate and blood pressure data every second and records video at 30 frames per second using the built-in camera. This data is sent to a server via Bluetooth or Wi-Fi.

[1156] Input: User's biometric information, walking speed, camera footage

[1157] Output: Real-time data sent to the server

[1158] Step 2: Real-time analysis of data

[1159] The server receives the data sent from the device and analyzes it in real time. It uses a Python script to monitor fluctuations in heart rate and walking speed to check for any abnormalities. It also uses the OpenCV library to analyze camera footage and identify road undulations and obstacles.

[1160] Input: Real-time data sent from the device (biometric information, walking speed, camera footage)

[1161] Output: Anomaly detection results, road undulations and obstacles identification results

[1162] Step 3: Generate audio guidance

[1163] Based on the analysis results, the server generates voice guidance and musical rhythms to stabilize the user's walking pace. Using Text-to-Speech (TTS) software, it generates voice guidance such as "Please slow down a bit" and provides this to the user via their device.

[1164] Input: Analysis results (user's walking pace, road undulations and obstacle identification results)

[1165] Output: Voice guidance and musical rhythm

[1166] Step 4: Personalized Coaching

[1167] Based on the analysis of the user's biometric information and walking speed, the server will suggest rest periods and provide audio guidance on exercises to prevent falls. The server will analyze the user's past data using a machine learning algorithm to provide optimal advice. For example, it may suggest, "Your heart rate has increased, so please take a short break."

[1168] Input: User vital data and historical data

[1169] Output: Audio suggestions for rest and exercise

[1170] Step 5: Emergency response

[1171] The device uses a built-in accelerometer to detect sudden movements or falls. In this case, the device immediately emits a loud alert and simultaneously transmits the location of the fall and camera footage to a server. The server then receives this information and automatically sends an emergency alert message to pre-defined contacts using services such as Twilio or AWS SNS.

[1172] Input: Acceleration sensor data, fall location information, camera footage

[1173] Output: Loud alert, emergency alert message

[1174] (Application example 1)

[1175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1176] Conventional automated driving systems do not adequately address the need to ensure the safety of elderly drivers. In particular, they lack features such as real-time monitoring of the driver's biometric information and driving pace, emergency detection, automatic stopping, and notification to emergency contacts. As a result, elderly people find it difficult to use automated vehicles with confidence, and they remain anxious about traveling alone. The present invention aims to enable elderly people to use automated vehicles with confidence and significantly improve driving safety.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1178] In this invention, the server includes means for collecting the user's biometric information, driving pace, and rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for detecting an emergency and automatically stopping the vehicle, means for automatically notifying emergency contacts, means for adjusting the vehicle's driving pace based on the driver's vital data, and means for understanding the situation around the vehicle based on video analysis. This allows even elderly people to use self-driving vehicles with peace of mind, and significantly improves safety while driving.

[1179] "User" refers to a driver who uses the system or an individual for whom the system is provided.

[1180] "Biometric information" refers to data that shows the driver's health condition in real time, such as heart rate, blood pressure, and body temperature.

[1181] "Driving pace" refers to fluctuations in the speed and rhythm of a driver's vehicle operation.

[1182] "Rhythm" refers to the regularity or periodicity of a driver's driving behavior.

[1183] "Camera footage" refers to video data captured by a camera used to monitor the inside and outside of a vehicle.

[1184] "Analysis" refers to the process of processing collected data to extract useful information.

[1185] "Road irregularities and obstacles" refers to physical undulations and objects that obstruct movement in front of the driver.

[1186] "Warning" refers to the act of notifying drivers of danger.

[1187] "Guidance" refers to instructions to encourage drivers to drive safely and behave appropriately.

[1188] An "emergency situation" refers to a situation that threatens the safety of the driver or vehicle.

[1189] "Automatic stop" means that the system stops the vehicle's operation without driver intervention.

[1190] "Emergency contacts" refer to the people you should contact first in the event of an emergency.

[1191] "Notification" refers to the act of a system communicating specific information to a designated recipient.

[1192] "Driving pace adjustment" refers to automatically changing the vehicle's speed and operation based on the driver's biometric information.

[1193] "Surrounding conditions" refers to all physical conditions and events inside and outside the vehicle and in the driving environment.

[1194] "Means" refers to devices or methods used to achieve a particular purpose.

[1195] The present invention provides an "automated driving vehicle support system for seniors" that enables elderly people to use automated driving vehicles safely and with peace of mind. Specific embodiments will be described below.

[1196] Hardware Configuration

[1197] 1. Wearable devices: fitness bands and smartwatches worn by seniors that collect real-time biometric information such as heart rate, blood pressure, and body temperature.

[1198] 2. Cameras inside and outside the vehicle: These cameras are used to monitor the inside and outside of the vehicle and collect video data.

[1199] 3. Smartphone or tablet: A device for data processing.

[1200] 4. Server: A central system for analyzing and processing data.

[1201] Software Configuration

[1202] 1. Biometric monitoring system: Software for collecting and analyzing driver vital data, such as Apple HealthKit or Google Fit API.

[1203] 2. Video analysis system: Using OpenCV, the system analyzes camera images from inside and outside the vehicle to detect road irregularities and obstacles.

[1204] 3. Voice Guidance System: Uses Google Text-to-Speech API and Amazon Polly to provide voice guidance, reminders, and pacing.

[1205] 4. Emergency Response System: Uses the Twilio API to automatically notify emergency contacts in the event of an emergency.

[1206] Processing flow

[1207] 1. Data Collection:

[1208] The wearable device collects the user's biometric information in real time and transmits it to a server.

[1209] Cameras inside and outside the vehicle collect video data in real time, which is also sent to a server.

[1210] 2. Data Analysis:

[1211] The server analyzes the collected vital data and video data to monitor the driver's physical condition and the driving environment.

[1212] 3. Real-time support:

[1213] It adjusts the driving pace and provides voice guidance to encourage the driver to take appropriate driving actions.

[1214] In the event of an emergency, the system will automatically shut down and notify emergency contacts.

[1215] Specific examples

[1216] For example, consider a scenario in which elderly person B goes out in a self-driving vehicle. B is wearing a wearable device, and an in-car camera monitors footage while driving, collecting and analyzing fluctuations in his or her physical condition. If B's ​​heart rate rises significantly while driving, a voice prompt will automatically advise him or her, saying, "Try not to push yourself, and take a break." Furthermore, if the external camera detects a danger ahead, a voice prompt will warn him or her, saying, "There is an obstacle ahead. Be careful."

[1217] Prompt Sentence Examples

[1218] "We are considering a system that will enable elderly people to safely use self-driving vehicles, by collecting biometric information in real time and analyzing driving behavior. Specifically, the system will monitor heart rate and driving pace, and issue audio warnings if it detects danger based on video analysis. What kind of functions and software should we use?"

[1219] Based on the above description, the present invention provides specific technical means for improving the safety of elderly people when using self-driving vehicles.

[1220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1221] Step 1:

[1222] Data collection:

[1223] The terminal (wearable device) collects the user's biometric information (heart rate, blood pressure, body temperature, etc.) in real time and sends it to a server. In addition, cameras inside and outside the vehicle collect video data in real time and send it to the server.

[1224] Input: User vital data, camera footage from inside and outside the vehicle.

[1225] Output: Biometric information and video data sent to the server.

[1226] Step 2:

[1227] Data Analysis:

[1228] The server analyzes the received biometric information and video data to monitor the driver's physical condition and the driving environment, including analyzing fluctuations in biometric information and analyzing the road environment using image processing technology.

[1229] Input: Biometric information and video data sent to the server.

[1230] Output: Analyzed user's physical condition and the situation around the vehicle.

[1231] Step 3:

[1232] Real-time guidance provided:

[1233] The server provides voice guidance to the driver based on the analysis results. For example, if the driver's heart rate rises, a message is sent urging them to take a break. If there is an obstacle ahead, a voice message will be sent saying, "There is an obstacle ahead. Be careful."

[1234] Input: Analyzed physical condition and vehicle surroundings.

[1235] Output: Voice guidance (break suggestions and obstacle notifications).

[1236] Step 4:

[1237] Pace your driving:

[1238] The server adjusts the vehicle's driving pace based on the user's vital data, taking measures such as slowing down if the user's heart rate is high.

[1239] Input: User's biometric information.

[1240] Output: Adjusted driving pace instructions.

[1241] Step 5:

[1242] Emergency automatic stop:

[1243] If the server detects an emergency (for example, a sudden change in the driver's health), it issues an instruction to automatically stop the vehicle.

[1244] Input: Detection of sudden changes in vital signs.

[1245] Output: Auto-stop command for vehicle.

[1246] Step 6:

[1247] Emergency notification:

[1248] The server automatically notifies the user's emergency contacts when an emergency shutdown occurs, including details about the user's location and current status.

[1249] Input: Detection of sudden changes in vital signs, location information.

[1250] Output: Notification message to emergency contacts.

[1251] Through these steps, the system will operate to enable elderly people to safely use self-driving vehicles.

[1252] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1253] This invention combines an emotion engine with a walking support system for seniors to recognize the user's emotional state in real time and provide pacemaking, warnings, and guidance based on this information, allowing users to enjoy going out with greater peace of mind and reducing the risk of falling.

[1254] Main components

[1255] 1. Terminal

[1256] It is a wearable device worn by the user that has the ability to collect heart rate, walking pace, rhythm, and camera footage in real time.

[1257] The device is also equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state.

[1258] The terminal is responsible for transmitting this data to the server.

[1259] 2. Server

[1260] It receives data sent from the terminal and analyzes it in real time.

[1261] To stabilize walking pace and rhythm, audio guidance and musical rhythms are generated and provided to the user via the device.

[1262] Based on the recognition results of the emotion engine, advice and warnings are given taking into account the emotional state.

[1263] 3. Users

[1264] Users wear the wearable device and receive real-time support from the system while going about their daily lives.

[1265] Explanation of program processing

[1266] Environmental sensor data collection

[1267] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. In addition, the device analyzes the camera footage and the user's facial expressions and voice to recognize their emotional state. This data is then sent to a server.

[1268] Real-time data analysis

[1269] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. It also analyzes camera footage to detect road irregularities and obstacles. It also analyzes the results of the emotion engine to understand the user's emotional state.

[1270] Pacing and Guidance

[1271] Based on the analysis results, the server performs pacemaking to stabilize the user's walking pace. This includes generating music and rhythms and providing voice guidance at appropriate times. For example, it will say, "There is a step ahead. Please be careful." In addition, based on the emotional state obtained from the emotion engine, if the user is feeling anxious, it will provide guidance in a soft voice, saying, "Please walk in a relaxed manner."

[1272] Personalized Coaching

[1273] The server suggests to the user to take a break or slow down based on the analysis of the user's vital data and walking pace. For example, if the user's heart rate rises, the server will prompt the user by voice, "Take a short break." It also provides advice based on the user's emotional state, such as "Try taking a deep breath to refresh yourself." Furthermore, the server will recommend exercises to prevent falls and provide audio instructions on how to do them.

[1274] Emergency response

[1275] The device constantly monitors data from the accelerometer and gyroscope to detect when the user falls. If it detects a possible fall, it will sound a loud alarm to alert those around it.

[1276] When the device detects a fall, it immediately sends the location of the fall and the camera footage at that time to the server, allowing the situation to be accurately understood.

[1277] Based on the information received by the server, a message containing the location and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing necessary assistance to be provided quickly.

[1278] Specific examples

[1279] For example, imagine a scenario in which a senior user, Person B, goes out for a walk. Person B is wearing a wearable device, which collects his / her heart rate, walking pace, camera footage, and emotional state in real time from facial expressions and voice. The server receives this data, and if it determines that Person B is walking too fast, it suggests that he / she slow down. This suggestion is made through voice, saying, "Slow down a bit." If the server also analyzes the camera footage and detects that there is a step ahead of Person B, it warns him / her with a voice saying, "There is a step ahead. Please be careful." On the other hand, if it recognizes that Person B is anxious from his / her facial expression, it provides guidance in a soft voice saying, "Please walk relaxedly."

[1280] In this way, this system provides multifunctional support that takes into account the user's emotional state, creating an environment in which users can go out with greater peace of mind.

[1281] The processing flow will be explained below.

[1282] Step 1:

[1283] The device collects the user's vital data (heart rate, blood pressure, body temperature, etc.), walking pace, and rhythm in real time. The device also collects camera footage and the user's facial expressions and voice, and analyzes these with an emotion engine to recognize the user's emotional state.

[1284] Step 2:

[1285] The device sends the collected data (vital data, walking pace, rhythm, camera footage, and emotional data) to a server. The data is sent securely and in real time using a communication protocol.

[1286] Step 3:

[1287] The server analyzes the received data in real time. Specifically, the server checks the user's heart rate and walking pace to determine whether they are within normal ranges. At the same time, it analyzes the camera footage to detect road irregularities and obstacles, and recognizes the user's emotional state based on the results of the emotion engine.

[1288] Step 4:

[1289] The server will pace itself, taking into account the user's emotional state. For example, if the user feels anxious, it will provide relaxing music and rhythms and give a voice guidance saying, "Walk in a relaxed manner." Under normal circumstances, it will provide appropriate music and rhythms based on the analysis results.

[1290] Step 5:

[1291] The server will then warn the user based on the results of the camera image analysis. For example, if there is a step ahead, the server will say, "There is a step ahead. Please be careful." If the user's emotional state is unstable, the server will give additional advice such as, "Walk slowly."

[1292] Step 6:

[1293] The server analyzes the user's vital data and walking pace and suggests that the user take a break or slow down. For example, if the user's heart rate rises, the server will prompt the user by voice, saying, "Take a short break." It also takes into account the user's emotional state and provides advice on how to change their mood (e.g., "Try taking a deep breath").

[1294] Step 7:

[1295] The device constantly monitors the user's fall using data from the accelerometer and gyroscope, and if it detects any signs of a possible fall, it will sound a loud noise to alert those around it.

[1296] Step 8:

[1297] When the device detects a fall, it immediately sends the location of the fall and camera footage to the server, allowing the server to accurately grasp the situation.

[1298] Step 9:

[1299] Based on the information received by the server, a message containing location information and video of the fall is automatically sent to pre-registered family members and emergency contacts, allowing for prompt assistance.

[1300] Step 10:

[1301] The server waits for feedback from family members or emergency contacts and provides additional support as needed, such as notifying medical facilities or making additional contact.

[1302] This series of processes allows users to receive personalized support that takes into account their emotional state, allowing them to enjoy going out with peace of mind. It also effectively reduces the risk of falling.

[1303] Example 2

[1304] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1305] To enable elderly people to go out and walk safely and with peace of mind, it is important to monitor not only vital signs and walking status but also emotional status in real time and provide appropriate guidance and warnings. However, current walking support systems do not adequately recognize emotional status or respond dynamically based on it. A system that can quickly respond to the risk of falls is also needed.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1307] In this invention, the server includes means for collecting the user's biometric data, walking speed, and movement rhythm in real time, means for analyzing camera images to detect road irregularities and obstacles, means for analyzing the user's facial expressions and voice to recognize the user's emotional state, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, and means for detecting the user's falls. This makes it possible to comprehensively monitor the elderly person's emotional and physical condition and provide appropriate guidance and warnings in real time.

[1308] "Biometric data" refers to data that indicates a person's physical condition, such as heart rate, blood pressure, and body temperature.

[1309] "Walking speed" refers to the speed at which the user walks, and is data indicating the walking pace.

[1310] "Movement rhythm" refers to the periodic pattern of the user's movement while walking, and is data that indicates the tempo and rhythm of walking.

[1311] "Camera images" refer to visual data acquired by a video recording device, and are data for visually capturing the surrounding situation.

[1312] "Emotional state" refers to the user's state of mind or feelings, including emotions such as joy, sadness, and anxiety.

[1313] "Warning" refers to warning or calling for caution to the user, and is a way to avoid danger.

[1314] "Guidance" means providing appropriate instructions and directions to the user.

[1315] The "means for detecting falls" is part of the system for monitoring a user's falls in real time and detecting them immediately when they occur.

[1316] "Means of collecting data in real time" refers to a method for obtaining data immediately without delay and providing it to the system.

[1317] "Means of analysis" are methods for analyzing collected data and converting it into meaningful information.

[1318] This invention is a system for supporting elderly people to go out and walk safely and with peace of mind. This system has the function of collecting the user's biometric data, walking speed, and movement rhythm in real time, and providing appropriate guidance and warnings based on that data.

[1319] The system consists of the following main components: The terminal is a wearable device worn by the user that collects heart rate, walking speed, rhythm, and camera images in real time. Furthermore, the terminal is equipped with an emotion engine that analyzes the user's facial expressions and voice to recognize their emotional state. This data is sequentially sent to the server.

[1320] The server receives data sent from the device and analyzes it in real time. Specific analysis includes algorithms for checking fluctuations in the user's heart rate and walking speed, image recognition technology for detecting road irregularities and obstacles from camera images, and an integrated analysis system that comprehensively evaluates the emotion engine's analysis results. Based on the analysis results, the server generates voice guidance and musical rhythms and provides them to the user via the device at the appropriate time.

[1321] In addition, for emergency response, the device uses an accelerometer and gyroscope to monitor the user's falls in real time. If it detects a movement that could indicate a fall, it will sound a loud alert and send the location of the fall and camera images taken at the time to a server. The server also has the function of automatically sending notifications to emergency contacts based on the information it receives.

[1322] For example, when an elderly user, Person A, goes out for a walk, Person A is wearing a wearable device. This device instantly collects Person A's heart rate, walking speed, camera images, facial expressions, and voice, and sends them to the server in real time. The server analyzes this data, and if it determines that Person A is walking too fast, it provides voice guidance such as "Slow down a bit." If the server detects a step ahead from the camera image, it warns, "There is a step ahead. Be careful." Furthermore, if it detects that Person A has an anxious expression, it sends guidance that takes emotions into consideration, such as "Walk relaxedly."

[1323] Below is an example of a prompt sentence.

[1324] "Generate a program that collects the user's biometric data and analyzes their walking speed."

[1325] "How can I detect a fall using an accelerometer and gyroscope?"

[1326] "Explain how you can use an emotion engine to analyze emotional states and provide dynamic guidance."

[1327] In this way, the system can provide multifunctional walking assistance that supports the user's safety and security.

[1328] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1329] Step 1:

[1330] The device collects the user's biometric data, walking speed, and movement rhythm in real time.

[1331] Input: Heart rate, blood pressure, body temperature, walking speed, and rhythm data obtained from the user's body.

[1332] Processing: These data are stored and recorded using sensors built into the device. Biometric data is acquired every second, and rhythm is quantified by analyzing walking tempo.

[1333] Output: Collected biometric data, walking speed, and rhythm data are sent to the server.

[1334] Step 2:

[1335] The device captures camera images and analyzes the user's facial expressions and voice.

[1336] Input: Video data input to the device's camera, user's voice data.

[1337] Processing: Using video recognition technology and voice recognition algorithms, the device analyzes the user's emotional state from their facial expressions and voice. The device's emotion engine identifies emotions such as smile, anxiety, and anger.

[1338] Output: Send the parsed emotional state data to the server.

[1339] Step 3:

[1340] The server receives the biometric data, walking speed, rhythm data, and emotional state data sent from the terminal and analyzes them in real time.

[1341] Input: Biometric data, walking speed, rhythm data, and emotional state data obtained from the device.

[1342] Processing: Analyzes data fluctuations over time to determine whether they are within the normal range. For example, if your heart rate rises sharply, it is determined that you are under exercise stress. This data is then comprehensively analyzed using an AI model.

[1343] Output: The analysis results are sent to the voice guidance generation unit.

[1344] Step 4:

[1345] Based on the analysis results, the server generates pacemaking and guidance for the user.

[1346] Input: The analyzed data and its evaluation results.

[1347] Processing: Taking into account the user's situation, appropriate music and voice guidance are created. For example, if it is determined that the walking speed is too fast, voice guidance such as "Please slow down a little" is created. If there is a step ahead, a warning such as "There is a step ahead. Be careful." If an anxious emotional state is detected from the emotion engine, guidance is generated in a soft voice such as "Please walk in a relaxed manner."

[1348] Output: Send the generated voice guidance to the device.

[1349] Step 5:

[1350] The terminal provides the user with the voice guidance received from the server.

[1351] Input: Voice guidance data sent from the server.

[1352] Processing: Play back audio guidance to the user through the device's built-in speaker.

[1353] Output: The user receives the audio prompt.

[1354] Step 6:

[1355] The device detects when the user falls and takes emergency action.

[1356] Input: Motion data from the accelerometer and gyroscope.

[1357] Processing: The system constantly monitors the user's movements and posture, and applies an algorithm to detect potential falls. If detected, it issues a loud alert and sends the user's location and camera images to a server.

[1358] Output: The location information of the fall and the camera image are sent to the server, and an alert is issued to those in the vicinity.

[1359] Step 7:

[1360] The server receives the fall information and sends a notification to emergency contacts.

[1361] Input: Fall location information and camera images sent from the device.

[1362] Processing: The received fall information is analyzed and a message is automatically sent to pre-registered emergency contacts. The message includes the location and video of the fall.

[1363] Output: Send a notification to emergency contacts with information about the fall.

[1364] (Application example 2)

[1365] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1366] In current food delivery services, staff are often at risk of accidents and mistakes due to excessive stress, fatigue, and an unstable walking pace. A system is needed that can monitor the mental and physical state of employees in real time and provide appropriate guidance.

[1367] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's vital signs, walking pace, and rhythm in real time, means for analyzing camera images and detecting road irregularities and obstacles, means for providing the user with audio warnings and guidance based on the data collected by the above means and the analysis results, means for recognizing the emotional state in real time and providing pacemaking and warnings based thereon, and means for detecting falls and obtaining and notifying location information. This makes it possible to monitor the stress and fatigue felt by staff during work in real time and encourage breaks and give warnings at appropriate times.

[1368] "Vital data" refers to data that indicates a user's vital signs, such as heart rate, blood pressure, and body temperature, in real time.

[1369] "Walking pace" is data on the speed and rhythm when the user walks.

[1370] "Camera footage" refers to real-time visual information acquired by the camera of a wearable device.

[1371] An "emotional state" is a state related to a user's feelings such as stress, anxiety, happiness, etc.

[1372] "Pacemaking" refers to adjusting the pace in real time so that the user can continue walking safely and efficiently.

[1373] "Attention" refers to alerting the user by audio or visual information.

[1374] "Guidance" refers to providing users with appropriate information and instructions to encourage safe and efficient behavior.

[1375] "Fall detection" means recognizing in real time that a user has fallen.

[1376] "Location information" is data indicating the current location of the user.

[1377] "Notification" means sending important information to the user and people around them.

[1378] This invention provides a wearable assistance system that helps food delivery staff perform their work more safely and efficiently. The wearable device worn by the delivery staff collects vital signs, walking pace and rhythm in real time, and analyzes camera footage. The device also has an emotion engine that can recognize the user's emotional state in real time.

[1379] The data collected by the wearable device is sent to a server, which analyzes the data in real time. In particular, the server analyzes vital signs, walking pace and rhythm, as well as emotional state recognition. This allows the server to provide appropriate pacing, alerts, and guidance to the user.

[1380] The hardware used includes wearable devices (e.g., smart glasses and head-mounted displays) equipped with heart rate sensors and cameras, and the software used includes HeartRateMonitor for collecting vital data, EmotionRecognizer for analyzing facial expressions, and VoiceAssistant for providing voice guidance.

[1381] Furthermore, the server also has the means to respond to cases such as when the user falls or other emergencies. For example, if the server detects a fall, it can immediately obtain the user's location information and notify pre-registered emergency contacts, which allows for a rapid response.

[1382] For example, if a delivery staff member feels stressed while walking, EmotionRecognizer will recognize this and the server will provide guidance to relax. Also, if the heart rate rises, the server can provide specific voice guidance such as "Your heart rate is high. Please take a short break."

[1383] Example prompt sentence:

[1384] "Develop a system that recognizes the emotional state of food delivery staff in real time and provides pacemaking, warnings, and guidance based on that. Emotional states include stress, anxiety, and fatigue. Heart rate data will also be used to prompt staff to take a break if fatigue is detected. Hardware to be used includes a wearable device, heart rate sensor, and camera, and software libraries include OpenCV and a voice assistant library."

[1385] In this way, the present invention is a system that provides an environment in which food delivery staff can work with peace of mind.

[1386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1387] Step 1:

[1388] The user wears a wearable device, which collects the user's vital data (e.g., heart rate), walking pace, rhythm, and camera footage in real time. The input here is raw data obtained from the sensors, and the output is the collection of these data.

[1389] Step 2:

[1390] The device sends the collected data to the server. At this point, the input is the vital signs, walking pace, rhythm, and camera footage collected by the device, and the output is this data sent to the server.

[1391] Step 3:

[1392] The server analyzes the received data. First, it analyzes the vital data and determines whether the heart rate and blood pressure are within normal ranges. The input is the vital data received by the server, and the output is the analysis result regarding the user's health condition.

[1393] Step 4:

[1394] The server analyzes the walking pace and rhythm and determines whether the user is walking safely and efficiently. The input is the walking pace and rhythm data, and the output is the analysis result.

[1395] Step 5:

[1396] The server analyzes the camera images and detects road irregularities and obstacles. The input is the camera images, and the output is the analysis results on the road conditions.

[1397] Step 6:

[1398] The device's emotion engine analyzes the user's facial expressions and voice to recognize their emotional state. The input is real-time facial and voice data from the wearable device, and the output is the analysis result of the user's emotional state.

[1399] Step 7:

[1400] The server determines appropriate guidance for the user based on vital data, walking pace, rhythm, analysis of camera footage, and recognition of emotional state. For example, if the heart rate is high or there is an obstacle on the road, the server generates appropriate voice guidance. The input is the results of various analyses, and the output is the guidance content provided to the user.

[1401] Step 8:

[1402] The terminal conveys the guidance content provided by the server to the user as voice. Voice guidance includes content such as "Please slow down a little" or "There is a step ahead. Please be careful." The input is the guidance content generated by the server, and the output is voice guidance.

[1403] Step 9:

[1404] The device constantly monitors the data from the accelerometer and gyroscope to detect falls. The input is real-time data from these sensors, and the output is the fall detection result.

[1405] Step 10:

[1406] When the device detects a fall, it immediately sends location information and camera footage to the server. The input is the fall detection result, location information, and camera footage, and the output is the transmission of this information to the server.

[1407] Step 11:

[1408] The server sends a notification to the emergency contact based on the received fall information. The input here is the fall information and the emergency contact information, and the output is a notification message.

[1409] This series of processes enables food delivery staff to carry out their duties safely and efficiently, and enables quick response in emergencies.

[1410] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1412] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1413] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1414] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1415] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1416] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1417] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1418] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1419] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1420] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1421] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1422] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1423] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1424] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1425] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1426] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1427] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1428] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1429] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1430] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1431] The following is further disclosed regarding the above embodiment.

[1432] (Claim 1)

[1433] a means for collecting the user's vital data, walking pace, and rhythm in real time;

[1434] A means of analyzing camera images and detecting road irregularities and obstacles,

[1435] A system including a means for providing audio warnings and guidance to a user based on the data collected by the means and the analysis results.

[1436] (Claim 2)

[1437] 2. The system according to claim 1, further comprising means for providing the user with music and rhythm for appropriate pacing based on the analysis results of said vital data and said walking pace and rhythm.

[1438] (Claim 3)

[1439] 2. The system according to claim 1, further comprising: means for suggesting to the user to take a break or slow down their pace based on the vital data and walking pace; and means for recommending exercises to prevent falls.

[1440] "Example 1"

[1441] (Claim 1)

[1442] means for collecting biometric information, walking speed, and walking pattern of a user in real time;

[1443] A means of analyzing the video and detecting road irregularities and obstacles,

[1444] means for providing audible warnings and instructions to a user based on the data collected by said means and the analysis results;

[1445] A means for detecting a sudden movement or fall of a user, issuing an alert immediately, and transmitting information on the position of the fall and video to a server;

[1446] a means for the server to send an emergency alert message;

[1447] A system including:

[1448] (Claim 2)

[1449] 2. The system according to claim 1, further comprising means for providing a user with music or rhythm that stabilizes an appropriate walking speed based on the analysis results of the biometric information and the walking speed and walking pattern.

[1450] (Claim 3)

[1451] means for suggesting to the user to take a break or slow down the walking speed based on the biometric information and the walking speed, and means for recommending exercise to prevent falls;

[1452] 10. The system of claim 1, further comprising means for analyzing historical data and providing optimal guidance.

[1453] "Application Example 1"

[1454] (Claim 1)

[1455] a means for collecting a user's biometric information, driving pace, and rhythm in real time;

[1456] A means of analyzing camera images and detecting road irregularities and obstacles,

[1457] means for providing a user with audio warnings and guidance based on the data collected by said means and the analysis results;

[1458] means for detecting an emergency situation and automatically stopping the vehicle;

[1459] A means of automatically notifying emergency contacts;

[1460] means for adjusting the driving pace of the vehicle based on the vital data of the driver;

[1461] A means for understanding the situation around the vehicle based on video analysis;

[1462] A system including a system combining the above means.

[1463] (Claim 2)

[1464] 2. The system according to claim 1, further comprising means for providing the user with music and rhythm for appropriate pacing based on the analysis results of the biological information and the driving pace and rhythm.

[1465] (Claim 3)

[1466] 2. The system according to claim 1, further comprising: means for suggesting to the user to take a break or slow down their driving pace, and means for recommending exercises to prevent falls, based on the biometric information and driving pace.

[1467] "Example 2: Combining Emotion Engines"

[1468] (Claim 1)

[1469] means for collecting biometric data, walking speed, and movement rhythm of a user in real time;

[1470] A means of analyzing camera images to detect road irregularities and obstacles;

[1471] means for analyzing a user's facial expression and voice to recognize the user's emotional state;

[1472] means for providing a user with audio warnings and guidance based on the data collected by said means and the analysis results;

[1473] A system including means for detecting a fall of a user.

[1474] (Claim 2)

[1475] 2. The system according to claim 1, further comprising means for providing the user with music or rhythm for appropriate pacing based on the analysis results of said biological data and said walking speed and movement rhythm.

[1476] (Claim 3)

[1477] 2. The system according to claim 1, further comprising: means for suggesting to the user to take a break or slow down, and means for recommending exercises to prevent falls, based on the biometric data and walking speed.

[1478] "Application example 2 when combining emotion engines"

[1479] (Claim 1)

[1480] a means for collecting the user's vital data, walking pace, and rhythm in real time;

[1481] A means of analyzing camera images and detecting road irregularities and obstacles,

[1482] means for providing a user with audio warnings and guidance based on the data collected by said means and the analysis results;

[1483] A means for recognizing emotional states in real time and providing pacemaking and reminders based thereon;

[1484] A means for detecting a fall, acquiring location information, and notifying the user;

[1485] A system including:

[1486] (Claim 2)

[1487] 2. The system according to claim 1, further comprising means for providing the user with music and rhythms for appropriate pacing based on the analysis results of said vital data, said walking pace and rhythm, and said emotional state recognition results.

[1488] (Claim 3)

[1489] means for suggesting to the user to take a break or slow down the pace based on the vital data and the walking pace;

[1490] Measures to recommend exercises to prevent falls;

[1491] 10. The system of claim 1, further comprising means for providing relaxation advice based on emotional state. [Explanation of symbols]

[1492] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting the user's vital data, walking pace, and rhythm in real time; A means of analyzing camera images and detecting road irregularities and obstacles, A system including a means for providing audio warnings and guidance to a user based on the data collected by the means and the analysis results.

2. 2. The system according to claim 1, further comprising means for providing the user with music and rhythm for appropriate pacing based on the analysis results of said vital data and said walking pace and rhythm.

3. The system according to claim 1 , further comprising: means for suggesting to the user to take a break or slow down their walking pace, and means for recommending exercises to prevent falls, based on the vital data and walking pace.

Images