System

Abstract

A system is provided.SOLUTION: A system comprising: means for preprocessing motion data received from a biomechanical sensor; means for analyzing the preprocessed motion data and using generative artificial intelligence to detect inappropriate motion or danger; and means for providing instructions to a user based on the results of the analysis by the generative artificial intelligence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, external support is essential for physically disabled and elderly people to carry out their daily lives safely and efficiently. Ensuring stability of movement is particularly important for those who use prosthetic limbs. However, current technology does not adequately provide systems that can detect inappropriate movements or risk of falling during movement in real time and provide appropriate instructions. As a result, users must constantly monitor their own movements, which places a great physical and mental burden on them. To solve this problem, a system is needed that can detect and analyze user movement data in real time and automatically recognize inappropriate movements and dangers. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides a system that combines a bionic sensor, a means for preprocessing motion data, generative artificial intelligence, and a means for notifying a user. The system of the present invention includes the following means.

[0006] 1. A means for bionic sensors to detect user movements in real time and transmit that movement data to a device.

[0007] 2. A means for the device to preprocess the motion data received from the sensors, remove noise, and normalize the data.

[0008] 3. A means of inputting preprocessed data into generative artificial intelligence to analyze behavior and detect inappropriate behavior or danger.

[0009] 4. A means of generating instructions for the user based on the analysis results of generative artificial intelligence and sending those instructions to the terminal.

[0010] 5. Means for informing the user of instructions, specifically means for informing the user of instructions including at least one of a visual display and an audio notification.

[0011] By integrating these methods, the system of the present invention can analyze the user's actions in real time, automatically detect inappropriate actions and dangers, and provide appropriate instructions, thereby significantly improving the safety and efficiency of the living environment for the physically disabled and elderly.

[0012] A "bionic sensor" is a device that detects and collects data on bodily movements and physiological states.

[0013] "Motion data" refers to data including information such as body position, angle, and speed captured by sensors.

[0014] "Preprocessing" refers to processing to remove noise from and normalize the motion data acquired from the sensor.

[0015] "Generative artificial intelligence" is an artificial intelligence model that learns from large amounts of data and makes predictions and analyses.

[0016] "Analysis" is the process of evaluating data preprocessed by generative artificial intelligence to detect inappropriate behavior or dangers.

[0017] "Users" refers to physically disabled people and elderly people who use prosthetic limbs.

[0018] "Instructions" means advice or warnings provided to the User based on the results of the analysis.

[0019] "Notification means" means a method for communicating the generated instructions to the user, including a visual display or an audio notification. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9]1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention provides a system that combines bionic sensors, motion data preprocessing means, generative artificial intelligence, and user notification means to monitor, analyze, and provide appropriate instructions to the movements of physically disabled and elderly people using prosthetic limbs in real time.

[0042] System Overview

[0043] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies the user.

[0044] First, sensors detect the user's movements in real time. The sensors capture specific movements, such as walking or lifting, and transmit the data to a device. For example, a sensor collects information on the angle and position of the user's foot while walking with a prosthetic leg.

[0045] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data. The preprocessed data is then sent to a server and input into the generative artificial intelligence.

[0046] The server then analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects improper behavior and risk of falling. From this analysis, it generates instructions to provide to the user.

[0047] The generated instructions are sent from the server to the terminal, which provides the instructions to the user by visual display (e.g., on-screen text message) or audio notification.

[0048] Specific examples

[0049] Example 1: Falling Hazard

[0050] If a user's foot becomes unstable while walking using a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device preprocesses the data and sends it to the server. The server analyzes it using generative artificial intelligence and recognizes that there is a high risk of falling. As a result, an instruction is generated saying, "There is a risk of falling. Please correct your posture," and the user is notified via the device. The notification is displayed as a text message on the screen and a voice message is broadcast at the same time.

[0051] Example 2: Incorrect behavior

[0052] If a user makes an inappropriate movement when lifting an object with their prosthetic arm, a sensor detects this movement and sends the data to the device. The device removes noise, normalizes the data, and then sends it to the server. The server's generative artificial intelligence analyzes the data and detects that the movement is inappropriate. As a result, an instruction is generated saying, "The movement is inappropriate. Please check again," and the user is notified via the device. This is also communicated to the user via a screen display and voice message.

[0053] In this way, this system can significantly improve the quality of life for the physically disabled and elderly by analyzing the user's movements in real time and providing appropriate instructions quickly.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] A user performs an action using a prosthetic arm or leg. For example, consider a situation in which a user walks with a prosthetic leg.

[0057] Step 2:

[0058] The sensors detect the user's movements in real time and collect specific movement data (for example, foot angle and position information over time).

[0059] Step 3:

[0060] The sensors collect and transmit motion data, including real-time location and angle data, to the device.

[0061] Step 4:

[0062] The device preprocesses the raw data received from the sensors, which includes denoising and normalizing the data, converting it into a format that is easier to analyze.

[0063] Step 5:

[0064] The device sends the preprocessed data to the server, where the data has been denoised and normalised.

[0065] Step 6:

[0066] The server feeds the preprocessed data into a generative artificial intelligence (AI) model that includes advanced learning algorithms such as recurrent neural networks.

[0067] Step 7:

[0068] The server uses generative artificial intelligence to analyze the movement data, detecting inappropriate movements and dangers from the data. For example, if a walking pattern differs from normal, it will be detected as an abnormality.

[0069] Step 8:

[0070] Based on the analysis results, the server generates instructions for the user, including specific content such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[0071] Step 9:

[0072] The server generates instructions and sends them to the terminal. The instructions may be in text or audio format.

[0073] Step 10:

[0074] The terminal notifies the user of the instructions received from the server. Notification methods include displaying a text message on the screen or an audio announcement. For example, the user may be shown a message saying "Danger of falling. Please correct your posture.", and simultaneously notified by an audio message.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] In technology that monitors the movements of physically disabled and elderly people using prosthetic limbs in real time and provides appropriate instructions quickly, current systems have issues with insufficient noise removal and normalization of movement data, making it difficult to obtain accurate analysis results.In addition, there are limited means of notifying users, which makes it difficult to provide sufficient warnings in certain environments.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for removing noise from the motion data and normalizing the motion data, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect inappropriate motions or dangers, and means for providing instructions to the user by visual display or audio notification based on the analysis results of the generative artificial intelligence, thereby enabling more accurate analysis of the user's motions in real time and providing more appropriate and prompt instructions.

[0080] A "bionic sensor" is a device that detects the movement or physiological state of a living organism and acquires this information as data.

[0081] "Motion data" refers to data that includes information about specific movements performed by a user, such as walking or lifting an object, recorded in real time.

[0082] "Preprocessing means" refers to the process of processing raw data obtained from a sensor, such as by removing noise and normalizing it, and converting it into a format that is easy to analyze.

[0083] The "means for removing noise and normalizing" refers to a process for removing unnecessary fluctuations and outliers contained in the operational data and converting the data to a uniform scale.

[0084] "Generative AI" is a type of AI that has the ability to analyze the data it receives and generate new instructions or results.

[0085] A "recurrent neural network" is a neural network model used to analyze time-series data, and is capable of storing past information and using it in current analysis.

[0086] "Means for giving instructions to the user based on the analysis results" refers to a means for issuing instructions to the user to encourage appropriate actions or corrections based on the results of the generative artificial intelligence's analysis of data.

[0087] "Visual display" refers to a method of conveying analytical results or instructions to the user using text or graphics, typically displayed as a message on a screen.

[0088] "Voice notification" is a method of conveying analysis results and instructions to the user by voice, and is provided as a voice message through a speaker or headset.

[0089] The present invention relates to a system that combines bionic sensors, pre-processing means, generative artificial intelligence, and user notification means, and is capable of monitoring, analyzing, and providing appropriate instructions in real time to the movements of physically disabled and elderly people when using prosthetic limbs.

[0090] This system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies users. Specific examples of use are shown below.

[0091] Hardware and software configuration and data processing

[0092] 1. Sensor

[0093] Hardware name: Accelerometer, Gyroscope, etc.

[0094] Role: Detects user behavior in real time and sends that data to the device.

[0095] Example: Collecting foot angle and position information while a user walks with a prosthetic leg.

[0096] 2. Terminal

[0097] Software name: Python script, NumPy, SciPy

[0098] Role: Preprocesses the motion data received from the sensors, specifically removing noise and normalizing the data.

[0099] Example: Data preprocessing removes unwanted shaking and blurring from walking data to generate a standardized walking pattern.

[0100] 3. Server

[0101] Software name: TENSORFLOW (registered trademark), Keras

[0102] Role: Analyzes pre-processed data and uses generative artificial intelligence models (e.g., recurrent neural network models) to detect improper behavior or dangers and generate appropriate instructions.

[0103] Example: Using a recurrent neural network to analyze a user's walking pattern in real time and detect the risk of falling.

[0104] 4. Means of notification

[0105] Software name: ANDROID (registered trademark) app, iOS app

[0106] Role: Notifies the user of instructions sent by the server. Notifications include visual indications (text messages on the screen) and audio notifications.

[0107] Example: A smartphone app displays the message "Risk of falling. Please correct your posture." on the screen and also announces the same message in voice.

[0108] Specific examples

[0109] Example 1: Falling Hazard

[0110] User: Walking with a prosthetic leg

[0111] Sensor: Detects unstable foot movement

[0112] Terminal: Preprocessing the data, removing noise, and normalizing the data

[0113] Server: Performs analysis and detects risk of falling. Generates instruction such as "Danger of falling. Please correct your posture."

[0114] Notification method: Notify the user via the device. Uses screen display and voice message.

[0115] Example 2: Incorrect behavior

[0116] User: Lifting an object with a prosthetic arm

[0117] Sensors: Detect inappropriate behavior

[0118] Terminal: Data preprocessing, noise removal and data normalization

[0119] Server: Analyzes data and detects inappropriate behavior. Generates an instruction saying, "The movement is inappropriate. Please check again."

[0120] Notification method: Notify the user via the device. Uses screen display and voice message.

[0121] Prompt Sentence Examples

[0122] The system analyzes the user's movement data and notifies them in real time of inappropriate movements or dangers, generating instructions such as, "Unstable foot movements were detected while walking with a prosthetic leg. There is a risk of falling, so please correct your posture."

[0123] This system monitors the user's actions in real time through a series of processes using the above-mentioned hardware and software, and provides appropriate instructions to ensure safety.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] The user performs a movement using the prosthetic limb.

[0127] Input: User actions (walking, lifting, etc.)

[0128] Specific action: The user walks using the prosthetic leg.

[0129] Output: Movement data (foot angle, position, acceleration, etc.)

[0130] Step 2:

[0131] The sensor detects the user's movements and transmits the data to the device.

[0132] Input: Real-time user actions

[0133] Specific movements: Bionic sensors capture and generate data on user movements in real time.

[0134] Output: Raw data (unprocessed motion data)

[0135] Step 3:

[0136] The device preprocesses the raw data received from the sensors.

[0137] Input: Raw data sent from the sensor

[0138] What it does: The terminal script performs noise reduction and data normalization, e.g., removes outliers and converts the data to a consistent scale.

[0139] Output: Preprocessed data (denoised and normalized data)

[0140] Step 4:

[0141] The terminal sends the preprocessed data to the server.

[0142] Input: Preprocessed data

[0143] Specific operation: The device sends preprocessed data to the server's API endpoint.

[0144] Output: Preprocessed data sent to the server

[0145] Step 5:

[0146] The server analyzes the received pre-processed data.

[0147] Input: Preprocessed data

[0148] Specific operation: The server uses TensorFlow and Keras to analyze data with a recurrent neural network model to identify specific patterns and detect inappropriate behavior or danger.

[0149] Output: Analysis results (behavior evaluation, risk detection results)

[0150] Step 6:

[0151] The server generates appropriate instructions based on the analysis results.

[0152] Input: Analysis results

[0153] Specific operation: The server uses a generative AI model to generate instructions for the detected problem, for example, creating a message such as "There is a risk of falling. Please correct your posture."

[0154] Output: Instruction message (specific instructions to the user)

[0155] Step 7:

[0156] The server sends the generated instructions to the terminal.

[0157] Input: prompt message

[0158] Specific operation: The server's API sends an instruction message to the device.

[0159] Output: Instruction message sent to the terminal

[0160] Step 8:

[0161] The terminal notifies the user of the instruction received from the server.

[0162] Input: prompt message

[0163] What it does: The device displays a message on the screen and gives a voice notification. For example, a smartphone displays a text message and a voice message saying, "Danger of falling. Please correct your posture."

[0164] Output: User notification (visual and audio instructions)

[0165] Step 9:

[0166] The user acknowledges the notification and corrects the behavior.

[0167] Input: Notification message

[0168] Specific action: The user checks the smartphone screen or voice message and adjusts their walking pace, for example.

[0169] Output: Improved behavior (corrected walking pattern)

[0170] (Application example 1)

[0171] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0172] In autonomous vehicles, if the driver's health condition is not properly monitored in real time, the possibility of dangerous situations occurring increases. Therefore, there is a need for a means to monitor the driver's biological information in real time, quickly detect abnormalities, and provide appropriate instructions. In particular, it is necessary to immediately detect situations such as a sudden increase in the driver's heart rate while driving or delayed reactions due to accumulated fatigue, thereby improving safety.

[0173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0174] In this invention, the server includes means for preprocessing motion data received from the biometric sensor, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect abnormal motion or danger, means for providing instructions to a user based on the analysis results of the generative artificial intelligence, means for monitoring biometric information of the driver of the autonomous vehicle in real time, and means for detecting and notifying an abnormality from the driver's biometric information, thereby making it possible to monitor the driver's biometric information in real time and quickly detect and notify an abnormality.

[0175] A "bionic sensor" is a sensor device for measuring physiological activity or conditions of the human body.

[0176] A "pre-processing means" is a processing system for converting raw data received from a sensor into a format that is easy to analyze.

[0177] "Generative AI" is an AI system that has the ability to generate new insights based on data and provide instructions for specific purposes.

[0178] A "recurrent neural network" is a deep learning model used to capture the characteristics of time series data.

[0179] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[0180] "Driver's biometric information" refers to data that indicates the driver's physical condition and state, such as heart rate, skin temperature, and muscle movement.

[0181] "Means for real-time monitoring" refers to a system that instantly analyzes information obtained from sensors and monitors the driver's condition.

[0182] The "means for detecting and notifying abnormalities" is a system that issues an alert when an abnormality is detected based on the driver's biometric information.

[0183] This invention is a system for monitoring the biological information of a driver of an autonomous vehicle in real time and quickly detecting and notifying an abnormality. A specific embodiment of the system is described below.

[0184] System Configuration

[0185] The system includes the following components:

[0186] 1. Bionic sensors: Sensor devices that measure the driver's heart rate, skin temperature, muscle movement, etc.

[0187] 2. Terminal: A device that processes data received from sensors.

[0188] 3. Server: A computer with generative artificial intelligence to analyze incoming data and generate appropriate instructions.

[0189] 4. Notification means: Visual display device and audio output device to notify the driver of abnormalities.

[0190] Operation overview

[0191] 1. Data collection: When a driver gets into an autonomous vehicle, biometric sensors measure the driver's vital signs in real time and transmit the data to a terminal.

[0192] 2. Data preprocessing: The terminal performs preprocessing on the received data, such as noise removal and normalization, to convert it into a format suitable for analysis.

[0193] 3. Data analysis: The server analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects driver anomalies.

[0194] 4. Notification: If an abnormality is detected, the server generates appropriate instructions and notifies the driver visually or audibly through the terminal.

[0195] Hardware and software used

[0196] Biometric sensors: heart rate monitors, skin thermometers, electromyography sensors, etc.

[0197] Data Preprocessor: A computer / device that performs data denoising and normalization.

[0198] Generative artificial intelligence: Uses deep learning frameworks such as TensorFlow.

[0199] Notification Device: A display, speaker, or any type of alert device.

[0200] Specific scenes and prompts

[0201] 1. Example:

[0202] If a driver's heart rate suddenly rises while driving and the recurrent neural network model determines this to be an abnormality, the system will notify the driver with a message saying, "An abnormality has been detected. Please take a break immediately."

[0203] 2. Example prompt:

[0204] "Predict the probability that the driver's heart rate will reach an abnormal value and notify the driver of the abnormality."

[0205] effect

[0206] This system will ensure the safety of drivers of self-driving vehicles and enable rapid response in emergencies, thereby improving the safety and reliability of self-driving vehicles.

[0207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0208] Step 1:

[0209] Data collection

[0210] When a user gets into an autonomous vehicle, bionic sensors measure their vital signs, such as heart rate, skin temperature, and muscle activity, in real time, and transmit this data to a device.

[0211] Input: Biometric data from biometric sensors

[0212] Output: Raw data sent to the terminal

[0213] Step 2:

[0214] Data Preprocessing

[0215] The device receives the received biometric data and performs noise reduction and normalization on it. Noise reduction removes unnecessary parts of the data, and normalization converts the data into a format suitable for analysis, allowing for smoother analysis.

[0216] Input: Raw data received from the sensor

[0217] Output: Preprocessed data

[0218] Step 3:

[0219] Data analysis

[0220] The preprocessed data is sent from the device to a server, where it is analyzed by a generative artificial intelligence (e.g., a recurrent neural network model). The purpose of the analysis is to detect abnormalities in the driver's biometric information.

[0221] Input: Preprocessed data

[0222] Output: Analysis results (whether anomalies were detected)

[0223] Step 4:

[0224] Generate notifications

[0225] The server generates a notification to the driver based on the analysis results. For example, if the driver's heart rate suddenly rises, it generates a message saying, "An abnormality has been detected. Please take a break immediately."

[0226] Input: Analysis results

[0227] Output: Notification content (message)

[0228] Step 5:

[0229] Sending notifications

[0230] The generated notification is sent to the terminal, which then notifies the driver visually or audibly. Specifically, it is possible to display a message on the display and play an audio message through the speaker.

[0231] Input: Notification content

[0232] Output: Notification to the driver (visual and audio)

[0233] In this way, the system can ensure safety by monitoring the driver's biometric information in real time and immediately notifying the driver if any abnormalities are detected.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] The present invention provides a system that combines bionic sensors, a means for preprocessing motion data, generative artificial intelligence, a means for notifying users, and an emotion engine that recognizes users' emotions. This system not only monitors and analyzes the motions of physically disabled or elderly people using prosthetic limbs in real time and provides appropriate instructions, but also takes into account the user's emotional state to reduce the psychological burden.

[0236] System Overview

[0237] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[0238] Operation flow

[0239] First, the sensor detects the user's movements in real time. For example, when the user walks with a prosthetic leg, the sensor collects information on the angle and position of the walking leg. The sensor then transmits this movement data to the device.

[0240] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data, and then sends the preprocessed data to the server.

[0241] On the server, generative AI analyzes the preprocessed data and detects inappropriate behavior or danger. Based on the analysis results, instructions are generated for the user. For example, "There is a risk of falling. Please correct your posture."

[0242] The generated instructions are sent from the server to the device, which then communicates the instructions to the user through visual displays and audio notifications, allowing the user to receive appropriate advice in real time.

[0243] Emotion Engine Operation

[0244] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. The emotion engine utilizes data obtained from sensors, cameras, and microphones. The results of emotion recognition are added as additional information to the behavior analysis results of generative AI.

[0245] Based on the results of the emotion engine, the instructions to the user can be adjusted accordingly. For example, if the user feels anxious about the risk of falling, a message such as "Please walk slowly and calmly" will be generated to reduce the psychological burden.

[0246] Specific examples

[0247] Example 1: Falling Hazard

[0248] If a user makes unsteady movements while walking with a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device then preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes it and detects the risk of falling. If the emotion engine recognizes anxiety from the user's facial expression, it generates an instruction saying, "There is a risk of falling. Please slowly correct your posture," and notifies the user via the device. The notification is displayed on the screen and by voice.

[0249] Example 2: Incorrect behavior

[0250] If a user makes an inappropriate movement when lifting an object with their prosthetic hand, a sensor detects this movement and sends the data to the device. The device preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes the data and detects the inappropriate movement. Furthermore, if the emotion engine recognizes irritation from the user's tone of voice, it generates an instruction saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the device.

[0251] In this way, the system can detect and analyze the user's movements and emotions in real time, and provide appropriate instructions and psychological support, further improving the quality of life for the physically disabled and elderly.

[0252] The processing flow will be explained below.

[0253] Step 1:

[0254] The user performs an action using a prosthetic arm or leg, for example, the user starts walking with the prosthetic leg.

[0255] Step 2:

[0256] Sensors detect the user's movements in real time and collect movement data, such as the angle and position of the feet while walking.

[0257] Step 3:

[0258] The sensors collect motion data and transmit it to the device, including real-time location and angle data.

[0259] Step 4:

[0260] The device preprocesses the raw data received from the sensors, specifically removing noise from the data and normalizing it, making the data easier to analyze.

[0261] Step 5:

[0262] The device sends the preprocessed data to the server, where it is in a denoised and normalized form.

[0263] Step 6:

[0264] The server feeds the preprocessed data into a generative artificial intelligence (AI) model, which uses advanced algorithms such as recurrent neural networks.

[0265] Step 7:

[0266] The server uses generative artificial intelligence to analyze the movement data and detect inappropriate movements or dangers. For example, if the user's walking pattern differs from normal, it will be detected as an abnormality.

[0267] Step 8:

[0268] Based on the analysis results, the server generates instructions for the user, such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[0269] Step 9:

[0270] The server generates and sends the instructions to the device, which may be in the form of a text or audio file.

[0271] Step 10:

[0272] The terminal notifies the user of the instruction received from the server, for example by displaying a message on the screen and playing a voice message.

[0273] Step 11:

[0274] The device collects the user's facial expressions and tone of voice and inputs them into the emotion engine as emotional data. For example, the device collects the user's reactions through a camera or microphone.

[0275] Step 12:

[0276] The server's emotion engine analyzes the emotion data and recognizes the user's emotional state, for example, determining whether the user is tense or relaxed.

[0277] Step 13:

[0278] Based on the analysis results of the emotion engine, the server adjusts the instructions to correspond to the user's psychological state. For example, if the user is feeling anxious, it generates instructions that give a sense of security.

[0279] Step 14:

[0280] The server then sends the adjusted instructions back to the device, and the device notifies the user. For example, the device displays and speaks the instruction "Please walk slowly and calmly."

[0281] Example 2

[0282] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] Conventional motion assistance systems can monitor a user's motion and provide necessary instructions, but they cannot consider the user's real-time emotional state and lack the ability to reduce psychological stress. This has led to problems such as increased psychological stress when physically disabled people and elderly people use prosthetic limbs. Therefore, the problem that the present invention aims to solve is to reduce psychological stress by recognizing the user's emotional state in real time and adjusting instructions based on that.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0285] In this invention, the server includes a device for preprocessing motion data received from a bionic sensor, a device using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motions or dangers, a device for giving instructions to the user based on the analysis results of the generative artificial intelligence, a device for recognizing the emotional state of the user by analyzing the user's facial expressions and tone of voice, a device for adjusting the instructions taking the emotional state into consideration, and a device for notifying the user of the instructions, thereby making it possible to monitor the user's motions and emotions in real time and provide appropriate instructions and psychological support at the same time.

[0286] A "bionic sensor" is a device that detects and measures human movement and physiological data in real time.

[0287] "Motion data" refers to information about the movements, position information, angle information, and other movements when a user uses a prosthetic arm or leg.

[0288] "Preprocessing" is the process of converting raw data received from sensors into a format suitable for analysis by performing processes such as noise removal and normalization.

[0289] "Generative AI" is a machine learning model that analyzes input data and learns and predicts specific patterns and trends. It often uses advanced algorithms, such as recurrent neural networks.

[0290] "Analysis results" are information obtained by generative artificial intelligence analyzing behavioral data, and indicate whether or not there is inappropriate behavior or danger.

[0291] A "device that gives instructions to the user" is a device that provides specific instructions and advice to the user based on the analysis results of generative artificial intelligence.

[0292] "Emotional state" refers to the user's current psychological state and is analyzed based on data such as facial expressions and tone of voice.

[0293] An "emotion engine" is software or algorithm that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[0294] "Adjusting the content of instructions" means changing the content of the instructions provided based on the analysis results of the emotion engine, taking into account the user's psychological state.

[0295] "Notification" refers to the act of conveying analysis results or instructions to the user visually or audibly.

[0296] "Real-time" is a term that indicates that sensors and systems instantly recognize the user's actions and emotional state, and then analyze and respond immediately.

[0297] This system combines bionic sensors, a motion data preprocessing device, generative artificial intelligence, a user notification device, and an emotion engine. The system aims to monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, provide appropriate instructions, and reduce the psychological burden by taking into account the user's emotional state.

[0298] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[0299] Hardware and software used

[0300] Sensors: Accelerometer, Gyroscope, Camera, Microphone

[0301] Devices: Mobile devices, tablets, wearable devices

[0302] Server: High-performance computing server, cloud computing platform

[0303] Generative AI model: Recurrent Neural Network (RNN)

[0304] Emotion engine: facial expression recognition algorithm, voice tone analysis algorithm

[0305] Data processing and calculation

[0306] 1. Sensor operation data collection

[0307] As the user uses the prosthetic limb, bionic sensors collect real-time motion data, capturing acceleration, angle, and position information at high frequency, which is then transmitted to a device.

[0308] 2. Preprocessing by the terminal

[0309] The device performs noise reduction and data normalization on the raw data received from the sensors. Denoising is done to remove outliers and unnecessary data, and normalization is done to put the data into a consistent format, making it suitable for analysis by generative artificial intelligence.

[0310] 3. Data analysis by the server

[0311] The preprocessed data is sent to a server and input into a generative artificial intelligence (RNN model). The model analyzes the user's movements to detect inappropriate behavior and danger. For example, it can predict the risk of falling based on the movement of the feet while walking.

[0312] 4. Emotion Recognition by Emotion Engine

[0313] The server's emotion engine uses data acquired from the camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state in real time, for example, detecting anxiety or irritation.

[0314] 5. Instruction Generation and Notification

[0315] Based on the analysis results of the generative AI and the recognition results of the emotion engine, the server generates instructions to give to the user. These instructions are sent to the device in real time, and the device conveys the instructions to the user through visual displays and audio notifications, allowing the user to take appropriate action immediately.

[0316] Specific examples

[0317] Example 1: Falling Hazard

[0318] "Prompt: The user is walking with a prosthetic leg and has become unsteady on their feet. The sensor detects this information and sends it to the device. The generative AI analyzes this data and detects the risk of falling. If anxiety is recognized from the user's facial expression, generate appropriate instructions."

[0319] An example of a generated instruction might be: "Danger of falling. Please slowly regain your balance."

[0320] Example 2: Incorrect behavior

[0321] "Prompt: The user performed an inappropriate action while lifting an object with the prosthetic arm. The sensor detected this information and sent it to the device. The generative AI analyzed this data and detected the inappropriate action. If frustration is detected in the user's tone of voice, generate appropriate instructions."

[0322] An example of a generated instruction would be: "Inappropriate behavior. Please calm down and try again."

[0323] By using this system, users can receive appropriate instructions and psychological support in real time, thereby improving the quality of life for the physically disabled and elderly.

[0324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0325] Step 1:

[0326] Sensors collect user movement data in real time.

[0327] Input: The user's prosthetic arm or leg movements

[0328] Movement: Accelerometer and gyroscope sensors measure your movement at a rate of 50 times per second.

[0329] Output: Raw data (foot angle, position information, etc.)

[0330] Step 2:

[0331] The device preprocesses the motion data received from the sensors.

[0332] Input: Raw data from the sensor

[0333] What it does: It removes noise and normalizes the data, eliminating outliers and scaling the data range to 0-1.

[0334] Output: Preprocessed data

[0335] Step 3:

[0336] The device sends the preprocessed data to the server.

[0337] Input: Preprocessed data

[0338] How it works: Sends data using an HTTP POST request.

[0339] Output: Preprocessed data arriving at the server

[0340] Step 4:

[0341] The server analyzes the preprocessed data using generative artificial intelligence.

[0342] Input: Preprocessed data

[0343] Behavior: Data is fed into a generative artificial intelligence (RNN model) to detect inappropriate behavior and danger.

[0344] Output: Analysis results (e.g., whether there is a risk of falling)

[0345] Step 5:

[0346] The server's emotion engine recognizes the user's emotional state in real time.

[0347] Input: Data from camera and microphone (facial expressions, tone of voice)

[0348] How it works: Recognizes emotional states using facial expression recognition and audio tone analysis algorithms.

[0349] Output: Emotion recognition result (e.g., anxiety, irritation)

[0350] Step 6:

[0351] The server generates instructions based on the analysis results and emotion recognition results.

[0352] Input: Analysis results, emotion recognition results

[0353] Action: The analysis results are combined with the emotion recognition results to generate instructions such as "There is a risk of falling. Please slowly correct your posture."

[0354] Output: Generated instructions

[0355] Step 7:

[0356] The server sends the generated instructions to the terminal.

[0357] Input: Generated instructions

[0358] Operation: Sends instructions to the device in real time using WebSocket.

[0359] Output: Instructions that arrive at the terminal

[0360] Step 8:

[0361] The device notifies the user of the instructions.

[0362] Input: Instructions that arrive at the terminal

[0363] Operation: Instructions are displayed on the device screen and communicated aloud through the speaker.

[0364] Output: Instructions to the user

[0365] In this way, the processing flow of the entire system is realized by a chain of specific operations and their inputs and outputs at each step.

[0366] (Application example 2)

[0367] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0368] While conventional systems can monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, they are unable to provide psychological support based on the user's emotional state. Furthermore, particularly in factory robot operating environments, there is a need for emotion analysis and appropriate instructions to reduce worker stress and anxiety, in addition to the ability to detect inappropriate movements and dangers.

[0369] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0370] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motion or danger, means for using an emotion engine to analyze emotional information such as the user's facial expression and voice, and means for providing instructions to the user based on the results of the motion analysis by the generative artificial intelligence and the results of the emotion analysis by the emotion engine. This not only improves the safety and efficiency of workers in the operating environment of factory robots, but also reduces the psychological burden on workers and provides a comfortable working environment.

[0371] definition statement

[0372] A "bionic sensor" is a device that detects a user's physical movements and physiological information in real time and collects them as data.

[0373] The "preprocessing means" refers to a processing method for removing noise and normalizing the raw data received from the sensor and converting it into a format suitable for analysis.

[0374] "Generative AI" is an AI technology that learns from large amounts of data and analyzes and judges specific actions and situations.

[0375] The "emotion engine" is a system that analyzes changes in a user's facial expressions and voice to determine their emotional state in real time.

[0376] "Means for providing instructions" refers to a method for notifying the user of information that prompts the user to take appropriate action based on the analysis results, including visual displays and audio notifications.

[0377] A "terminal" is a device that acts as an interface with sensors and users, and is responsible for preprocessing and notifying data.

[0378] "Using a model" means using a mathematical or computational method to analyze data based on a particular calculation method or algorithm.

[0379] "Detecting real-time actions" means instantly detecting a user's actions and acquiring them as data without delay.

[0380] MODE FOR CARRYING OUT THE INVENTION

[0381] The present invention is a system for improving worker safety and efficiency in a factory robotic operating environment, which combines bionic sensors, a means for pre-processing motion data, generative artificial intelligence, an emotion engine for recognizing user emotions, and a means for notifying the user.

[0382] System Overview

[0383] This system consists of the following components:

[0384] Bionic sensors: Detect the user's physical movements in real time and transmit the data to the device.

[0385] Device: Receives motion data and performs preprocessing such as noise removal and normalization. The preprocessed data is sent to the server.

[0386] Server: Analyzes the pre-processed data using generative artificial intelligence to detect inappropriate behavior and danger. The server also has an emotion engine that analyzes the user's facial expressions and voice.

[0387] Notification method: Based on the analysis results, instructions are given to the user through visual displays and audio notifications.

[0388] Operation flow

[0389] First, a bionic sensor detects the user's movements in real time. Next, the device preprocesses the movement data received from the sensor, removing noise and normalizing the data. This preprocessed data is sent to a server where a generative AI analyzes it. Based on the analysis results, inappropriate movements and dangers are detected. Furthermore, an emotion engine analyzes the user's facial expressions and voice fluctuations to recognize the user's emotional state. Finally, appropriate instructions are generated based on the movement analysis results by the generative AI and the emotion analysis results by the emotion engine, and the instructions are communicated to the user via notification means.

[0390] Hardware and Software

[0391] Bionic sensors: devices for detecting user movements

[0392] Terminal: A device that preprocesses data and sends it to the server.

[0393] Server: A server equipped with generative artificial intelligence and emotion engine using TensorFlow and Keras libraries.

[0394] Notification method: Uses the pyttsx3 library for audio output and the display device for visual display

[0395] Specific examples

[0396] Example 1: Detecting dangerous behavior

[0397] If a worker adopts an improper posture while using a robotic arm to lift a heavy object, a bionic sensor detects this. The data sent from the sensor is preprocessed on the terminal to remove noise and normalize it. It is then sent to the server, where generative artificial intelligence detects dangerous movements. At the same time, an emotion engine analyzes the worker's facial expressions and voice and recognizes that his emotional state is unstable. Based on this information, an instruction is generated: "It's dangerous. Please slowly correct your posture." This is communicated to the worker through visual displays and audio notifications.

[0398] Example 2: Fixing improper behavior

[0399] If a worker makes an inappropriate move when using the robotic arm to lift an object, and the emotion engine recognizes irritation from the worker's tone of voice, it generates a message saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the terminal. The notification is displayed on the screen and by voice.

[0400] This improves the safety and efficiency of workers in a factory robot operation environment, reduces psychological stress, and provides a comfortable working environment.

[0401] The system can use prompts such as "Dangerous. Please slowly correct your position" or "Inappropriate movement. Please calm down and try again."

[0402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0403] Processing Steps

[0404] Step 1:

[0405] Bionic sensors detect user movements in real time.

[0406] Input: User's physical movements

[0407] Output: Motion data (e.g., foot angle and position information)

[0408] Specific operation: The sensor detects motion data and sends it to the device.

[0409] Step 2:

[0410] The device preprocesses the motion data received from the sensors.

[0411] Input: Motion data received from sensors

[0412] Output: Preprocessed motion data

[0413] Specific operations: Denoise and normalize the data, then format it, and send the preprocessed data to the server.

[0414] Step 3:

[0415] The server analyzes the preprocessed data using generative artificial intelligence.

[0416] Input: Preprocessed motion data

[0417] Output: Results of behavior analysis (e.g., whether there is danger or not, detection of inappropriate behavior)

[0418] Specific behavior: A generative AI model analyzes the behavior data and evaluates the risk. The analysis results are saved.

[0419] Step 4:

[0420] The server analyzes the user's facial expressions and voice using an emotion engine.

[0421] Input: Video and audio data from the camera and microphone

[0422] Output: Sentiment analysis results (e.g., user's emotional state)

[0423] Specific operation: The emotion engine analyzes video and audio data to recognize the user's emotions. The analysis results are saved.

[0424] Step 5:

[0425] The server generates instructions for the user based on the results of motion analysis and emotion analysis.

[0426] Input: Motion analysis results and emotion analysis results

[0427] Output: Instructions (prompt)

[0428] Specific actions: Generative AI combines the results of motion analysis and emotion analysis to generate an appropriate prompt, such as "It's dangerous. Please slowly correct your posture."

[0429] Step 6:

[0430] The server sends instructions to the device, and the device notifies the user.

[0431] Input: Generated prompt statement

[0432] Output: User notification (visual and audio)

[0433] Specific operation: The server sends the generated instructions to the device, and the device communicates the instructions to the user through visual display and audio notification.

[0434] This allows users to receive proper instructions and psychological support regarding their actions in real time, improving safety and efficiency.

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0436] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0438] [Second embodiment]

[0439] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0440] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0441] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0443] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0445] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0446] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0447] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0450] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0451] The present invention provides a system that combines bionic sensors, motion data preprocessing means, generative artificial intelligence, and user notification means to monitor, analyze, and provide appropriate instructions to the movements of physically disabled and elderly people using prosthetic limbs in real time.

[0452] System Overview

[0453] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies the user.

[0454] First, sensors detect the user's movements in real time. The sensors capture specific movements, such as walking or lifting, and transmit the data to a device. For example, a sensor collects information on the angle and position of the user's foot while walking with a prosthetic leg.

[0455] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data. The preprocessed data is then sent to a server and input into the generative artificial intelligence.

[0456] The server then analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects improper behavior and risk of falling. From this analysis, it generates instructions to provide to the user.

[0457] The generated instructions are sent from the server to the terminal, which provides the instructions to the user by visual display (e.g., on-screen text message) or audio notification.

[0458] Specific examples

[0459] Example 1: Falling Hazard

[0460] If a user's foot becomes unstable while walking using a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device preprocesses the data and sends it to the server. The server analyzes it using generative artificial intelligence and recognizes that there is a high risk of falling. As a result, an instruction is generated saying, "There is a risk of falling. Please correct your posture," and the user is notified via the device. The notification is displayed as a text message on the screen and a voice message is broadcast at the same time.

[0461] Example 2: Incorrect behavior

[0462] If a user makes an inappropriate movement when lifting an object with their prosthetic arm, a sensor detects this movement and sends the data to the device. The device removes noise, normalizes the data, and then sends it to the server. The server's generative artificial intelligence analyzes the data and detects that the movement is inappropriate. As a result, an instruction is generated saying, "The movement is inappropriate. Please check again," and the user is notified via the device. This is also communicated to the user via a screen display and voice message.

[0463] In this way, this system can significantly improve the quality of life for the physically disabled and elderly by analyzing the user's movements in real time and providing appropriate instructions quickly.

[0464] The processing flow will be explained below.

[0465] Step 1:

[0466] A user performs an action using a prosthetic arm or leg. For example, consider a situation in which a user walks with a prosthetic leg.

[0467] Step 2:

[0468] The sensors detect the user's movements in real time and collect specific movement data (for example, foot angle and position information over time).

[0469] Step 3:

[0470] The sensors collect and transmit motion data, including real-time location and angle data, to the device.

[0471] Step 4:

[0472] The device preprocesses the raw data received from the sensors, which includes denoising and normalizing the data, converting it into a format that is easier to analyze.

[0473] Step 5:

[0474] The device sends the preprocessed data to the server, where the data has been denoised and normalised.

[0475] Step 6:

[0476] The server feeds the preprocessed data into a generative artificial intelligence (AI) model that includes advanced learning algorithms such as recurrent neural networks.

[0477] Step 7:

[0478] The server uses generative artificial intelligence to analyze the movement data, detecting inappropriate movements and dangers from the data. For example, if a walking pattern differs from normal, it will be detected as an abnormality.

[0479] Step 8:

[0480] Based on the analysis results, the server generates instructions for the user, including specific content such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[0481] Step 9:

[0482] The server generates instructions and sends them to the terminal. The instructions may be in text or audio format.

[0483] Step 10:

[0484] The terminal notifies the user of the instructions received from the server. Notification methods include displaying a text message on the screen or an audio announcement. For example, the user may be shown a message saying "Danger of falling. Please correct your posture.", and simultaneously notified by an audio message.

[0485] Example 1

[0486] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0487] In technology that monitors the movements of physically disabled and elderly people using prosthetic limbs in real time and provides appropriate instructions quickly, current systems have issues with insufficient noise removal and normalization of movement data, making it difficult to obtain accurate analysis results.In addition, there are limited means of notifying users, which makes it difficult to provide sufficient warnings in certain environments.

[0488] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0489] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for removing noise from the motion data and normalizing the motion data, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect inappropriate motions or dangers, and means for providing instructions to the user by visual display or audio notification based on the analysis results of the generative artificial intelligence, thereby enabling more accurate analysis of the user's motions in real time and providing more appropriate and prompt instructions.

[0490] A "bionic sensor" is a device that detects the movement or physiological state of a living organism and acquires this information as data.

[0491] "Motion data" refers to data that includes information about specific movements performed by a user, such as walking or lifting an object, recorded in real time.

[0492] "Preprocessing means" refers to the process of processing raw data obtained from a sensor, such as by removing noise and normalizing it, and converting it into a format that is easy to analyze.

[0493] The "means for removing noise and normalizing" refers to a process for removing unnecessary fluctuations and outliers contained in the operational data and converting the data to a uniform scale.

[0494] "Generative AI" is a type of AI that has the ability to analyze the data it receives and generate new instructions or results.

[0495] A "recurrent neural network" is a neural network model used to analyze time-series data, and is capable of storing past information and using it in current analysis.

[0496] "Means for giving instructions to the user based on the analysis results" refers to a means for issuing instructions to the user to encourage appropriate actions or corrections based on the results of the generative artificial intelligence's analysis of data.

[0497] "Visual display" refers to a method of conveying analytical results or instructions to the user using text or graphics, typically displayed as a message on a screen.

[0498] "Voice notification" is a method of conveying analysis results and instructions to the user by voice, and is provided as a voice message through a speaker or headset.

[0499] The present invention relates to a system that combines bionic sensors, pre-processing means, generative artificial intelligence, and user notification means, and is capable of monitoring, analyzing, and providing appropriate instructions in real time to the movements of physically disabled and elderly people when using prosthetic limbs.

[0500] This system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies users. Specific examples of use are shown below.

[0501] Hardware and software configuration and data processing

[0502] 1. Sensor

[0503] Hardware name: Accelerometer, Gyroscope, etc.

[0504] Role: Detects user behavior in real time and sends that data to the device.

[0505] Example: Collecting foot angle and position information while a user walks with a prosthetic leg.

[0506] 2. Terminal

[0507] Software name: Python script, NumPy, SciPy

[0508] Role: Preprocesses the motion data received from the sensors, specifically removing noise and normalizing the data.

[0509] Example: Data preprocessing removes unwanted shaking and blurring from walking data to generate a standardized walking pattern.

[0510] 3. Server

[0511] Software name: TensorFlow, Keras

[0512] Role: Analyzes pre-processed data and uses generative artificial intelligence models (e.g., recurrent neural network models) to detect improper behavior or dangers and generate appropriate instructions.

[0513] Example: Using a recurrent neural network to analyze a user's walking pattern in real time and detect the risk of falling.

[0514] 4. Means of notification

[0515] Software name: Android app, iOS app

[0516] Role: Notifies the user of instructions sent by the server. Notifications include visual indications (text messages on the screen) and audio notifications.

[0517] Example: A smartphone app displays the message "Risk of falling. Please correct your posture." on the screen and also announces the same message in voice.

[0518] Specific examples

[0519] Example 1: Falling Hazard

[0520] User: Walking with a prosthetic leg

[0521] Sensor: Detects unstable foot movement

[0522] Terminal: Preprocessing the data, removing noise, and normalizing the data

[0523] Server: Performs analysis and detects risk of falling. Generates instruction such as "Danger of falling. Please correct your posture."

[0524] Notification method: Notify the user via the device. Uses screen display and voice message.

[0525] Example 2: Incorrect behavior

[0526] User: Lifting an object with a prosthetic arm

[0527] Sensors: Detect inappropriate behavior

[0528] Terminal: Data preprocessing, noise removal and data normalization

[0529] Server: Analyzes data and detects inappropriate behavior. Generates an instruction saying, "The movement is inappropriate. Please check again."

[0530] Notification method: Notify the user via the device. Uses screen display and voice message.

[0531] Prompt Sentence Examples

[0532] The system analyzes the user's movement data and notifies them in real time of inappropriate movements or dangers, generating instructions such as, "Unstable foot movements were detected while walking with a prosthetic leg. There is a risk of falling, so please correct your posture."

[0533] This system monitors the user's actions in real time through a series of processes using the above-mentioned hardware and software, and provides appropriate instructions to ensure safety.

[0534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0535] Step 1:

[0536] The user performs a movement using the prosthetic limb.

[0537] Input: User actions (walking, lifting, etc.)

[0538] Specific action: The user walks using the prosthetic leg.

[0539] Output: Movement data (foot angle, position, acceleration, etc.)

[0540] Step 2:

[0541] The sensor detects the user's movements and transmits the data to the device.

[0542] Input: Real-time user actions

[0543] Specific movements: Bionic sensors capture and generate data on user movements in real time.

[0544] Output: Raw data (unprocessed motion data)

[0545] Step 3:

[0546] The device preprocesses the raw data received from the sensors.

[0547] Input: Raw data sent from the sensor

[0548] What it does: The terminal script performs noise reduction and data normalization, e.g., removes outliers and converts the data to a consistent scale.

[0549] Output: Preprocessed data (denoised and normalized data)

[0550] Step 4:

[0551] The terminal sends the preprocessed data to the server.

[0552] Input: Preprocessed data

[0553] Specific operation: The device sends preprocessed data to the server's API endpoint.

[0554] Output: Preprocessed data sent to the server

[0555] Step 5:

[0556] The server analyzes the received pre-processed data.

[0557] Input: Preprocessed data

[0558] Specific operation: The server uses TensorFlow and Keras to analyze data with a recurrent neural network model to identify specific patterns and detect inappropriate behavior or danger.

[0559] Output: Analysis results (behavior evaluation, risk detection results)

[0560] Step 6:

[0561] The server generates appropriate instructions based on the analysis results.

[0562] Input: Analysis results

[0563] Specific operation: The server uses a generative AI model to generate instructions for the detected problem, for example, creating a message such as "There is a risk of falling. Please correct your posture."

[0564] Output: Instruction message (specific instructions to the user)

[0565] Step 7:

[0566] The server sends the generated instructions to the terminal.

[0567] Input: prompt message

[0568] Specific operation: The server's API sends an instruction message to the device.

[0569] Output: Instruction message sent to the terminal

[0570] Step 8:

[0571] The terminal notifies the user of the instruction received from the server.

[0572] Input: prompt message

[0573] What it does: The device displays a message on the screen and gives a voice notification. For example, a smartphone displays a text message and a voice message saying, "Danger of falling. Please correct your posture."

[0574] Output: User notification (visual and audio instructions)

[0575] Step 9:

[0576] The user acknowledges the notification and corrects the behavior.

[0577] Input: Notification message

[0578] Specific action: The user checks the smartphone screen or voice message and adjusts their walking pace, for example.

[0579] Output: Improved behavior (corrected walking pattern)

[0580] (Application example 1)

[0581] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] In autonomous vehicles, if the driver's health condition is not properly monitored in real time, the possibility of dangerous situations occurring increases. Therefore, there is a need for a means to monitor the driver's biological information in real time, quickly detect abnormalities, and provide appropriate instructions. In particular, it is necessary to immediately detect situations such as a sudden increase in the driver's heart rate while driving or delayed reactions due to accumulated fatigue, thereby improving safety.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0584] In this invention, the server includes means for preprocessing motion data received from the biometric sensor, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect abnormal motion or danger, means for providing instructions to a user based on the analysis results of the generative artificial intelligence, means for monitoring biometric information of the driver of the autonomous vehicle in real time, and means for detecting and notifying an abnormality from the driver's biometric information, thereby making it possible to monitor the driver's biometric information in real time and quickly detect and notify an abnormality.

[0585] A "bionic sensor" is a sensor device for measuring physiological activity or conditions of the human body.

[0586] A "pre-processing means" is a processing system for converting raw data received from a sensor into a format that is easy to analyze.

[0587] "Generative AI" is an AI system that has the ability to generate new insights based on data and provide instructions for specific purposes.

[0588] A "recurrent neural network" is a deep learning model used to capture the characteristics of time series data.

[0589] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[0590] "Driver's biometric information" refers to data that indicates the driver's physical condition and state, such as heart rate, skin temperature, and muscle movement.

[0591] "Means for real-time monitoring" refers to a system that instantly analyzes information obtained from sensors and monitors the driver's condition.

[0592] The "means for detecting and notifying abnormalities" is a system that issues an alert when an abnormality is detected based on the driver's biometric information.

[0593] This invention is a system for monitoring the biological information of a driver of an autonomous vehicle in real time and quickly detecting and notifying an abnormality. A specific embodiment of the system is described below.

[0594] System Configuration

[0595] The system includes the following components:

[0596] 1. Bionic sensors: Sensor devices that measure the driver's heart rate, skin temperature, muscle movement, etc.

[0597] 2. Terminal: A device that processes data received from sensors.

[0598] 3. Server: A computer with generative artificial intelligence to analyze incoming data and generate appropriate instructions.

[0599] 4. Notification means: Visual display device and audio output device to notify the driver of abnormalities.

[0600] Operation overview

[0601] 1. Data collection: When a driver gets into an autonomous vehicle, biometric sensors measure the driver's vital signs in real time and transmit the data to a terminal.

[0602] 2. Data preprocessing: The terminal performs preprocessing on the received data, such as noise removal and normalization, to convert it into a format suitable for analysis.

[0603] 3. Data analysis: The server analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects driver anomalies.

[0604] 4. Notification: If an abnormality is detected, the server generates appropriate instructions and notifies the driver visually or audibly through the terminal.

[0605] Hardware and software used

[0606] Biometric sensors: heart rate monitors, skin thermometers, electromyography sensors, etc.

[0607] Data Preprocessor: A computer / device that performs data denoising and normalization.

[0608] Generative artificial intelligence: Uses deep learning frameworks such as TensorFlow.

[0609] Notification Device: A display, speaker, or any type of alert device.

[0610] Specific scenes and prompts

[0611] 1. Example:

[0612] If a driver's heart rate suddenly rises while driving and the recurrent neural network model determines this to be an abnormality, the system will notify the driver with a message saying, "An abnormality has been detected. Please take a break immediately."

[0613] 2. Example prompt:

[0614] "Predict the probability that the driver's heart rate will reach an abnormal value and notify the driver of the abnormality."

[0615] effect

[0616] This system will ensure the safety of drivers of self-driving vehicles and enable rapid response in emergencies, thereby improving the safety and reliability of self-driving vehicles.

[0617] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0618] Step 1:

[0619] Data collection

[0620] When a user gets into an autonomous vehicle, bionic sensors measure their vital signs, such as heart rate, skin temperature, and muscle activity, in real time, and transmit this data to a device.

[0621] Input: Biometric data from biometric sensors

[0622] Output: Raw data sent to the terminal

[0623] Step 2:

[0624] Data Preprocessing

[0625] The device receives the received biometric data and performs noise reduction and normalization on it. Noise reduction removes unnecessary parts of the data, and normalization converts the data into a format suitable for analysis, allowing for smoother analysis.

[0626] Input: Raw data received from the sensor

[0627] Output: Preprocessed data

[0628] Step 3:

[0629] Data analysis

[0630] The preprocessed data is sent from the device to a server, where it is analyzed by a generative artificial intelligence (e.g., a recurrent neural network model). The purpose of the analysis is to detect abnormalities in the driver's biometric information.

[0631] Input: Preprocessed data

[0632] Output: Analysis results (whether anomalies were detected)

[0633] Step 4:

[0634] Generate notifications

[0635] The server generates a notification to the driver based on the analysis results. For example, if the driver's heart rate suddenly rises, it generates a message saying, "An abnormality has been detected. Please take a break immediately."

[0636] Input: Analysis results

[0637] Output: Notification content (message)

[0638] Step 5:

[0639] Sending notifications

[0640] The generated notification is sent to the terminal, which then notifies the driver visually or audibly. Specifically, it is possible to display a message on the display and play an audio message through the speaker.

[0641] Input: Notification content

[0642] Output: Notification to the driver (visual and audio)

[0643] In this way, the system can ensure safety by monitoring the driver's biometric information in real time and immediately notifying the driver if any abnormalities are detected.

[0644] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0645] The present invention provides a system that combines bionic sensors, a means for preprocessing motion data, generative artificial intelligence, a means for notifying users, and an emotion engine that recognizes users' emotions. This system not only monitors and analyzes the motions of physically disabled or elderly people using prosthetic limbs in real time and provides appropriate instructions, but also takes into account the user's emotional state to reduce the psychological burden.

[0646] System Overview

[0647] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[0648] Operation flow

[0649] First, the sensor detects the user's movements in real time. For example, when the user walks with a prosthetic leg, the sensor collects information on the angle and position of the walking leg. The sensor then transmits this movement data to the device.

[0650] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data, and then sends the preprocessed data to the server.

[0651] On the server, generative AI analyzes the preprocessed data and detects inappropriate behavior or danger. Based on the analysis results, instructions are generated for the user. For example, "There is a risk of falling. Please correct your posture."

[0652] The generated instructions are sent from the server to the device, which then communicates the instructions to the user through visual displays and audio notifications, allowing the user to receive appropriate advice in real time.

[0653] Emotion Engine Operation

[0654] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. The emotion engine utilizes data obtained from sensors, cameras, and microphones. The results of emotion recognition are added as additional information to the behavior analysis results of generative AI.

[0655] Based on the results of the emotion engine, the instructions to the user can be adjusted accordingly. For example, if the user feels anxious about the risk of falling, a message such as "Please walk slowly and calmly" will be generated to reduce the psychological burden.

[0656] Specific examples

[0657] Example 1: Falling Hazard

[0658] If a user makes unsteady movements while walking with a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device then preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes it and detects the risk of falling. If the emotion engine recognizes anxiety from the user's facial expression, it generates an instruction saying, "There is a risk of falling. Please slowly correct your posture," and notifies the user via the device. The notification is displayed on the screen and by voice.

[0659] Example 2: Incorrect behavior

[0660] If a user makes an inappropriate movement when lifting an object with their prosthetic hand, a sensor detects this movement and sends the data to the device. The device preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes the data and detects the inappropriate movement. Furthermore, if the emotion engine recognizes irritation from the user's tone of voice, it generates an instruction saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the device.

[0661] In this way, the system can detect and analyze the user's movements and emotions in real time, and provide appropriate instructions and psychological support, further improving the quality of life for the physically disabled and elderly.

[0662] The processing flow will be explained below.

[0663] Step 1:

[0664] The user performs an action using a prosthetic arm or leg, for example, the user starts walking with the prosthetic leg.

[0665] Step 2:

[0666] Sensors detect the user's movements in real time and collect movement data, such as the angle and position of the feet while walking.

[0667] Step 3:

[0668] The sensors collect motion data and transmit it to the device, including real-time location and angle data.

[0669] Step 4:

[0670] The device preprocesses the raw data received from the sensors, specifically removing noise from the data and normalizing it, making the data easier to analyze.

[0671] Step 5:

[0672] The device sends the preprocessed data to the server, where it is in a denoised and normalized form.

[0673] Step 6:

[0674] The server feeds the preprocessed data into a generative artificial intelligence (AI) model, which uses advanced algorithms such as recurrent neural networks.

[0675] Step 7:

[0676] The server uses generative artificial intelligence to analyze the movement data and detect inappropriate movements or dangers. For example, if the user's walking pattern differs from normal, it will be detected as an abnormality.

[0677] Step 8:

[0678] Based on the analysis results, the server generates instructions for the user, such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[0679] Step 9:

[0680] The server generates and sends the instructions to the device, which may be in the form of a text or audio file.

[0681] Step 10:

[0682] The terminal notifies the user of the instruction received from the server, for example by displaying a message on the screen and playing a voice message.

[0683] Step 11:

[0684] The device collects the user's facial expressions and tone of voice and inputs them into the emotion engine as emotional data. For example, the device collects the user's reactions through a camera or microphone.

[0685] Step 12:

[0686] The server's emotion engine analyzes the emotion data and recognizes the user's emotional state, for example, determining whether the user is tense or relaxed.

[0687] Step 13:

[0688] Based on the analysis results of the emotion engine, the server adjusts the instructions to correspond to the user's psychological state. For example, if the user is feeling anxious, it generates instructions that give a sense of security.

[0689] Step 14:

[0690] The server then sends the adjusted instructions back to the device, and the device notifies the user. For example, the device displays and speaks the instruction "Please walk slowly and calmly."

[0691] Example 2

[0692] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] Conventional motion assistance systems can monitor a user's motion and provide necessary instructions, but they cannot consider the user's real-time emotional state and lack the ability to reduce psychological stress. This has led to problems such as increased psychological stress when physically disabled people and elderly people use prosthetic limbs. Therefore, the problem that the present invention aims to solve is to reduce psychological stress by recognizing the user's emotional state in real time and adjusting instructions based on that.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0695] In this invention, the server includes a device for preprocessing motion data received from a bionic sensor, a device using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motions or dangers, a device for giving instructions to the user based on the analysis results of the generative artificial intelligence, a device for recognizing the emotional state of the user by analyzing the user's facial expressions and tone of voice, a device for adjusting the instructions taking the emotional state into consideration, and a device for notifying the user of the instructions, thereby making it possible to monitor the user's motions and emotions in real time and provide appropriate instructions and psychological support at the same time.

[0696] A "bionic sensor" is a device that detects and measures human movement and physiological data in real time.

[0697] "Motion data" refers to information about the movements, position information, angle information, and other movements when a user uses a prosthetic arm or leg.

[0698] "Preprocessing" is the process of converting raw data received from sensors into a format suitable for analysis by performing processes such as noise removal and normalization.

[0699] "Generative AI" is a machine learning model that analyzes input data and learns and predicts specific patterns and trends. It often uses advanced algorithms, such as recurrent neural networks.

[0700] "Analysis results" are information obtained by generative artificial intelligence analyzing behavioral data, and indicate whether or not there is inappropriate behavior or danger.

[0701] A "device that gives instructions to the user" is a device that provides specific instructions and advice to the user based on the analysis results of generative artificial intelligence.

[0702] "Emotional state" refers to the user's current psychological state and is analyzed based on data such as facial expressions and tone of voice.

[0703] An "emotion engine" is software or algorithm that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[0704] "Adjusting the content of instructions" means changing the content of the instructions provided based on the analysis results of the emotion engine, taking into account the user's psychological state.

[0705] "Notification" refers to the act of conveying analysis results or instructions to the user visually or audibly.

[0706] "Real-time" is a term that indicates that sensors and systems instantly recognize the user's actions and emotional state, and then analyze and respond immediately.

[0707] This system combines bionic sensors, a motion data preprocessing device, generative artificial intelligence, a user notification device, and an emotion engine. The system aims to monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, provide appropriate instructions, and reduce the psychological burden by taking into account the user's emotional state.

[0708] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[0709] Hardware and software used

[0710] Sensors: Accelerometer, Gyroscope, Camera, Microphone

[0711] Devices: Mobile devices, tablets, wearable devices

[0712] Server: High-performance computing server, cloud computing platform

[0713] Generative AI model: Recurrent Neural Network (RNN)

[0714] Emotion engine: facial expression recognition algorithm, voice tone analysis algorithm

[0715] Data processing and calculation

[0716] 1. Sensor operation data collection

[0717] As the user uses the prosthetic limb, bionic sensors collect real-time motion data, capturing acceleration, angle, and position information at high frequency, which is then transmitted to a device.

[0718] 2. Preprocessing by the terminal

[0719] The device performs noise reduction and data normalization on the raw data received from the sensors. Denoising is done to remove outliers and unnecessary data, and normalization is done to put the data into a consistent format, making it suitable for analysis by generative artificial intelligence.

[0720] 3. Data analysis by the server

[0721] The preprocessed data is sent to a server and input into a generative artificial intelligence (RNN model). The model analyzes the user's movements to detect inappropriate behavior and danger. For example, it can predict the risk of falling based on the movement of the feet while walking.

[0722] 4. Emotion Recognition by Emotion Engine

[0723] The server's emotion engine uses data acquired from the camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state in real time, for example, detecting anxiety or irritation.

[0724] 5. Instruction Generation and Notification

[0725] Based on the analysis results of the generative AI and the recognition results of the emotion engine, the server generates instructions to give to the user. These instructions are sent to the device in real time, and the device conveys the instructions to the user through visual displays and audio notifications, allowing the user to take appropriate action immediately.

[0726] Specific examples

[0727] Example 1: Falling Hazard

[0728] "Prompt: The user is walking with a prosthetic leg and has become unsteady on their feet. The sensor detects this information and sends it to the device. The generative AI analyzes this data and detects the risk of falling. If anxiety is recognized from the user's facial expression, generate appropriate instructions."

[0729] An example of a generated instruction might be: "Danger of falling. Please slowly regain your balance."

[0730] Example 2: Incorrect behavior

[0731] "Prompt: The user performed an inappropriate action while lifting an object with the prosthetic arm. The sensor detected this information and sent it to the device. The generative AI analyzed this data and detected the inappropriate action. If frustration is detected in the user's tone of voice, generate appropriate instructions."

[0732] An example of a generated instruction would be: "Inappropriate behavior. Please calm down and try again."

[0733] By using this system, users can receive appropriate instructions and psychological support in real time, thereby improving the quality of life for the physically disabled and elderly.

[0734] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0735] Step 1:

[0736] Sensors collect user movement data in real time.

[0737] Input: The user's prosthetic arm or leg movements

[0738] Movement: Accelerometer and gyroscope sensors measure your movement at a rate of 50 times per second.

[0739] Output: Raw data (foot angle, position information, etc.)

[0740] Step 2:

[0741] The device preprocesses the motion data received from the sensors.

[0742] Input: Raw data from the sensor

[0743] What it does: It removes noise and normalizes the data, eliminating outliers and scaling the data range to 0-1.

[0744] Output: Preprocessed data

[0745] Step 3:

[0746] The device sends the preprocessed data to the server.

[0747] Input: Preprocessed data

[0748] How it works: Sends data using an HTTP POST request.

[0749] Output: Preprocessed data arriving at the server

[0750] Step 4:

[0751] The server analyzes the preprocessed data using generative artificial intelligence.

[0752] Input: Preprocessed data

[0753] Behavior: Data is fed into a generative artificial intelligence (RNN model) to detect inappropriate behavior and danger.

[0754] Output: Analysis results (e.g., whether there is a risk of falling)

[0755] Step 5:

[0756] The server's emotion engine recognizes the user's emotional state in real time.

[0757] Input: Data from camera and microphone (facial expressions, tone of voice)

[0758] How it works: Recognizes emotional states using facial expression recognition and audio tone analysis algorithms.

[0759] Output: Emotion recognition result (e.g., anxiety, irritation)

[0760] Step 6:

[0761] The server generates instructions based on the analysis results and emotion recognition results.

[0762] Input: Analysis results, emotion recognition results

[0763] Action: The analysis results are combined with the emotion recognition results to generate instructions such as "There is a risk of falling. Please slowly correct your posture."

[0764] Output: Generated instructions

[0765] Step 7:

[0766] The server sends the generated instructions to the terminal.

[0767] Input: Generated instructions

[0768] Operation: Sends instructions to the device in real time using WebSocket.

[0769] Output: Instructions that arrive at the terminal

[0770] Step 8:

[0771] The device notifies the user of the instructions.

[0772] Input: Instructions that arrive at the terminal

[0773] Operation: Instructions are displayed on the device screen and communicated aloud through the speaker.

[0774] Output: Instructions to the user

[0775] In this way, the processing flow of the entire system is realized by a chain of specific operations and their inputs and outputs at each step.

[0776] (Application example 2)

[0777] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] While conventional systems can monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, they are unable to provide psychological support based on the user's emotional state. Furthermore, particularly in factory robot operating environments, there is a need for emotion analysis and appropriate instructions to reduce worker stress and anxiety, in addition to the ability to detect inappropriate movements and dangers.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0780] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motion or danger, means for using an emotion engine to analyze emotional information such as the user's facial expression and voice, and means for providing instructions to the user based on the results of the motion analysis by the generative artificial intelligence and the results of the emotion analysis by the emotion engine. This not only improves the safety and efficiency of workers in the operating environment of factory robots, but also reduces the psychological burden on workers and provides a comfortable working environment.

[0781] definition statement

[0782] A "bionic sensor" is a device that detects a user's physical movements and physiological information in real time and collects them as data.

[0783] The "preprocessing means" refers to a processing method for removing noise and normalizing the raw data received from the sensor and converting it into a format suitable for analysis.

[0784] "Generative AI" is an AI technology that learns from large amounts of data and analyzes and judges specific actions and situations.

[0785] The "emotion engine" is a system that analyzes changes in a user's facial expressions and voice to determine their emotional state in real time.

[0786] "Means for providing instructions" refers to a method for notifying the user of information that prompts the user to take appropriate action based on the analysis results, including visual displays and audio notifications.

[0787] A "terminal" is a device that acts as an interface with sensors and users, and is responsible for preprocessing and notifying data.

[0788] "Using a model" means using a mathematical or computational method to analyze data based on a particular calculation method or algorithm.

[0789] "Detecting real-time actions" means instantly detecting a user's actions and acquiring them as data without delay.

[0790] MODE FOR CARRYING OUT THE INVENTION

[0791] The present invention is a system for improving worker safety and efficiency in a factory robotic operating environment, which combines bionic sensors, a means for pre-processing motion data, generative artificial intelligence, an emotion engine for recognizing user emotions, and a means for notifying the user.

[0792] System Overview

[0793] This system consists of the following components:

[0794] Bionic sensors: Detect the user's physical movements in real time and transmit the data to the device.

[0795] Device: Receives motion data and performs preprocessing such as noise removal and normalization. The preprocessed data is sent to the server.

[0796] Server: Analyzes the pre-processed data using generative artificial intelligence to detect inappropriate behavior and danger. The server also has an emotion engine that analyzes the user's facial expressions and voice.

[0797] Notification method: Based on the analysis results, instructions are given to the user through visual displays and audio notifications.

[0798] Operation flow

[0799] First, a bionic sensor detects the user's movements in real time. Next, the device preprocesses the movement data received from the sensor, removing noise and normalizing the data. This preprocessed data is sent to a server where a generative AI analyzes it. Based on the analysis results, inappropriate movements and dangers are detected. Furthermore, an emotion engine analyzes the user's facial expressions and voice fluctuations to recognize the user's emotional state. Finally, appropriate instructions are generated based on the movement analysis results by the generative AI and the emotion analysis results by the emotion engine, and the instructions are communicated to the user via notification means.

[0800] Hardware and Software

[0801] Bionic sensors: devices for detecting user movements

[0802] Terminal: A device that preprocesses data and sends it to the server.

[0803] Server: A server equipped with generative artificial intelligence and emotion engine using TensorFlow and Keras libraries.

[0804] Notification method: Uses the pyttsx3 library for audio output and the display device for visual display

[0805] Specific examples

[0806] Example 1: Detecting dangerous behavior

[0807] If a worker adopts an improper posture while using a robotic arm to lift a heavy object, a bionic sensor detects this. The data sent from the sensor is preprocessed on the terminal to remove noise and normalize it. It is then sent to the server, where generative artificial intelligence detects dangerous movements. At the same time, an emotion engine analyzes the worker's facial expressions and voice and recognizes that his emotional state is unstable. Based on this information, an instruction is generated: "It's dangerous. Please slowly correct your posture." This is communicated to the worker through visual displays and audio notifications.

[0808] Example 2: Fixing improper behavior

[0809] If a worker makes an inappropriate move when using the robotic arm to lift an object, and the emotion engine recognizes irritation from the worker's tone of voice, it generates a message saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the terminal. The notification is displayed on the screen and by voice.

[0810] This improves the safety and efficiency of workers in a factory robot operation environment, reduces psychological stress, and provides a comfortable working environment.

[0811] The system can use prompts such as "Dangerous. Please slowly correct your position" or "Inappropriate movement. Please calm down and try again."

[0812] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0813] Processing Steps

[0814] Step 1:

[0815] Bionic sensors detect user movements in real time.

[0816] Input: User's physical movements

[0817] Output: Motion data (e.g., foot angle and position information)

[0818] Specific operation: The sensor detects motion data and sends it to the device.

[0819] Step 2:

[0820] The device preprocesses the motion data received from the sensors.

[0821] Input: Motion data received from sensors

[0822] Output: Preprocessed motion data

[0823] Specific operations: Denoise and normalize the data, then format it, and send the preprocessed data to the server.

[0824] Step 3:

[0825] The server analyzes the preprocessed data using generative artificial intelligence.

[0826] Input: Preprocessed motion data

[0827] Output: Results of behavior analysis (e.g., whether there is danger or not, detection of inappropriate behavior)

[0828] Specific behavior: A generative AI model analyzes the behavior data and evaluates the risk. The analysis results are saved.

[0829] Step 4:

[0830] The server analyzes the user's facial expressions and voice using an emotion engine.

[0831] Input: Video and audio data from the camera and microphone

[0832] Output: Sentiment analysis results (e.g., user's emotional state)

[0833] Specific operation: The emotion engine analyzes video and audio data to recognize the user's emotions. The analysis results are saved.

[0834] Step 5:

[0835] The server generates instructions for the user based on the results of motion analysis and emotion analysis.

[0836] Input: Motion analysis results and emotion analysis results

[0837] Output: Instructions (prompt)

[0838] Specific actions: Generative AI combines the results of motion analysis and emotion analysis to generate an appropriate prompt, such as "It's dangerous. Please slowly correct your posture."

[0839] Step 6:

[0840] The server sends instructions to the device, and the device notifies the user.

[0841] Input: Generated prompt statement

[0842] Output: User notification (visual and audio)

[0843] Specific operation: The server sends the generated instructions to the device, and the device communicates the instructions to the user through visual display and audio notification.

[0844] This allows users to receive proper instructions and psychological support regarding their actions in real time, improving safety and efficiency.

[0845] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0846] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0847] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0848] [Third embodiment]

[0849] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0850] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0851] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0852] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0853] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0854] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0855] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0856] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0857] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0858] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0859] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0860] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0861] The present invention provides a system that combines bionic sensors, motion data preprocessing means, generative artificial intelligence, and user notification means to monitor, analyze, and provide appropriate instructions to the movements of physically disabled and elderly people using prosthetic limbs in real time.

[0862] System Overview

[0863] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies the user.

[0864] First, sensors detect the user's movements in real time. The sensors capture specific movements, such as walking or lifting, and transmit the data to a device. For example, a sensor collects information on the angle and position of the user's foot while walking with a prosthetic leg.

[0865] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data. The preprocessed data is then sent to a server and input into the generative artificial intelligence.

[0866] The server then analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects improper behavior and risk of falling. From this analysis, it generates instructions to provide to the user.

[0867] The generated instructions are sent from the server to the terminal, which provides the instructions to the user by visual display (e.g., on-screen text message) or audio notification.

[0868] Specific examples

[0869] Example 1: Falling Hazard

[0870] If a user's foot becomes unstable while walking using a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device preprocesses the data and sends it to the server. The server analyzes it using generative artificial intelligence and recognizes that there is a high risk of falling. As a result, an instruction is generated saying, "There is a risk of falling. Please correct your posture," and the user is notified via the device. The notification is displayed as a text message on the screen and a voice message is broadcast at the same time.

[0871] Example 2: Incorrect behavior

[0872] If a user makes an inappropriate movement when lifting an object with their prosthetic arm, a sensor detects this movement and sends the data to the device. The device removes noise, normalizes the data, and then sends it to the server. The server's generative artificial intelligence analyzes the data and detects that the movement is inappropriate. As a result, an instruction is generated saying, "The movement is inappropriate. Please check again," and the user is notified via the device. This is also communicated to the user via a screen display and voice message.

[0873] In this way, this system can significantly improve the quality of life for the physically disabled and elderly by analyzing the user's movements in real time and providing appropriate instructions quickly.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] A user performs an action using a prosthetic arm or leg. For example, consider a situation in which a user walks with a prosthetic leg.

[0877] Step 2:

[0878] The sensors detect the user's movements in real time and collect specific movement data (for example, foot angle and position information over time).

[0879] Step 3:

[0880] The sensors collect and transmit motion data, including real-time location and angle data, to the device.

[0881] Step 4:

[0882] The device preprocesses the raw data received from the sensors, which includes denoising and normalizing the data, converting it into a format that is easier to analyze.

[0883] Step 5:

[0884] The device sends the preprocessed data to the server, where the data has been denoised and normalised.

[0885] Step 6:

[0886] The server feeds the preprocessed data into a generative artificial intelligence (AI) model that includes advanced learning algorithms such as recurrent neural networks.

[0887] Step 7:

[0888] The server uses generative artificial intelligence to analyze the movement data, detecting inappropriate movements and dangers from the data. For example, if a walking pattern differs from normal, it will be detected as an abnormality.

[0889] Step 8:

[0890] Based on the analysis results, the server generates instructions for the user, including specific content such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[0891] Step 9:

[0892] The server generates instructions and sends them to the terminal. The instructions may be in text or audio format.

[0893] Step 10:

[0894] The terminal notifies the user of the instructions received from the server. Notification methods include displaying a text message on the screen or an audio announcement. For example, the user may be shown a message saying "Danger of falling. Please correct your posture.", and simultaneously notified by an audio message.

[0895] Example 1

[0896] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0897] In technology that monitors the movements of physically disabled and elderly people using prosthetic limbs in real time and provides appropriate instructions quickly, current systems have issues with insufficient noise removal and normalization of movement data, making it difficult to obtain accurate analysis results.In addition, there are limited means of notifying users, which makes it difficult to provide sufficient warnings in certain environments.

[0898] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0899] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for removing noise from the motion data and normalizing the motion data, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect inappropriate motions or dangers, and means for providing instructions to the user by visual display or audio notification based on the analysis results of the generative artificial intelligence, thereby enabling more accurate analysis of the user's motions in real time and providing more appropriate and prompt instructions.

[0900] A "bionic sensor" is a device that detects the movement or physiological state of a living organism and acquires this information as data.

[0901] "Motion data" refers to data that includes information about specific movements performed by a user, such as walking or lifting an object, recorded in real time.

[0902] "Preprocessing means" refers to the process of processing raw data obtained from a sensor, such as by removing noise and normalizing it, and converting it into a format that is easy to analyze.

[0903] The "means for removing noise and normalizing" refers to a process for removing unnecessary fluctuations and outliers contained in the operational data and converting the data to a uniform scale.

[0904] "Generative AI" is a type of AI that has the ability to analyze the data it receives and generate new instructions or results.

[0905] A "recurrent neural network" is a neural network model used to analyze time-series data, and is capable of storing past information and using it in current analysis.

[0906] "Means for giving instructions to the user based on the analysis results" refers to a means for issuing instructions to the user to encourage appropriate actions or corrections based on the results of the generative artificial intelligence's analysis of data.

[0907] "Visual display" refers to a method of conveying analytical results or instructions to the user using text or graphics, typically displayed as a message on a screen.

[0908] "Voice notification" is a method of conveying analysis results and instructions to the user by voice, and is provided as a voice message through a speaker or headset.

[0909] The present invention relates to a system that combines bionic sensors, pre-processing means, generative artificial intelligence, and user notification means, and is capable of monitoring, analyzing, and providing appropriate instructions in real time to the movements of physically disabled and elderly people when using prosthetic limbs.

[0910] This system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies users. Specific examples of use are shown below.

[0911] Hardware and software configuration and data processing

[0912] 1. Sensor

[0913] Hardware name: Accelerometer, Gyroscope, etc.

[0914] Role: Detects user behavior in real time and sends that data to the device.

[0915] Example: Collecting foot angle and position information while a user walks with a prosthetic leg.

[0916] 2. Terminal

[0917] Software name: Python script, NumPy, SciPy

[0918] Role: Preprocesses the motion data received from the sensors, specifically removing noise and normalizing the data.

[0919] Example: Data preprocessing removes unwanted shaking and blurring from walking data to generate a standardized walking pattern.

[0920] 3. Server

[0921] Software name: TensorFlow, Keras

[0922] Role: Analyzes pre-processed data and uses generative artificial intelligence models (e.g., recurrent neural network models) to detect improper behavior or dangers and generate appropriate instructions.

[0923] Example: Using a recurrent neural network to analyze a user's walking pattern in real time and detect the risk of falling.

[0924] 4. Means of notification

[0925] Software name: Android app, iOS app

[0926] Role: Notifies the user of instructions sent by the server. Notifications include visual indications (text messages on the screen) and audio notifications.

[0927] Example: A smartphone app displays the message "Risk of falling. Please correct your posture." on the screen and also announces the same message in voice.

[0928] Specific examples

[0929] Example 1: Falling Hazard

[0930] User: Walking with a prosthetic leg

[0931] Sensor: Detects unstable foot movement

[0932] Terminal: Preprocessing the data, removing noise, and normalizing the data

[0933] Server: Performs analysis and detects risk of falling. Generates instruction such as "Danger of falling. Please correct your posture."

[0934] Notification method: Notify the user via the device. Uses screen display and voice message.

[0935] Example 2: Incorrect behavior

[0936] User: Lifting an object with a prosthetic arm

[0937] Sensors: Detect inappropriate behavior

[0938] Terminal: Data preprocessing, noise removal and data normalization

[0939] Server: Analyzes data and detects inappropriate behavior. Generates an instruction saying, "The movement is inappropriate. Please check again."

[0940] Notification method: Notify the user via the device. Uses screen display and voice message.

[0941] Prompt Sentence Examples

[0942] The system analyzes the user's movement data and notifies them in real time of inappropriate movements or dangers, generating instructions such as, "Unstable foot movements were detected while walking with a prosthetic leg. There is a risk of falling, so please correct your posture."

[0943] This system monitors the user's actions in real time through a series of processes using the above-mentioned hardware and software, and provides appropriate instructions to ensure safety.

[0944] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0945] Step 1:

[0946] The user performs a movement using the prosthetic limb.

[0947] Input: User actions (walking, lifting, etc.)

[0948] Specific action: The user walks using the prosthetic leg.

[0949] Output: Movement data (foot angle, position, acceleration, etc.)

[0950] Step 2:

[0951] The sensor detects the user's movements and transmits the data to the device.

[0952] Input: Real-time user actions

[0953] Specific movements: Bionic sensors capture and generate data on user movements in real time.

[0954] Output: Raw data (unprocessed motion data)

[0955] Step 3:

[0956] The device preprocesses the raw data received from the sensors.

[0957] Input: Raw data sent from the sensor

[0958] What it does: The terminal script performs noise reduction and data normalization, e.g., removes outliers and converts the data to a consistent scale.

[0959] Output: Preprocessed data (denoised and normalized data)

[0960] Step 4:

[0961] The terminal sends the preprocessed data to the server.

[0962] Input: Preprocessed data

[0963] Specific operation: The device sends preprocessed data to the server's API endpoint.

[0964] Output: Preprocessed data sent to the server

[0965] Step 5:

[0966] The server analyzes the received pre-processed data.

[0967] Input: Preprocessed data

[0968] Specific operation: The server uses TensorFlow and Keras to analyze data with a recurrent neural network model to identify specific patterns and detect inappropriate behavior or danger.

[0969] Output: Analysis results (behavior evaluation, risk detection results)

[0970] Step 6:

[0971] The server generates appropriate instructions based on the analysis results.

[0972] Input: Analysis results

[0973] Specific operation: The server uses a generative AI model to generate instructions for the detected problem, for example, creating a message such as "There is a risk of falling. Please correct your posture."

[0974] Output: Instruction message (specific instructions to the user)

[0975] Step 7:

[0976] The server sends the generated instructions to the terminal.

[0977] Input: prompt message

[0978] Specific operation: The server's API sends an instruction message to the device.

[0979] Output: Instruction message sent to the terminal

[0980] Step 8:

[0981] The terminal notifies the user of the instruction received from the server.

[0982] Input: prompt message

[0983] What it does: The device displays a message on the screen and gives a voice notification. For example, a smartphone displays a text message and a voice message saying, "Danger of falling. Please correct your posture."

[0984] Output: User notification (visual and audio instructions)

[0985] Step 9:

[0986] The user acknowledges the notification and corrects the behavior.

[0987] Input: Notification message

[0988] Specific action: The user checks the smartphone screen or voice message and adjusts their walking pace, for example.

[0989] Output: Improved behavior (corrected walking pattern)

[0990] (Application example 1)

[0991] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0992] In autonomous vehicles, if the driver's health condition is not properly monitored in real time, the possibility of dangerous situations occurring increases. Therefore, there is a need for a means to monitor the driver's biological information in real time, quickly detect abnormalities, and provide appropriate instructions. In particular, it is necessary to immediately detect situations such as a sudden increase in the driver's heart rate while driving or delayed reactions due to accumulated fatigue, thereby improving safety.

[0993] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0994] In this invention, the server includes means for preprocessing motion data received from the biometric sensor, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect abnormal motion or danger, means for providing instructions to a user based on the analysis results of the generative artificial intelligence, means for monitoring biometric information of the driver of the autonomous vehicle in real time, and means for detecting and notifying an abnormality from the driver's biometric information, thereby making it possible to monitor the driver's biometric information in real time and quickly detect and notify an abnormality.

[0995] A "bionic sensor" is a sensor device for measuring physiological activity or conditions of the human body.

[0996] A "pre-processing means" is a processing system for converting raw data received from a sensor into a format that is easy to analyze.

[0997] "Generative AI" is an AI system that has the ability to generate new insights based on data and provide instructions for specific purposes.

[0998] A "recurrent neural network" is a deep learning model used to capture the characteristics of time series data.

[0999] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[1000] "Driver's biometric information" refers to data that indicates the driver's physical condition and state, such as heart rate, skin temperature, and muscle movement.

[1001] "Means for real-time monitoring" refers to a system that instantly analyzes information obtained from sensors and monitors the driver's condition.

[1002] The "means for detecting and notifying abnormalities" is a system that issues an alert when an abnormality is detected based on the driver's biometric information.

[1003] This invention is a system for monitoring the biological information of a driver of an autonomous vehicle in real time and quickly detecting and notifying an abnormality. A specific embodiment of the system is described below.

[1004] System Configuration

[1005] The system includes the following components:

[1006] 1. Bionic sensors: Sensor devices that measure the driver's heart rate, skin temperature, muscle movement, etc.

[1007] 2. Terminal: A device that processes data received from sensors.

[1008] 3. Server: A computer with generative artificial intelligence to analyze incoming data and generate appropriate instructions.

[1009] 4. Notification means: Visual display device and audio output device to notify the driver of abnormalities.

[1010] Operation overview

[1011] 1. Data collection: When a driver gets into an autonomous vehicle, biometric sensors measure the driver's vital signs in real time and transmit the data to a terminal.

[1012] 2. Data preprocessing: The terminal performs preprocessing on the received data, such as noise removal and normalization, to convert it into a format suitable for analysis.

[1013] 3. Data analysis: The server analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects driver anomalies.

[1014] 4. Notification: If an abnormality is detected, the server generates appropriate instructions and notifies the driver visually or audibly through the terminal.

[1015] Hardware and software used

[1016] Biometric sensors: heart rate monitors, skin thermometers, electromyography sensors, etc.

[1017] Data Preprocessor: A computer / device that performs data denoising and normalization.

[1018] Generative artificial intelligence: Uses deep learning frameworks such as TensorFlow.

[1019] Notification Device: A display, speaker, or any type of alert device.

[1020] Specific scenes and prompts

[1021] 1. Example:

[1022] If a driver's heart rate suddenly rises while driving and the recurrent neural network model determines this to be an abnormality, the system will notify the driver with a message saying, "An abnormality has been detected. Please take a break immediately."

[1023] 2. Example prompt:

[1024] "Predict the probability that the driver's heart rate will reach an abnormal value and notify the driver of the abnormality."

[1025] effect

[1026] This system will ensure the safety of drivers of self-driving vehicles and enable rapid response in emergencies, thereby improving the safety and reliability of self-driving vehicles.

[1027] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1028] Step 1:

[1029] Data collection

[1030] When a user gets into an autonomous vehicle, bionic sensors measure their vital signs, such as heart rate, skin temperature, and muscle activity, in real time, and transmit this data to a device.

[1031] Input: Biometric data from biometric sensors

[1032] Output: Raw data sent to the terminal

[1033] Step 2:

[1034] Data Preprocessing

[1035] The device receives the received biometric data and performs noise reduction and normalization on it. Noise reduction removes unnecessary parts of the data, and normalization converts the data into a format suitable for analysis, allowing for smoother analysis.

[1036] Input: Raw data received from the sensor

[1037] Output: Preprocessed data

[1038] Step 3:

[1039] Data analysis

[1040] The preprocessed data is sent from the device to a server, where it is analyzed by a generative artificial intelligence (e.g., a recurrent neural network model). The purpose of the analysis is to detect abnormalities in the driver's biometric information.

[1041] Input: Preprocessed data

[1042] Output: Analysis results (whether anomalies were detected)

[1043] Step 4:

[1044] Generate notifications

[1045] The server generates a notification to the driver based on the analysis results. For example, if the driver's heart rate suddenly rises, it generates a message saying, "An abnormality has been detected. Please take a break immediately."

[1046] Input: Analysis results

[1047] Output: Notification content (message)

[1048] Step 5:

[1049] Sending notifications

[1050] The generated notification is sent to the terminal, which then notifies the driver visually or audibly. Specifically, it is possible to display a message on the display and play an audio message through the speaker.

[1051] Input: Notification content

[1052] Output: Notification to the driver (visual and audio)

[1053] In this way, the system can ensure safety by monitoring the driver's biometric information in real time and immediately notifying the driver if any abnormalities are detected.

[1054] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1055] The present invention provides a system that combines bionic sensors, a means for preprocessing motion data, generative artificial intelligence, a means for notifying users, and an emotion engine that recognizes users' emotions. This system not only monitors and analyzes the motions of physically disabled or elderly people using prosthetic limbs in real time and provides appropriate instructions, but also takes into account the user's emotional state to reduce the psychological burden.

[1056] System Overview

[1057] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[1058] Operation flow

[1059] First, the sensor detects the user's movements in real time. For example, when the user walks with a prosthetic leg, the sensor collects information on the angle and position of the walking leg. The sensor then transmits this movement data to the device.

[1060] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data, and then sends the preprocessed data to the server.

[1061] On the server, generative AI analyzes the preprocessed data and detects inappropriate behavior or danger. Based on the analysis results, instructions are generated for the user. For example, "There is a risk of falling. Please correct your posture."

[1062] The generated instructions are sent from the server to the device, which then communicates the instructions to the user through visual displays and audio notifications, allowing the user to receive appropriate advice in real time.

[1063] Emotion Engine Operation

[1064] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. The emotion engine utilizes data obtained from sensors, cameras, and microphones. The results of emotion recognition are added as additional information to the behavior analysis results of generative AI.

[1065] Based on the results of the emotion engine, the instructions to the user can be adjusted accordingly. For example, if the user feels anxious about the risk of falling, a message such as "Please walk slowly and calmly" will be generated to reduce the psychological burden.

[1066] Specific examples

[1067] Example 1: Falling Hazard

[1068] If a user makes unsteady movements while walking with a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device then preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes it and detects the risk of falling. If the emotion engine recognizes anxiety from the user's facial expression, it generates an instruction saying, "There is a risk of falling. Please slowly correct your posture," and notifies the user via the device. The notification is displayed on the screen and by voice.

[1069] Example 2: Incorrect behavior

[1070] If a user makes an inappropriate movement when lifting an object with their prosthetic hand, a sensor detects this movement and sends the data to the device. The device preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes the data and detects the inappropriate movement. Furthermore, if the emotion engine recognizes irritation from the user's tone of voice, it generates an instruction saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the device.

[1071] In this way, the system can detect and analyze the user's movements and emotions in real time, and provide appropriate instructions and psychological support, further improving the quality of life for the physically disabled and elderly.

[1072] The processing flow will be explained below.

[1073] Step 1:

[1074] The user performs an action using a prosthetic arm or leg, for example, the user starts walking with the prosthetic leg.

[1075] Step 2:

[1076] Sensors detect the user's movements in real time and collect movement data, such as the angle and position of the feet while walking.

[1077] Step 3:

[1078] The sensors collect motion data and transmit it to the device, including real-time location and angle data.

[1079] Step 4:

[1080] The device preprocesses the raw data received from the sensors, specifically removing noise from the data and normalizing it, making the data easier to analyze.

[1081] Step 5:

[1082] The device sends the preprocessed data to the server, where it is in a denoised and normalized form.

[1083] Step 6:

[1084] The server feeds the preprocessed data into a generative artificial intelligence (AI) model, which uses advanced algorithms such as recurrent neural networks.

[1085] Step 7:

[1086] The server uses generative artificial intelligence to analyze the movement data and detect inappropriate movements or dangers. For example, if the user's walking pattern differs from normal, it will be detected as an abnormality.

[1087] Step 8:

[1088] Based on the analysis results, the server generates instructions for the user, such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[1089] Step 9:

[1090] The server generates and sends the instructions to the device, which may be in the form of a text or audio file.

[1091] Step 10:

[1092] The terminal notifies the user of the instruction received from the server, for example by displaying a message on the screen and playing a voice message.

[1093] Step 11:

[1094] The device collects the user's facial expressions and tone of voice and inputs them into the emotion engine as emotional data. For example, the device collects the user's reactions through a camera or microphone.

[1095] Step 12:

[1096] The server's emotion engine analyzes the emotion data and recognizes the user's emotional state, for example, determining whether the user is tense or relaxed.

[1097] Step 13:

[1098] Based on the analysis results of the emotion engine, the server adjusts the instructions to correspond to the user's psychological state. For example, if the user is feeling anxious, it generates instructions that give a sense of security.

[1099] Step 14:

[1100] The server then sends the adjusted instructions back to the device, and the device notifies the user. For example, the device displays and speaks the instruction "Please walk slowly and calmly."

[1101] Example 2

[1102] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1103] Conventional motion assistance systems can monitor a user's motion and provide necessary instructions, but they cannot consider the user's real-time emotional state and lack the ability to reduce psychological stress. This has led to problems such as increased psychological stress when physically disabled people and elderly people use prosthetic limbs. Therefore, the problem that the present invention aims to solve is to reduce psychological stress by recognizing the user's emotional state in real time and adjusting instructions based on that.

[1104] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1105] In this invention, the server includes a device for preprocessing motion data received from a bionic sensor, a device using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motions or dangers, a device for giving instructions to the user based on the analysis results of the generative artificial intelligence, a device for recognizing the emotional state of the user by analyzing the user's facial expressions and tone of voice, a device for adjusting the instructions taking the emotional state into consideration, and a device for notifying the user of the instructions, thereby making it possible to monitor the user's motions and emotions in real time and provide appropriate instructions and psychological support at the same time.

[1106] A "bionic sensor" is a device that detects and measures human movement and physiological data in real time.

[1107] "Motion data" refers to information about the movements, position information, angle information, and other movements when a user uses a prosthetic arm or leg.

[1108] "Preprocessing" is the process of converting raw data received from sensors into a format suitable for analysis by performing processes such as noise removal and normalization.

[1109] "Generative AI" is a machine learning model that analyzes input data and learns and predicts specific patterns and trends. It often uses advanced algorithms, such as recurrent neural networks.

[1110] "Analysis results" are information obtained by generative artificial intelligence analyzing behavioral data, and indicate whether or not there is inappropriate behavior or danger.

[1111] A "device that gives instructions to the user" is a device that provides specific instructions and advice to the user based on the analysis results of generative artificial intelligence.

[1112] "Emotional state" refers to the user's current psychological state and is analyzed based on data such as facial expressions and tone of voice.

[1113] An "emotion engine" is software or algorithm that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[1114] "Adjusting the content of instructions" means changing the content of the instructions provided based on the analysis results of the emotion engine, taking into account the user's psychological state.

[1115] "Notification" refers to the act of conveying analysis results or instructions to the user visually or audibly.

[1116] "Real-time" is a term that indicates that sensors and systems instantly recognize the user's actions and emotional state, and then analyze and respond immediately.

[1117] This system combines bionic sensors, a motion data preprocessing device, generative artificial intelligence, a user notification device, and an emotion engine. The system aims to monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, provide appropriate instructions, and reduce the psychological burden by taking into account the user's emotional state.

[1118] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[1119] Hardware and software used

[1120] Sensors: Accelerometer, Gyroscope, Camera, Microphone

[1121] Devices: Mobile devices, tablets, wearable devices

[1122] Server: High-performance computing server, cloud computing platform

[1123] Generative AI model: Recurrent Neural Network (RNN)

[1124] Emotion engine: facial expression recognition algorithm, voice tone analysis algorithm

[1125] Data processing and calculation

[1126] 1. Sensor operation data collection

[1127] As the user uses the prosthetic limb, bionic sensors collect real-time motion data, capturing acceleration, angle, and position information at high frequency, which is then transmitted to a device.

[1128] 2. Preprocessing by the terminal

[1129] The device performs noise reduction and data normalization on the raw data received from the sensors. Denoising is done to remove outliers and unnecessary data, and normalization is done to put the data into a consistent format, making it suitable for analysis by generative artificial intelligence.

[1130] 3. Data analysis by the server

[1131] The preprocessed data is sent to a server and input into a generative artificial intelligence (RNN model). The model analyzes the user's movements to detect inappropriate behavior and danger. For example, it can predict the risk of falling based on the movement of the feet while walking.

[1132] 4. Emotion Recognition by Emotion Engine

[1133] The server's emotion engine uses data acquired from the camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state in real time, for example, detecting anxiety or irritation.

[1134] 5. Instruction Generation and Notification

[1135] Based on the analysis results of the generative AI and the recognition results of the emotion engine, the server generates instructions to give to the user. These instructions are sent to the device in real time, and the device conveys the instructions to the user through visual displays and audio notifications, allowing the user to take appropriate action immediately.

[1136] Specific examples

[1137] Example 1: Falling Hazard

[1138] "Prompt: The user is walking with a prosthetic leg and has become unsteady on their feet. The sensor detects this information and sends it to the device. The generative AI analyzes this data and detects the risk of falling. If anxiety is recognized from the user's facial expression, generate appropriate instructions."

[1139] An example of a generated instruction might be: "Danger of falling. Please slowly regain your balance."

[1140] Example 2: Incorrect behavior

[1141] "Prompt: The user performed an inappropriate action while lifting an object with the prosthetic arm. The sensor detected this information and sent it to the device. The generative AI analyzed this data and detected the inappropriate action. If frustration is detected in the user's tone of voice, generate appropriate instructions."

[1142] An example of a generated instruction would be: "Inappropriate behavior. Please calm down and try again."

[1143] By using this system, users can receive appropriate instructions and psychological support in real time, thereby improving the quality of life for the physically disabled and elderly.

[1144] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1145] Step 1:

[1146] Sensors collect user movement data in real time.

[1147] Input: The user's prosthetic arm or leg movements

[1148] Movement: Accelerometer and gyroscope sensors measure your movement at a rate of 50 times per second.

[1149] Output: Raw data (foot angle, position information, etc.)

[1150] Step 2:

[1151] The device preprocesses the motion data received from the sensors.

[1152] Input: Raw data from the sensor

[1153] What it does: It removes noise and normalizes the data, eliminating outliers and scaling the data range to 0-1.

[1154] Output: Preprocessed data

[1155] Step 3:

[1156] The device sends the preprocessed data to the server.

[1157] Input: Preprocessed data

[1158] How it works: Sends data using an HTTP POST request.

[1159] Output: Preprocessed data arriving at the server

[1160] Step 4:

[1161] The server analyzes the preprocessed data using generative artificial intelligence.

[1162] Input: Preprocessed data

[1163] Behavior: Data is fed into a generative artificial intelligence (RNN model) to detect inappropriate behavior and danger.

[1164] Output: Analysis results (e.g., whether there is a risk of falling)

[1165] Step 5:

[1166] The server's emotion engine recognizes the user's emotional state in real time.

[1167] Input: Data from camera and microphone (facial expressions, tone of voice)

[1168] How it works: Recognizes emotional states using facial expression recognition and audio tone analysis algorithms.

[1169] Output: Emotion recognition result (e.g., anxiety, irritation)

[1170] Step 6:

[1171] The server generates instructions based on the analysis results and emotion recognition results.

[1172] Input: Analysis results, emotion recognition results

[1173] Action: The analysis results are combined with the emotion recognition results to generate instructions such as "There is a risk of falling. Please slowly correct your posture."

[1174] Output: Generated instructions

[1175] Step 7:

[1176] The server sends the generated instructions to the terminal.

[1177] Input: Generated instructions

[1178] Operation: Sends instructions to the device in real time using WebSocket.

[1179] Output: Instructions that arrive at the terminal

[1180] Step 8:

[1181] The device notifies the user of the instructions.

[1182] Input: Instructions that arrive at the terminal

[1183] Operation: Instructions are displayed on the device screen and communicated aloud through the speaker.

[1184] Output: Instructions to the user

[1185] In this way, the processing flow of the entire system is realized by a chain of specific operations and their inputs and outputs at each step.

[1186] (Application example 2)

[1187] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1188] While conventional systems can monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, they are unable to provide psychological support based on the user's emotional state. Furthermore, particularly in factory robot operating environments, there is a need for emotion analysis and appropriate instructions to reduce worker stress and anxiety, in addition to the ability to detect inappropriate movements and dangers.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1190] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motion or danger, means for using an emotion engine to analyze emotional information such as the user's facial expression and voice, and means for providing instructions to the user based on the results of the motion analysis by the generative artificial intelligence and the results of the emotion analysis by the emotion engine. This not only improves the safety and efficiency of workers in the operating environment of factory robots, but also reduces the psychological burden on workers and provides a comfortable working environment.

[1191] definition statement

[1192] A "bionic sensor" is a device that detects a user's physical movements and physiological information in real time and collects them as data.

[1193] The "preprocessing means" refers to a processing method for removing noise and normalizing the raw data received from the sensor and converting it into a format suitable for analysis.

[1194] "Generative AI" is an AI technology that learns from large amounts of data and analyzes and judges specific actions and situations.

[1195] The "emotion engine" is a system that analyzes changes in a user's facial expressions and voice to determine their emotional state in real time.

[1196] "Means for providing instructions" refers to a method for notifying the user of information that prompts the user to take appropriate action based on the analysis results, including visual displays and audio notifications.

[1197] A "terminal" is a device that acts as an interface with sensors and users, and is responsible for preprocessing and notifying data.

[1198] "Using a model" means using a mathematical or computational method to analyze data based on a particular calculation method or algorithm.

[1199] "Detecting real-time actions" means instantly detecting a user's actions and acquiring them as data without delay.

[1200] MODE FOR CARRYING OUT THE INVENTION

[1201] The present invention is a system for improving worker safety and efficiency in a factory robotic operating environment, which combines bionic sensors, a means for pre-processing motion data, generative artificial intelligence, an emotion engine for recognizing user emotions, and a means for notifying the user.

[1202] System Overview

[1203] This system consists of the following components:

[1204] Bionic sensors: Detect the user's physical movements in real time and transmit the data to the device.

[1205] Device: Receives motion data and performs preprocessing such as noise removal and normalization. The preprocessed data is sent to the server.

[1206] Server: Analyzes the pre-processed data using generative artificial intelligence to detect inappropriate behavior and danger. The server also has an emotion engine that analyzes the user's facial expressions and voice.

[1207] Notification method: Based on the analysis results, instructions are given to the user through visual displays and audio notifications.

[1208] Operation flow

[1209] First, a bionic sensor detects the user's movements in real time. Next, the device preprocesses the movement data received from the sensor, removing noise and normalizing the data. This preprocessed data is sent to a server where a generative AI analyzes it. Based on the analysis results, inappropriate movements and dangers are detected. Furthermore, an emotion engine analyzes the user's facial expressions and voice fluctuations to recognize the user's emotional state. Finally, appropriate instructions are generated based on the movement analysis results by the generative AI and the emotion analysis results by the emotion engine, and the instructions are communicated to the user via notification means.

[1210] Hardware and Software

[1211] Bionic sensors: devices for detecting user movements

[1212] Terminal: A device that preprocesses data and sends it to the server.

[1213] Server: A server equipped with generative artificial intelligence and emotion engine using TensorFlow and Keras libraries.

[1214] Notification method: Uses the pyttsx3 library for audio output and the display device for visual display

[1215] Specific examples

[1216] Example 1: Detecting dangerous behavior

[1217] If a worker adopts an improper posture while using a robotic arm to lift a heavy object, a bionic sensor detects this. The data sent from the sensor is preprocessed on the terminal to remove noise and normalize it. It is then sent to the server, where generative artificial intelligence detects dangerous movements. At the same time, an emotion engine analyzes the worker's facial expressions and voice and recognizes that his emotional state is unstable. Based on this information, an instruction is generated: "It's dangerous. Please slowly correct your posture." This is communicated to the worker through visual displays and audio notifications.

[1218] Example 2: Fixing improper behavior

[1219] If a worker makes an inappropriate move when using the robotic arm to lift an object, and the emotion engine recognizes irritation from the worker's tone of voice, it generates a message saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the terminal. The notification is displayed on the screen and by voice.

[1220] This improves the safety and efficiency of workers in a factory robot operation environment, reduces psychological stress, and provides a comfortable working environment.

[1221] The system can use prompts such as "Dangerous. Please slowly correct your position" or "Inappropriate movement. Please calm down and try again."

[1222] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1223] Processing Steps

[1224] Step 1:

[1225] Bionic sensors detect user movements in real time.

[1226] Input: User's physical movements

[1227] Output: Motion data (e.g., foot angle and position information)

[1228] Specific operation: The sensor detects motion data and sends it to the device.

[1229] Step 2:

[1230] The device preprocesses the motion data received from the sensors.

[1231] Input: Motion data received from sensors

[1232] Output: Preprocessed motion data

[1233] Specific operations: Denoise and normalize the data, then format it, and send the preprocessed data to the server.

[1234] Step 3:

[1235] The server analyzes the preprocessed data using generative artificial intelligence.

[1236] Input: Preprocessed motion data

[1237] Output: Results of behavior analysis (e.g., whether there is danger or not, detection of inappropriate behavior)

[1238] Specific behavior: A generative AI model analyzes the behavior data and evaluates the risk. The analysis results are saved.

[1239] Step 4:

[1240] The server analyzes the user's facial expressions and voice using an emotion engine.

[1241] Input: Video and audio data from the camera and microphone

[1242] Output: Sentiment analysis results (e.g., user's emotional state)

[1243] Specific operation: The emotion engine analyzes video and audio data to recognize the user's emotions. The analysis results are saved.

[1244] Step 5:

[1245] The server generates instructions for the user based on the results of motion analysis and emotion analysis.

[1246] Input: Motion analysis results and emotion analysis results

[1247] Output: Instructions (prompt)

[1248] Specific actions: Generative AI combines the results of motion analysis and emotion analysis to generate an appropriate prompt, such as "It's dangerous. Please slowly correct your posture."

[1249] Step 6:

[1250] The server sends instructions to the device, and the device notifies the user.

[1251] Input: Generated prompt statement

[1252] Output: User notification (visual and audio)

[1253] Specific operation: The server sends the generated instructions to the device, and the device communicates the instructions to the user through visual display and audio notification.

[1254] This allows users to receive proper instructions and psychological support regarding their actions in real time, improving safety and efficiency.

[1255] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1256] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1257] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1258] [Fourth embodiment]

[1259] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1260] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1261] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1262] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1263] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1264] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1265] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1266] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1267] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1268] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1269] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1270] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1271] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1272] The present invention provides a system that combines bionic sensors, motion data preprocessing means, generative artificial intelligence, and user notification means to monitor, analyze, and provide appropriate instructions to the movements of physically disabled and elderly people using prosthetic limbs in real time.

[1273] System Overview

[1274] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies the user.

[1275] First, sensors detect the user's movements in real time. The sensors capture specific movements, such as walking or lifting, and transmit the data to a device. For example, a sensor collects information on the angle and position of the user's foot while walking with a prosthetic leg.

[1276] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data. The preprocessed data is then sent to a server and input into the generative artificial intelligence.

[1277] The server then analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects improper behavior and risk of falling. From this analysis, it generates instructions to provide to the user.

[1278] The generated instructions are sent from the server to the terminal, which provides the instructions to the user by visual display (e.g., on-screen text message) or audio notification.

[1279] Specific examples

[1280] Example 1: Falling Hazard

[1281] If a user's foot becomes unstable while walking using a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device preprocesses the data and sends it to the server. The server analyzes it using generative artificial intelligence and recognizes that there is a high risk of falling. As a result, an instruction is generated saying, "There is a risk of falling. Please correct your posture," and the user is notified via the device. The notification is displayed as a text message on the screen and a voice message is broadcast at the same time.

[1282] Example 2: Incorrect behavior

[1283] If a user makes an inappropriate movement when lifting an object with their prosthetic arm, a sensor detects this movement and sends the data to the device. The device removes noise, normalizes the data, and then sends it to the server. The server's generative artificial intelligence analyzes the data and detects that the movement is inappropriate. As a result, an instruction is generated saying, "The movement is inappropriate. Please check again," and the user is notified via the device. This is also communicated to the user via a screen display and voice message.

[1284] In this way, this system can significantly improve the quality of life for the physically disabled and elderly by analyzing the user's movements in real time and providing appropriate instructions quickly.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] A user performs an action using a prosthetic arm or leg. For example, consider a situation in which a user walks with a prosthetic leg.

[1288] Step 2:

[1289] The sensors detect the user's movements in real time and collect specific movement data (for example, foot angle and position information over time).

[1290] Step 3:

[1291] The sensors collect and transmit motion data, including real-time location and angle data, to the device.

[1292] Step 4:

[1293] The device preprocesses the raw data received from the sensors, which includes denoising and normalizing the data, converting it into a format that is easier to analyze.

[1294] Step 5:

[1295] The device sends the preprocessed data to the server, where the data has been denoised and normalised.

[1296] Step 6:

[1297] The server feeds the preprocessed data into a generative artificial intelligence (AI) model that includes advanced learning algorithms such as recurrent neural networks.

[1298] Step 7:

[1299] The server uses generative artificial intelligence to analyze the movement data, detecting inappropriate movements and dangers from the data. For example, if a walking pattern differs from normal, it will be detected as an abnormality.

[1300] Step 8:

[1301] Based on the analysis results, the server generates instructions for the user, including specific content such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[1302] Step 9:

[1303] The server generates instructions and sends them to the terminal. The instructions may be in text or audio format.

[1304] Step 10:

[1305] The terminal notifies the user of the instructions received from the server. Notification methods include displaying a text message on the screen or an audio announcement. For example, the user may be shown a message saying "Danger of falling. Please correct your posture.", and simultaneously notified by an audio message.

[1306] Example 1

[1307] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1308] In technology that monitors the movements of physically disabled and elderly people using prosthetic limbs in real time and provides appropriate instructions quickly, current systems have issues with insufficient noise removal and normalization of movement data, making it difficult to obtain accurate analysis results.In addition, there are limited means of notifying users, which makes it difficult to provide sufficient warnings in certain environments.

[1309] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1310] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for removing noise from the motion data and normalizing the motion data, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect inappropriate motions or dangers, and means for providing instructions to the user by visual display or audio notification based on the analysis results of the generative artificial intelligence, thereby enabling more accurate analysis of the user's motions in real time and providing more appropriate and prompt instructions.

[1311] A "bionic sensor" is a device that detects the movement or physiological state of a living organism and acquires this information as data.

[1312] "Motion data" refers to data that includes information about specific movements performed by a user, such as walking or lifting an object, recorded in real time.

[1313] "Preprocessing means" refers to the process of processing raw data obtained from a sensor, such as by removing noise and normalizing it, and converting it into a format that is easy to analyze.

[1314] The "means for removing noise and normalizing" refers to a process for removing unnecessary fluctuations and outliers contained in the operational data and converting the data to a uniform scale.

[1315] "Generative AI" is a type of AI that has the ability to analyze the data it receives and generate new instructions or results.

[1316] A "recurrent neural network" is a neural network model used to analyze time-series data, and is capable of storing past information and using it in current analysis.

[1317] "Means for giving instructions to the user based on the analysis results" refers to a means for issuing instructions to the user to encourage appropriate actions or corrections based on the results of the generative artificial intelligence's analysis of data.

[1318] "Visual display" refers to a method of conveying analytical results or instructions to the user using text or graphics, typically displayed as a message on a screen.

[1319] "Voice notification" is a method of conveying analysis results and instructions to the user by voice, and is provided as a voice message through a speaker or headset.

[1320] The present invention relates to a system that combines bionic sensors, pre-processing means, generative artificial intelligence, and user notification means, and is capable of monitoring, analyzing, and providing appropriate instructions in real time to the movements of physically disabled and elderly people when using prosthetic limbs.

[1321] This system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, and a terminal that notifies users. Specific examples of use are shown below.

[1322] Hardware and software configuration and data processing

[1323] 1. Sensor

[1324] Hardware name: Accelerometer, Gyroscope, etc.

[1325] Role: Detects user behavior in real time and sends that data to the device.

[1326] Example: Collecting foot angle and position information while a user walks with a prosthetic leg.

[1327] 2. Terminal

[1328] Software name: Python script, NumPy, SciPy

[1329] Role: Preprocesses the motion data received from the sensors, specifically removing noise and normalizing the data.

[1330] Example: Data preprocessing removes unwanted shaking and blurring from walking data to generate a standardized walking pattern.

[1331] 3. Server

[1332] Software name: TensorFlow, Keras

[1333] Role: Analyzes pre-processed data and uses generative artificial intelligence models (e.g., recurrent neural network models) to detect improper behavior or dangers and generate appropriate instructions.

[1334] Example: Using a recurrent neural network to analyze a user's walking pattern in real time and detect the risk of falling.

[1335] 4. Means of notification

[1336] Software name: Android app, iOS app

[1337] Role: Notifies the user of instructions sent by the server. Notifications include visual indications (text messages on the screen) and audio notifications.

[1338] Example: A smartphone app displays the message "Risk of falling. Please correct your posture." on the screen and also announces the same message in voice.

[1339] Specific examples

[1340] Example 1: Falling Hazard

[1341] User: Walking with a prosthetic leg

[1342] Sensor: Detects unstable foot movement

[1343] Terminal: Preprocessing the data, removing noise, and normalizing the data

[1344] Server: Performs analysis and detects risk of falling. Generates instruction such as "Danger of falling. Please correct your posture."

[1345] Notification method: Notify the user via the device. Uses screen display and voice message.

[1346] Example 2: Incorrect behavior

[1347] User: Lifting an object with a prosthetic arm

[1348] Sensors: Detect inappropriate behavior

[1349] Terminal: Data preprocessing, noise removal and data normalization

[1350] Server: Analyzes data and detects inappropriate behavior. Generates an instruction saying, "The movement is inappropriate. Please check again."

[1351] Notification method: Notify the user via the device. Uses screen display and voice message.

[1352] Prompt Sentence Examples

[1353] The system analyzes the user's movement data and notifies them in real time of inappropriate movements or dangers, generating instructions such as, "Unstable foot movements were detected while walking with a prosthetic leg. There is a risk of falling, so please correct your posture."

[1354] This system monitors the user's actions in real time through a series of processes using the above-mentioned hardware and software, and provides appropriate instructions to ensure safety.

[1355] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1356] Step 1:

[1357] The user performs a movement using the prosthetic limb.

[1358] Input: User actions (walking, lifting, etc.)

[1359] Specific action: The user walks using the prosthetic leg.

[1360] Output: Movement data (foot angle, position, acceleration, etc.)

[1361] Step 2:

[1362] The sensor detects the user's movements and transmits the data to the device.

[1363] Input: Real-time user actions

[1364] Specific movements: Bionic sensors capture and generate data on user movements in real time.

[1365] Output: Raw data (unprocessed motion data)

[1366] Step 3:

[1367] The device preprocesses the raw data received from the sensors.

[1368] Input: Raw data sent from the sensor

[1369] What it does: The terminal script performs noise reduction and data normalization, e.g., removes outliers and converts the data to a consistent scale.

[1370] Output: Preprocessed data (denoised and normalized data)

[1371] Step 4:

[1372] The terminal sends the preprocessed data to the server.

[1373] Input: Preprocessed data

[1374] Specific operation: The device sends preprocessed data to the server's API endpoint.

[1375] Output: Preprocessed data sent to the server

[1376] Step 5:

[1377] The server analyzes the received pre-processed data.

[1378] Input: Preprocessed data

[1379] Specific operation: The server uses TensorFlow and Keras to analyze data with a recurrent neural network model to identify specific patterns and detect inappropriate behavior or danger.

[1380] Output: Analysis results (behavior evaluation, risk detection results)

[1381] Step 6:

[1382] The server generates appropriate instructions based on the analysis results.

[1383] Input: Analysis results

[1384] Specific operation: The server uses a generative AI model to generate instructions for the detected problem, for example, creating a message such as "There is a risk of falling. Please correct your posture."

[1385] Output: Instruction message (specific instructions to the user)

[1386] Step 7:

[1387] The server sends the generated instructions to the terminal.

[1388] Input: prompt message

[1389] Specific operation: The server's API sends an instruction message to the device.

[1390] Output: Instruction message sent to the terminal

[1391] Step 8:

[1392] The terminal notifies the user of the instruction received from the server.

[1393] Input: prompt message

[1394] What it does: The device displays a message on the screen and gives a voice notification. For example, a smartphone displays a text message and a voice message saying, "Danger of falling. Please correct your posture."

[1395] Output: User notification (visual and audio instructions)

[1396] Step 9:

[1397] The user acknowledges the notification and corrects the behavior.

[1398] Input: Notification message

[1399] Specific action: The user checks the smartphone screen or voice message and adjusts their walking pace, for example.

[1400] Output: Improved behavior (corrected walking pattern)

[1401] (Application example 1)

[1402] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] In autonomous vehicles, if the driver's health condition is not properly monitored in real time, the possibility of dangerous situations occurring increases. Therefore, there is a need for a means to monitor the driver's biological information in real time, quickly detect abnormalities, and provide appropriate instructions. In particular, it is necessary to immediately detect situations such as a sudden increase in the driver's heart rate while driving or delayed reactions due to accumulated fatigue, thereby improving safety.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1405] In this invention, the server includes means for preprocessing motion data received from the biometric sensor, means for analyzing the preprocessed motion data and using generative artificial intelligence to detect abnormal motion or danger, means for providing instructions to a user based on the analysis results of the generative artificial intelligence, means for monitoring biometric information of the driver of the autonomous vehicle in real time, and means for detecting and notifying an abnormality from the driver's biometric information, thereby making it possible to monitor the driver's biometric information in real time and quickly detect and notify an abnormality.

[1406] A "bionic sensor" is a sensor device for measuring physiological activity or conditions of the human body.

[1407] A "pre-processing means" is a processing system for converting raw data received from a sensor into a format that is easy to analyze.

[1408] "Generative AI" is an AI system that has the ability to generate new insights based on data and provide instructions for specific purposes.

[1409] A "recurrent neural network" is a deep learning model used to capture the characteristics of time series data.

[1410] An "autonomous vehicle" is a vehicle that is capable of driving autonomously without the intervention of a human driver.

[1411] "Driver's biometric information" refers to data that indicates the driver's physical condition and state, such as heart rate, skin temperature, and muscle movement.

[1412] "Means for real-time monitoring" refers to a system that instantly analyzes information obtained from sensors and monitors the driver's condition.

[1413] The "means for detecting and notifying abnormalities" is a system that issues an alert when an abnormality is detected based on the driver's biometric information.

[1414] This invention is a system for monitoring the biological information of a driver of an autonomous vehicle in real time and quickly detecting and notifying an abnormality. A specific embodiment of the system is described below.

[1415] System Configuration

[1416] The system includes the following components:

[1417] 1. Bionic sensors: Sensor devices that measure the driver's heart rate, skin temperature, muscle movement, etc.

[1418] 2. Terminal: A device that processes data received from sensors.

[1419] 3. Server: A computer with generative artificial intelligence to analyze incoming data and generate appropriate instructions.

[1420] 4. Notification means: Visual display device and audio output device to notify the driver of abnormalities.

[1421] Operation overview

[1422] 1. Data collection: When a driver gets into an autonomous vehicle, biometric sensors measure the driver's vital signs in real time and transmit the data to a terminal.

[1423] 2. Data preprocessing: The terminal performs preprocessing on the received data, such as noise removal and normalization, to convert it into a format suitable for analysis.

[1424] 3. Data analysis: The server analyzes the pre-processed data using generative artificial intelligence (e.g., a recurrent neural network model). Based on the analysis results, it detects driver anomalies.

[1425] 4. Notification: If an abnormality is detected, the server generates appropriate instructions and notifies the driver visually or audibly through the terminal.

[1426] Hardware and software used

[1427] Biometric sensors: heart rate monitors, skin thermometers, electromyography sensors, etc.

[1428] Data Preprocessor: A computer / device that performs data denoising and normalization.

[1429] Generative artificial intelligence: Uses deep learning frameworks such as TensorFlow.

[1430] Notification Device: A display, speaker, or any type of alert device.

[1431] Specific scenes and prompts

[1432] 1. Example:

[1433] If a driver's heart rate suddenly rises while driving and the recurrent neural network model determines this to be an abnormality, the system will notify the driver with a message saying, "An abnormality has been detected. Please take a break immediately."

[1434] 2. Example prompt:

[1435] "Predict the probability that the driver's heart rate will reach an abnormal value and notify the driver of the abnormality."

[1436] effect

[1437] This system will ensure the safety of drivers of self-driving vehicles and enable rapid response in emergencies, thereby improving the safety and reliability of self-driving vehicles.

[1438] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1439] Step 1:

[1440] Data collection

[1441] When a user gets into an autonomous vehicle, bionic sensors measure their vital signs, such as heart rate, skin temperature, and muscle activity, in real time, and transmit this data to a device.

[1442] Input: Biometric data from biometric sensors

[1443] Output: Raw data sent to the terminal

[1444] Step 2:

[1445] Data Preprocessing

[1446] The device receives the received biometric data and performs noise reduction and normalization on it. Noise reduction removes unnecessary parts of the data, and normalization converts the data into a format suitable for analysis, allowing for smoother analysis.

[1447] Input: Raw data received from the sensor

[1448] Output: Preprocessed data

[1449] Step 3:

[1450] Data analysis

[1451] The preprocessed data is sent from the device to a server, where it is analyzed by a generative artificial intelligence (e.g., a recurrent neural network model). The purpose of the analysis is to detect abnormalities in the driver's biometric information.

[1452] Input: Preprocessed data

[1453] Output: Analysis results (whether anomalies were detected)

[1454] Step 4:

[1455] Generate notifications

[1456] The server generates a notification to the driver based on the analysis results. For example, if the driver's heart rate suddenly rises, it generates a message saying, "An abnormality has been detected. Please take a break immediately."

[1457] Input: Analysis results

[1458] Output: Notification content (message)

[1459] Step 5:

[1460] Sending notifications

[1461] The generated notification is sent to the terminal, which then notifies the driver visually or audibly. Specifically, it is possible to display a message on the display and play an audio message through the speaker.

[1462] Input: Notification content

[1463] Output: Notification to the driver (visual and audio)

[1464] In this way, the system can ensure safety by monitoring the driver's biometric information in real time and immediately notifying the driver if any abnormalities are detected.

[1465] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1466] The present invention provides a system that combines bionic sensors, a means for preprocessing motion data, generative artificial intelligence, a means for notifying users, and an emotion engine that recognizes users' emotions. This system not only monitors and analyzes the motions of physically disabled or elderly people using prosthetic limbs in real time and provides appropriate instructions, but also takes into account the user's emotional state to reduce the psychological burden.

[1467] System Overview

[1468] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[1469] Operation flow

[1470] First, the sensor detects the user's movements in real time. For example, when the user walks with a prosthetic leg, the sensor collects information on the angle and position of the walking leg. The sensor then transmits this movement data to the device.

[1471] The device then preprocesses the motion data received from the sensors, which includes removing noise and normalizing the data, and then sends the preprocessed data to the server.

[1472] On the server, generative AI analyzes the preprocessed data and detects inappropriate behavior or danger. Based on the analysis results, instructions are generated for the user. For example, "There is a risk of falling. Please correct your posture."

[1473] The generated instructions are sent from the server to the device, which then communicates the instructions to the user through visual displays and audio notifications, allowing the user to receive appropriate advice in real time.

[1474] Emotion Engine Operation

[1475] The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state. The emotion engine utilizes data obtained from sensors, cameras, and microphones. The results of emotion recognition are added as additional information to the behavior analysis results of generative AI.

[1476] Based on the results of the emotion engine, the instructions to the user can be adjusted accordingly. For example, if the user feels anxious about the risk of falling, a message such as "Please walk slowly and calmly" will be generated to reduce the psychological burden.

[1477] Specific examples

[1478] Example 1: Falling Hazard

[1479] If a user makes unsteady movements while walking with a prosthetic leg, a sensor detects this and sends the data to the device in real time. The device then preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes it and detects the risk of falling. If the emotion engine recognizes anxiety from the user's facial expression, it generates an instruction saying, "There is a risk of falling. Please slowly correct your posture," and notifies the user via the device. The notification is displayed on the screen and by voice.

[1480] Example 2: Incorrect behavior

[1481] If a user makes an inappropriate movement when lifting an object with their prosthetic hand, a sensor detects this movement and sends the data to the device. The device preprocesses the data and sends it to the server. The server's generative artificial intelligence analyzes the data and detects the inappropriate movement. Furthermore, if the emotion engine recognizes irritation from the user's tone of voice, it generates an instruction saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the device.

[1482] In this way, the system can detect and analyze the user's movements and emotions in real time, and provide appropriate instructions and psychological support, further improving the quality of life for the physically disabled and elderly.

[1483] The processing flow will be explained below.

[1484] Step 1:

[1485] The user performs an action using a prosthetic arm or leg, for example, the user starts walking with the prosthetic leg.

[1486] Step 2:

[1487] Sensors detect the user's movements in real time and collect movement data, such as the angle and position of the feet while walking.

[1488] Step 3:

[1489] The sensors collect motion data and transmit it to the device, including real-time location and angle data.

[1490] Step 4:

[1491] The device preprocesses the raw data received from the sensors, specifically removing noise from the data and normalizing it, making the data easier to analyze.

[1492] Step 5:

[1493] The device sends the preprocessed data to the server, where it is in a denoised and normalized form.

[1494] Step 6:

[1495] The server feeds the preprocessed data into a generative artificial intelligence (AI) model, which uses advanced algorithms such as recurrent neural networks.

[1496] Step 7:

[1497] The server uses generative artificial intelligence to analyze the movement data and detect inappropriate movements or dangers. For example, if the user's walking pattern differs from normal, it will be detected as an abnormality.

[1498] Step 8:

[1499] Based on the analysis results, the server generates instructions for the user, such as "There is a risk of falling. Please correct your posture" or "Your movements are inappropriate. Please check again."

[1500] Step 9:

[1501] The server generates and sends the instructions to the device, which may be in the form of a text or audio file.

[1502] Step 10:

[1503] The terminal notifies the user of the instruction received from the server, for example by displaying a message on the screen and playing a voice message.

[1504] Step 11:

[1505] The device collects the user's facial expressions and tone of voice and inputs them into the emotion engine as emotional data. For example, the device collects the user's reactions through a camera or microphone.

[1506] Step 12:

[1507] The server's emotion engine analyzes the emotion data and recognizes the user's emotional state, for example, determining whether the user is tense or relaxed.

[1508] Step 13:

[1509] Based on the analysis results of the emotion engine, the server adjusts the instructions to correspond to the user's psychological state. For example, if the user is feeling anxious, it generates instructions that give a sense of security.

[1510] Step 14:

[1511] The server then sends the adjusted instructions back to the device, and the device notifies the user. For example, the device displays and speaks the instruction "Please walk slowly and calmly."

[1512] Example 2

[1513] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1514] Conventional motion assistance systems can monitor a user's motion and provide necessary instructions, but they cannot consider the user's real-time emotional state and lack the ability to reduce psychological stress. This has led to problems such as increased psychological stress when physically disabled people and elderly people use prosthetic limbs. Therefore, the problem that the present invention aims to solve is to reduce psychological stress by recognizing the user's emotional state in real time and adjusting instructions based on that.

[1515] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1516] In this invention, the server includes a device for preprocessing motion data received from a bionic sensor, a device using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motions or dangers, a device for giving instructions to the user based on the analysis results of the generative artificial intelligence, a device for recognizing the emotional state of the user by analyzing the user's facial expressions and tone of voice, a device for adjusting the instructions taking the emotional state into consideration, and a device for notifying the user of the instructions, thereby making it possible to monitor the user's motions and emotions in real time and provide appropriate instructions and psychological support at the same time.

[1517] A "bionic sensor" is a device that detects and measures human movement and physiological data in real time.

[1518] "Motion data" refers to information about the movements, position information, angle information, and other movements when a user uses a prosthetic arm or leg.

[1519] "Preprocessing" is the process of converting raw data received from sensors into a format suitable for analysis by performing processes such as noise removal and normalization.

[1520] "Generative AI" is a machine learning model that analyzes input data and learns and predicts specific patterns and trends. It often uses advanced algorithms, such as recurrent neural networks.

[1521] "Analysis results" are information obtained by generative artificial intelligence analyzing behavioral data, and indicate whether or not there is inappropriate behavior or danger.

[1522] A "device that gives instructions to the user" is a device that provides specific instructions and advice to the user based on the analysis results of generative artificial intelligence.

[1523] "Emotional state" refers to the user's current psychological state and is analyzed based on data such as facial expressions and tone of voice.

[1524] An "emotion engine" is software or algorithm that analyzes a user's facial expressions, tone of voice, etc. to recognize their emotional state.

[1525] "Adjusting the content of instructions" means changing the content of the instructions provided based on the analysis results of the emotion engine, taking into account the user's psychological state.

[1526] "Notification" refers to the act of conveying analysis results or instructions to the user visually or audibly.

[1527] "Real-time" is a term that indicates that sensors and systems instantly recognize the user's actions and emotional state, and then analyze and respond immediately.

[1528] This system combines bionic sensors, a motion data preprocessing device, generative artificial intelligence, a user notification device, and an emotion engine. The system aims to monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, provide appropriate instructions, and reduce the psychological burden by taking into account the user's emotional state.

[1529] The system consists of bionic sensors, terminals, a server equipped with generative artificial intelligence, a terminal that notifies the user, and an emotion engine.

[1530] Hardware and software used

[1531] Sensors: Accelerometer, Gyroscope, Camera, Microphone

[1532] Devices: Mobile devices, tablets, wearable devices

[1533] Server: High-performance computing server, cloud computing platform

[1534] Generative AI model: Recurrent Neural Network (RNN)

[1535] Emotion engine: facial expression recognition algorithm, voice tone analysis algorithm

[1536] Data processing and calculation

[1537] 1. Sensor operation data collection

[1538] As the user uses the prosthetic limb, bionic sensors collect real-time motion data, capturing acceleration, angle, and position information at high frequency, which is then transmitted to a device.

[1539] 2. Preprocessing by the terminal

[1540] The device performs noise reduction and data normalization on the raw data received from the sensors. Denoising is done to remove outliers and unnecessary data, and normalization is done to put the data into a consistent format, making it suitable for analysis by generative artificial intelligence.

[1541] 3. Data analysis by the server

[1542] The preprocessed data is sent to a server and input into a generative artificial intelligence (RNN model). The model analyzes the user's movements to detect inappropriate behavior and danger. For example, it can predict the risk of falling based on the movement of the feet while walking.

[1543] 4. Emotion Recognition by Emotion Engine

[1544] The server's emotion engine uses data acquired from the camera and microphone to analyze the user's facial expressions and tone of voice to recognize their emotional state in real time, for example, detecting anxiety or irritation.

[1545] 5. Instruction Generation and Notification

[1546] Based on the analysis results of the generative AI and the recognition results of the emotion engine, the server generates instructions to give to the user. These instructions are sent to the device in real time, and the device conveys the instructions to the user through visual displays and audio notifications, allowing the user to take appropriate action immediately.

[1547] Specific examples

[1548] Example 1: Falling Hazard

[1549] "Prompt: The user is walking with a prosthetic leg and has become unsteady on their feet. The sensor detects this information and sends it to the device. The generative AI analyzes this data and detects the risk of falling. If anxiety is recognized from the user's facial expression, generate appropriate instructions."

[1550] An example of a generated instruction might be: "Danger of falling. Please slowly regain your balance."

[1551] Example 2: Incorrect behavior

[1552] "Prompt: The user performed an inappropriate action while lifting an object with the prosthetic arm. The sensor detected this information and sent it to the device. The generative AI analyzed this data and detected the inappropriate action. If frustration is detected in the user's tone of voice, generate appropriate instructions."

[1553] An example of a generated instruction would be: "Inappropriate behavior. Please calm down and try again."

[1554] By using this system, users can receive appropriate instructions and psychological support in real time, thereby improving the quality of life for the physically disabled and elderly.

[1555] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1556] Step 1:

[1557] Sensors collect user movement data in real time.

[1558] Input: The user's prosthetic arm or leg movements

[1559] Movement: Accelerometer and gyroscope sensors measure your movement at a rate of 50 times per second.

[1560] Output: Raw data (foot angle, position information, etc.)

[1561] Step 2:

[1562] The device preprocesses the motion data received from the sensors.

[1563] Input: Raw data from the sensor

[1564] What it does: It removes noise and normalizes the data, eliminating outliers and scaling the data range to 0-1.

[1565] Output: Preprocessed data

[1566] Step 3:

[1567] The device sends the preprocessed data to the server.

[1568] Input: Preprocessed data

[1569] How it works: Sends data using an HTTP POST request.

[1570] Output: Preprocessed data arriving at the server

[1571] Step 4:

[1572] The server analyzes the preprocessed data using generative artificial intelligence.

[1573] Input: Preprocessed data

[1574] Behavior: Data is fed into a generative artificial intelligence (RNN model) to detect inappropriate behavior and danger.

[1575] Output: Analysis results (e.g., whether there is a risk of falling)

[1576] Step 5:

[1577] The server's emotion engine recognizes the user's emotional state in real time.

[1578] Input: Data from camera and microphone (facial expressions, tone of voice)

[1579] How it works: Recognizes emotional states using facial expression recognition and audio tone analysis algorithms.

[1580] Output: Emotion recognition result (e.g., anxiety, irritation)

[1581] Step 6:

[1582] The server generates instructions based on the analysis results and emotion recognition results.

[1583] Input: Analysis results, emotion recognition results

[1584] Action: The analysis results are combined with the emotion recognition results to generate instructions such as "There is a risk of falling. Please slowly correct your posture."

[1585] Output: Generated instructions

[1586] Step 7:

[1587] The server sends the generated instructions to the terminal.

[1588] Input: Generated instructions

[1589] Operation: Sends instructions to the device in real time using WebSocket.

[1590] Output: Instructions that arrive at the terminal

[1591] Step 8:

[1592] The device notifies the user of the instructions.

[1593] Input: Instructions that arrive at the terminal

[1594] Operation: Instructions are displayed on the device screen and communicated aloud through the speaker.

[1595] Output: Instructions to the user

[1596] In this way, the processing flow of the entire system is realized by a chain of specific operations and their inputs and outputs at each step.

[1597] (Application example 2)

[1598] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] While conventional systems can monitor and analyze the movements of physically disabled and elderly people using prosthetic limbs in real time, they are unable to provide psychological support based on the user's emotional state. Furthermore, particularly in factory robot operating environments, there is a need for emotion analysis and appropriate instructions to reduce worker stress and anxiety, in addition to the ability to detect inappropriate movements and dangers.

[1600] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1601] In this invention, the server includes means for preprocessing motion data received from the bionic sensor, means for using generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motion or danger, means for using an emotion engine to analyze emotional information such as the user's facial expression and voice, and means for providing instructions to the user based on the results of the motion analysis by the generative artificial intelligence and the results of the emotion analysis by the emotion engine. This not only improves the safety and efficiency of workers in the operating environment of factory robots, but also reduces the psychological burden on workers and provides a comfortable working environment.

[1602] definition statement

[1603] A "bionic sensor" is a device that detects a user's physical movements and physiological information in real time and collects them as data.

[1604] The "preprocessing means" refers to a processing method for removing noise and normalizing the raw data received from the sensor and converting it into a format suitable for analysis.

[1605] "Generative AI" is an AI technology that learns from large amounts of data and analyzes and judges specific actions and situations.

[1606] The "emotion engine" is a system that analyzes changes in a user's facial expressions and voice to determine their emotional state in real time.

[1607] "Means for providing instructions" refers to a method for notifying the user of information that prompts the user to take appropriate action based on the analysis results, including visual displays and audio notifications.

[1608] A "terminal" is a device that acts as an interface with sensors and users, and is responsible for preprocessing and notifying data.

[1609] "Using a model" means using a mathematical or computational method to analyze data based on a particular calculation method or algorithm.

[1610] "Detecting real-time actions" means instantly detecting a user's actions and acquiring them as data without delay.

[1611] MODE FOR CARRYING OUT THE INVENTION

[1612] The present invention is a system for improving worker safety and efficiency in a factory robotic operating environment, which combines bionic sensors, a means for pre-processing motion data, generative artificial intelligence, an emotion engine for recognizing user emotions, and a means for notifying the user.

[1613] System Overview

[1614] This system consists of the following components:

[1615] Bionic sensors: Detect the user's physical movements in real time and transmit the data to the device.

[1616] Device: Receives motion data and performs preprocessing such as noise removal and normalization. The preprocessed data is sent to the server.

[1617] Server: Analyzes the pre-processed data using generative artificial intelligence to detect inappropriate behavior and danger. The server also has an emotion engine that analyzes the user's facial expressions and voice.

[1618] Notification method: Based on the analysis results, instructions are given to the user through visual displays and audio notifications.

[1619] Operation flow

[1620] First, a bionic sensor detects the user's movements in real time. Next, the device preprocesses the movement data received from the sensor, removing noise and normalizing the data. This preprocessed data is sent to a server where a generative AI analyzes it. Based on the analysis results, inappropriate movements and dangers are detected. Furthermore, an emotion engine analyzes the user's facial expressions and voice fluctuations to recognize the user's emotional state. Finally, appropriate instructions are generated based on the movement analysis results by the generative AI and the emotion analysis results by the emotion engine, and the instructions are communicated to the user via notification means.

[1621] Hardware and Software

[1622] Bionic sensors: devices for detecting user movements

[1623] Terminal: A device that preprocesses data and sends it to the server.

[1624] Server: A server equipped with generative artificial intelligence and emotion engine using TensorFlow and Keras libraries.

[1625] Notification method: Uses the pyttsx3 library for audio output and the display device for visual display

[1626] Specific examples

[1627] Example 1: Detecting dangerous behavior

[1628] If a worker adopts an improper posture while using a robotic arm to lift a heavy object, a bionic sensor detects this. The data sent from the sensor is preprocessed on the terminal to remove noise and normalize it. It is then sent to the server, where generative artificial intelligence detects dangerous movements. At the same time, an emotion engine analyzes the worker's facial expressions and voice and recognizes that his emotional state is unstable. Based on this information, an instruction is generated: "It's dangerous. Please slowly correct your posture." This is communicated to the worker through visual displays and audio notifications.

[1629] Example 2: Fixing improper behavior

[1630] If a worker makes an inappropriate move when using the robotic arm to lift an object, and the emotion engine recognizes irritation from the worker's tone of voice, it generates a message saying, "Your movement is inappropriate. Please calm down and try again," and notifies the user via the terminal. The notification is displayed on the screen and by voice.

[1631] This improves the safety and efficiency of workers in a factory robot operation environment, reduces psychological stress, and provides a comfortable working environment.

[1632] The system can use prompts such as "Dangerous. Please slowly correct your position" or "Inappropriate movement. Please calm down and try again."

[1633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1634] Processing Steps

[1635] Step 1:

[1636] Bionic sensors detect user movements in real time.

[1637] Input: User's physical movements

[1638] Output: Motion data (e.g., foot angle and position information)

[1639] Specific operation: The sensor detects motion data and sends it to the device.

[1640] Step 2:

[1641] The device preprocesses the motion data received from the sensors.

[1642] Input: Motion data received from sensors

[1643] Output: Preprocessed motion data

[1644] Specific operations: Denoise and normalize the data, then format it, and send the preprocessed data to the server.

[1645] Step 3:

[1646] The server analyzes the preprocessed data using generative artificial intelligence.

[1647] Input: Preprocessed motion data

[1648] Output: Results of behavior analysis (e.g., whether there is danger or not, detection of inappropriate behavior)

[1649] Specific behavior: A generative AI model analyzes the behavior data and evaluates the risk. The analysis results are saved.

[1650] Step 4:

[1651] The server analyzes the user's facial expressions and voice using an emotion engine.

[1652] Input: Video and audio data from the camera and microphone

[1653] Output: Sentiment analysis results (e.g., user's emotional state)

[1654] Specific operation: The emotion engine analyzes video and audio data to recognize the user's emotions. The analysis results are saved.

[1655] Step 5:

[1656] The server generates instructions for the user based on the results of motion analysis and emotion analysis.

[1657] Input: Motion analysis results and emotion analysis results

[1658] Output: Instructions (prompt)

[1659] Specific actions: Generative AI combines the results of motion analysis and emotion analysis to generate an appropriate prompt, such as "It's dangerous. Please slowly correct your posture."

[1660] Step 6:

[1661] The server sends instructions to the device, and the device notifies the user.

[1662] Input: Generated prompt statement

[1663] Output: User notification (visual and audio)

[1664] Specific operation: The server sends the generated instructions to the device, and the device communicates the instructions to the user through visual display and audio notification.

[1665] This allows users to receive proper instructions and psychological support regarding their actions in real time, improving safety and efficiency.

[1666] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1667] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1668] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1669] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1670] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1671] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1672] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1673] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1674] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1675] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1676] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1677] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1678] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1679] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1680] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1681] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1682] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1683] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1684] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1685] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1686] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1687] The following is further disclosed regarding the above embodiment.

[1688] (Claim 1)

[1689] means for preprocessing the motion data received from the bionic sensor;

[1690] a means for analyzing the pre-processed motion data and using generative artificial intelligence to detect inappropriate motions and hazards;

[1691] means for giving instructions to a user based on the analysis results by the generative artificial intelligence;

[1692] A system including:

[1693] (Claim 2)

[1694] 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

[1695] (Claim 3)

[1696] 2. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network.

[1697] (Claim 4)

[1698] 10. The system of claim 1, wherein the means for notifying the user of the instruction includes at least one of a visual display and an audio notification.

[1699] (Claim 5)

[1700] 10. The system of claim 1, wherein the inappropriate behavior is determined by comparing the inappropriate behavior to a predefined standard.

[1701] "Example 1"

[1702] (Claim 1)

[1703] means for preprocessing the motion data received from the bionic sensor;

[1704] a means for analyzing the pre-processed motion data and using generative artificial intelligence to detect inappropriate motions and hazards;

[1705] A means for giving instructions to a user by visual display or voice notification based on the analysis results by the generative artificial intelligence;

[1706] means for removing noise from and normalizing the motion data;

[1707] A system including:

[1708] (Claim 2)

[1709] 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

[1710] (Claim 3)

[1711] 2. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network.

[1712] "Application Example 1"

[1713] (Claim 1)

[1714] means for preprocessing the motion data received from the bionic sensor;

[1715] A means for analyzing the pre-processed motion data and using generative artificial intelligence to detect abnormal motion or danger;

[1716] means for giving instructions to a user based on the analysis results by the generative artificial intelligence;

[1717] means for monitoring biometric information of a driver of an autonomous vehicle in real time;

[1718] means for detecting an abnormality from the driver's biological information and notifying the same;

[1719] A system including:

[1720] (Claim 2)

[1721] 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

[1722] (Claim 3)

[1723] 2. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network.

[1724] "Example 2: Combining Emotion Engines"

[1725] (Claim 1)

[1726] an apparatus for preprocessing motion data received from the bionic sensor;

[1727] A device that uses generative artificial intelligence to analyze the preprocessed motion data and detect inappropriate motions and dangers;

[1728] a device that gives instructions to a user based on the analysis results of the generative artificial intelligence;

[1729] A device that recognizes the user's emotional state by analyzing their facial expressions and tone of voice,

[1730] a device for adjusting the instruction content in consideration of the emotional state;

[1731] a device for notifying a user of the instruction;

[1732] A system including:

[1733] (Claim 2)

[1734] 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

[1735] (Claim 3)

[1736] 2. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network.

[1737] "Application example 2 when combining emotion engines"

[1738] Claims

[1739] (Claim 1)

[1740] means for preprocessing the motion data received from the bionic sensor;

[1741] a means for analyzing the pre-processed motion data and using generative artificial intelligence to detect inappropriate motions and hazards;

[1742] A method using an emotion engine that analyzes emotional information such as the user's facial expressions and voice,

[1743] a means for giving instructions to a user based on the results of the motion analysis by the generative artificial intelligence and the results of the emotion analysis by the emotion engine;

[1744] A system including:

[1745] (Claim 2)

[1746] 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

[1747] (Claim 3)

[1748] 2. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network. [Explanation of symbols]

[1749] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for preprocessing the motion data received from the bionic sensor; a means for analyzing the pre-processed motion data and using generative artificial intelligence to detect inappropriate motions and hazards; means for giving instructions to a user based on the analysis results by the generative artificial intelligence; A system including:

2. 10. The system of claim 1, wherein the bionic sensor detects real-time movements of the user and transmits the data to the terminal.

3. 10. The system of claim 1, wherein the generative artificial intelligence uses a model including a recurrent neural network.

4. 10. The system of claim 1, wherein the means for notifying the user of the instruction includes at least one of a visual display and an audio notification.

5. The system of claim 1 , wherein the inappropriate behavior is determined by comparing the inappropriate behavior to predefined criteria.

Images