Information processing system

Abstract

The present invention provides an information processing system comprising: means for preprocessing motion data received from a bioengineering sensor; a means for analyzing the preprocessed motion data using generative artificial intelligence to detect an inappropriate motion or danger; and the device is used for providing an instruction to a user according to an analysis result of the generative artificial intelligence.

Description

Technical Field

[0001] The technology disclosed herein relates to an information processing system. Background Technology

[0002] Japanese Patent Application Publication No. 2022-180282 discloses a method for controlling a role-based chatbot executed by at least one processor. The method includes the following steps: receiving a user's speech; adding the user's speech to a prompt word, the prompt word containing instruction statements associated with an explanation of the chatbot's role; encoding the prompt word; and inputting the encoded prompt word into a language model to generate a chatbot response to the user's speech.

[0003] In existing technologies, there is a lack of real-time and effective motion monitoring and safety alert systems for people with disabilities or the elderly when using prostheses (such as prosthetic hands and feet). When users' movements are not standardized or danger occurs, it is often impossible to detect and deal with the situation in time, which can easily lead to accidents such as falls and injuries, making it difficult to ensure the personal safety of users and improve their self-care ability and quality of life. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides an information processing system, comprising: a device for preprocessing motion data from bioengineering sensors; a device for analyzing the preprocessed motion data using generative artificial intelligence and detecting inappropriate actions or dangers; and a device for providing instructions to the user based on the analysis results of the generative artificial intelligence. Through this system, the user's motion status can be monitored in real time, anomalies or dangers can be analyzed and detected promptly, and warnings and instructions can be issued to the user through visual displays or voice notifications, effectively improving user safety and convenience.

[0005] "Bioengineering sensors" refer to sensing devices used to detect and collect biometric information such as user limb movements, positions, and angles in real time.

[0006] "Motion data" refers to various parameters and signals collected by bioengineering sensors that can reflect the user's motion state and behavioral characteristics.

[0007] "Preprocessing" refers to the process of removing noise, normalizing, and extracting features from the collected raw motion data in order to facilitate subsequent analysis.

[0008] "Generative artificial intelligence" refers to artificial intelligence systems that can perform pattern recognition, anomaly detection, and result generation based on input action data, including but not limited to models such as recurrent neural networks.

[0009] "Analysis" refers to the process by which generative artificial intelligence identifies, judges, and classifies pre-processed action data in order to detect inappropriate actions or dangerous situations.

[0010] "Inappropriate movements" refer to user body movements that are judged to be abnormal or risky through generative artificial intelligence analysis and based on pre-set standards.

[0011] "Danger" refers to a potential risk state that may lead to a fall, equipment malfunction, or other personal injury due to abnormal user behavior patterns.

[0012] "Instructions" refer to information issued by the system to users based on analysis results, including warnings, operational suggestions, or safety reminders.

[0013] "Visual display" refers to the way instructions are presented to users in the form of text, images, etc., through devices such as terminal screens.

[0014] "Voice notification" refers to the method of broadcasting instructions to users in the form of sound through audio devices such as speakers. Attached Figure Description

[0015] Figure 1 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the first embodiment.

[0016] Figure 2 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and smart device according to the first embodiment.

[0017] Figure 3 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the second embodiment.

[0018] Figure 4 This is a conceptual diagram illustrating an example of the main functions of the data processing device and smart glasses according to the second embodiment.

[0019] Figure 5 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the third embodiment.

[0020] Figure 6 This is a conceptual diagram illustrating an example of the main functions of the data processing apparatus and head-mounted terminal according to the third embodiment.

[0021] Figure 7 This is a conceptual diagram illustrating an example of the configuration of the data processing system according to the fourth embodiment.

[0022] Figure 8 This is a conceptual diagram illustrating an example of the main functions of the data processing device and robot according to the fourth embodiment.

[0023] Figure 9 This represents an emotion map that maps multiple emotions.

[0024] Figure 10 This represents an emotion map that maps multiple emotions.

[0025] Figure 11 This is a sequence diagram illustrating the processing flow of the data processing system of the first embodiment.

[0026] Figure 12 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 1.

[0027] Figure 13 This is a sequence diagram illustrating the processing flow of the data processing system of the second embodiment.

[0028] Figure 14 This is a sequence diagram illustrating the processing flow of the data processing system in Application Example 2. Detailed Implementation

[0029] Hereinafter, an example of an implementation of the system according to the present disclosure will be described with reference to the accompanying drawings.

[0030] First, let me explain the terminology used in the following instructions.

[0031] In the following embodiments, the processor (hereinafter referred to as "processor") with reference numerals may be a single computing device or a combination of multiple computing devices. Furthermore, the processor may be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0032] In the following embodiments, RAM (Random Access Memory), as indicated in the figures, is a memory that temporarily stores information and is used as working memory by the processor.

[0033] In the following embodiments, the memory, as indicated by the reference numerals, is one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), disks (e.g., hard disks), or magnetic tapes.

[0034] In the following embodiments, the communication I / F (Interface) with reference numerals is an interface that includes a communication processor and an antenna, etc. The communication I / F is responsible for communication between multiple computers. As an example of a communication specification applicable to the communication I / F, wireless communication specifications such as 5G (5th Generation Mobile Communication System), Wi-Fi (wireless fidelity) (registered trademark), or Bluetooth (registered trademark) can be listed.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" connects to express more than three items, the same interpretation as "A and / or B" applies.

[0036] First Implementation Method

[0037] Figure 1 An example of the configuration of the data processing system 10 according to the first embodiment is shown.

[0038] like Figure 1 As shown, the data processing system 10 includes a data processing device 12 and an intelligent device 14. A server can be cited as an example of the data processing device 12.

[0039] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0040] The smart device 14 includes a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. In addition, the receiving device 38, output device 40, camera 42, and communication I / F 44 are also connected to the bus 52.

[0041] The receiving device 38 includes a touchscreen 38A and a microphone 38B, and receives user input. The touchscreen 38A receives user input via touch by detecting contact with an indicator (e.g., a pen or finger). The microphone 38B receives user input via sound by detecting the user's voice. The control unit 46A in the processor 46 sends data representing the user input received by the touchscreen 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data representing the user input.

[0042] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting data in a form perceptible to the user 20 (e.g., sound and / or text). The display 40A displays visual information such as text and images according to instructions from the processor 46. The speaker 40B outputs sound according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0043] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for sending and receiving various information between processor 46 and processor 28 via network 54.

[0044] Figure 2 The diagram shows an example of the main functions of the data processing device 12 and the smart device 14.

[0045] like Figure 2 As shown, in the data processing apparatus 12, specific processing is performed by the processor 28. A specific processing program 56 is stored in the memory 32. The specific processing program 56 is an example of a "program" as understood in this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0046] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0047] In the smart device 14, the processor 46 performs the acceptance output processing. The memory 50 stores the acceptance output program 60. The acceptance output program 60 is used in conjunction with the data processing system 10 and the specific processing program 56. The processor 46 reads the acceptance output program 60 from the memory 50 and executes the read acceptance output program 60 on the RAM 48. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48. Furthermore, the smart device 14 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290. The acceptance output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance output program 60 executed on the RAM 48.

[0048] Alternatively, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains the processing results (prediction results, etc.) using the data generation model 58 by communicating with the server device that has the data generation model 58. Furthermore, the data processing device 12 may be a server device or a user-held terminal device (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of the processing of the data processing system 10 of the first embodiment will be described.

[0049] Example 1

[0050] The flow of a specific process in Example 1 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. Furthermore, the data processing device 12 is referred to as the "server," and the smart device 14 is referred to as the "terminal."

[0051] Existing systems for monitoring user movements and providing risk alerts for assistive devices (such as prostheses and prosthetic hands) suffer from problems such as excessive noise in the movement data, insufficient data normalization, and inadequate real-time performance and accuracy. This results in a low detection rate for abnormal movements or dangerous situations, limited means of effectively notifying users, and difficulty in ensuring the user's immediate safety and the correctness of their movements.

[0052] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 1 is achieved by the following means.

[0053] In this invention, the server includes a signal processing module for noise removal and data normalization of user action information acquired from a bioinformatics detection device; an intelligent analysis module for performing anomaly or hazard analysis on the processed action data using a generative artificial intelligence model incorporating time-series analysis; an instruction generation module for automatically generating instructions based on the analysis results; and a communication and notification module for sending instructions to the user terminal via communication and providing real-time notifications to the user via visual display or voice on the terminal. This significantly improves the accuracy and real-time performance of action data, enhances the detection capability of abnormal actions and dangerous states, and enables users to receive timely, clear, and intuitive notifications and react correctly, thereby effectively ensuring user safety and the effectiveness of assistive devices.

[0054] "Bioinformation detection device" refers to a device that can detect a user's actions or physiological state in real time and convert this data into electronic information output.

[0055] "Signal processing device" refers to equipment or module that performs preprocessing operations such as noise removal, filtering, and normalization on the raw data obtained by the bioinformatics detection device.

[0056] "Information processing device" refers to a device or module used to receive pre-processed action data and analyze and judge the data based on an artificial intelligence model.

[0057] "Generative artificial intelligence models" refer to artificial intelligence algorithms that can automatically generate output content such as analysis results and instructions based on input data, including but not limited to recurrent neural networks.

[0058] "Command generation and communication device" refers to a device or module that generates user commands based on analysis results and sends the commands to the user terminal through communication methods such as networks.

[0059] "User terminal" refers to an electronic device used to receive server instructions and notify users visually or by voice, including portable devices, computers, etc.

[0060] A "notification device" refers to a device or module implemented on a user terminal that notifies the user of server instructions through screen display or sound playback.

[0061] A “reference comparison device” refers to a device or module that stores reference information for judging whether an action is appropriate and compares it with current action data to help determine whether the action is abnormal.

[0062] A "feedback device" refers to a device or module that enables users to adjust or correct their actions in a timely manner after receiving a notification, thereby achieving human-computer interaction feedback.

[0063] This invention relates to a user action monitoring and risk alert system based on a generative artificial intelligence model. The system mainly consists of a bioinformatics detection device, a signal processing device, an information processing device, a user terminal, and a notification module.

[0064] In a specific implementation, the user wears or has a biometric detection device, such as an accelerometer or gyroscope, which can detect the user's movement data in real time. For example, when a user is walking with a prosthesis, the biometric detection device can collect data such as the acceleration and angle changes of the user's legs.

[0065] Terminals (such as smartphones and wearable devices) acquire the raw data via wired or wireless means and use signal processing devices based on Python scripts, combined with commonly used open-source scientific computing libraries such as NumPy and SciPy, to perform noise removal, outlier removal, and normalization on the acquired data. For example, abnormal accelerations appearing in the data stream are removed, and the data is transformed to a standardized numerical range.

[0066] After preprocessing, the terminal uploads standardized motion data to the information processing device (i.e., the server) via API. The server uses deep learning frameworks such as TensorFlow and Keras to run pre-trained generative artificial intelligence models, especially recurrent neural networks (RNN, LSTM, etc.), to perform temporal analysis of user actions and determine whether there is a risk of falling, abnormal posture, or other risky conditions.

[0067] The server automatically generates user instructions based on the model's output. For example, when a fall risk is detected, the server generates instructions such as "Abnormal gait detected, fall risk present, please pause activity and adjust your posture," and returns this information to the terminal via the network.

[0068] After receiving information from the server, the user terminal provides text prompts on the display screen and simultaneously broadcasts voice messages through a speaker or headphones. For example, while walking, a user receives a pop-up notification and a voice prompt on the terminal, reminding them to adjust their movements accordingly.

[0069] In addition, the system can store and call preset judgment benchmark information and compare it with real-time collected data to further improve the accuracy of anomaly detection and provide adaptive action judgment standards for special user scenarios (such as rehabilitation assistance training, elderly care, etc.).

[0070] Specific usage examples:

[0071] For example, an elderly person uses a smart prosthetic leg to walk at home, and the terminal continuously collects foot movement data. After processing, the data is uploaded to a server. The server, through a recurrent neural network model, detects that the person's gait speed has suddenly slowed down and their steps are unsteady. The server generates a warning: "Unsteady gait, risk of falling, please be careful." The terminal immediately displays a pop-up warning and provides a synchronized voice announcement. Upon hearing the warning, the user immediately stops and adjusts their posture, effectively avoiding the potential risk.

[0072] Example of prompts for generative artificial intelligence models:

[0073] "Based on the following acceleration and angle data, please determine whether the user's action poses a risk and output the corresponding user command. Data: Acceleration = [0.3, 0.8, 1.0], Angle = [30, 28, 35]."

[0074] In summary, the system described in this invention can realize real-time acquisition of multi-source motion data, standardized processing, intelligent analysis based on recurrent neural networks, and multi-channel, real-time user risk alerts, effectively improving the personal safety and motion standardization of users of prostheses, rehabilitation aids, etc.

[0075] use Figure 11 The processing flow is explained.

[0076] Step 1:

[0077] Users perform everyday actions, such as walking or picking up objects, by wearing biometric detection devices (such as accelerometers and gyroscopes).

[0078] Input: The user's actual actions.

[0079] Output: Raw motion data (such as acceleration, angle values, etc.) collected by the bioinformatics detection device.

[0080] Step 2:

[0081] The terminal receives raw motion data from the bioinformatics detection device.

[0082] Input: Raw data stream sent by the bioinformatics detection device.

[0083] The terminal performs noise removal, outlier removal, and data normalization on the raw data. Specific operations include using tools such as NumPy and SciPy to smooth the data, filter out outliers, and scale the data to a standard range.

[0084] Output: Preprocessed standardized motion data.

[0085] Step 3:

[0086] The terminal uploads the pre-processed standardized action data to the server via the network.

[0087] Input: Standardized motion data.

[0088] The terminal calls the API interface to upload data to the remote server in a structured format (such as JSON).

[0089] Output: Processed action data received by the server.

[0090] Step 4:

[0091] The server uses generative artificial intelligence models, especially recurrent neural networks, built on deep learning platforms (such as TensorFlow and Keras) to perform time-series analysis on the received data.

[0092] Input: Standardized motion data from the terminal.

[0093] The server performs multi-dimensional time-series analysis on motion data, such as determining the continuity and stability of gait, and detecting any risk of falling or abnormal behavior.

[0094] Output: Analysis results of user action status (such as whether there is a risk of falling, whether the action is abnormal, etc.).

[0095] Step 5:

[0096] The server generates corresponding prompts based on the analysis results and sends the instructions to the terminal.

[0097] Input: User action analysis results.

[0098] The server automatically generates alarm or suggestion text messages (prompt statements) based on different risk categories (such as fall risk, abnormal movement, etc.), such as "Abnormal gait detected, risk of fall exists, please adjust your posture".

[0099] Output: User-oriented instruction information.

[0100] Step 6:

[0101] The terminal receives instruction information from the server and displays text prompts on the screen, while simultaneously broadcasting voice messages through a speaker or headphones.

[0102] Input: Command information sent by the server.

[0103] The terminal displays a pop-up notification on the user interface and plays voice messages, while also providing vibration alerts.

[0104] Output: Risk warnings and operational suggestions received by the user.

[0105] Step 7:

[0106] Users can adjust and correct their actions in real time based on the prompts on the terminal.

[0107] Input: Risk warnings and suggestions provided by the terminal.

[0108] Users react quickly to visual and auditory cues, such as pausing walking, adjusting posture, and slowing down.

[0109] Output: Improved user behavior and reduced risk level.

[0110] Application Example 1

[0111] The process flow corresponding to the specific processing in Use Case 1 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. Furthermore, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0112] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is achieved by the following means.

[0113] In this invention, the server includes a device for noise removal and standardization preprocessing of physiological information from a physiological information detection device; a device for anomaly identification and analysis of the processed physiological information based on a generative artificial intelligence model; a device for generating and adjusting prompt content by combining the analysis results with the emotional state inferred from the user's facial expressions or voice information; and a device for providing instructions or suggestions to the user through a terminal in the form of visual or audio information. This enables real-time and efficient monitoring of the user's physiological and emotional state, intelligent anomaly detection, and diverse personalized proactive prompts, thereby improving health and safety protection and the human-computer interaction experience.

[0114] "Physiological information detection device" refers to sensor equipment used to collect biological signals such as user heart rate, skin temperature, electromyography signals, and posture in real time.

[0115] "Noise removal" refers to the process of processing the collected raw physiological information data to remove outliers, errors, or irrelevant interference signals in order to improve data quality.

[0116] "Standardized preprocessing" refers to the process of standardizing raw data to transform it into a data format with uniform dimensions that is easy to analyze and model.

[0117] "Generative artificial intelligence models" refer to artificial intelligence systems that can be trained using large amounts of data and generate analytical results or predictive conclusions through the model, including deep learning models such as recurrent neural networks.

[0118] "Anomaly detection analysis" refers to the process of analyzing a user's physiological or behavioral data through a model to detect abnormal events or risk states that are significantly different from the normal state.

[0119] "Emotional state estimation" refers to the process of using algorithms to infer a user's current psychological or emotional state based on their facial expressions or voice information.

[0120] "Prompt content" refers to the information generated based on system analysis results and the user's current status to convey suggestions, warnings, or instructions to the user.

[0121] A "terminal" refers to an information display device, such as a monitor or speaker, used to receive instructions from a server and present relevant content to the user in various ways, including visual and audio methods.

[0122] "Visual information" refers to information content presented to users in the form of pictures, images, text, etc.

[0123] "Audio information" refers to information content output to users in the form of sound, speech, etc.

[0124] "Deep learning algorithms" refer to machine learning methods based on artificial neural networks that perform nonlinear modeling and feature extraction on large amounts of data, and are often used for complex pattern recognition.

[0125] A "recurrent neural network" is an artificial neural network structure that can process time-series data and has memory capabilities in sequence modeling.

[0126] The specific embodiments of the present invention are as follows.

[0127] This system includes a physiological information detection device, a terminal, a server, and display and voice output devices for user notifications, as well as an emotion recognition module for analyzing the user's emotional state.

[0128] Physiological information detection devices can be hardware devices such as heart rate sensors, skin temperature sensors, electromyography (EMG) sensors, and posture sensors. For example, general-purpose heart rate belts, skin thermometers, and EMG acquisition armbands can be used. These devices can continuously collect heart rate, body temperature, EMG signals, and movement data in real time while the user is driving, walking, exercising, or engaging in other activities.

[0129] Terminals (such as in-vehicle central control systems, smartphones, wearable devices, etc.) connect to sensors via Bluetooth or wired connections to receive raw physiological data in real time. The terminals are equipped with data processing software (such as Python scripts, scikit-learn libraries, etc.) to perform noise removal (e.g., using median filtering, mean filtering, etc.) and standardization preprocessing (e.g., normalization, standard deviation standardization, etc.) on the raw data to generate a data format suitable for subsequent analysis.

[0130] The terminal transmits preprocessed data to the server via secure communication protocols (such as HTTPS, MQTT, etc.). The server runs generative artificial intelligence models, such as recurrent neural networks (RNN, LSTM, and other time-series recognition models) applied to TensorFlow or Keras frameworks, to perform deep learning analysis on the preprocessed time-series data. The server can accurately determine whether the user is currently in an inappropriate physiological state, such as detecting a sudden increase in heart rate, abnormal gait, or uncoordinated movements, or potential risks such as falls.

[0131] Based on the AI ​​model's analysis results, the server automatically invokes natural language generation algorithms and combines them with pre-set prompt templates to generate specific instructions, suggestions, or alarm messages. If the server receives user facial expressions or voice tone information transmitted from the terminal, it will also analyze the user's current emotional state using an emotion recognition module (such as using Face++ cloud API, PyTorch CNN model, etc.) and dynamically adjust the tone of the prompts and the feedback method. For example, if the system detects that the user is anxious or nervous, it will use reassuring language.

[0132] The generated notification content is sent to the terminal via the network. The terminal uses a display screen (such as a car screen, mobile phone screen, etc.) and a TTS speech synthesis engine (such as iFlytek TTS module, etc.) to inform the user of the notification in both text and audio formats.

[0133] This system is particularly suitable for real-world scenarios such as health management for drivers of autonomous vehicles, gait anomaly detection for users of prosthetics / suppositories, and mobility assistance for frail elderly individuals. Through timely and intelligent detection and notification, it can effectively improve user safety and interactive experience.

[0134] Practical application examples:

[0135] 1. When the driver's heart rate in an autonomous vehicle is detected to be higher than the safety threshold for 5 consecutive seconds, the server generates a "Heart rate abnormality detected, please rest immediately" prompt and notifies the user simultaneously through the in-vehicle screen and speaker.

[0136] 2. The prosthetic foot user's gait was analyzed and found to deviate significantly from the normal template in the database. At the same time, anxiety was detected in the user's voice. The server performed emotional adaptation on the notification content and pushed a voice and text message to the user: "A risk of falling has been detected. Please slow down and remain calm."

[0137] 3. When a prosthetic hand user moves an object, abnormal electromyographic signals and a tense expression appear. The system prompts, "The moving motion is not standard. Please slow down and don't rush."

[0138] Example of prompts for generative artificial intelligence models:

[0139] "Based on the heart rate sequence of the last 30 seconds, please determine whether the user has an abnormal heartbeat and provide specific prompts."

[0140] “Analyze the gait data of prosthetic users, identify the risk of falls, and generate warning prompts in a considerate tone.”

[0141] "Detect the user's current emotional state, analyze it in conjunction with physiological data, and generate AI prompts that are most suitable for the current situation."

[0142] This invention combines multiple hardware sensors, terminals and servers with powerful data processing and deep learning capabilities, and intelligent interaction strategies tailored to specific user emotional states to achieve comprehensive monitoring and feedback of physiological, motor, and psychological states, significantly improving user health, safety, and human-computer interaction experience.

[0143] use Figure 12 The processing flow is explained.

[0144] Step 1:

[0145] Users wear physiological information monitoring devices and begin daily activities. Inputs are the user's real-time physiological signals (such as heart rate, skin temperature, electromyography, and gait angles), and outputs are raw data collected by the sensors. Users can then activate the sensors and perform driving, walking, or carrying actions according to their needs.

[0146] Step 2:

[0147] The terminal receives raw data collected in real time from the physiological information detection device and performs noise removal and data standardization processing. The input is raw data transmitted by the sensor; the terminal uses filtering algorithms to remove impulse noise and outliers, and normalization algorithms to adjust the data format. The output is a cleaned and standardized dataset. The terminal reads data via Bluetooth or a wired connection and performs data preprocessing locally.

[0148] Step 3:

[0149] The terminal sends preprocessed data to the server via encrypted communication. The input is the standardized data output from the terminal, and the output is a structured physiological information data packet sent to the server. The terminal calls a REST API or MQTT protocol to upload the data to the server in formats such as JSON, and retains a record of the transmission.

[0150] Step 4:

[0151] The server receives preprocessed physiological information data and performs time-series data analysis using a generative artificial intelligence model. The input is the received standardized physiological information sequence. The server uses LSTM or other deep learning models based on platforms such as TensorFlow to analyze abnormal patterns in the data, such as sudden increases in heart rate and abnormal gait. The output is the anomaly determination result and anomaly category. The server automates the analysis process and generates analysis conclusions based on set thresholds.

[0152] Step 5:

[0153] The server analyzes the results of the AI ​​model to determine whether to generate a prompt message, and combines this with user emotion recognition (such as facial expressions or voice) analysis to adjust the notification content. The inputs are the anomaly detection result and the emotion recognition result; the output is the targeted prompt content. The server calls a natural language generation algorithm, combines it with the user's emotional state, and selects an appropriate template to generate personalized suggestions or warnings.

[0154] Step 6:

[0155] The server sends the notification message to the terminal over the network. The input is the generated notification message (text and optional audio), and the output is the message data packet sent to the terminal. The server pushes the message to the specified terminal via WebSocket or HTTP push, and records the notification sending time and content.

[0156] Step 7:

[0157] The terminal receives notification messages pushed by the server and notifies the user visually and / or audioly through the display screen and TTS (Text-to-Speech) engine. The input is the notification message generated by the server; the terminal displays the text content on the screen and invokes the speech synthesis module to read it aloud; the output is the multimodal notification received by the user. The terminal uses different display methods depending on the urgency, such as full-screen pop-ups, color warnings, and highlighted alarms.

[0158] Step 8:

[0159] Users adjust their behavior or actions based on information fed back from the terminal. The input is the received notification content, and the output is the corresponding countermeasures taken by the user, such as slowing down, stopping driving, or adjusting posture. Users react in real-time to ensure their safety and normal functioning in real-world scenarios.

[0160] Alternatively, an emotion engine for inferring user emotions can be combined. That is, the specific processing unit 290 can also use the emotion-specific model 59 to infer user emotions and perform specific processing using user emotions.

[0161] Example 2

[0162] The flow of a specific process in Example 2 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart device 14. The data processing device 12 will be referred to as the "server," and the smart device 14 as the "terminal."

[0163] While existing assistive systems can monitor users' limb movements and provide necessary instructions, they cannot recognize users' emotional states in real time and lack the function of alleviating users' psychological burden. When using prostheses and other assistive devices, people with disabilities or the elderly often experience increased psychological stress due to improper operation or emotional fluctuations, which can even lead to safety risks. Therefore, current technologies cannot simultaneously address both safe movement guidance and emotional psychological support, making it difficult to effectively improve user experience and safety.

[0164] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Embodiment 2 is achieved by the following means.

[0165] In this invention, the server includes a processing device for preprocessing biometric data from a motion detection device; a computing device for performing behavioral analysis and risk assessment on the preprocessed data using a generative artificial intelligence model; a recognition device for identifying the user's psychological state based on facial or voice information; and a prompting device for generating personalized prompts by combining the user's motion assessment results and psychological state, and promptly notifying the user through visual or auditory means. This enables the integration of user-motion-based safety monitoring and user-emotion-oriented psychological intervention, achieving a synergy between motion guidance and emotional care, significantly improving the safety and quality of life for assistive device users.

[0166] "Motion detection device" refers to hardware equipment that can detect a user's body movements, biosignals and related physical quantities in real time, including accelerometers, gyroscopes, cameras, microphones, etc.

[0167] "Bioinformatics data" refers to various digital information related to a user's limb movements and physiological state collected by motion detection devices, including raw data such as position, speed, acceleration, angle, facial expression, and voice.

[0168] "Processing device" refers to a computing or control unit used to receive and preprocess raw biological information data (such as noise removal, data normalization, etc.).

[0169] "Generative artificial intelligence models" refer to intelligent algorithm systems that can perform pattern recognition, prediction, and judgment based on input data. Typical implementations include machine learning models such as recurrent neural networks.

[0170] "Computing device" refers to a computer device or server unit used to execute generative artificial intelligence models and perform behavioral evaluation and risk assessment on preprocessed data.

[0171] "Sentiment analysis" refers to the computational process of automatically judging and recognizing a user's current emotional state (such as anxiety, anger, calmness, etc.) by analyzing the user's facial expressions or voice information.

[0172] "Identification device" refers to a computational or auxiliary unit that combines emotion analysis results to identify a user's psychological state.

[0173] A "prompt device" refers to a device that generates personalized prompts based on behavioral assessment results and psychological state recognition results, and uses them to output notifications.

[0174] "Visual information output device" refers to an output terminal that can display prompts to users in a visual manner (such as a screen, display, etc.).

[0175] "Audio information output device" refers to an output system that can inform users of prompts through auditory means such as sound and speech.

[0176] "Notification device" refers to a terminal device that is responsible for transmitting prompts to the user in real time through visual information output devices and / or auditory information output devices.

[0177] This invention can be implemented in the following ways.

[0178] First, the system includes a server, terminals, and multiple motion detection devices. These devices can be accelerometers, gyroscopes, cameras, and microphones, used to collect user motion data and emotion-related information (such as facial expressions and voice) in real time. The terminals are portable smart devices, such as mobile devices, tablets, or wearable devices. The server can be a high-performance computing server or a cloud computing platform, used to perform complex calculations such as data processing and emotion recognition.

[0179] The terminal is responsible for receiving raw biometric data from the motion detection device and using data processing software (such as NumPy, scikit-learn, etc.) to remove noise, filter outliers, and normalize the data to obtain a structured and standardized data format. This preprocessed data is then uploaded to the server in real time via communication protocols such as HTTP POST requests.

[0180] The server utilizes generative artificial intelligence models, such as recurrent neural networks (RNNs) developed using the Tensorflow or PyTorch frameworks in Python, to analyze data and assess whether the user's actual actions involve any irregularities or security risks. Simultaneously, the server employs sentiment analysis algorithms, including facial expression recognition algorithms (such as those from the OpenCV library) and voice emotion recognition algorithms (such as those from the Librosa library), to perform real-time analysis of user emotion-related data collected by the camera and microphone, identifying whether the user is in a psychological state such as anxiety, anger, or calmness.

[0181] After obtaining the results of the action risk and psychological state, the server merges the two and automatically generates a personalized prompt (the prompt can be automatically generated using natural language generation technology or templates). Then, the server simultaneously pushes the generated prompt to the terminal via real-time communication mechanisms such as WebSocket.

[0182] After receiving a prompt, the terminal notifies the user in real time of warnings, suggestions, or reassurances through screen display (visual output) and speaker voice broadcast (auditory output). Users can adjust their actions or emotions accordingly to ensure safety and reduce psychological stress.

[0183] In practical implementation, for example, when a user is walking with a prosthetic foot, if the motion detection device detects unstable walking or abnormal acceleration changes, the terminal preprocesses the data and uploads it to the server. The server identifies this as a "high risk of falling" using a generative artificial intelligence model, and simultaneously, sentiment analysis determines that the user's expression is "anxious." At this point, the server sends a prompt to the user stating, "There is a risk of falling; please slow down and remain calm." Upon seeing or hearing the prompt, the user adjusts their movements promptly to avoid danger.

[0184] For example, when a user operates the prosthetic hand to grasp an object, the sensors record that the grasping action is not standard. The terminal processes the data and uploads it to the server. The server comprehensively analyzes the abnormal action and the user's agitated voice, and then pushes an encouraging prompt such as "The action is not standard, please relax and try again" to help the user rationally adjust the operation and reduce emotional stress.

[0185] Typical input prompts for generative artificial intelligence models are as follows:

[0186] The user experienced unsteadiness while walking with a prosthetic foot; the sensors detected the anomaly and reported it. Please analyze the fall risk using generative artificial intelligence and, combined with the user's anxious facial expressions, generate gentle yet specific safety prompts.

[0187] When a user uses their prosthetic hand to grasp an object, their movements are incorrect, and the sensors have detected the corresponding data. Please use generative artificial intelligence to determine the cause of the problem, and consider the user's current impatient tone, to generate encouraging and specific corrective instructions.

[0188] Through the above-mentioned hardware and software configuration and data processing methods, this invention can achieve comprehensive monitoring and evaluation of user action data and psychological state, and output comprehensive prompts to users in a personalized and real-time manner, greatly improving the safety and comfort of users in the process of using assistive devices.

[0189] use Figure 13 The processing flow is explained.

[0190] Step 1:

[0191] Users' limb movements and emotional states in real time are collected by motion detection devices (such as accelerometers, gyroscopes, cameras, and microphones) through assistive devices (such as prosthetics or prosthetic hands).

[0192] The input consists of raw data on the user's actions (such as angle, acceleration, and position) and raw data on emotions (such as facial expressions and voice signals).

[0193] The output is a raw data stream containing action and emotion signals.

[0194] Specifically, the motion detection device automatically collects user data multiple times per second and transmits it to the terminal in real time.

[0195] Step 2:

[0196] After receiving the raw data collected by the sensor, the terminal performs data preprocessing operations, including noise removal, abnormal data filtering, and normalization.

[0197] The input consists of raw values ​​such as gait and acceleration, as well as biological signals such as facial expressions and speech.

[0198] The terminal performs noise reduction and normalization on each piece of data, scaling parameters such as angle and acceleration to a uniform standard range, while removing abnormal or erroneous sampling points.

[0199] The output is structured, standardized preprocessed data.

[0200] The specific action involves the terminal executing a data cleaning algorithm to format the cleaned data into a standard package.

[0201] Step 3:

[0202] The terminal uploads the pre-processed data to the server in real time via HTTP POST or WebSocket.

[0203] The input is the standardized preprocessed data packet.

[0204] Data transmission employs a secure communication protocol to ensure the integrity and timeliness of uploaded data.

[0205] The output is the structured data packet received by the server.

[0206] Specifically, the terminal automatically packages and transmits data at fixed intervals, and the server synchronously receives and verifies the integrity of the data.

[0207] Step 4:

[0208] The server uses generative artificial intelligence models (such as RNN models) to analyze action data and determine whether the user's actions are irregular or pose a security risk.

[0209] The input is preprocessed structured action data.

[0210] The server calls the trained RNN model, inputs the data into the model, and the model outputs evaluation results such as "normal action", "non-standard action" or "risk of falling".

[0211] The output is the evaluation conclusion data of motion analysis.

[0212] Specifically, the server automatically selects the latest set of action data, inputs it into the model in batches, and obtains risk predictions or behavioral evaluations.

[0213] Step 5:

[0214] The server uses sentiment analysis algorithms to identify a user's emotional state by analyzing facial expressions and voice signals.

[0215] The input consists of standardized facial and voice data from the camera and microphone.

[0216] The server runs facial expression recognition algorithms (such as OpenCV) and speech emotion recognition tools (such as Librosa) to extract features and determine emotional tendencies from the data, resulting in specific emotion labels such as "anxiety," "anger," and "calm."

[0217] The output is the recognition result of the user's current emotional state.

[0218] Specifically, the server automatically captures sentiment data that is consistent with the timing of the action data and runs multi-threaded sentiment recognition tasks in parallel.

[0219] Step 6:

[0220] The server correlates the action evaluation results and emotion recognition results of the generative artificial intelligence model to automatically generate personalized prompts.

[0221] The input consists of action analysis results and sentiment labels.

[0222] The server automatically matches a prompt template or calls the prompt generation module based on different combinations to generate prompt text suitable for the current scenario.

[0223] The output consists of one or more prompt statements.

[0224] The specific actions involve the server querying the prompt template library or automatically generating new prompts through the natural language processing module.

[0225] Step 7:

[0226] The server pushes the generated prompts to the terminal in real time via communication methods such as WebSocket.

[0227] The input is the text of the generated prompt statement.

[0228] The server calls the message push interface to send text to the display or broadcast module specified on the terminal.

[0229] The output is the prompt message received by the terminal.

[0230] Specifically, the server identifies currently active terminals and pushes point-to-point real-time text messages.

[0231] Step 8:

[0232] After receiving a prompt, the terminal will display the prompt on the screen and / or through the speaker, conveying warnings, suggestions, or reassurances to the user.

[0233] The input is the prompt message pushed by the server.

[0234] The terminal calls the display driver module to display the prompt content on the screen and calls the audio driver module to perform voice broadcast.

[0235] The output is visible and audible guidance information for the user.

[0236] The specific actions include displaying a prompt box on the terminal and playing audio simultaneously to ensure that users receive information in a timely manner.

[0237] Step 9:

[0238] Users can adjust their actions or mental state in a timely manner based on the prompts they receive, thereby optimizing the effectiveness of assistive devices and ensuring safety.

[0239] The input is the perceived multimodal prompt information.

[0240] Users can adjust their walking pace, posture, or hand movements based on the suggested actions, and proactively relieve their tension or emotions when necessary.

[0241] The output is an improvement in the user's safety behavior and psychological state.

[0242] The specific actions involve the user responding to the prompts, such as slowing down, re-gripping, or relaxing.

[0243] Application Example 2

[0244] The process flow corresponding to the specific processing in Use Case 2 will be described below. The various parts of the system described below are implemented by the data processing device 12 and the intelligent device 14. In addition, the data processing device 12 is referred to as the "server" and the intelligent device 14 is referred to as the "terminal".

[0245] While existing work environments can monitor and analyze users' physical movements in real time using biometric sensors, they cannot provide targeted psychological support by incorporating users' emotional states. Furthermore, they cannot generate appropriate prompts in a timely manner when inappropriate movements or dangers are detected, based on emotional states, to effectively improve user operational safety and psychological comfort. Therefore, how to develop an intelligent prompting system based on comprehensive analysis of movement and emotion to simultaneously ensure the safety and mental well-being of workers is a pressing technical challenge in this field.

[0246] The specific processing performed by the specific processing unit 290 of the data processing apparatus 12 in Application Example 2 is achieved by the following means.

[0247] In this invention, the server includes: a device for preprocessing biological information acquired from a bioinformatics detection device; a device for analyzing the preprocessed biological information using a generative artificial intelligence model to detect inappropriate actions or dangers; an emotion inference device for analyzing image and audio information to infer the user's emotional state; and a control device for generating appropriate prompts based on action and emotion analysis results and notifying the user visually or auditorily. This enables comprehensive intelligent analysis of the operator's actions and emotional state, and timely issuance of personalized, contextualized prompts, thereby effectively improving work safety and mental health.

[0248] "Bioinformatics detection device" refers to a device that can collect and acquire biological data such as the user's body movements and physiological parameters in real time.

[0249] "Preprocessing" refers to the process of performing operations such as denoising, normalization, and format conversion on the acquired raw bioinformatics data to make it suitable for subsequent analysis and processing.

[0250] "Information processing device" refers to a computing device or server used to receive, store and analyze data, and to execute algorithms and data processing tasks.

[0251] "Generative artificial intelligence models" refer to artificial intelligence algorithm systems that can automatically generate analysis results or judgment results through big data training.

[0252] "Inappropriate actions" refer to user behaviors that are inconsistent with pre-set operating standards or safety regulations and may lead to danger or affect work efficiency.

[0253] "Hazard" refers to a risk factor that exists during operation or work that could lead to personal injury, equipment damage, or a safety accident.

[0254] "Emotion inference device" refers to a hardware or software system that infers a user's current emotional state by analyzing multimodal data such as images and voice.

[0255] "Prompt statements" refer to reminders, warnings, or suggestions generated by the system for users based on action analysis and emotion inference results.

[0256] "Control device" refers to the computing and management device that generates prompts based on analysis results and coordinates the collaborative work of various functional units.

[0257] "Prompt device" refers to an interface device that can output prompts or information to users through visual, auditory, or other means.

[0258] "Judgment benchmark" refers to the operational standards, rules, or thresholds that the system uses to distinguish whether an action is compliant.

[0259] "Terminal device" refers to a data processing device used to receive biological information and server instructions, and to interact with users.

[0260] The system of this invention achieves comprehensive intelligent analysis and personalized prompts for the actions and emotions of workers through the coordinated work of a bio-information detection device, a terminal device, a server, and a prompting device.

[0261] The server includes hardware capable of performing information processing and software components that support artificial intelligence analysis. Specifically, the server can be a general-purpose computer server, on which generative AI modeling and inference tools such as TensorFlow and Keras are deployed, along with the OpenCV library for facial expression data analysis and the SpeechRecognition library for speech emotion analysis. The server also has a built-in module for automatically generating and managing prompts.

[0262] The terminal device includes data acquisition and communication equipment with processing capabilities, such as smartphones, tablets, and industrial control all-in-one computers. The terminal device connects in real-time with bio-information detection devices (such as inertial motion capture sensors and heart rate sensors) via Bluetooth, Wi-Fi, etc., to acquire the user's motion information, facial expression video, and voice / audio data. It then performs preliminary data preprocessing (such as noise filtering and normalization), primarily using data processing libraries such as NumPy and Pandas, before uploading the data to the server.

[0263] The bioinformatics detection device is mainly responsible for collecting users' motion data and physiological signals in real time, ensuring the originality and timeliness of the data. These motion data, such as limb angles, positions, and accelerations, are periodically sampled and sent to the terminal.

[0264] In typical application scenarios, when users operate robots or production equipment in a factory, the terminal device collects the user's motion information and audio / video data. This data, in collaboration with the server, determines whether the user is engaging in inappropriate actions or posing a safety hazard, while also assessing the user's psychological state. The server uses a generative artificial intelligence model to recognize and judge the received pre-processed data, and simultaneously analyzes the user's emotions using an emotion inference module. Once the system detects a potential danger in the user's actions and their emotional state is anxious or agitated, it automatically generates personalized prompts and immediately notifies the user through the terminal's speaker and display screen. Examples include: "Danger, please adjust your posture slowly."; "Inappropriate action, please calm down and double-check."

[0265] Through the above methods, the present invention can not only support users' safe and efficient operation in real time, but also provide users with psychological support, significantly improving the safety and comfort of industrial work sites.

[0266] use Figure 14 The processing flow is explained.

[0267] Step 1:

[0268] The terminal collects users' motion data, facial expressions, and voice data in real time through biometric detection devices. Inputs include motion parameters such as limb angles and accelerations, as well as video data captured by the camera and audio data captured by the microphone. The terminal stores and performs preliminary processing on this raw data, such as timestamping the motion data and standardizing the format of the video and audio, and outputs structured multimodal raw data.

[0269] Step 2:

[0270] The terminal preprocesses the collected motion data. The input is the raw motion data. The terminal uses data processing libraries such as NumPy or Pandas to perform preprocessing operations such as noise filtering, normalization, and missing value imputation to remove outliers and standardize the units, and outputs preprocessed motion data.

[0271] Step 3:

[0272] The terminal uploads the pre-processed motion data, facial expression video, and audio data to the server via a secure communication protocol. The input is the pre-processed data packet, and the output is the data file or data stream successfully uploaded to the server.

[0273] Step 4:

[0274] The server receives data uploaded from the terminal. Inputs include structured motion data, facial expression video, and audio. The server first verifies the integrity and validity of the data, then sends it to the appropriate analysis module. The data is stored in a temporary storage area and assigned a unique identifier; the output is a categorized data stream and metadata.

[0275] Step 5:

[0276] The server uses generative artificial intelligence models to analyze motion data. The input is validated motion data. The server calls deep learning models such as TensorFlow and Keras to identify, classify, and assess the risks of user actions, analyzing whether the user is performing inappropriate or dangerous actions. The model's output includes analysis results such as action type and risk level.

[0277] Step 6:

[0278] The server synchronously analyzes facial expression videos and audio recordings using a sentiment inference module. The input consists of raw image and audio files. The server acquires facial expression feature points from the user using OpenCV, analyzes the emotional nuances of the speech using SpeechRecognition, and ultimately infers the user's current emotional state (e.g., anxiety, focus). The output is an analysis label and score for the emotional state.

[0279] Step 7:

[0280] The server automatically generates personalized prompts based on a comprehensive assessment of motion and sentiment analysis results. Inputs include the motion analysis report and sentiment tags. The server invokes the prompt generation algorithm, selecting or automatically generating appropriate text prompts from a pre-defined library, such as "Danger, please adjust your posture slowly." Output is the text data of the prompt.

[0281] Step 8:

[0282] The server sends the generated prompt message to the terminal over the network. The input is the text prompt message generated by the server, and the output is the prompt message received by the terminal.

[0283] Step 9:

[0284] After receiving the prompt, the terminal uses a local speech synthesis library (such as pyttsx3) to convert the text prompt into speech, plays it through the speaker, and simultaneously displays the prompt content in a large-font pop-up window on the screen. The input is the text prompt, and the output is the intuitive prompt that the user can see and hear.

[0285] Step 10:

[0286] Users adjust their behavior and posture in a timely manner based on the prompts provided by the terminal. Input consists of visual and auditory cues; users then make judgments and decisions before taking appropriate actions. Output includes the user's adjusted actions and improved mental state.

[0287] The specific processing unit 290 sends the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires sound representing user input regarding the result of the specific processing. The control unit 46A sends the sound data representing user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0288] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0289] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or external devices, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or external devices.

[0290] For example, the collection unit is implemented by the control unit 46A of the smart device 14 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart device 14 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the output device 40 of the smart device 14 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0291] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart device 14.

[0292] Second Implementation Method

[0293] Figure 3 An example of the configuration of the data processing system 210 according to the second embodiment is shown.

[0294] like Figure 3 As shown, the data processing system 210 includes a data processing device 12 and smart glasses 214. A server can be cited as an example of the data processing device 12.

[0295] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0296] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, and communication I / F 44 are also connected to the bus 52.

[0297] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0298] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0299] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0300] Figure 4 This illustrates an example of the main functions of the data processing device 12 and the smart glasses 214. For example... Figure 4 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0301] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0302] The memory 32 stores a data generation model 58 and an emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290. The specific processing unit 290 can use the emotion-specific model 59 to infer the user's emotions and perform specific processing based on the user's emotions. In the emotion inference function (emotion-specific function) using the emotion-specific model 59, various inferences and predictions related to the user's emotions are performed, including inferences and predictions of the user's emotions, but this is not limited to this example. Furthermore, emotion inference and prediction may also include, for example, emotion analysis (parsing).

[0303] In the smart glasses 214, the processor 46 performs reception and output processing. The memory 50 stores the reception and output program 60. The processor 46 reads the reception and output program 60 from the memory 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is implemented by the processor 46 operating as a control unit 46A according to the reception and output program 60 executed on the RAM 48. Furthermore, the smart glasses 214 has the same data generation model and emotion-specific model as the data generation model 58 and the emotion-specific model 59, and these models can also be used to perform the same processing as the specific processing unit 290.

[0304] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the smart glasses 214. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0305] Example 1

[0306] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0307] Application Example 1

[0308] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0309] Example 2

[0310] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0311] Application Example 2

[0312] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0313] The specific processing unit 290 sends the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A outputs the result of the specific processing to the speaker 240. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0314] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0315] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or external devices, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or external devices.

[0316] For example, the collection unit is implemented by the control unit 46A of the smart glasses 214 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the smart glasses 214 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the smart glasses 214 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0317] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the smart glasses 214.

[0318] Third Implementation Method

[0319] Figure 5 An example of the configuration of the data processing system 310 according to the third embodiment is shown.

[0320] like Figure 5 As shown, the data processing system 310 includes a data processing device 12 and a head-mounted terminal 314. A server can be cited as an example of the data processing device 12.

[0321] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0322] The head-mounted terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, display 343, and communication I / F 44 are also connected to the bus 52.

[0323] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0324] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to capture images of the user 20's surroundings (e.g., the field of view defined by an angle equivalent to the field of vision of an average healthy person).

[0325] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0326] Figure 6 This illustrates an example of the main functions of the data processing device 12 and the head-mounted terminal 314. For example... Figure 6 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0327] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0328] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0329] In the head-mounted terminal 314, the processor 46 performs the acceptance / output processing. The memory 50 stores the acceptance / output program 60. The processor 46 reads the acceptance / output program 60 from the memory 50 and executes the read acceptance / output program 60 on the RAM 48. The acceptance / output processing is implemented by the processor 46 operating as a control unit 46A according to the acceptance / output program 60 executed on the RAM 48.

[0330] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the head-mounted terminal 314. In the following description, the data processing device 12 will be referred to as the "server" and the head-mounted terminal 314 will be referred to as the "terminal".

[0331] Example 1

[0332] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0333] Application Example 1

[0334] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0335] Example 2

[0336] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0337] Application Example 2

[0338] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0339] The specific processing unit 290 sends the result of the specific processing to the head-mounted terminal 314. In the head-mounted terminal 314, the control unit 46A outputs the result of the specific processing to the speaker 240 and the display 343. The microphone 238 acquires sound input representing the user's input regarding the result of the specific processing. The control unit 46A sends the sound data representing the user's input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0340] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0341] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the head-mounted terminal 314, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the head-mounted terminal 314. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the head-mounted terminal 314 or external devices, and the head-mounted terminal 314 acquires or collects information required for processing from the data processing device 12 or external devices.

[0342] For example, the collection unit is implemented by the control unit 46A of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the head-mounted terminal 314 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12 to analyze the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12 to generate a menu using a generation AI. For example, the serving unit is implemented by the speaker 240 and display 343 of the head-mounted terminal 314 or the specific processing unit 290 of the data processing device 12 to provide the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0343] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the head-mounted terminal 314.

[0344] Fourth Implementation Method

[0345] Figure 7 An example of the configuration of the data processing system 410 according to the fourth embodiment is shown.

[0346] like Figure 7 As shown, the data processing system 410 includes a data processing device 12 and a robot 414. A server can be cited as an example of the data processing device 12.

[0347] The data processing apparatus 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" as understood in this disclosure. The computer 22 includes a processor 28, RAM 30, and memory 32. The processor 28, RAM 30, and memory 32 are connected to a bus 34. Furthermore, the database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0348] Robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and memory 50. The processor 46, RAM 48, and memory 50 are connected to a bus 52. Furthermore, the microphone 238, speaker 240, camera 42, controlled object 443, and communication I / F 44 are also connected to the bus 52.

[0349] Microphone 238 receives instructions from user 20 by receiving sounds emitted by user 20. Microphone 238 captures sounds emitted by user 20 and converts the captured sounds into sound data, which is then output to processor 46. Speaker 240 outputs sound according to instructions from processor 46.

[0350] Camera 42 is a small digital camera equipped with an optical system such as a lens, aperture and shutter, and imaging elements such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, to photograph the area around robot 414 (e.g., the field of view defined by a perspective equivalent to the field of vision of an average healthy person).

[0351] Communication I / F44 is connected to network 54. Communication I / F44 and 26 are responsible for the transmission and reception of various information between processor 46 and processor 28 via network 54. The transmission and reception of various information between processor 46 and processor 28 using communication I / F44 and 26 is performed in a secure state.

[0352] The controlled object 443 includes a display device, LEDs (light-emitting diodes) for the eyes, and motors for driving the arms, hands, and feet. The posture or movement of the robot 414 is controlled by controlling the motors in the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. In addition, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0353] Figure 8 This illustrates an example of the main functions of the data processing device 12 and the robot 414. For example... Figure 8 As shown, in the data processing device 12, specific processing is performed by the processor 28. The specific processing program 56 is stored in the memory 32.

[0354] The specific processing program 56 is an example of a "program" involved in the technology of this disclosure. The processor 28 reads the specific processing program 56 from the memory 32 and executes the read specific processing program 56 on the RAM 30. Specific processing is implemented by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0355] The memory 32 stores the data generation model 58 and the emotion-specific model 59. The data generation model 58 and the emotion-specific model 59 are used by the specific processing unit 290.

[0356] In robot 414, the processor 46 performs the acceptance and output processing. The memory 50 stores the acceptance and output program 60. The processor 46 reads the acceptance and output program 60 from the memory 50 and executes the read acceptance and output program 60 on RAM 48. The acceptance and output processing is implemented by the processor 46 acting as the control unit 46A according to the acceptance and output program 60 executed on RAM 48.

[0357] Next, the specific processing of the specific processing unit 290 of the data processing device 12 will be described. Each part of the system described below is implemented by the data processing device 12 and the robot 414. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 will be referred to as the "terminal".

[0358] Example 1

[0359] The process is the same as that of the specific process described in Embodiment 1 in the first embodiment above, so the description is omitted.

[0360] Application Example 1

[0361] The process is the same as that in the specific processing described in Application Example 1 of the first embodiment above, so the description is omitted.

[0362] Example 2

[0363] The process is the same as that of the specific process in Embodiment 2 described in the first embodiment above, so the description is omitted.

[0364] Application Example 2

[0365] The process is the same as that in the specific processing described in Application Example 2 of the first embodiment above, so the description is omitted.

[0366] The specific processing unit 290 sends the result of the specific processing to the robot 414. In the robot 414, the control unit 46A outputs the result of the specific processing to the speaker 240 and the controlled object 443. The microphone 238 acquires sound input representing the result of the specific processing. The control unit 46A sends the sound data representing the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the sound data.

[0367] Data generation model 58 is a so-called generative AI (Artificial Intelligence). Examples of data generation models 58 include ChatGPT (registered trademark) (accessible via the internet (URL: https: / / openai.com / blog / chatgpt)). Data generation model 58 is obtained through deep learning on a neural network. Input to data generation model 58 are prompt words containing instructions, and inference data such as sound data representing sound, text data representing text, and image data representing images (e.g., still image data or animation data). Data generation model 58 infers from the input inference data based on the instructions represented by the prompt words and outputs the inference result in one or more data forms, such as sound data, text data, and image data. Data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or induction. The specific processing unit 290 performs the aforementioned specific processing while using data generation model 58. The data generation model 58 can also be a model finely tuned to output inference results from prompts that do not contain instructions. In this case, the data generation model 58 can output inference results based on prompts that do not contain instructions. The data processing apparatus 12, etc., includes various data generation models 58, including AI other than the generation AI. AI other than the generation AI can be, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or Naive Bayes, and can perform various processes, but is not limited to this example. Furthermore, the AI ​​can also be an AI agent. Furthermore, when the processing of the above-mentioned parts is performed by AI, the processing can be partially or entirely performed by AI, but is not limited to this example. Furthermore, the processing performed by the AI ​​including the generation AI can be replaced by processing in the rule base, and the processing in the rule base can also be replaced by processing performed by the AI ​​including the generation AI.

[0368] Furthermore, the processing of the aforementioned data processing system 10 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it can also be performed by both the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. Additionally, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or external devices, and the robot 414 acquires or collects information required for processing from the data processing device 12 or external devices.

[0369] For example, the collection unit is implemented by the control unit 46A of the robot 414 or the specific processing unit 290 of the data processing device 12. For example, the acquisition unit uses the camera 42 or communication I / F 44 of the robot 414 to acquire step data, which is then processed by the specific processing unit 290 of the data processing device 12. For example, the analysis unit is implemented by the specific processing unit 290 of the data processing device 12, which analyzes the data from the collection unit and the acquisition unit. For example, the generation unit is implemented by the specific processing unit 290 of the data processing device 12, which uses a generation AI to generate a menu. For example, the serving unit is implemented by the speaker 240 of the robot 414 and the control object 443 or the specific processing unit 290 of the data processing device 12, which provides the generated menu to the user. The correspondence between each unit and the device or control unit is not limited to the above examples and various changes can be made.

[0370] In the above embodiments, examples of specific processing by the data processing device 12 are given, but the technology disclosed herein is not limited to this, and specific processing may also be performed by the robot 414.

[0371] Furthermore, the emotion-specific model 59, acting as an emotion engine, can determine a user's emotion based on a specific mapping. Specifically, the emotion-specific model 59 can determine a user's emotion based on an emotion graph that serves as a specific mapping (see [reference]). Figure 9 The system determines the user's emotions. Furthermore, the emotion-specific model 59 can similarly determine the robot's emotions, and the specific processing unit 290 performs specific processing based on the robot's emotions.

[0372] Figure 9 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. On the outer side of the concentric circles, emotions representing states or behaviors arising from mood are arranged. Emotions are concepts that include feelings and mental states. Emotions generated by reactions occurring in the brain are arranged roughly to the left of the concentric circles. Emotions derived from situational judgments are arranged roughly to the right of the concentric circles. Emotions generated by reactions occurring in the brain and derived from situational judgments are arranged roughly above and below the concentric circles. Furthermore, "pleasant" emotions are arranged above the concentric circles, and "unpleasant" emotions are arranged below them. Thus, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to occur simultaneously are mapped close to each other.

[0373] These emotions are distributed at the three o'clock position of the emotion map 400, typically fluctuating between peace and anxiety. In the right half of the emotion map 400, situational awareness dominates over internal sensation, thus resulting in an impression of calm.

[0374] The inner side of the emotion map 400 represents the inner state, while the outer side represents behavior. Therefore, the further outward you are from the emotion map 400, the more visible the emotion becomes (manifested in behavior).

[0375] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotions in robots, cars, motorcycles, etc., can also be created in the following way: based on various balances such as posture and remaining battery power, when these balances deviate from an ideal state, it indicates an unpleasant state; when they approach the ideal state, it indicates a pleasant state. Emotion maps can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory-dominated region, called "response," are arranged. Furthermore, in the right half of the emotion map, emotions belonging to the situational cognition-dominated region, called "situation," are arranged.

[0376] In the emotion map, two types of emotions that promote learning are defined. One is a negative emotion on the situational side, in the middle or peripheral region of "repentance" or "reflection." This occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be blamed again." The other is a positive emotion on the response side, near the "desire" region. This occurs when there are positive feelings such as "wanting more" or "wanting to know more."

[0377] The emotion-specific model 59 inputs user input into a pre-trained neural network to obtain emotion values ​​representing each emotion shown in the emotion map 400, thereby determining the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 10 As shown in the sentiment graph 900, it was trained in a way that sentiments that are configured close to each other have similar values. Figure 10 The text shows examples of emotions such as "peace of mind", "stability", and "reassurance" that have similar emotion values.

[0378] The above description focuses on the functions of the data processing device 12, but the system of this disclosure is not necessarily installed on a server. The system of this disclosure can also be installed as a general information processing system. This disclosure can also be installed, for example, as a software program running on a personal computer, an application running on a smartphone, etc. The method of this disclosure can also be provided to users in the form of SaaS (Software as a Service).

[0379] In the above embodiments, an example of a specific process being performed by a single computer 22 is given. However, the technology disclosed herein is not limited to this, and the specific process can also be distributed among multiple computers, including computer 22. For example, the data generation model 58 can be located on an external device of the data processing apparatus 12, where data is generated based on the input data.

[0380] In the above embodiments, examples of storing a specific processing program 56 in the memory 32 have been described, but the technology disclosed herein is not limited thereto. For example, the specific processing program 56 may also be stored in a portable computer-readable non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed into the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0381] Alternatively, a specific processing program 56 may be pre-stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 according to the requirements of the data processing device 12.

[0382] In addition, it is not necessary to store all the specific processing program 56 in the storage device such as the server connected to the data processing device 12 via the network 54 or in the memory 32; a portion of the specific processing program 56 may be stored in advance.

[0383] As hardware resources for performing specific processes, various processors, as shown below, can be used. For example, a CPU can be listed as a processor, which functions as a general-purpose processor that performs specific processes by executing software, i.e., a program. Furthermore, processors can be listed as special-purpose circuits such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application-Specific Integrated Circuits), which are processors with circuitry specifically designed to perform specific processes. Each processor has built-in or connected memory, and each processor executes specific processes using that memory.

[0384] The hardware resources for performing a specific process can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources for performing a specific process can be a single processor.

[0385] As an example of a single processor, there are two approaches: First, a processor is composed of a combination of one or more CPUs and software, which functions as a hardware resource to perform a specific process; second, as represented by a SoC (System-on-a-chip), a processor is used to implement the functionality of the entire system, which includes multiple hardware resources for performing a specific process, using a single IC (Integrated Circuit) chip. In this way, the specific process is implemented by using one or more of the aforementioned processors as hardware resources.

[0386] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits that combine semiconductor elements and other circuit components. Moreover, the specific process described above is just one example. Therefore, without departing from the main point, unnecessary steps can certainly be deleted, new steps added, or the processing order changed.

[0387] The descriptions and illustrations above are detailed explanations of a portion of the technology disclosed herein, and are merely one example of the technology disclosed herein. For example, the above descriptions of the structure, function, effect, and results are just one example of the structure, function, effect, and results of a portion of the technology disclosed herein. Therefore, without departing from the spirit of the technology disclosed herein, unnecessary parts may be deleted, new elements added, or replacements may be made to the descriptions and illustrations above. Furthermore, to avoid confusion and facilitate understanding of a portion of the technology disclosed herein, explanations of common technical knowledge that do not require special explanation under the premise of being able to implement the technology disclosed herein have been omitted from the descriptions and illustrations above.

[0388] All documents, patent applications and technical specifications set forth in this specification are incorporated herein by reference to the same extent that each document, patent application and technical specification is specifically and individually described therein and referenced by reference.

[0389] In addition, the following notes are provided in response to the above explanation.

[0390] Example 1

[0391] (Note 1)

[0392] An information processing system includes: a signal processing device for preprocessing biological motion information acquired from a bioinformatics detection device, such as noise cancellation and data normalization; an information processing device for sending the preprocessed biological motion information to the information processing device and evaluating and judging abnormal actions or dangerous states using a generative artificial intelligence model including time-series analysis; an instruction generation and communication device for generating instruction information based on the analysis results of the generative artificial intelligence model and sending it to a user terminal via a communication device; a notification device for visual display on the user terminal via a display device or voice notification via an audio output device; a benchmark comparison device for pre-storing benchmark information used for analysis and comparing the benchmark information with the collected motion data to determine the appropriateness of the action when judging abnormal actions; and a feedback device for enabling the user to correct their actions in real time after receiving a notification.

[0393] (Note 2)

[0394] According to the information processing system described in Appendix 1, the bio-information detection device continuously monitors the user's actions and transmits the bio-action information to the signal processing device via wireless or wired communication.

[0395] (Note 3)

[0396] According to the information processing system described in Appendix 1, the generative artificial intelligence model includes a recurrent artificial intelligence network structure for analyzing temporal biological action information.

[0397] Application Example 1

[0398] (Note 1)

[0399] An information processing system includes: a device for performing noise removal and standardization preprocessing on physiological information from a physiological information detection device; a device for analyzing the preprocessed physiological information and using a generative artificial intelligence model to determine whether a user is in an unsuitable physiological state or a risky state; a device for generating and adjusting notification content based on the analysis results and the user's emotional state inference, and providing instructions or suggestions to the user; a device for presenting the notification content to the user via a terminal in the form of at least one of visual or audio information; a device for analysis using a generative artificial intelligence model with a deep learning algorithm that includes time-series information analysis capabilities; and an emotion recognition device for inferring the user's emotional state based on the user's facial expression information or voice information.

[0400] (Note 2)

[0401] According to the information processing system described in Appendix 1, the physiological information detection device can collect the user's physiological information in real time and send the physiological information to the terminal.

[0402] (Note 3)

[0403] According to the information processing system described in Note 1, the generative artificial intelligence model is an algorithm that includes a recurrent neural network for temporal learning.

[0404] Example 2

[0405] (Note 1)

[0406] An information processing system includes: a processing device for preprocessing biometric data from a motion detection device; a computing device for inputting the preprocessed biometric data into a generative artificial intelligence model for behavior assessment and risk determination; a recognition device for identifying a user's psychological state based on the evaluation results of the computing device and in conjunction with emotion analysis based on user facial expression information or voice information; and a prompting device for generating personalized prompt statements based on the evaluation results and psychological state recognition results and conveying them to the user; wherein the prompting device uses a visual information output device and / or an auditory information output device to notify the user of the prompt statements in real time.

[0407] (Note 2)

[0408] According to the information processing system described in Appendix 1, the motion detection device can simultaneously collect the user's motion state and emotional state in real time, and send multiple information signals to the processing device.

[0409] (Note 3)

[0410] According to the information processing system described in Note 1, the generative artificial intelligence model uses a recurrent neural network machine learning algorithm to achieve behavior assessment and risk prediction.

[0411] Application Example 2

[0412] (Note 1)

[0413] An information processing system includes: a device for preprocessing biological information acquired from a bio-information detection device; a device for analyzing the preprocessed biological information and detecting inappropriate actions or dangers using a generative artificial intelligence model employed in the information processing device; an emotion inference device for acquiring image and audio information of a user and inferring emotional state through analysis of facial expression and voice data; a control device for generating appropriate prompt statements based on the action analysis results of the generative artificial intelligence model and the emotion analysis results of the emotion inference device; a prompting device for notifying the user of the prompt statements through at least one of visual or auditory information; and a device for judging the inappropriate action by comparing it with a pre-stored judgment criterion.

[0414] (Note 2)

[0415] According to the information processing system described in Appendix 1, the bio-information detection device can detect the user's bio-information in real time and send the information to the terminal device.

[0416] (Note 3)

[0417] According to the information processing system described in Note 1, the generative artificial intelligence model is analyzed using an algorithm that includes a recursive artificial intelligence model.

Claims

1. An information processing system, characterized in that, include: A device for preprocessing motion data received from bioengineered sensors; Devices used to analyze pre-processed motion data using generative artificial intelligence to detect inappropriate actions or dangers. and A device for providing instructions to a user based on the analysis results of the generative artificial intelligence.

2. The information processing system according to claim 1, characterized in that, The bioengineered sensor is used to detect the user's actions in real time and send the data to the terminal.

3. The information processing system according to claim 1, characterized in that, The generative artificial intelligence employs a model that includes recurrent neural networks.

4. The information processing system according to claim 1, characterized in that, The device for providing instructions to a user provides instructions to a user in at least one of visual display and voice notification.

5. The information processing system according to claim 1, characterized in that, The inappropriate action is determined by comparing it with a predefined standard.

Images