System

Abstract

To provide a system.SOLUTION: A system for providing guidance about a body movement from a guidance database in real-time so as to allow a user to understand a movement or activity performed in daily life and perform a safe movement or activity includes: means for accepting a question from the user about the movement or activity; means for extracting information related to the question from the guidance database and generating the guidance in real-time; and means for providing the generated guidance to the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Accurate guidance on mobility and activity is necessary to support safe mobility and appropriate physical activity in daily life for seniors. However, seniors may have difficulty obtaining information on mobility and activity, for example, they may have difficulty consulting with experts or finding appropriate guidance. [Means for solving the problem]

[0005] The present disclosure provides appropriate guidance in real time to questions about movements and activities that seniors perform in their daily lives. Specifically, the present disclosure provides a system including the following means.

[0006] The system includes a means for accepting questions from a user about movement and activities. The system enables the user to accept questions about movement and activities through the terminal. This allows users, especially seniors, to easily ask questions when they have doubts or concerns.

[0007] The system includes means for extracting information relevant to the query from a guidance database and generating guidance in real time. The system searches the guidance database based on the received query to extract relevant information related to movements and activities, and generates guidance based on the information in real time.

[0008] The system includes a means for providing the generated guidance to a user. The system provides the generated guidance to the user through a terminal. The user can practice safe movement and appropriate body movements by referring to the guidance.

[0009] Through these means, seniors can easily access guidance to support safe mobility and proper physical activity in their daily lives.

[0010] "Questions about movement or activity" refer to questions that express doubts or concerns about the movement or activities that the user engages in in their daily life.

[0011] The "guidance database" is a database that contains information to support safe mobility and proper physical movement for seniors. This database includes specific guidance on mobility and activity, such as how to maintain balance when going up and down stairs and how to walk with correct posture.

[0012] "Generating guidance in real time" means generating appropriate guidance immediately in response to a question from a user. The system extracts information from a guidance database based on the user's question and generates appropriate guidance on the spot.

[0013] "Providing to the user" refers to conveying the guidance generated by the system to the user through the terminal. The user can refer to the guidance displayed on the terminal to practice safe movement and appropriate body movements. [Brief description of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12] 11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0018] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0035] The embodiment for implementing the present disclosure includes the following elements.

[0036] (1) Server

[0037] The server receives a question from the user regarding a movement or activity, extracts appropriate information from the guidance database, generates guidance in real time, and transmits the generated guidance to the terminal. The server may take into account the characteristics and requirements of the user's movement or activity when extracting appropriate information for the question from the guidance database.

[0038] (2) Terminal

[0039] The terminal allows the user to input questions about movements or activities and transmits them to the server, and also displays and provides guidance received from the server to the user.

[0040] (3) Guidance database

[0041] The guidance database stores specific guidance on movements and activities. The server searches the guidance database based on the user's question and extracts appropriate information.

[0042] (4) User

[0043] The user sends questions about movement and activities to the server through the device, and receives guidance from the device to practice safe movement and appropriate physical movements.

[0044] The above elements are combined to realize a system that supports users in safe movement and appropriate physical movement in daily life. The system understands the movements or activities that the user performs in daily life, and provides real-time guidance on physical movements from a guidance database to enable safe movement or activity. The server extracts information from the guidance database, generates guidance in real-time, and transmits it to the terminal. The terminal provides guidance to the user, and the user acts based on that guidance. The system is constructed according to this form to enable users to achieve safe movement and appropriate physical movement.

[0045] The process flow will be explained below.

[0046] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0047] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0048] Step 3: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. Specifically, the server compiles specific procedures and precautions for movement or activity and prepares them as guidance.

[0049] Step 4: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0050] Step 5: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal screen. The user then acts while referring to the guidance. The user refers to the guidance displayed on the terminal and practices safe movement and appropriate physical movements. Specifically, the user moves and engages in activities by following the guidance, such as holding onto handrails and paying attention to where they are stepping.

[0051] Through the above process steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance.

[0052] Example 1

[0053] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0054] Conventional guidance systems have difficulty in providing accurate real-time guidance for users' movements and activities in daily life. In addition, there is a lack of means to provide customized guidance that fully takes into account the specific circumstances and requirements of users, which has led to a demand for improved safety and efficiency.

[0055] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0056] In this invention, the server includes a means for accepting a question from a user regarding movement or activity via a terminal, a server means for analyzing the content of the question, extracting information related to the question from the guidance database, and generating the guidance in real time using a generative AI model, and a terminal means for providing the generated guidance to the user, thereby making it possible to provide customized guidance in real time that takes into account the specific situation and requirements of the user.

[0057] "User" refers to an individual who utilizes the system to input travel and activity-related questions and receive guidance.

[0058] A "terminal" is a device that is operated by a user and provides the means for sending questions to a server and displaying guidance.

[0059] "Server" refers to a device or system that receives and analyzes questions sent by users, extracts information from a guidance database, generates guidance using a generative AI model, and transmits it to a terminal.

[0060] "Guidance Database" refers to a database that stores information about movements and activities and allows the server to search and extract appropriate information based on a user's question.

[0061] A "generative AI model" is an artificial intelligence model used by the server to generate guidance, and is capable of generating guidance in natural language that is appropriate to the user's question.

[0062] A "prompt sentence" is text to be input into a generative AI model and is used as input data when the server generates guidance based on a user's question.

[0063] "Real-time" refers to the fact that the process from when a user inputs a question to when the server generates guidance and sends it to the terminal is carried out almost instantly.

[0064] "Guidance" refers to specific instructions or advice that is generated by the server and provided to the user via the terminal in order to support the user's movements or activities.

[0065] The system of the present invention aims to provide appropriate guidance in real time to questions about movements and activities that a user performs in daily life. How to implement this system will be described below.

[0066] Configuring the Server

[0067] The server is configured with high-performance hardware and appropriate software. For example, a high-performance server can be a server commonly known as a "data center server," and the software used is Apache (registered trademark) Tomcat, MySQL (registered trademark), Python, etc. The server receives questions sent by users, performs natural language processing, and searches the guidance database based on the results.

[0068] 1. Natural Language Processing: The server analyzes the questions received from the users and extracts key keywords and context. For this, a natural language processing library implemented in Python can be used.

[0069] 2. Guidance database: The server searches a guidance database built using MySQL or ElasticSearch (registered trademark) and extracts information appropriate to the user's question.

[0070] 3. Generative AI model: Based on the extracted information, a generative AI model is used to generate guidance in real time. The generative AI model can be, for example, a GPT-based model.

[0071] Example prompt:

[0072] Please tell me the correct way to walk to avoid back pain.

[0073] "I would like to know some exercises that I can do regularly to avoid back pain."

[0074] Terminal configuration

[0075] The terminal provides a user interface for users to input questions and receive guidance. The terminal is typically a smartphone or tablet, and requires a dedicated app to run on it. This app is often developed with React Native.

[0076] 1. Question input: A user inputs a question about their movement or activity using a dedicated app on their smartphone or tablet. Once a question is entered, the device sends the question to the server as an HTTP POST request.

[0077] 2. Guidance display: Receives the guidance returned from the server and visually displays it to the user. HTTP communication is used to receive the data.

[0078] User Actions

[0079] A user interacts with the system using a terminal, specifically by performing the following operations:

[0080] 1. Question input: The user inputs a question about their movement or activity into a dedicated app. For example, they might input, "Please tell me the route to work if it rains tomorrow."

[0081] 2. Receiving guidance: Check the guidance sent from the server and act accordingly. For example, in response to a question such as "Please give me some advice about my commute route tomorrow. I would like to know the route to take if it rains," the server will provide guidance such as "On rainy days, we recommend avoiding route A and using route B."

[0082] Working Example

[0083] Hardware: Server equipment - Data center servers, high-capacity storage servers (NAS)

[0084] Software: Apache Tomcat, MySQL, Elasticsearch, generative AI model (GPT-based model)

[0085] User environment: Smartphones - iPhone (registered trademark), Samsung Galaxy, dedicated app (developed with React Native)

[0086] Using this system, users can instantly resolve any queries regarding their mobility or activities, enabling them to live their daily lives safely and efficiently.

[0087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0088] Step 1:

[0089] The user operates the terminal to input a question.

[0090] Specific actions: A user opens a smartphone app and inputs a question about mobility and activity. For example, the user types, "Please tell me the correct way to walk to avoid back pain," and presses the send button.

[0091] Input: The question text entered by the user.

[0092] Output: The device is ready to send question text.

[0093] Step 2:

[0094] The terminal transmits the entered question to the server.

[0095] What happens: The app sends an HTTP POST request to the server containing the text "Please tell me the correct way to walk to avoid back pain."

[0096] Input: The question text entered by the user.

[0097] Output: The HTTP POST request received by the server.

[0098] Step 3:

[0099] The server receives the query and begins parsing it.

[0100] Specific operation: The server receives an HTTP POST request and parses the question text, "Please tell me the correct way to walk to avoid back pain."

[0101] Input: HTTP POST request, question text.

[0102] Output: Keywords extracted as analysis results (e.g. "lower back pain", "correct walking method").

[0103] Step 4:

[0104] The server searches the guidance database.

[0105] Specific operation: The server searches the guidance database for information highly related to "lower back pain" and "correct walking method."

[0106] Input: Keywords from the analysis results.

[0107] Output: Information extracted from the guidance database (e.g., information on how to walk to avoid back pain).

[0108] Step 5:

[0109] The server extracts the appropriate information and generates guidance.

[0110] Specific operation: Based on the extracted information, the server uses a generative AI model to generate guidance in a form that is easy for the user to understand.

[0111] Input: Extract information from the guidance database.

[0112] Output: The generated guidance text (e.g., "To avoid back pain, it is important to keep your back straight and use both legs evenly when walking.").

[0113] Step 6:

[0114] The server transmits the generated guidance to the terminal.

[0115] Specific operation: The server sends the generated guidance to the terminal as an HTTP response.

[0116] Input: The generated guidance text.

[0117] Output: Guidance text sent as HTTP response.

[0118] Step 7:

[0119] The terminal displays the received guidance to the user.

[0120] What happens: The device receives the HTTP response and the app displays guidance to the user, saying, "To avoid back pain, it's important to keep your back straight and use both legs evenly when walking."

[0121] Input: Guidance text received as HTTP response.

[0122] Output: The guidance text that is displayed to the user.

[0123] Step 8:

[0124] The user takes action based on the guidance.

[0125] Specific behaviors: Users should follow the guidance provided by the app to practice safe movement and proper body movements, such as walking with a straight back and using both legs evenly.

[0126] Input: The guidance shown in the app.

[0127] Output: User behavior (standing straight and using both legs evenly when walking).

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Conventional systems were able to provide guidance for safe movement and activities in users' daily lives, but the scope of application was limited to individual users. In particular, there was no real-time guidance system for robots to perform work safely and efficiently in work environments such as factories. This posed a risk of reducing work safety and efficiency.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this disclosure, the server includes a means for receiving questions from a user regarding movements or activities in order to understand movements or activities performed by the user in daily life and provide guidance regarding body movements from a guidance database in real time so that the user can move or engage in safe movements or activities, a means for extracting information related to the questions from the guidance database and generating the guidance in real time, a means for providing the generated guidance to the user, and a means for transmitting the guidance to a robot control system and causing the robot to perform work based on an appropriate path and operation procedure. This enables safe and efficient work guidance not only for users but also in work environments such as factories.

[0133] A "user" refers to a human being who uses the system and who asks questions about movements and activities.

[0134] "Movement" refers to the general action of a user moving from one place to another.

[0135] "Activity" refers to the actions and tasks that a user performs during daily life or work.

[0136] "Guidance database" refers to a database that stores specific guidance information regarding movements and activities.

[0137] "Real-time" refers to providing immediate responses and guidance to users' questions.

[0138] A "question" refers to an inquiry about information that a user wants to know or confirm regarding travel or activity.

[0139] A "server" refers to a device that accepts questions, generates corresponding guidance, and transmits the necessary information to each terminal or system.

[0140] A "robot control system" refers to a control unit that receives and executes commands for a robot's movements and tasks.

[0141] A "route" refers to the path a robot or user takes to reach a destination.

[0142] "Operational procedure" refers to the sequence of specific operations or actions when moving or performing work.

[0143] "Generative AI model" refers to an artificial intelligence model that generates prompt sentences based on input from a user and provides appropriate guidance.

[0144] A "prompt sentence" refers to an input sentence created by a generative AI model to derive guidance information.

[0145] The embodiment of the present invention is to build a guidance system for improving the safety and efficiency of robot work in a factory. The system mainly includes a server, a terminal, a guidance database, a robot control system and a generative AI model.

[0146] First, the server accepts questions about movement or activity from the user. The user uses a smartphone or tablet to input questions about safe movement routes and operating procedures for the robot when it works in the factory, and sends the questions to the server.

[0147] The server searches the guidance database based on the received question, extracts relevant information, and generates more appropriate guidance based on the characteristics and requirements of the user's movements and activities. The generated guidance is sent to the user's terminal and the robot's control system in real time.

[0148] The robot's control system selects and executes the optimal route and operating procedure based on the guidance received from the server, enabling the robot to work safely and efficiently within the factory.

[0149] The server also uses the generative AI model to create prompts and provide more appropriate guidance for the user's questions. When the user enters a question, the server uses the generative AI model to analyze the question, extracts the necessary information from the guidance database, and generates the optimal guidance.

[0150] As a concrete example, if a user wants to know the best path for a robot to move heavy machinery around a factory, they can ask the following:

[0151] "Please tell me the path the robot will follow within the factory."

[0152] Or as a prompt:

[0153] "What is the best route for a robot to move heavy machinery so that it can avoid obstacles?"

[0154] The system is implemented using Python and the requests module and communicates through a server API. Other required hardware includes a smartphone or tablet, a server and a factory robot.

[0155] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0156] Step 1:

[0157] A user uses a smartphone or tablet device to input questions about movement and activity. The input questions are sent to the server. The input includes information the user wants to know or confirm (e.g., "Please tell me the path the robot will follow in the factory"). The output is the question data that is sent to the server.

[0158] Step 2:

[0159] The server analyzes the received question and searches the guidance database. The input is the submitted question data, and the output is the relevant information extracted from the guidance database. Specifically, the server extracts keywords from the question and searches the guidance database based on the keywords.

[0160] Step 3:

[0161] The server generates guidance in real time based on the information extracted from the guidance database. The input is the information extracted from the guidance database, and the output is the generated guidance. In concrete terms, the server appropriately combines the extracted data to generate guidance in a format that is easy for the user to understand.

[0162] Step 4:

[0163] The server transmits the generated guidance to the user's terminal and the robot's control system. The input is the generated guidance, and the output is the guidance data transmitted to the terminal and the robot's control system. Specifically, the server distributes data to the user's terminal and the robot's control system via a network.

[0164] Step 5:

[0165] The robot's control system selects the optimal path and operation procedure based on the received guidance. The input is the transmitted guidance data, and the output is a control signal based on the optimal path and operation procedure. Specifically, the control system analyzes the content of the guidance and controls the robot's sensors and actuators to execute the operation.

[0166] Step 6:

[0167] The server uses the generative AI model to create a prompt sentence and provide more appropriate guidance for the user's question. The input is the user's question and related information, and the output is a prompt sentence and guidance based on it. In concrete terms, the server uses the generative AI model to analyze the question in natural language and generate an appropriate prompt sentence.

[0168] Step 7:

[0169] The user checks the guidance received from the server and performs appropriate movements and activities based on it. The input is the guidance displayed on the terminal, and the output is the user's actual behavior. In terms of specific actions, the user follows the guidance to move safely and supervises the robot to complete a task.

[0170] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0171] The embodiment for implementing the present disclosure includes the following elements.

[0172] (1) Server

[0173] The server receives questions from users about their movements or activities, extracts appropriate information from a guidance database, and generates guidance in real time.Furthermore, it combines an emotion engine to recognize the user's emotions and utilizes the information in generating guidance.

[0174] (2) Terminal

[0175] The terminal receives user input questions about movement or activity and sends them to the server. The terminal also displays and provides guidance received from the server to the user. Furthermore, if the emotion engine recognizes the user's emotion, it can convey the emotion to the user by display or voice.

[0176] (3) Guidance database

[0177] The guidance database stores specific guidance on movement and activities. The server searches the guidance database based on the user's question and extracts appropriate information. In addition, the guidance database also stores information for generating guidance according to the user's emotion when the emotion engine recognizes the emotion.

[0178] (4) User

[0179] The user sends questions about movement and activities to the server through the device. The user also practices safe movement and appropriate body movements based on the guidance received from the device. Furthermore, if the emotion engine recognizes the user's emotions, the user can receive guidance and feedback according to the emotions.

[0180] By combining the above elements, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the device. The device provides guidance to the user, and the user refers to the guidance and practices safe movement and appropriate body movements. In addition, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0181] The process flow will be explained below.

[0182] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0183] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0184] Step 3: The server uses the emotion engine to recognize the user's emotions. The server uses the emotion engine to recognize emotions from the user's speech and facial expressions, etc. This makes it possible to grasp the user's emotional state.

[0185] Step 4: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. In addition, if the emotion engine recognizes the user's emotions, it can use that information to generate guidance. For example, if the user is feeling anxious or nervous, it can generate more polite guidance or guidance that encourages relaxation.

[0186] Step 5: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0187] Step 6: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal's screen. Furthermore, if the emotion engine recognizes the user's emotions, it can convey this to the user by display or voice. The user then acts based on the guidance. The user refers to the guidance displayed on the terminal to move safely and practice appropriate physical movements. Furthermore, if the emotion engine recognizes the user's emotions, they can receive guidance and feedback according to the emotions.

[0188] Through the above processing steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance. Furthermore, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0189] Example 2

[0190] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0191] In modern society, users move and engage in many activities in their daily lives, but it is difficult to find safe and efficient ways to do so. In addition, there is a lack of systems that can respond individually to the user's emotions and situations, making it difficult to provide guidance that is optimal for the user's needs. In particular, there are few systems that provide appropriate guidance in real time, and there is a need to create an environment where users can act with peace of mind.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0193] In this invention, the server includes a means for receiving a question from a user regarding movement or activity, a means for analyzing the content of the question and recognizing the user's emotion, a means for extracting information related to the question from a database and generating guidance according to the user's emotion, and a means for providing the generated guidance to the user. This makes it possible to provide appropriate and safe guidance for the user's movement or activity in real time. Furthermore, since an individual response can be made taking into account the user's emotion, the user's sense of security and satisfaction can be increased.

[0194] "User" refers to an individual who utilizes the system to receive guidance regarding travel and activities.

[0195] "Movement" refers to the act of a user physically moving from one location to another.

[0196] An "activity" refers to a specific action or behavior that a user performs in their daily life.

[0197] "Question" refers to information or a question about a trip or activity that a User inputs into the System.

[0198] "Server" refers to a central processing unit that receives and analyzes a user's question, generates appropriate guidance, and provides it to the user.

[0199] The term "receiving means" refers to a mechanism by which the system receives a question sent by a user.

[0200] "Means of analysis" refers to the process of understanding the question received and analyzing its content and intent.

[0201] "Means for recognizing emotions" refers to a mechanism for analyzing emotions from a user's input and understanding the user's state based on the results.

[0202] "Database" refers to an information system for storing and managing information on movements and activities.

[0203] "Means of extracting" refers to the process of retrieving the relevant information from the database.

[0204] The "means for generating" refers to a mechanism for generating optimal guidance based on the user's question and the results of sentiment analysis.

[0205] "Means for providing" refers to a mechanism for displaying or communicating the generated guidance to a user.

[0206] The present invention provides a system for providing a user with safe and effective guidance for movements or activities that the user performs in daily life. Specific embodiments for implementing the system will be described below.

[0207] Components

[0208] server

[0209] The Server is a central processing unit that accepts and analyzes user queries about movements or activities and provides appropriate guidance. The Server includes the following functions:

[0210] Question reception function: Receives questions from the terminal and analyzes their contents.

[0211] Emotion recognition: Uses natural language processing tools such as Google® Cloud Natural Language API to analyze emotions from user questions and context.

[0212] Guidance generation function: Searches a guidance database based on the analyzed question and emotion information to generate appropriate guidance.

[0213] Terminal

[0214] The terminal is the device where the user enters the question and receives guidance from the server. Typical terminals are smartphones or tablets and have the following features:

[0215] Question input function: Provides an interface for users to input questions.

[0216] Guidance display function: Displays guidance received from the server to the user in text or audio.

[0217] Guidance Database

[0218] The guidance database stores information about movements and activities that the server uses to generate appropriate guidance. A database system such as MongoDB is used.

[0219] Specific examples of implementation

[0220] For example, if a user types a question into the device such as "How can I get to the station while avoiding crowds?", the device sends the question to the server. The server uses the Google Cloud Natural Language API to analyze the question and extracts the keywords "crowded" and "get to the station." At the same time, it uses emotion recognition to determine whether the user is feeling anxious.

[0221] The server then searches the guidance database to extract guidance on the appropriate travel method for the question. If the user feels anxious, it can add an encouraging message to reassure the user. The generated guidance is sent to the terminal and displayed on the screen for the user to confirm.

[0222] Examples of prompt statements

[0223] Here are some example prompts to input to a generative AI model:

[0224] Prompt text (movement):

[0225] "Users are currently feeling anxious and want to know how to get to the station without crowding. Please explain the specific means of transportation and the reasons for doing so to ease users' anxiety."

[0226] Prompt (Activity):

[0227] "The user feels tired and wants to know a simple stretching method to relax. Please explain the specific stretching method and its effect, taking into consideration the user's fatigue."

[0228] The present invention can provide individualized support according to the needs of a user and provide appropriate guidance in real time, allowing the user to move around and perform activities safely and efficiently.

[0229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0230] Step 1:

[0231] The user inputs a question into the terminal.

[0232] Specific behavior:

[0233] The user opens their smartphone at home and types in the text field, "How can I get to the station without crowding?" Once they've finished typing, they press the "Send" button.

[0234] Input: The question entered by the user (in text format)

[0235] Output: The question data is generated on the terminal.

[0236] Step 2:

[0237] The terminal sends a question to the server.

[0238] Specific behavior:

[0239] When the "Submit" button is pressed, the terminal converts the entered question into JSON format and sends it to the server as an HTTP request.

[0240] Input: Question data entered by the user (JSON format)

[0241] Output: The question data is sent to the server.

[0242] Step 3:

[0243] The server receives and parses the query.

[0244] Specific behavior:

[0245] The server analyzes the received question data and calls the Google Cloud Natural Language API to extract keywords ("crowded" and "go to the station").

[0246] Input: Question data sent from the terminal (JSON format)

[0247] Output: Parsed keywords (in text format)

[0248] Step 4:

[0249] The server uses an emotion engine to recognize the user's emotions.

[0250] Specific behavior:

[0251] Use the Google Cloud Natural Language API to recognize emotions (e.g. anxiety) from user input.

[0252] Input: Question data sent from the terminal (JSON format)

[0253] Output: Recognized emotion information (text format)

[0254] Step 5:

[0255] The server searches the guidance database and extracts the appropriate guidance.

[0256] Specific behavior:

[0257] The server queries the MongoDB guidance database based on the parsed keywords to extract relevant guidance information, and then takes sentiment information into account to generate customized guidance.

[0258] Input: Analyzed keywords, recognized emotion information

[0259] Output: Customized guidance information (text format)

[0260] Step 6:

[0261] The server transmits the generated guidance to the terminal.

[0262] Specific behavior:

[0263] The server converts the generated guidance information into JSON format and sends it to the terminal as an HTTP response.

[0264] Input: Customized Guidance Information

[0265] Output: Guidance data (JSON format) is sent to the device.

[0266] Step 7:

[0267] The terminal displays the received guidance to the user.

[0268] Specific behavior:

[0269] The terminal analyzes the guidance data received from the server and displays it as text on the screen, and may also convey it to the user as a voice message.

[0270] Input: Guidance data sent from the server (JSON format)

[0271] Output: Guidance displayed to the user (in text and audio format)

[0272] Step 8:

[0273] The user refers to the guidance when moving around or performing activities.

[0274] Specific behavior:

[0275] The user refers to the guidance displayed on the device, avoids crowds, and travels safely to the station. The user also follows the guidance to take appropriate action.

[0276] Input: Guidance displayed on the terminal

[0277] Output: Safe movement or appropriate activity of the user

[0278] (Application example 2)

[0279] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0280] Conventional travel and activity guidance systems provide uniform guidance without considering the user's emotional state, and therefore lack support that reflects the user's individual needs and emotions. The present invention aims to solve this problem and provide more personalized travel and activity guidance in real time based on the user's emotions.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0282] In this invention, the server includes a means for receiving a question from a user regarding a movement or activity, a means for extracting information related to the question from a guidance database and generating guidance in real time, a means for providing the generated guidance to the user, and a means for recognizing an emotion of the user and customizing the guidance based on the emotion, thereby making it possible to provide accurate and individualized guidance corresponding to the emotion of the user.

[0283] A "user" is an entity that submits a travel or activity related question to the server and receives guidance.

[0284] "Movement or activity" refers to physical movement or bodily actions that a user performs in daily life.

[0285] A "guidance database" is a database that stores specific guidance information regarding a user's movements and activities.

[0286] The "means for accepting a question" is an interface for sending a question about a movement or activity from a user to the server.

[0287] The "means for extracting information and generating guidance in real time" refers to a mechanism for extracting information related to a question from a guidance database and generating guidance in real time based on the information.

[0288] A "means for providing guidance to a user" is a method or device for communicating the generated guidance to a user.

[0289] A "means for recognizing emotions" is a technology or device for detecting and recognizing a user's emotional state.

[0290] The "means for customizing guidance based on emotions" is a mechanism for individualizing the content and method of guidance according to the recognized emotions of the user.

[0291] The present invention relates to a system for providing real-time guidance to a user to support the user's movements or activities in daily life. Specific embodiments of the system are described below.

[0292] System Configuration

[0293] This system consists of a server, a terminal, a guidance database, an emotion engine, and a user.

[0294] server

[0295] The server has the following functions:

[0296] Question acceptance: Accepts questions about travel or activity sent from the user via the terminal.

[0297] Information Extraction: Extract information relevant to the question from the guidance database.

[0298] Guidance generation: Generate guidance in real time based on the extracted information.

[0299] Emotion Recognition: Use an emotion engine to recognize a user's emotions and customize guidance based on those emotions.

[0300] Terminal

[0301] The terminal functions as a user interface and has the following roles:

[0302] Question input: The user inputs a question about a movement or activity and sends it to the server.

[0303] Display guidance: Displays the guidance received from the server and provides it to the user in the form of audio or video.

[0304] Emotion transmission: The user's emotions are captured using the built-in camera and transmitted to the server.

[0305] Guidance Database

[0306] The guidance database stores specific guidance information related to the user's movements and activities, and also includes guidance information corresponding to the user's emotions.

[0307] Emotion Engine

[0308] The emotion engine processes image data and analyzes the user's emotions. This information is used by the server to customize guidance.

[0309] Data processing and calculation

[0310] This system uses the following hardware and software.

[0311] Camera: Used to take a picture of the user's face and recognize emotions.

[0312] Server (Python framework): Receives questions and emotion data, searches the guidance database, and generates appropriate guidance.

[0313] Emotion engine: A software module that acts as a user interface and analyzes the emotional state using image data as input.

[0314] Guidance database: Contains specific advice and guide data on movement and activities.

[0315] The server first receives the question sent by the terminal, converts it into an appropriate format, and searches the guidance database. Then, it generates suitable guidance in real time based on the obtained guidance information. At the same time, it analyzes the user's emotions through an emotion engine and customizes the content of the guidance based on the results. Finally, the server sends the customized guidance to the terminal, which then provides it to the user by display or voice.

[0316] Specific examples and generated AI model prompts

[0317] For example, if a user enters a question such as "I want to go to Shinjuku Station by the shortest route" in an autonomous vehicle, the server will search the guidance database and provide the optimal route. At the same time, if the user feels anxious about the displayed guidance, the emotion engine will detect this and provide additional guidance to relax.

[0318] Example prompt:

[0319] I am currently heading from my house to Shibuya Station, but I would like to know the shortest route. Also, I am feeling a bit anxious. What driving advice would you give me based on this feeling?

[0320] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0321] Step 1:

[0322] The user inputs a question about a movement or activity into the terminal. For example, the user inputs "I want to go to Shinjuku Station by the shortest route," and the question is sent to the terminal. This question becomes important data for generating guidance later.

[0323] Step 2:

[0324] The terminal sends the question entered by the user to the server. At this time, the terminal converts the question content into an appropriate format and transmits the data to the server using a secure communication method. Specifically, the terminal constructs the question in JSON format and transmits it via an HTTP request.

[0325] Step 3:

[0326] The server receives the question data sent from the terminal and searches the guidance database. The server generates a query to extract related information from the database based on the question content, and executes the query. For example, it extracts information related to "Shinjuku Station" and "shortest route".

[0327] Step 4:

[0328] The server generates guidance in real time based on information extracted from the guidance database. At this time, the server converts the extracted information into a form that is easy for the user to understand and creates a guidance message. For example, it creates specific instructions such as "The shortest route to Shinjuku Station is via XX Road and YY Street."

[0329] Step 5:

[0330] The terminal uses a camera to capture a face image of the user and transmits it to the server as emotion data. The emotion data is important information for recognizing the current emotion state of the user. For example, the image data is converted into an appropriate format before transmission.

[0331] Step 6:

[0332] The server uses an emotion engine to analyze the user's emotions. The received image data is input into the emotion engine, and the emotion state (e.g., anxiety, tension, relief, etc.) is output. Based on this output data, data is organized to adjust the guidance content.

[0333] Step 7:

[0334] The server customizes the guidance message based on the emotion data, for example adding an additional message to relax the user ("Take a deep breath and relax") to the guidance if the user feels anxious.

[0335] Step 8:

[0336] The server sends the customized guidance to the terminal. The server converts the tailored guidance message into a suitable format and transmits the data to the terminal using a secure communication means.

[0337] Step 9:

[0338] The terminal displays or provides the customized guidance received from the server to the user by voice. The user performs optimal movement or activity based on the customized guidance. For example, a voice synthesis system can be used to read out the guidance message to make it easier for the user to understand the instructions.

[0339] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0340] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0341] In the above embodiment, an example has been given in which the specific process is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0342] [Second embodiment]

[0343] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0344] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0345] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0346] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0347] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0348] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0349] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0350] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0351] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0352] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0353] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0354] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0355] The embodiment for implementing the present disclosure includes the following elements.

[0356] (1) Server

[0357] The server accepts queries from the user regarding travel or activities, extracts appropriate information from a guidance database, generates guidance in real time, and transmits the generated guidance to the terminal.

[0358] (2) Terminal

[0359] The terminal allows the user to input questions about movements or activities and transmits them to the server, and also displays and provides guidance received from the server to the user.

[0360] (3) Guidance database

[0361] The guidance database stores specific guidance on movements and activities. The server searches the guidance database based on the user's question and extracts appropriate information.

[0362] (4) User

[0363] The user sends questions about movement and activities to the server through the device, and receives guidance from the device to practice safe movement and appropriate physical movements.

[0364] The combination of the above elements realizes a system that supports users in moving safely and making appropriate body movements in their daily lives. The server extracts information from the guidance database, generates guidance in real time, and sends it to the terminal. The terminal provides guidance to the user, who then acts based on that guidance. The system is built according to this format, enabling users to move safely and make appropriate body movements.

[0365] The process flow will be explained below.

[0366] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0367] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0368] Step 3: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. Specifically, the server compiles specific procedures and precautions for movement or activity and prepares them as guidance.

[0369] Step 4: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0370] Step 5: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal screen. The user then acts while referring to the guidance. The user refers to the guidance displayed on the terminal and practices safe movement and appropriate physical movements. Specifically, the user moves and engages in activities by following the guidance, such as holding onto handrails and paying attention to where they are stepping.

[0371] Through the above process steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance.

[0372] Example 1

[0373] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0374] Conventional guidance systems have difficulty in providing accurate real-time guidance for users' movements and activities in daily life. In addition, there is a lack of means to provide customized guidance that fully takes into account the specific circumstances and requirements of users, which has led to a demand for improved safety and efficiency.

[0375] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0376] In this invention, the server includes a means for accepting a question from a user regarding movement or activity via a terminal, a server means for analyzing the content of the question, extracting information related to the question from the guidance database, and generating the guidance in real time using a generative AI model, and a terminal means for providing the generated guidance to the user, thereby making it possible to provide customized guidance in real time that takes into account the specific situation and requirements of the user.

[0377] "User" refers to an individual who utilizes the system to input travel and activity-related questions and receive guidance.

[0378] A "terminal" is a device that is operated by a user and provides the means for sending questions to a server and displaying guidance.

[0379] "Server" refers to a device or system that receives and analyzes questions sent by users, extracts information from a guidance database, generates guidance using a generative AI model, and transmits it to a terminal.

[0380] "Guidance Database" refers to a database that stores information about movements and activities and allows the server to search and extract appropriate information based on a user's question.

[0381] A "generative AI model" is an artificial intelligence model used by the server to generate guidance, and is capable of generating guidance in natural language that is appropriate to the user's question.

[0382] A "prompt sentence" is text to be input into a generative AI model and is used as input data when the server generates guidance based on a user's question.

[0383] "Real-time" refers to the fact that the process from when a user inputs a question to when the server generates guidance and sends it to the terminal is carried out almost instantly.

[0384] "Guidance" refers to specific instructions or advice that is generated by the server and provided to the user via the terminal in order to support the user's movements or activities.

[0385] The system of the present invention aims to provide appropriate guidance in real time to questions about movements and activities that a user performs in daily life. How to implement this system will be described below.

[0386] Configuring the Server

[0387] The server is configured with high-performance hardware and appropriate software. For example, a high-performance server can be a "data center server" with Apache Tomcat, MySQL, Python, etc. as software. The server receives questions sent by users, performs natural language processing, and searches the guidance database based on the results.

[0388] 1. Natural Language Processing: The server analyzes the questions received from the users and extracts key keywords and context. For this, a natural language processing library implemented in Python can be used.

[0389] 2. Guidance database: The server searches a guidance database built using MySQL and Elasticsearch to extract information appropriate to the user's question.

[0390] 3. Generative AI model: Based on the extracted information, a generative AI model is used to generate guidance in real time. The generative AI model can be, for example, a GPT-based model.

[0391] Example prompt:

[0392] Please tell me the correct way to walk to avoid back pain.

[0393] "I would like to know some exercises that I can do regularly to avoid back pain."

[0394] Device configuration

[0395] The terminal provides a user interface for users to input questions and receive guidance. The terminal is typically a smartphone or tablet, and requires a dedicated app to run on it. This app is often developed with React Native.

[0396] 1. Question input: A user inputs a question about their movement or activity using a dedicated app on their smartphone or tablet. Once a question is entered, the device sends the question to the server as an HTTP POST request.

[0397] 2. Guidance display: Receives the guidance returned from the server and visually displays it to the user. HTTP communication is used to receive the data.

[0398] User Actions

[0399] A user interacts with the system using a terminal, specifically by performing the following operations:

[0400] 1. Question input: The user inputs a question about their movement or activity into a dedicated app. For example, they might input, "Please tell me the route to work if it rains tomorrow."

[0401] 2. Receiving guidance: Check the guidance sent from the server and act accordingly. For example, in response to a question such as "Please give me some advice about my commute route tomorrow. I would like to know the route to take if it rains," the server will provide guidance such as "On rainy days, we recommend avoiding route A and using route B."

[0402] Working Example

[0403] Hardware: Server equipment - Data center servers, high-capacity storage servers (NAS)

[0404] Software: Apache Tomcat, MySQL, Elasticsearch, generative AI model (GPT-based model)

[0405] User environment: Smartphones - iPhone, Samsung Galaxy, dedicated app (developed with React Native)

[0406] Using this system, users can instantly resolve any queries regarding their mobility or activities, enabling them to live their daily lives safely and efficiently.

[0407] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0408] Step 1:

[0409] The user operates the terminal to input a question.

[0410] Specific actions: A user opens a smartphone app and inputs a question about mobility and activity. For example, the user types, "Please tell me the correct way to walk to avoid back pain," and presses the send button.

[0411] Input: The question text entered by the user.

[0412] Output: The device is ready to send question text.

[0413] Step 2:

[0414] The terminal transmits the entered question to the server.

[0415] What happens: The app sends an HTTP POST request to the server containing the text "Please tell me the correct way to walk to avoid back pain."

[0416] Input: The question text entered by the user.

[0417] Output: The HTTP POST request received by the server.

[0418] Step 3:

[0419] The server receives the query and begins parsing it.

[0420] Specific operation: The server receives an HTTP POST request and parses the question text, "Please tell me the correct way to walk to avoid back pain."

[0421] Input: HTTP POST request, question text.

[0422] Output: Keywords extracted as analysis results (e.g. "lower back pain", "correct walking method").

[0423] Step 4:

[0424] The server searches the guidance database.

[0425] Specific operation: The server searches the guidance database for information highly related to "lower back pain" and "correct walking method."

[0426] Input: Keywords from the analysis results.

[0427] Output: Information extracted from the guidance database (e.g., information on how to walk to avoid back pain).

[0428] Step 5:

[0429] The server extracts the appropriate information and generates guidance.

[0430] Specific operation: Based on the extracted information, the server uses a generative AI model to generate guidance in a form that is easy for the user to understand.

[0431] Input: Extract information from the guidance database.

[0432] Output: The generated guidance text (e.g., "To avoid back pain, it is important to keep your back straight and use both legs evenly when walking.").

[0433] Step 6:

[0434] The server transmits the generated guidance to the terminal.

[0435] Specific operation: The server sends the generated guidance to the terminal as an HTTP response.

[0436] Input: The generated guidance text.

[0437] Output: Guidance text sent as HTTP response.

[0438] Step 7:

[0439] The terminal displays the received guidance to the user.

[0440] What happens: The device receives the HTTP response and the app displays guidance to the user, saying, "To avoid back pain, it's important to keep your back straight and use both legs evenly when walking."

[0441] Input: Guidance text received as HTTP response.

[0442] Output: The guidance text that is displayed to the user.

[0443] Step 8:

[0444] The user takes action based on the guidance.

[0445] Specific behaviors: Users should follow the guidance provided by the app to practice safe movement and proper body movements, such as walking with a straight back and using both legs evenly.

[0446] Input: The guidance shown in the app.

[0447] Output: User behavior (standing straight and using both legs evenly when walking).

[0448] (Application example 1)

[0449] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0450] Conventional systems were able to provide guidance for safe movement and activities in users' daily lives, but the scope of application was limited to individual users. In particular, there was no real-time guidance system for robots to perform work safely and efficiently in work environments such as factories. This posed a risk of reducing work safety and efficiency.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0452] In this disclosure, the server includes a means for receiving questions from a user regarding movements or activities in order to understand movements or activities performed by the user in daily life and provide guidance regarding body movements from a guidance database in real time so that the user can move or engage in safe movements or activities, a means for extracting information related to the questions from the guidance database and generating the guidance in real time, a means for providing the generated guidance to the user, and a means for transmitting the guidance to a robot control system and causing the robot to perform work based on an appropriate path and operation procedure. This enables safe and efficient work guidance not only for users but also in work environments such as factories.

[0453] A "user" refers to a human being who uses the system and who asks questions about movements and activities.

[0454] "Movement" refers to the general action of a user moving from one place to another.

[0455] "Activity" refers to the actions and tasks that a user performs during daily life or work.

[0456] "Guidance database" refers to a database that stores specific guidance information regarding movements and activities.

[0457] "Real-time" refers to providing immediate responses and guidance to users' questions.

[0458] A "question" refers to an inquiry about information that a user wants to know or confirm regarding travel or activity.

[0459] A "server" refers to a device that accepts questions, generates corresponding guidance, and transmits the necessary information to each terminal or system.

[0460] A "robot control system" refers to a control unit that receives and executes commands for a robot's movements and tasks.

[0461] A "route" refers to the path a robot or user takes to reach a destination.

[0462] "Operational procedure" refers to the sequence of specific operations or actions when moving or performing work.

[0463] "Generative AI model" refers to an artificial intelligence model that generates prompt sentences based on input from a user and provides appropriate guidance.

[0464] A "prompt sentence" refers to an input sentence created by a generative AI model to derive guidance information.

[0465] This embodiment is to build a guidance system to improve the safety and efficiency of robot work in factories. The system mainly includes a server, a terminal, a guidance database, a robot control system and a generative AI model.

[0466] First, the server accepts questions about movement or activity from the user. The user uses a smartphone or tablet to input questions about safe movement routes and operating procedures for the robot when it works in the factory, and sends the questions to the server.

[0467] The server searches the guidance database based on the received question, extracts relevant information, and generates more appropriate guidance based on the characteristics and requirements of the user's movements and activities. The generated guidance is sent to the user's terminal and the robot's control system in real time.

[0468] The robot's control system selects and executes the optimal route and operating procedure based on the guidance received from the server, enabling the robot to work safely and efficiently within the factory.

[0469] The server also uses the generative AI model to create prompts and provide more appropriate guidance for the user's questions. When the user enters a question, the server uses the generative AI model to analyze the question, extracts the necessary information from the guidance database, and generates the optimal guidance.

[0470] As a concrete example, if a user wants to know the best path for a robot to move heavy machinery around a factory, they can ask the following:

[0471] "Please tell me the path the robot will follow within the factory."

[0472] Or as a prompt:

[0473] "What is the best route for a robot to move heavy machinery so that it can avoid obstacles?"

[0474] The system is implemented using Python and the requests module and communicates through a server API. Other required hardware includes a smartphone or tablet, a server and a factory robot.

[0475] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0476] Step 1:

[0477] A user uses a smartphone or tablet device to input a question about movement or activity. The input question is sent to the server. The input includes information the user wants to know or confirm (e.g., "Please tell me the path the robot will follow in the factory"). The output is the question data that is sent to the server.

[0478] Step 2:

[0479] The server analyzes the received question and searches the guidance database. The input is the submitted question data, and the output is the relevant information extracted from the guidance database. Specifically, the server extracts keywords from the question and searches the guidance database based on the keywords.

[0480] Step 3:

[0481] The server generates guidance in real time based on the information extracted from the guidance database. The input is the information extracted from the guidance database, and the output is the generated guidance. In concrete terms, the server appropriately combines the extracted data to generate guidance in a format that is easy for the user to understand.

[0482] Step 4:

[0483] The server transmits the generated guidance to the user's terminal and the robot's control system. The input is the generated guidance, and the output is the guidance data transmitted to the terminal and the robot's control system. Specifically, the server distributes data to the user's terminal and the robot's control system via a network.

[0484] Step 5:

[0485] The robot's control system selects the optimal path and operation procedure based on the received guidance. The input is the transmitted guidance data, and the output is a control signal based on the optimal path and operation procedure. Specifically, the control system analyzes the content of the guidance and controls the robot's sensors and actuators to execute the operation.

[0486] Step 6:

[0487] The server uses the generative AI model to create a prompt sentence and provide more appropriate guidance for the user's question. The input is the user's question and related information, and the output is a prompt sentence and guidance based on it. In concrete terms, the server uses the generative AI model to analyze the question in natural language and generate an appropriate prompt sentence.

[0488] Step 7:

[0489] The user checks the guidance received from the server and performs appropriate movements and activities based on it. The input is the guidance displayed on the terminal, and the output is the user's actual behavior. In terms of specific actions, the user follows the guidance to move safely and supervises the robot to complete a task.

[0490] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0491] The embodiment for implementing the present disclosure includes the following elements.

[0492] (1) Server

[0493] The server receives questions from users about their movements or activities, extracts appropriate information from a guidance database, and generates guidance in real time.Furthermore, it combines an emotion engine to recognize the user's emotions and utilizes the information in generating guidance.

[0494] (2) Terminal

[0495] The terminal receives user input questions about movement or activity and sends them to the server. The terminal also displays and provides guidance received from the server to the user. Furthermore, if the emotion engine recognizes the user's emotion, it can convey the emotion to the user by display or voice.

[0496] (3) Guidance database

[0497] The guidance database stores specific guidance on movement and activities. The server searches the guidance database based on the user's question and extracts appropriate information. In addition, the guidance database also stores information for generating guidance according to the user's emotion when the emotion engine recognizes the emotion.

[0498] (4) User

[0499] The user sends questions about movement and activities to the server through the device. The user also practices safe movement and appropriate body movements based on the guidance received from the device. Furthermore, if the emotion engine recognizes the user's emotions, the user can receive guidance and feedback according to the emotions.

[0500] By combining the above elements, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the device. The device provides guidance to the user, and the user refers to the guidance and practices safe movement and appropriate body movements. In addition, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0501] The process flow will be explained below.

[0502] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0503] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0504] Step 3: The server uses the emotion engine to recognize the user's emotions. The server uses the emotion engine to recognize emotions from the user's speech and facial expressions, etc. This makes it possible to grasp the user's emotional state.

[0505] Step 4: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. In addition, if the emotion engine recognizes the user's emotions, it can use that information to generate guidance. For example, if the user is feeling anxious or nervous, it can generate more polite guidance or guidance that encourages relaxation.

[0506] Step 5: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0507] Step 6: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal's screen. Furthermore, if the emotion engine recognizes the user's emotions, it can convey this to the user by display or voice. The user then acts based on the guidance. The user refers to the guidance displayed on the terminal to move safely and practice appropriate physical movements. Furthermore, if the emotion engine recognizes the user's emotions, they can receive guidance and feedback according to the emotions.

[0508] Through the above processing steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance. Furthermore, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0509] Example 2

[0510] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0511] In modern society, users move and engage in many activities in their daily lives, but it is difficult to find safe and efficient ways to do so. In addition, there is a lack of systems that can respond individually to the user's emotions and situations, making it difficult to provide guidance that is optimal for the user's needs. In particular, there are few systems that provide appropriate guidance in real time, and there is a need to create an environment where users can act with peace of mind.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0513] In this invention, the server includes a means for receiving a question from a user regarding movement or activity, a means for analyzing the content of the question and recognizing the user's emotion, a means for extracting information related to the question from a database and generating guidance according to the user's emotion, and a means for providing the generated guidance to the user. This makes it possible to provide appropriate and safe guidance for the user's movement or activity in real time. Furthermore, since an individual response can be made taking into account the user's emotion, the user's sense of security and satisfaction can be increased.

[0514] "User" refers to an individual who utilizes the system to receive guidance regarding travel and activities.

[0515] "Movement" refers to the act of a user physically moving from one location to another.

[0516] An "activity" refers to a specific action or behavior that a user performs in their daily life.

[0517] "Question" refers to information or a question about a trip or activity that a User inputs into the System.

[0518] "Server" refers to a central processing unit that receives and analyzes a user's question, generates appropriate guidance, and provides it to the user.

[0519] The term "receiving means" refers to a mechanism by which the system receives a question sent by a user.

[0520] "Means of analysis" refers to the process of understanding the question received and analyzing its content and intent.

[0521] "Means for recognizing emotions" refers to a mechanism for analyzing emotions from a user's input and understanding the user's state based on the results.

[0522] "Database" refers to an information system for storing and managing information on movements and activities.

[0523] "Means of extracting" refers to the process of retrieving the relevant information from the database.

[0524] The "means for generating" refers to a mechanism for generating optimal guidance based on the user's question and the results of sentiment analysis.

[0525] "Means for providing" refers to a mechanism for displaying or communicating the generated guidance to a user.

[0526] The present invention provides a system for providing a user with safe and effective guidance for movements or activities that the user performs in daily life. Specific embodiments for implementing the system will be described below.

[0527] Components

[0528] server

[0529] The Server is a central processing unit that accepts and analyzes user queries about movements or activities and provides appropriate guidance. The Server includes the following functions:

[0530] Question reception function: Receives questions from the terminal and analyzes their contents.

[0531] Emotion recognition: Use natural language processing tools such as Google Cloud Natural Language API to analyze emotions from user questions and context.

[0532] Guidance generation function: Searches a guidance database based on the analyzed question and emotion information to generate appropriate guidance.

[0533] Terminal

[0534] The terminal is the device where the user enters the question and receives guidance from the server. Typical terminals are smartphones or tablets and have the following features:

[0535] Question input function: Provides an interface for users to input questions.

[0536] Guidance display function: Displays guidance received from the server to the user in text or audio.

[0537] Guidance Database

[0538] The guidance database stores information about movements and activities that the server uses to generate appropriate guidance. A database system such as MongoDB is used.

[0539] Specific examples of implementation

[0540] For example, if a user types a question into the device such as "How can I get to the station while avoiding crowds?", the device sends the question to the server. The server uses the Google Cloud Natural Language API to analyze the question and extracts the keywords "crowded" and "get to the station." At the same time, it uses emotion recognition to determine whether the user is feeling anxious.

[0541] The server then searches the guidance database to extract guidance on the appropriate travel method for the question. If the user feels anxious, it can add an encouraging message to reassure the user. The generated guidance is sent to the terminal and displayed on the screen for the user to confirm.

[0542] Examples of prompt statements

[0543] Here are some example prompts to input to a generative AI model:

[0544] Prompt text (movement):

[0545] "Users are currently feeling anxious and want to know how to get to the station without crowding. Please explain the specific means of transportation and the reasons for doing so to ease users' anxiety."

[0546] Prompt (Activity):

[0547] "The user feels tired and wants to know a simple stretching method to relax. Please explain the specific stretching method and its effect, taking into consideration the user's fatigue."

[0548] This disclosure provides personalized, real-time guidance tailored to the user's needs, allowing the user to move and perform activities safely and efficiently.

[0549] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0550] Step 1:

[0551] The user inputs a question into the terminal.

[0552] Specific behavior:

[0553] The user opens their smartphone at home and types in the text field, "How can I get to the station without crowding?" Once they've finished typing, they press the "Send" button.

[0554] Input: The question entered by the user (in text format)

[0555] Output: The question data is generated on the terminal.

[0556] Step 2:

[0557] The terminal sends a question to the server.

[0558] Specific behavior:

[0559] When the "Submit" button is pressed, the terminal converts the entered question into JSON format and sends it to the server as an HTTP request.

[0560] Input: Question data entered by the user (JSON format)

[0561] Output: The question data is sent to the server.

[0562] Step 3:

[0563] The server receives and parses the query.

[0564] Specific behavior:

[0565] The server analyzes the received question data and calls the Google Cloud Natural Language API to extract keywords ("crowded" and "go to the station").

[0566] Input: Question data sent from the terminal (JSON format)

[0567] Output: Parsed keywords (in text format)

[0568] Step 4:

[0569] The server recognizes the user's emotions using an emotion engine.

[0570] Specific behavior:

[0571] Use the Google Cloud Natural Language API to recognize emotions (e.g. anxiety) from user input.

[0572] Input: Question data sent from the terminal (JSON format)

[0573] Output: Recognized emotion information (text format)

[0574] Step 5:

[0575] The server searches the guidance database and extracts the appropriate guidance.

[0576] Specific behavior:

[0577] The server queries the MongoDB guidance database based on the parsed keywords to extract relevant guidance information, and then takes sentiment information into account to generate customized guidance.

[0578] Input: Analyzed keywords, recognized emotion information

[0579] Output: Customized guidance information (text format)

[0580] Step 6:

[0581] The server transmits the generated guidance to the terminal.

[0582] Specific behavior:

[0583] The server converts the generated guidance information into JSON format and sends it to the terminal as an HTTP response.

[0584] Input: Customized Guidance Information

[0585] Output: Guidance data (JSON format) is sent to the device.

[0586] Step 7:

[0587] The terminal displays the received guidance to the user.

[0588] Specific behavior:

[0589] The terminal analyzes the guidance data received from the server and displays it as text on the screen, and may also convey it to the user as a voice message.

[0590] Input: Guidance data sent from the server (JSON format)

[0591] Output: Guidance displayed to the user (in text and audio format)

[0592] Step 8:

[0593] The user refers to the guidance when moving around or performing activities.

[0594] Specific behavior:

[0595] The user refers to the guidance displayed on the device, avoids crowds, and travels safely to the station. The user also follows the guidance to take appropriate action.

[0596] Input: Guidance displayed on the terminal

[0597] Output: Safe movement or appropriate activity of the user

[0598] (Application example 2)

[0599] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0600] Conventional travel and activity guidance systems provide uniform guidance without considering the user's emotional state, and therefore lack support that reflects the user's individual needs and emotions. The present invention aims to solve this problem and provide more personalized travel and activity guidance in real time based on the user's emotions.

[0601] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0602] In this invention, the server includes a means for receiving a question from a user regarding a movement or activity, a means for extracting information related to the question from a guidance database and generating guidance in real time, a means for providing the generated guidance to the user, and a means for recognizing an emotion of the user and customizing the guidance based on the emotion, thereby making it possible to provide accurate and individualized guidance corresponding to the emotion of the user.

[0603] A "user" is an entity that submits a travel or activity related question to the server and receives guidance.

[0604] "Movement or activity" refers to physical movement or bodily actions that a user performs in daily life.

[0605] A "guidance database" is a database that stores specific guidance information regarding a user's movements and activities.

[0606] The "means for accepting a question" is an interface for sending a question about a movement or activity from a user to the server.

[0607] The "means for extracting information and generating guidance in real time" refers to a mechanism for extracting information related to a question from a guidance database and generating guidance in real time based on the information.

[0608] A "means for providing guidance to a user" is a method or device for communicating the generated guidance to a user.

[0609] A "means for recognizing emotions" is a technology or device for detecting and recognizing a user's emotional state.

[0610] The "means for customizing guidance based on emotions" is a mechanism for individualizing the content and method of guidance according to the recognized emotions of the user.

[0611] The present disclosure relates to a system for providing real-time guidance to a user to support the movements or activities performed in daily life. A specific embodiment of the system is described below.

[0612] System Configuration

[0613] This system consists of a server, a terminal, a guidance database, an emotion engine, and a user.

[0614] server

[0615] The server has the following functions:

[0616] Question acceptance: Accepts questions about travel or activity sent from the user via the terminal.

[0617] Information Extraction: Extract information relevant to the question from the guidance database.

[0618] Guidance generation: Generate guidance in real time based on the extracted information.

[0619] Emotion Recognition: Uses an emotion engine to recognize a user's emotions and customize guidance based on those emotions.

[0620] Terminal

[0621] The terminal functions as a user interface and has the following roles:

[0622] Question input: The user inputs a question about a movement or activity and sends it to the server.

[0623] Display guidance: Displays the guidance received from the server and provides it to the user in the form of audio or video.

[0624] Emotion transmission: The user's emotions are captured using the built-in camera and transmitted to the server.

[0625] Guidance Database

[0626] The guidance database stores specific guidance information related to the user's movements and activities, and also includes guidance information corresponding to the user's emotions.

[0627] Emotion Engine

[0628] The emotion engine processes image data and analyzes the user's emotions. This information is used by the server to customize guidance.

[0629] Data processing and calculation

[0630] This system uses the following hardware and software.

[0631] Camera: Used to take a picture of the user's face and recognize emotions.

[0632] Server (Python framework): Receives questions and emotion data, searches the guidance database, and generates appropriate guidance.

[0633] Emotion engine: A software module that acts as a user interface and analyzes the emotional state using image data as input.

[0634] Guidance database: Contains specific advice and guide data on movement and activities.

[0635] The server first receives the question sent by the terminal, converts it into an appropriate format, and searches the guidance database. Then, it generates suitable guidance in real time based on the obtained guidance information. At the same time, it analyzes the user's emotions through an emotion engine and customizes the content of the guidance based on the results. Finally, the server sends the customized guidance to the terminal, which then provides it to the user by display or voice.

[0636] Specific examples and generated AI model prompts

[0637] For example, if a user enters a question such as "I want to go to Shinjuku Station by the shortest route" in an autonomous vehicle, the server will search the guidance database and provide the optimal route. At the same time, if the user feels anxious about the displayed guidance, the emotion engine will detect this and provide additional guidance to relax.

[0638] Example prompt:

[0639] I am currently heading from my house to Shibuya Station, but I would like to know the shortest route. Also, I am feeling a bit anxious. What driving advice would you give me based on this feeling?

[0640] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0641] Step 1:

[0642] The user inputs a question about a movement or activity into the terminal. For example, the user inputs "I want to go to Shinjuku Station by the shortest route," and the question is sent to the terminal. This question becomes important data for generating guidance later.

[0643] Step 2:

[0644] The terminal sends the question entered by the user to the server. At this time, the terminal converts the question content into an appropriate format and transmits the data to the server using a secure communication method. Specifically, the terminal constructs the question in JSON format and transmits it via an HTTP request.

[0645] Step 3:

[0646] The server receives the question data sent from the terminal and searches the guidance database. The server generates a query to extract related information from the database based on the question content, and executes the query. For example, it extracts information related to "Shinjuku Station" and "shortest route".

[0647] Step 4:

[0648] The server generates guidance in real time based on information extracted from the guidance database. At this time, the server converts the extracted information into a form that is easy for the user to understand and creates a guidance message. For example, it creates specific instructions such as "The shortest route to Shinjuku Station is via XX Road and YY Street."

[0649] Step 5:

[0650] The terminal uses a camera to capture a face image of the user and transmits it to the server as emotion data. The emotion data is important information for recognizing the current emotion state of the user. For example, the image data is converted into an appropriate format before transmission.

[0651] Step 6:

[0652] The server uses an emotion engine to analyze the user's emotions. The received image data is input into the emotion engine, and the emotion state (e.g., anxiety, tension, relief, etc.) is output. Based on this output data, data is organized to adjust the guidance content.

[0653] Step 7:

[0654] The server customizes the guidance message based on the emotion data, for example adding an additional message to relax the user ("Take a deep breath and relax") to the guidance if the user feels anxious.

[0655] Step 8:

[0656] The server sends the customized guidance to the terminal. The server converts the tailored guidance message into a suitable format and transmits the data to the terminal using a secure communication means.

[0657] Step 9:

[0658] The terminal displays or provides the customized guidance received from the server to the user by voice. The user performs optimal movement or activity based on the customized guidance. For example, a voice synthesis system can be used to read out the guidance message to make it easier for the user to understand the instructions.

[0659] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0660] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0661] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0662] [Third embodiment]

[0663] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0664] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0665] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0666] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0667] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0668] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0669] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0670] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0671] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0672] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0673] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0674] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0675] The embodiment for implementing the present disclosure includes the following elements.

[0676] (1) Server

[0677] The server accepts queries from the user regarding travel or activities, extracts appropriate information from a guidance database, generates guidance in real time, and transmits the generated guidance to the terminal.

[0678] (2) Terminal

[0679] The terminal allows the user to input questions about movements or activities and transmits them to the server, and also displays and provides guidance received from the server to the user.

[0680] (3) Guidance database

[0681] The guidance database stores specific guidance on movements and activities. The server searches the guidance database based on the user's question and extracts appropriate information.

[0682] (4) User

[0683] The user sends questions about movement and activities to the server through the device, and receives guidance from the device to practice safe movement and appropriate physical movements.

[0684] The combination of the above elements realizes a system that supports users in moving safely and making appropriate body movements in their daily lives. The server extracts information from the guidance database, generates guidance in real time, and sends it to the terminal. The terminal provides guidance to the user, who then acts based on that guidance. The system is built according to this format, enabling users to move safely and make appropriate body movements.

[0685] The process flow will be explained below.

[0686] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0687] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0688] Step 3: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. Specifically, the server compiles specific procedures and precautions for movement or activity and prepares them as guidance.

[0689] Step 4: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0690] Step 5: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal screen. The user then acts while referring to the guidance. The user refers to the guidance displayed on the terminal and practices safe movement and appropriate physical movements. Specifically, the user moves and engages in activities by following the guidance, such as holding onto handrails and paying attention to where they are stepping.

[0691] Through the above process steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance.

[0692] Example 1

[0693] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0694] Conventional guidance systems have difficulty in providing accurate real-time guidance for users' movements and activities in daily life. In addition, there is a lack of means to provide customized guidance that fully takes into account the specific circumstances and requirements of users, which has led to a demand for improved safety and efficiency.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0696] In this invention, the server includes a means for accepting a question from a user regarding movement or activity via a terminal, a server means for analyzing the content of the question, extracting information related to the question from the guidance database, and generating the guidance in real time using a generative AI model, and a terminal means for providing the generated guidance to the user, thereby making it possible to provide customized guidance in real time that takes into account the specific situation and requirements of the user.

[0697] "User" refers to an individual who utilizes the system to input travel and activity-related questions and receive guidance.

[0698] A "terminal" is a device that is operated by a user and provides the means for sending questions to a server and displaying guidance.

[0699] "Server" refers to a device or system that receives and analyzes questions sent by users, extracts information from a guidance database, generates guidance using a generative AI model, and transmits it to a terminal.

[0700] "Guidance Database" refers to a database that stores information about movements and activities and allows the server to search and extract appropriate information based on a user's question.

[0701] A "generative AI model" is an artificial intelligence model used by the server to generate guidance, and is capable of generating guidance in natural language that is appropriate to the user's question.

[0702] A "prompt sentence" is text to be input into a generative AI model and is used as input data when the server generates guidance based on a user's question.

[0703] "Real-time" refers to the fact that the process from when a user inputs a question to when the server generates guidance and sends it to the terminal is carried out almost instantly.

[0704] "Guidance" refers to specific instructions or advice that is generated by the server and provided to the user via the terminal in order to support the user's movements or activities.

[0705] The system of the present invention aims to provide appropriate guidance in real time to questions about movements and activities that a user performs in daily life. How to implement this system will be described below.

[0706] Configuring the Server

[0707] The server is configured with high-performance hardware and appropriate software. For example, a high-performance server can be a "data center server" with Apache Tomcat, MySQL, Python, etc. as software. The server receives questions sent by users, performs natural language processing, and searches the guidance database based on the results.

[0708] 1. Natural Language Processing: The server analyzes the questions received from the users and extracts key keywords and context. For this, a natural language processing library implemented in Python can be used.

[0709] 2. Guidance database: The server searches a guidance database built using MySQL and Elasticsearch to extract information appropriate to the user's question.

[0710] 3. Generative AI model: Based on the extracted information, a generative AI model is used to generate guidance in real time. The generative AI model can be, for example, a GPT-based model.

[0711] Example prompt:

[0712] Please tell me the correct way to walk to avoid back pain.

[0713] "I would like to know some exercises that I can do regularly to avoid back pain."

[0714] Device configuration

[0715] The terminal provides a user interface for users to input questions and receive guidance. The terminal is typically a smartphone or tablet, and requires a dedicated app to run on it. This app is often developed with React Native.

[0716] 1. Question input: A user inputs a question about their movement or activity using a dedicated app on their smartphone or tablet. Once a question is entered, the device sends the question to the server as an HTTP POST request.

[0717] 2. Guidance display: Receives the guidance returned from the server and visually displays it to the user. HTTP communication is used to receive the data.

[0718] User Actions

[0719] A user interacts with the system using a terminal, specifically by performing the following operations:

[0720] 1. Question input: The user inputs a question about their movement or activity into a dedicated app. For example, they might input, "Please tell me the route to work if it rains tomorrow."

[0721] 2. Receiving guidance: Check the guidance sent from the server and act accordingly. For example, in response to a question such as "Please give me some advice about my commute route tomorrow. I would like to know the route to take if it rains," the server will provide guidance such as "On rainy days, we recommend avoiding route A and using route B."

[0722] Working Example

[0723] Hardware: Server equipment - Data center servers, high-capacity storage servers (NAS)

[0724] Software: Apache Tomcat, MySQL, Elasticsearch, generative AI model (GPT-based model)

[0725] User environment: Smartphone, dedicated app (developed with React Native)

[0726] Using this system, users can instantly resolve any queries regarding their mobility or activities, enabling them to live their daily lives safely and efficiently.

[0727] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0728] Step 1:

[0729] The user operates the terminal to input a question.

[0730] Specific actions: A user opens a smartphone app and inputs a question about mobility and activity. For example, the user types, "Please tell me the correct way to walk to avoid back pain," and presses the send button.

[0731] Input: The question text entered by the user.

[0732] Output: The device is ready to send question text.

[0733] Step 2:

[0734] The terminal transmits the entered question to the server.

[0735] What happens: The app sends an HTTP POST request to the server containing the text "Please tell me the correct way to walk to avoid back pain."

[0736] Input: The question text entered by the user.

[0737] Output: The HTTP POST request received by the server.

[0738] Step 3:

[0739] The server receives the query and begins parsing it.

[0740] Specific operation: The server receives an HTTP POST request and parses the question text, "Please tell me the correct way to walk to avoid back pain."

[0741] Input: HTTP POST request, question text.

[0742] Output: Keywords extracted as analysis results (e.g. "lower back pain", "correct walking method").

[0743] Step 4:

[0744] The server searches the guidance database.

[0745] Specific operation: The server searches the guidance database for information highly related to "lower back pain" and "correct walking method."

[0746] Input: Keywords from the analysis results.

[0747] Output: Information extracted from the guidance database (e.g., information on how to walk to avoid back pain).

[0748] Step 5:

[0749] The server extracts the appropriate information and generates guidance.

[0750] Specific operation: Based on the extracted information, the server uses a generative AI model to generate guidance in a form that is easy for the user to understand.

[0751] Input: Extract information from the guidance database.

[0752] Output: The generated guidance text (e.g., "To avoid back pain, it is important to keep your back straight and use both legs evenly when walking.").

[0753] Step 6:

[0754] The server transmits the generated guidance to the terminal.

[0755] Specific operation: The server sends the generated guidance to the terminal as an HTTP response.

[0756] Input: The generated guidance text.

[0757] Output: Guidance text sent as HTTP response.

[0758] Step 7:

[0759] The terminal displays the received guidance to the user.

[0760] What happens: The device receives the HTTP response and the app displays guidance to the user, saying, "To avoid back pain, it's important to keep your back straight and use both legs evenly when walking."

[0761] Input: Guidance text received as HTTP response.

[0762] Output: The guidance text that is displayed to the user.

[0763] Step 8:

[0764] The user takes action based on the guidance.

[0765] Specific behaviors: Users should follow the guidance provided by the app to practice safe movement and proper body movements, such as walking with a straight back and using both legs evenly.

[0766] Input: The guidance shown in the app.

[0767] Output: User behavior (standing straight and using both legs evenly when walking).

[0768] (Application example 1)

[0769] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0770] Conventional systems were able to provide guidance for safe movement and activities in users' daily lives, but the scope of application was limited to individual users. In particular, there was no real-time guidance system for robots to perform work safely and efficiently in work environments such as factories. This posed a risk of reducing work safety and efficiency.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0772] In this disclosure, the server includes a means for receiving questions from a user regarding movements or activities in order to understand movements or activities performed by the user in daily life and provide guidance regarding body movements from a guidance database in real time so that the user can move or engage in safe movements or activities, a means for extracting information related to the questions from the guidance database and generating the guidance in real time, a means for providing the generated guidance to the user, and a means for transmitting the guidance to a robot control system and causing the robot to perform work based on an appropriate path and operation procedure. This enables safe and efficient work guidance not only for users but also in work environments such as factories.

[0773] A "user" refers to a human being who uses the system and who asks questions about movements and activities.

[0774] "Movement" refers to the general action of a user moving from one place to another.

[0775] "Activity" refers to the actions and tasks that a user performs during daily life or work.

[0776] "Guidance database" refers to a database that stores specific guidance information regarding movements and activities.

[0777] "Real-time" refers to providing immediate responses and guidance to users' questions.

[0778] A "question" refers to an inquiry about information that a user wants to know or confirm regarding travel or activity.

[0779] A "server" refers to a device that accepts questions, generates corresponding guidance, and transmits the necessary information to each terminal or system.

[0780] A "robot control system" refers to a control unit that receives and executes commands for a robot's movements and tasks.

[0781] A "route" refers to the path a robot or user takes to reach a destination.

[0782] "Operational procedure" refers to the sequence of specific operations or actions when moving or performing work.

[0783] "Generative AI model" refers to an artificial intelligence model that generates prompt sentences based on input from a user and provides appropriate guidance.

[0784] A "prompt sentence" refers to an input sentence created by a generative AI model to derive guidance information.

[0785] The embodiment of the present invention is to build a guidance system for improving the safety and efficiency of robot work in a factory. The system mainly includes a server, a terminal, a guidance database, a robot control system and a generative AI model.

[0786] First, the server accepts questions about movement or activity from the user. The user uses a smartphone or tablet to input questions about safe movement routes and operating procedures for the robot when it works in the factory, and sends the questions to the server.

[0787] The server searches the guidance database based on the received question, extracts relevant information, and generates more appropriate guidance based on the characteristics and requirements of the user's movements and activities. The generated guidance is sent to the user's terminal and the robot's control system in real time.

[0788] The robot's control system selects and executes the optimal route and operating procedure based on the guidance received from the server, enabling the robot to work safely and efficiently within the factory.

[0789] The server also uses the generative AI model to create prompts and provide more appropriate guidance for the user's questions. When the user enters a question, the server uses the generative AI model to analyze the question, extracts the necessary information from the guidance database, and generates the optimal guidance.

[0790] As a concrete example, if a user wants to know the best path for a robot to move heavy machinery around a factory, they can ask the following:

[0791] "Please tell me the path the robot will follow within the factory."

[0792] Or as a prompt:

[0793] "What is the best route for a robot to move heavy machinery so that it can avoid obstacles?"

[0794] The system is implemented using Python and the requests module and communicates through a server API. Other required hardware includes a smartphone or tablet, a server and a factory robot.

[0795] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0796] Step 1:

[0797] A user uses a smartphone or tablet device to input a question about movement or activity. The input question is sent to the server. The input includes information the user wants to know or confirm (e.g., "Please tell me the path the robot will follow in the factory"). The output is the question data that is sent to the server.

[0798] Step 2:

[0799] The server analyzes the received question and searches the guidance database. The input is the submitted question data, and the output is the relevant information extracted from the guidance database. Specifically, the server extracts keywords from the question and searches the guidance database based on the keywords.

[0800] Step 3:

[0801] The server generates guidance in real time based on the information extracted from the guidance database. The input is the information extracted from the guidance database, and the output is the generated guidance. In concrete terms, the server appropriately combines the extracted data to generate guidance in a format that is easy for the user to understand.

[0802] Step 4:

[0803] The server transmits the generated guidance to the user's terminal and the robot's control system. The input is the generated guidance, and the output is the guidance data transmitted to the terminal and the robot's control system. Specifically, the server distributes data to the user's terminal and the robot's control system via a network.

[0804] Step 5:

[0805] The robot's control system selects the optimal path and operation procedure based on the received guidance. The input is the transmitted guidance data, and the output is a control signal based on the optimal path and operation procedure. Specifically, the control system analyzes the content of the guidance and controls the robot's sensors and actuators to execute the operation.

[0806] Step 6:

[0807] The server uses the generative AI model to create a prompt sentence and provide more appropriate guidance for the user's question. The input is the user's question and related information, and the output is a prompt sentence and guidance based on it. In concrete terms, the server uses the generative AI model to analyze the question in natural language and generate an appropriate prompt sentence.

[0808] Step 7:

[0809] The user checks the guidance received from the server and performs appropriate movements and activities based on it. The input is the guidance displayed on the terminal, and the output is the user's actual behavior. In terms of specific actions, the user follows the guidance to move safely and supervises the robot to complete a task.

[0810] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0811] The embodiment for implementing the present disclosure includes the following elements.

[0812] (1) Server

[0813] The server receives questions from users about their movements or activities, extracts appropriate information from a guidance database, and generates guidance in real time.Furthermore, it combines an emotion engine to recognize the user's emotions and utilizes the information in generating guidance.

[0814] (2) Terminal

[0815] The terminal receives user input questions about movement or activity and sends them to the server. The terminal also displays and provides guidance received from the server to the user. Furthermore, if the emotion engine recognizes the user's emotion, it can convey the emotion to the user by display or voice.

[0816] (3) Guidance database

[0817] The guidance database stores specific guidance on movement and activities. The server searches the guidance database based on the user's question and extracts appropriate information. In addition, the guidance database also stores information for generating guidance according to the user's emotion when the emotion engine recognizes the emotion.

[0818] (4) User

[0819] The user sends questions about movement and activities to the server through the device. The user also practices safe movement and appropriate body movements based on the guidance received from the device. Furthermore, if the emotion engine recognizes the user's emotions, the user can receive guidance and feedback according to the emotions.

[0820] By combining the above elements, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the device. The device provides guidance to the user, and the user refers to the guidance and practices safe movement and appropriate body movements. In addition, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0821] The process flow will be explained below.

[0822] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[0823] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[0824] Step 3: The server uses the emotion engine to recognize the user's emotions. The server uses the emotion engine to recognize emotions from the user's speech and facial expressions, etc. This makes it possible to grasp the user's emotional state.

[0825] Step 4: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. In addition, if the emotion engine recognizes the user's emotions, it can use that information to generate guidance. For example, if the user is feeling anxious or nervous, it can generate more polite guidance or guidance that encourages relaxation.

[0826] Step 5: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[0827] Step 6: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal's screen. Furthermore, if the emotion engine recognizes the user's emotions, it can convey this to the user by display or voice. The user then acts based on the guidance. The user refers to the guidance displayed on the terminal to move safely and practice appropriate physical movements. Furthermore, if the emotion engine recognizes the user's emotions, they can receive guidance and feedback according to the emotions.

[0828] Through the above processing steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance. Furthermore, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[0829] Example 2

[0830] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0831] In modern society, users move and engage in many activities in their daily lives, but it is difficult to find safe and efficient ways to do so. In addition, there is a lack of systems that can respond individually to the user's emotions and situations, making it difficult to provide guidance that is optimal for the user's needs. In particular, there are few systems that provide appropriate guidance in real time, and there is a need to create an environment where users can act with peace of mind.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0833] In this invention, the server includes a means for receiving a question from a user regarding movement or activity, a means for analyzing the content of the question and recognizing the user's emotion, a means for extracting information related to the question from a database and generating guidance according to the user's emotion, and a means for providing the generated guidance to the user. This makes it possible to provide appropriate and safe guidance for the user's movement or activity in real time. Furthermore, since an individual response can be made taking into account the user's emotion, the user's sense of security and satisfaction can be increased.

[0834] "User" refers to an individual who utilizes the system to receive guidance regarding travel and activities.

[0835] "Movement" refers to the act of a user physically moving from one location to another.

[0836] An "activity" refers to a specific action or behavior that a user performs in their daily life.

[0837] "Question" refers to information or a question about a trip or activity that a User inputs into the System.

[0838] "Server" refers to a central processing unit that receives and analyzes a user's question, generates appropriate guidance, and provides it to the user.

[0839] The term "receiving means" refers to a mechanism by which the system receives a question sent by a user.

[0840] "Means of analysis" refers to the process of understanding the question received and analyzing its content and intent.

[0841] "Means for recognizing emotions" refers to a mechanism for analyzing emotions from a user's input and understanding the user's state based on the results.

[0842] "Database" refers to an information system for storing and managing information on movements and activities.

[0843] "Means of extracting" refers to the process of retrieving the relevant information from the database.

[0844] The "means for generating" refers to a mechanism for generating optimal guidance based on the user's question and the results of sentiment analysis.

[0845] "Means for providing" refers to a mechanism for displaying or communicating the generated guidance to a user.

[0846] The present invention provides a system for providing a user with safe and effective guidance for movements or activities that the user performs in daily life. Specific embodiments for implementing the system will be described below.

[0847] Components

[0848] server

[0849] The Server is a central processing unit that accepts and analyzes user queries about movements or activities and provides appropriate guidance. The Server includes the following functions:

[0850] Question reception function: Receives questions from the terminal and analyzes their contents.

[0851] Emotion recognition: Use natural language processing tools such as Google Cloud Natural Language API to analyze emotions from user questions and context.

[0852] Guidance generation function: Searches a guidance database based on the analyzed question and emotion information to generate appropriate guidance.

[0853] The terminal is the device where the user enters the question and receives guidance from the server. Typical terminals are smartphones or tablets and have the following features:

[0854] Question input function: Provides an interface for users to input questions.

[0855] Guidance display function: Displays guidance received from the server to the user in text or audio.

[0856] Guidance Database

[0857] The guidance database stores information about movements and activities that the server uses to generate appropriate guidance. A database system such as MongoDB is used.

[0858] Specific examples of implementation

[0859] For example, if a user types a question into the device such as "How can I get to the station while avoiding crowds?", the device sends the question to the server. The server uses the Google Cloud Natural Language API to analyze the question and extracts the keywords "crowded" and "get to the station." At the same time, it uses emotion recognition to determine whether the user is feeling anxious.

[0860] The server then searches the guidance database to extract guidance on the appropriate travel method for the question. If the user feels anxious, it can add an encouraging message to reassure the user. The generated guidance is sent to the terminal and displayed on the screen for the user to confirm.

[0861] Examples of prompt statements

[0862] Here are some example prompts to input to a generative AI model:

[0863] Prompt text (movement):

[0864] "Users are currently feeling anxious and want to know how to get to the station without crowding. Please explain the specific means of transportation and the reasons for doing so to ease users' anxiety."

[0865] Prompt (Activity):

[0866] "The user feels tired and wants to know a simple stretching method to relax. Please explain the specific stretching method and its effect, taking into consideration the user's fatigue."

[0867] The present invention can provide individualized support according to the needs of the user and provide appropriate guidance in real time, allowing the user to move around and perform activities safely and efficiently.

[0868] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0869] Step 1:

[0870] The user inputs a question into the terminal.

[0871] Specific behavior:

[0872] The user opens their smartphone at home and types in the text field, "How can I get to the station without crowding?" Once they've finished typing, they press the "Send" button.

[0873] Input: The question entered by the user (in text format)

[0874] Output: The question data is generated on the terminal.

[0875] Step 2:

[0876] The terminal sends a question to the server.

[0877] Specific behavior:

[0878] When the "Submit" button is pressed, the terminal converts the entered question into JSON format and sends it to the server as an HTTP request.

[0879] Input: Question data entered by the user (JSON format)

[0880] Output: The question data is sent to the server.

[0881] Step 3:

[0882] The server receives and parses the query.

[0883] Specific behavior:

[0884] The server analyzes the received question data and calls the Google Cloud Natural Language API to extract keywords ("crowded" and "go to the station").

[0885] Input: Question data sent from the terminal (JSON format)

[0886] Output: Parsed keywords (in text format)

[0887] Step 4:

[0888] The server recognizes the user's emotions using an emotion engine.

[0889] Specific behavior:

[0890] Use the Google Cloud Natural Language API to recognize emotions (e.g. anxiety) from user input.

[0891] Input: Question data sent from the terminal (JSON format)

[0892] Output: Recognized emotion information (text format)

[0893] Step 5:

[0894] The server searches the guidance database and extracts the appropriate guidance.

[0895] Specific behavior:

[0896] The server queries the MongoDB guidance database based on the parsed keywords to extract relevant guidance information, and then takes sentiment information into account to generate customized guidance.

[0897] Input: Analyzed keywords, recognized emotion information

[0898] Output: Customized guidance information (text format)

[0899] Step 6:

[0900] The server transmits the generated guidance to the terminal.

[0901] Specific behavior:

[0902] The server converts the generated guidance information into JSON format and sends it to the terminal as an HTTP response.

[0903] Input: Customized Guidance Information

[0904] Output: Guidance data (JSON format) is sent to the device.

[0905] Step 7:

[0906] The terminal displays the received guidance to the user.

[0907] Specific behavior:

[0908] The terminal analyzes the guidance data received from the server and displays it as text on the screen, and may also convey it to the user as a voice message.

[0909] Input: Guidance data sent from the server (JSON format)

[0910] Output: Guidance displayed to the user (in text and audio format)

[0911] Step 8:

[0912] The user refers to the guidance when moving around or performing activities.

[0913] Specific behavior:

[0914] The user refers to the guidance displayed on the device, avoids crowds, and travels safely to the station. The user also follows the guidance to take appropriate action.

[0915] Input: Guidance displayed on the terminal

[0916] Output: Safe movement or appropriate activity of the user

[0917] (Application example 2)

[0918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0919] Conventional travel and activity guidance systems provide uniform guidance without considering the user's emotional state, and therefore lack support that reflects the user's individual needs and emotions. The present invention aims to solve this problem and provide more personalized travel and activity guidance in real time based on the user's emotions.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0921] In this invention, the server includes a means for receiving a question from a user regarding a movement or activity, a means for extracting information related to the question from a guidance database and generating guidance in real time, a means for providing the generated guidance to the user, and a means for recognizing an emotion of the user and customizing the guidance based on the emotion, thereby making it possible to provide accurate and individualized guidance corresponding to the emotion of the user.

[0922] A "user" is an entity that submits a travel or activity related question to the server and receives guidance.

[0923] "Movement or activity" refers to physical movement or bodily actions that a user performs in daily life.

[0924] A "guidance database" is a database that stores specific guidance information regarding a user's movements and activities.

[0925] The "means for accepting a question" is an interface for sending a question about a movement or activity from a user to the server.

[0926] The "means for extracting information and generating guidance in real time" refers to a mechanism for extracting information related to a question from a guidance database and generating guidance in real time based on the information.

[0927] A "means for providing guidance to a user" is a method or device for communicating the generated guidance to a user.

[0928] A "means for recognizing emotions" is a technology or device for detecting and recognizing a user's emotional state.

[0929] The "means for customizing guidance based on emotions" is a mechanism for individualizing the content and method of guidance according to the recognized emotions of the user.

[0930] The present invention relates to a system for providing real-time guidance to a user to support the user's movements or activities in daily life. Specific embodiments of the system are described below.

[0931] System Configuration

[0932] This system consists of a server, a terminal, a guidance database, an emotion engine, and a user.

[0933] server

[0934] The server has the following functions:

[0935] Question acceptance: Accepts questions about travel or activity sent from the user via the terminal.

[0936] Information Extraction: Extract information relevant to the question from the guidance database.

[0937] Guidance generation: Generate guidance in real time based on the extracted information.

[0938] Emotion Recognition: Uses an emotion engine to recognize a user's emotions and customize guidance based on those emotions.

[0939] Terminal

[0940] The terminal functions as a user interface and has the following roles:

[0941] Question input: The user inputs a question about a movement or activity and sends it to the server.

[0942] Display guidance: Displays the guidance received from the server and provides it to the user in the form of audio or video.

[0943] Emotion transmission: The user's emotions are captured using the built-in camera and transmitted to the server.

[0944] Guidance Database

[0945] The guidance database stores specific guidance information related to the user's movements and activities, and also includes guidance information corresponding to the user's emotions.

[0946] Emotion Engine

[0947] The emotion engine processes image data and analyzes the user's emotions. This information is used by the server to customize guidance.

[0948] Data processing and calculation

[0949] This system uses the following hardware and software.

[0950] Camera: Used to take a picture of the user's face and recognize emotions.

[0951] Server (Python framework): Receives questions and emotion data, searches the guidance database, and generates appropriate guidance.

[0952] Emotion engine: A software module that acts as a user interface and analyzes the emotional state using image data as input.

[0953] Guidance database: Contains specific advice and guide data on movement and activities.

[0954] The server first receives the question sent by the terminal, converts it into an appropriate format, and searches the guidance database. Then, it generates suitable guidance in real time based on the obtained guidance information. At the same time, it analyzes the user's emotions through an emotion engine and customizes the content of the guidance based on the results. Finally, the server sends the customized guidance to the terminal, which then provides it to the user by display or voice.

[0955] Specific examples and generated AI model prompts

[0956] For example, if a user enters a question such as "I want to go to Shinjuku Station by the shortest route" in an autonomous vehicle, the server will search the guidance database and provide the optimal route. At the same time, if the user feels anxious about the displayed guidance, the emotion engine will detect this and provide additional guidance to relax.

[0957] Example prompt:

[0958] I am currently heading from my house to Shibuya Station, but I would like to know the shortest route. Also, I am feeling a bit anxious. What driving advice would you give me based on this feeling?

[0959] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0960] Step 1:

[0961] The user inputs a question about a movement or activity into the terminal. For example, the user inputs "I want to go to Shinjuku Station by the shortest route," and the question is sent to the terminal. This question becomes important data for generating guidance later.

[0962] Step 2:

[0963] The terminal sends the question entered by the user to the server. At this time, the terminal converts the question content into an appropriate format and transmits the data to the server using a secure communication method. Specifically, the terminal constructs the question in JSON format and transmits it via an HTTP request.

[0964] Step 3:

[0965] The server receives the question data sent from the terminal and searches the guidance database. The server generates a query to extract related information from the database based on the question content, and executes the query. For example, it extracts information related to "Shinjuku Station" and "shortest route".

[0966] Step 4:

[0967] The server generates guidance in real time based on information extracted from the guidance database. At this time, the server converts the extracted information into a form that is easy for the user to understand and creates a guidance message. For example, it creates specific instructions such as "The shortest route to Shinjuku Station is via XX Road and YY Street."

[0968] Step 5:

[0969] The terminal uses a camera to capture a face image of the user and transmits it to the server as emotion data. The emotion data is important information for recognizing the current emotion state of the user. For example, the image data is converted into an appropriate format before transmission.

[0970] Step 6:

[0971] The server uses an emotion engine to analyze the user's emotions. The received image data is input into the emotion engine, and the emotion state (e.g., anxiety, tension, relief, etc.) is output. Based on this output data, data is organized to adjust the guidance content.

[0972] Step 7:

[0973] The server customizes the guidance message based on the emotion data, for example adding an additional message to relax the user ("Take a deep breath and relax") to the guidance if the user feels anxious.

[0974] Step 8:

[0975] The server sends the customized guidance to the terminal. The server converts the tailored guidance message into a suitable format and transmits the data to the terminal using a secure communication means.

[0976] Step 9:

[0977] The terminal displays or provides the customized guidance received from the server to the user by voice. The user performs optimal movement or activity based on the customized guidance. For example, a voice synthesis system can be used to read out the guidance message to make it easier for the user to understand the instructions.

[0978] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0979] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0981] [Fourth embodiment]

[0982] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0983] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0985] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0989] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0990] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0991] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0992] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0993] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0994] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[0995] The embodiment for implementing the present disclosure includes the following elements.

[0996] (1) Server

[0997] The server accepts queries from the user regarding travel or activities, extracts appropriate information from a guidance database, generates guidance in real time, and transmits the generated guidance to the terminal.

[0998] (2) Terminal

[0999] The terminal allows the user to input questions about movements or activities and transmits them to the server, and also displays and provides guidance received from the server to the user.

[1000] (3) Guidance database

[1001] The guidance database stores specific guidance on movements and activities. The server searches the guidance database based on the user's question and extracts appropriate information.

[1002] (4) User

[1003] The user sends questions about movement and activities to the server through the device, and receives guidance from the device to practice safe movement and appropriate physical movements.

[1004] The combination of the above elements realizes a system that supports users in moving safely and making appropriate body movements in their daily lives. The server extracts information from the guidance database, generates guidance in real time, and sends it to the terminal. The terminal provides guidance to the user, who then acts based on that guidance. The system is built according to this format, enabling users to move safely and make appropriate body movements.

[1005] The process flow will be explained below.

[1006] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[1007] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[1008] Step 3: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. Specifically, the server compiles specific procedures and precautions for movement or activity and prepares them as guidance.

[1009] Step 4: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[1010] Step 5: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal screen. The user then acts while referring to the guidance. The user refers to the guidance displayed on the terminal and practices safe movement and appropriate physical movements. Specifically, the user moves and engages in activities by following the guidance, such as holding onto handrails and paying attention to where they are stepping.

[1011] Through the above process steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance.

[1012] Example 1

[1013] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[1014] Conventional guidance systems have difficulty in providing accurate real-time guidance for users' movements and activities in daily life. In addition, there is a lack of means to provide customized guidance that fully takes into account the specific circumstances and requirements of users, which has led to a demand for improved safety and efficiency.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1016] In this invention, the server includes a means for accepting a question from a user regarding movement or activity via a terminal, a server means for analyzing the content of the question, extracting information related to the question from the guidance database, and generating the guidance in real time using a generative AI model, and a terminal means for providing the generated guidance to the user, thereby making it possible to provide customized guidance in real time that takes into account the specific situation and requirements of the user.

[1017] "User" refers to an individual who utilizes the system to input travel and activity-related questions and receive guidance.

[1018] A "terminal" is a device that is operated by a user and provides the means for sending questions to a server and displaying guidance.

[1019] "Server" refers to a device or system that receives and analyzes questions sent by users, extracts information from a guidance database, generates guidance using a generative AI model, and transmits it to a terminal.

[1020] "Guidance Database" refers to a database that stores information about movements and activities and allows the server to search and extract appropriate information based on a user's question.

[1021] A "generative AI model" is an artificial intelligence model used by the server to generate guidance, and is capable of generating guidance in natural language that is appropriate to the user's question.

[1022] A "prompt sentence" is text to be input into a generative AI model and is used as input data when the server generates guidance based on a user's question.

[1023] "Real-time" refers to the fact that the process from when a user inputs a question to when the server generates guidance and sends it to the terminal is carried out almost instantly.

[1024] "Guidance" refers to specific instructions or advice that is generated by the server and provided to the user via the terminal in order to support the user's movements or activities.

[1025] The system of the present invention aims to provide appropriate guidance in real time to questions about movements and activities that a user performs in daily life. How to implement this system will be described below.

[1026] Configuring the Server

[1027] The server is configured with high-performance hardware and appropriate software. For example, a high-performance server can be a "data center server" with Apache Tomcat, MySQL, Python, etc. as software. The server receives questions sent by users, performs natural language processing, and searches the guidance database based on the results.

[1028] 1. Natural Language Processing: The server analyzes the questions received from the users and extracts key keywords and context. For this, a natural language processing library implemented in Python can be used.

[1029] 2. Guidance database: The server searches a guidance database built using MySQL and Elasticsearch to extract information appropriate to the user's question.

[1030] 3. Generative AI model: Based on the extracted information, a generative AI model is used to generate guidance in real time. The generative AI model can be, for example, a GPT-based model.

[1031] Example prompt:

[1032] Please tell me the correct way to walk to avoid back pain.

[1033] "I would like to know some exercises that I can do regularly to avoid back pain."

[1034] Device configuration

[1035] The terminal provides a user interface for users to input questions and receive guidance. The terminal is typically a smartphone or tablet, and requires a dedicated app to run on it. This app is often developed with React Native.

[1036] 1. Question input: A user inputs a question about their movement or activity using a dedicated app on their smartphone or tablet. Once a question is entered, the device sends the question to the server as an HTTP POST request.

[1037] 2. Guidance display: Receives the guidance returned from the server and visually displays it to the user. HTTP communication is used to receive the data.

[1038] User Actions

[1039] A user interacts with the system using a terminal, specifically by performing the following operations:

[1040] 1. Question input: The user inputs a question about their movement or activity into a dedicated app. For example, they might input, "Please tell me the route to work if it rains tomorrow."

[1041] 2. Receiving guidance: Check the guidance sent from the server and act accordingly. For example, in response to a question such as "Please give me some advice about my commute route tomorrow. I would like to know the route to take if it rains," the server will provide guidance such as "On rainy days, we recommend avoiding route A and using route B."

[1042] Working Example

[1043] Hardware: Server equipment - Data center servers, high-capacity storage servers (NAS)

[1044] Software: Apache Tomcat, MySQL, Elasticsearch, generative AI model (GPT-based model)

[1045] User environment: Smartphones - iPhone, Samsung Galaxy, dedicated app (developed with React Native)

[1046] Using this system, users can instantly resolve any queries regarding their mobility or activities, enabling them to live their daily lives safely and efficiently.

[1047] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1048] Step 1:

[1049] The user operates the terminal to input a question.

[1050] Specific actions: A user opens a smartphone app and inputs a question about mobility and activity. For example, the user types, "Please tell me the correct way to walk to avoid back pain," and presses the send button.

[1051] Input: The question text entered by the user.

[1052] Output: The device is ready to send question text.

[1053] Step 2:

[1054] The terminal transmits the entered question to the server.

[1055] What happens: The app sends an HTTP POST request to the server containing the text "Please tell me the correct way to walk to avoid back pain."

[1056] Input: The question text entered by the user.

[1057] Output: The HTTP POST request received by the server.

[1058] Step 3:

[1059] The server receives the query and begins parsing it.

[1060] Specific operation: The server receives an HTTP POST request and parses the question text, "Please tell me the correct way to walk to avoid back pain."

[1061] Input: HTTP POST request, question text.

[1062] Output: Keywords extracted as analysis results (e.g. "lower back pain", "correct walking method").

[1063] Step 4:

[1064] The server searches the guidance database.

[1065] Specific operation: The server searches the guidance database for information highly related to "lower back pain" and "correct walking method."

[1066] Input: Keywords from the analysis results.

[1067] Output: Information extracted from the guidance database (e.g., information on how to walk to avoid back pain).

[1068] Step 5:

[1069] The server extracts the appropriate information and generates guidance.

[1070] Specific operation: Based on the extracted information, the server uses a generative AI model to generate guidance in a form that is easy for the user to understand.

[1071] Input: Extract information from the guidance database.

[1072] Output: The generated guidance text (e.g., "To avoid back pain, it is important to keep your back straight and use both legs evenly when walking.").

[1073] Step 6:

[1074] The server transmits the generated guidance to the terminal.

[1075] Specific operation: The server sends the generated guidance to the terminal as an HTTP response.

[1076] Input: The generated guidance text.

[1077] Output: Guidance text sent as HTTP response.

[1078] Step 7:

[1079] The terminal displays the received guidance to the user.

[1080] What happens: The device receives the HTTP response and the app displays guidance to the user, saying, "To avoid back pain, it's important to keep your back straight and use both legs evenly when walking."

[1081] Input: Guidance text received as HTTP response.

[1082] Output: The guidance text that is displayed to the user.

[1083] Step 8:

[1084] The user takes action based on the guidance.

[1085] Specific behaviors: Users should follow the guidance provided by the app to practice safe movement and proper body movements, such as walking with a straight back and using both legs evenly.

[1086] Input: The guidance shown in the app.

[1087] Output: User behavior (standing straight and using both legs evenly when walking).

[1088] (Application example 1)

[1089] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1090] Conventional systems were able to provide guidance for safe movement and activities in users' daily lives, but the scope of application was limited to individual users. In particular, there was no real-time guidance system for robots to perform work safely and efficiently in work environments such as factories. This posed a risk of reducing work safety and efficiency.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1092] In this invention, the server includes a means for receiving questions from a user regarding movements or activities in order to understand movements or activities performed by the user in daily life and provide guidance regarding body movements from a guidance database in real time so that the user can move or engage in safe movements or activities, a means for extracting information related to the questions from the guidance database and generating the guidance in real time, a means for providing the generated guidance to the user, and a means for transmitting the guidance to a robot control system and causing the robot to perform work based on an appropriate path and operation procedure. This enables safe and efficient work guidance not only for users but also in work environments such as factories.

[1093] A "user" refers to a human being who uses the system and who asks questions about movements and activities.

[1094] "Movement" refers to the general action of a user moving from one place to another.

[1095] "Activity" refers to the actions and tasks that a user performs during daily life or work.

[1096] "Guidance database" refers to a database that stores specific guidance information regarding movements and activities.

[1097] "Real-time" refers to providing immediate responses and guidance to users' questions.

[1098] A "question" refers to an inquiry about information that a user wants to know or confirm regarding travel or activity.

[1099] A "server" refers to a device that accepts questions, generates corresponding guidance, and transmits the necessary information to each terminal or system.

[1100] A "robot control system" refers to a control unit that receives and executes commands for a robot's movements and tasks.

[1101] A "route" refers to the path a robot or user takes to reach a destination.

[1102] "Operational procedure" refers to the sequence of specific operations or actions when moving or performing work.

[1103] "Generative AI model" refers to an artificial intelligence model that generates prompt sentences based on input from a user and provides appropriate guidance.

[1104] A "prompt sentence" refers to an input sentence created by a generative AI model to derive guidance information.

[1105] The embodiment of the present invention is to build a guidance system for improving the safety and efficiency of robot work in a factory. The system mainly includes a server, a terminal, a guidance database, a robot control system and a generative AI model.

[1106] First, the server accepts questions about movement or activity from the user. The user uses a smartphone or tablet to input questions about safe movement routes and operating procedures for the robot when it works in the factory, and sends the questions to the server.

[1107] The server searches the guidance database based on the received question, extracts relevant information, and generates more appropriate guidance based on the characteristics and requirements of the user's movements and activities. The generated guidance is sent to the user's terminal and the robot's control system in real time.

[1108] The robot's control system selects and executes the optimal route and operating procedure based on the guidance received from the server, enabling the robot to work safely and efficiently within the factory.

[1109] The server also uses the generative AI model to create prompts and provide more appropriate guidance for the user's questions. When the user enters a question, the server uses the generative AI model to analyze the question, extracts the necessary information from the guidance database, and generates the optimal guidance.

[1110] As a concrete example, if a user wants to know the best path for a robot to move heavy machinery around a factory, they can ask the following:

[1111] "Please tell me the path the robot will follow within the factory."

[1112] Or as a prompt:

[1113] "What is the best route for a robot to move heavy machinery so that it can avoid obstacles?"

[1114] The system is implemented using Python and the requests module and communicates through a server API. Other required hardware includes a smartphone or tablet, a server and a factory robot.

[1115] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1116] Step 1:

[1117] A user uses a smartphone or tablet device to input questions about movement and activity. The input questions are sent to the server. The input includes information the user wants to know or confirm (e.g., "Please tell me the path the robot will follow in the factory"). The output is the question data that is sent to the server.

[1118] Step 2:

[1119] The server analyzes the received question and searches the guidance database. The input is the submitted question data, and the output is the relevant information extracted from the guidance database. Specifically, the server extracts keywords from the question and searches the guidance database based on the keywords.

[1120] Step 3:

[1121] The server generates guidance in real time based on the information extracted from the guidance database. The input is the information extracted from the guidance database, and the output is the generated guidance. In concrete terms, the server appropriately combines the extracted data to generate guidance in a format that is easy for the user to understand.

[1122] Step 4:

[1123] The server transmits the generated guidance to the user's terminal and the robot's control system. The input is the generated guidance, and the output is the guidance data transmitted to the terminal and the robot's control system. Specifically, the server distributes data to the user's terminal and the robot's control system via a network.

[1124] Step 5:

[1125] The robot's control system selects the optimal path and operation procedure based on the received guidance. The input is the transmitted guidance data, and the output is a control signal based on the optimal path and operation procedure. Specifically, the control system analyzes the content of the guidance and controls the robot's sensors and actuators to execute the operation.

[1126] Step 6:

[1127] The server uses the generative AI model to create a prompt sentence and provide more appropriate guidance for the user's question. The input is the user's question and related information, and the output is a prompt sentence and guidance based on it. In concrete terms, the server uses the generative AI model to analyze the question in natural language and generate an appropriate prompt sentence.

[1128] Step 7:

[1129] The user checks the guidance received from the server and performs appropriate movements and activities based on it. The input is the guidance displayed on the terminal, and the output is the user's actual behavior. In terms of specific actions, the user follows the guidance to move safely and supervises the robot to complete a task.

[1130] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1131] The embodiment for implementing the present disclosure includes the following elements.

[1132] (1) Server

[1133] The server receives questions from users about their movements or activities, extracts appropriate information from a guidance database, and generates guidance in real time.Furthermore, it combines an emotion engine to recognize the user's emotions and utilizes the information in generating guidance.

[1134] (2) Terminal

[1135] The terminal receives user input questions about movement or activity and sends them to the server. The terminal also displays and provides guidance received from the server to the user. Furthermore, if the emotion engine recognizes the user's emotion, it can convey the emotion to the user by display or voice.

[1136] (3) Guidance database

[1137] The guidance database stores specific guidance on movement and activities. The server searches the guidance database based on the user's question and extracts appropriate information. In addition, the guidance database also stores information for generating guidance according to the user's emotion when the emotion engine recognizes the emotion.

[1138] (4) User

[1139] The user sends questions about movement and activities to the server through the device. The user also practices safe movement and appropriate body movements based on the guidance received from the device. Furthermore, if the emotion engine recognizes the user's emotions, the user can receive guidance and feedback according to the emotions.

[1140] By combining the above elements, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the device. The device provides guidance to the user, and the user refers to the guidance and practices safe movement and appropriate body movements. In addition, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[1141] The process flow will be explained below.

[1142] Step 1: A user sends a question about a movement or activity to a server through a terminal. The user inputs a question about a movement or activity into a question form on the terminal and presses a send button.

[1143] Step 2: Server receives the query and extracts information from the guidance database. The server receives the query sent by the user. It then searches through the guidance database and extracts the appropriate information about the trip or activity.

[1144] Step 3: The server uses the emotion engine to recognize the user's emotions. The server uses the emotion engine to recognize emotions from the user's speech and facial expressions, etc. This makes it possible to grasp the user's emotional state.

[1145] Step 4: The server generates guidance in real time. The server generates guidance in real time based on the extracted information. In addition, if the emotion engine recognizes the user's emotions, it can use that information to generate guidance. For example, if the user is feeling anxious or nervous, it can generate more polite guidance or guidance that encourages relaxation.

[1146] Step 5: The server transmits the generated guidance to the terminal. The server transmits the generated guidance to the terminal, so that the user can receive the guidance on the terminal.

[1147] Step 6: The terminal provides guidance to the user. The terminal displays the guidance received from the server. Specifically, it provides the guidance to the user by displaying the guidance text and images on the terminal's screen. Furthermore, if the emotion engine recognizes the user's emotions, it can convey this to the user by display or voice. The user then acts based on the guidance. The user refers to the guidance displayed on the terminal to move safely and practice appropriate physical movements. Furthermore, if the emotion engine recognizes the user's emotions, they can receive guidance and feedback according to the emotions.

[1148] Through the above processing steps, the user sends a question about movement or activity to the server, and the server generates appropriate guidance in real time and sends it to the terminal. The terminal provides the guidance to the user, and the user practices safe movement and appropriate body movements by referring to the guidance. Furthermore, the emotion engine recognizes the user's emotions and provides corresponding guidance and feedback, making the user's experience more personalized and providing more effective support.

[1149] Example 2

[1150] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1151] In modern society, users move and engage in many activities in their daily lives, but it is difficult to find safe and efficient ways to do so. In addition, there is a lack of systems that can respond individually to the user's emotions and situations, making it difficult to provide guidance that is optimal for the user's needs. In particular, there are few systems that provide appropriate guidance in real time, and there is a need to create an environment where users can act with peace of mind.

[1152] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1153] In this disclosure, the server includes a means for receiving a question from a user regarding movement or activity, a means for analyzing the content of the question and recognizing the user's emotion, a means for extracting information related to the question from a database and generating guidance according to the user's emotion, and a means for providing the generated guidance to the user. This makes it possible to provide appropriate and safe guidance for the user's movement or activity in real time. Furthermore, since an individual response can be made taking into account the user's emotion, the user's sense of security and satisfaction can be increased.

[1154] "User" refers to an individual who utilizes the system to receive guidance regarding travel and activities.

[1155] "Movement" refers to the act of a user physically moving from one location to another.

[1156] An "activity" refers to a specific action or behavior that a user performs in their daily life.

[1157] "Question" refers to information or a question about a trip or activity that a User inputs into the System.

[1158] "Server" refers to a central processing unit that receives and analyzes a user's question, generates appropriate guidance, and provides it to the user.

[1159] The term "receiving means" refers to a mechanism by which the system receives a question sent by a user.

[1160] "Means of analysis" refers to the process of understanding the question received and analyzing its content and intent.

[1161] "Means for recognizing emotions" refers to a mechanism for analyzing emotions from a user's input and understanding the user's state based on the results.

[1162] "Database" refers to an information system for storing and managing information on movements and activities.

[1163] "Means of extracting" refers to the process of retrieving the relevant information from the database.

[1164] The "means for generating" refers to a mechanism for generating optimal guidance based on the user's question and the results of sentiment analysis.

[1165] "Means for providing" refers to a mechanism for displaying or communicating the generated guidance to a user.

[1166] The present invention provides a system for providing a user with safe and effective guidance for movements or activities that the user performs in daily life. Specific embodiments for implementing the system will be described below.

[1167] Components

[1168] server

[1169] The Server is a central processing unit that accepts and analyzes user queries about movements or activities and provides appropriate guidance. The Server includes the following functions:

[1170] Question reception function: Receives questions from the terminal and analyzes their contents.

[1171] Emotion recognition: Use natural language processing tools such as Google Cloud Natural Language API to analyze emotions from user questions and context.

[1172] Guidance generation function: Searches a guidance database based on the analyzed question and emotion information to generate appropriate guidance.

[1173] Terminal

[1174] The terminal is the device where the user enters the question and receives guidance from the server. Typical terminals are smartphones or tablets and have the following features:

[1175] Question input function: Provides an interface for users to input questions.

[1176] Guidance display function: Displays guidance received from the server to the user in text or audio.

[1177] Guidance Database

[1178] The guidance database stores information about movements and activities that the server uses to generate appropriate guidance. A database system such as MongoDB is used.

[1179] Specific examples of implementation

[1180] For example, if a user types a question into the device such as "How can I get to the station while avoiding crowds?", the device sends the question to the server. The server uses the Google Cloud Natural Language API to analyze the question and extracts the keywords "crowded" and "get to the station." At the same time, it uses emotion recognition to determine whether the user is feeling anxious.

[1181] The server then searches the guidance database to extract guidance on the appropriate travel method for the question. If the user feels anxious, it can add an encouraging message to reassure the user. The generated guidance is sent to the terminal and displayed on the screen for the user to confirm.

[1182] Examples of prompt statements

[1183] Here are some example prompts to input to a generative AI model:

[1184] Prompt text (movement):

[1185] "Users are currently feeling anxious and want to know how to get to the station without crowding. Please explain the specific means of transportation and the reasons for doing so to ease users' anxiety."

[1186] Prompt (Activity):

[1187] "The user feels tired and wants to know a simple stretching method to relax. Please explain the specific stretching method and its effect, taking into consideration the user's fatigue."

[1188] The present invention can provide individualized support according to the needs of the user and provide appropriate guidance in real time, allowing the user to move around and perform activities safely and efficiently.

[1189] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1190] Step 1:

[1191] The user inputs a question into the terminal.

[1192] Specific behavior:

[1193] The user opens their smartphone at home and types in the text field, "How can I get to the station without crowding?" Once they've finished typing, they press the "Send" button.

[1194] Input: The question entered by the user (in text format)

[1195] Output: The question data is generated on the terminal.

[1196] Step 2:

[1197] The terminal sends a question to the server.

[1198] Specific behavior:

[1199] When the "Submit" button is pressed, the terminal converts the entered question into JSON format and sends it to the server as an HTTP request.

[1200] Input: Question data entered by the user (JSON format)

[1201] Output: The question data is sent to the server.

[1202] Step 3:

[1203] The server receives and parses the query.

[1204] Specific behavior:

[1205] The server analyzes the received question data and calls the Google Cloud Natural Language API to extract keywords ("crowded" and "go to the station").

[1206] Input: Question data sent from the terminal (JSON format)

[1207] Output: Parsed keywords (in text format)

[1208] Step 4:

[1209] The server recognizes the user's emotions using an emotion engine.

[1210] Specific behavior:

[1211] Use the Google Cloud Natural Language API to recognize emotions (e.g. anxiety) from user input.

[1212] Input: Question data sent from the terminal (JSON format)

[1213] Output: Recognized emotion information (text format)

[1214] Step 5:

[1215] The server searches the guidance database and extracts the appropriate guidance.

[1216] Specific behavior:

[1217] The server queries the MongoDB guidance database based on the parsed keywords to extract relevant guidance information, and then takes sentiment information into account to generate customized guidance.

[1218] Input: Analyzed keywords, recognized emotion information

[1219] Output: Customized guidance information (text format)

[1220] Step 6:

[1221] The server transmits the generated guidance to the terminal.

[1222] Specific behavior:

[1223] The server converts the generated guidance information into JSON format and sends it to the terminal as an HTTP response.

[1224] Input: Customized Guidance Information

[1225] Output: Guidance data (JSON format) is sent to the device.

[1226] Step 7:

[1227] The terminal displays the received guidance to the user.

[1228] Specific behavior:

[1229] The terminal analyzes the guidance data received from the server and displays it as text on the screen, and may also convey it to the user as a voice message.

[1230] Input: Guidance data sent from the server (JSON format)

[1231] Output: Guidance displayed to the user (in text and audio format)

[1232] Step 8:

[1233] The user refers to the guidance when moving around or performing activities.

[1234] Specific behavior:

[1235] The user refers to the guidance displayed on the device, avoids crowds, and travels safely to the station. The user also follows the guidance to take appropriate action.

[1236] Input: Guidance displayed on the terminal

[1237] Output: Safe movement or appropriate activity of the user

[1238] (Application example 2)

[1239] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1240] Conventional travel and activity guidance systems provide uniform guidance without considering the user's emotional state, and therefore lack support that reflects the user's individual needs and emotions. The present invention aims to solve this problem and provide more personalized travel and activity guidance in real time based on the user's emotions.

[1241] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1242] In this invention, the server includes a means for receiving a question from a user regarding a movement or activity, a means for extracting information related to the question from a guidance database and generating guidance in real time, a means for providing the generated guidance to the user, and a means for recognizing an emotion of the user and customizing the guidance based on the emotion, thereby making it possible to provide accurate and individualized guidance corresponding to the emotion of the user.

[1243] A "user" is an entity that submits a travel or activity related question to the server and receives guidance.

[1244] "Movement or activity" refers to physical movement or bodily actions that a user performs in daily life.

[1245] A "guidance database" is a database that stores specific guidance information regarding a user's movements and activities.

[1246] The "means for accepting a question" is an interface for sending a question about a movement or activity from a user to the server.

[1247] The "means for extracting information and generating guidance in real time" refers to a mechanism for extracting information related to a question from a guidance database and generating guidance in real time based on the information.

[1248] A "means for providing guidance to a user" is a method or device for communicating the generated guidance to a user.

[1249] A "means for recognizing emotions" is a technology or device for detecting and recognizing a user's emotional state.

[1250] The "means for customizing guidance based on emotions" is a mechanism for individualizing the content and method of guidance according to the recognized emotions of the user.

[1251] The present invention relates to a system for providing real-time guidance to a user to support the user's movements or activities in daily life. Specific embodiments of the system are described below.

[1252] System Configuration

[1253] This system consists of a server, a terminal, a guidance database, an emotion engine, and a user.

[1254] server

[1255] The server has the following functions:

[1256] Question acceptance: Accepts questions about travel or activity sent from the user via the terminal.

[1257] Information Extraction: Extract information relevant to the question from the guidance database.

[1258] Guidance generation: Generate guidance in real time based on the extracted information.

[1259] Emotion Recognition: Use an emotion engine to recognize a user's emotions and customize guidance based on those emotions.

[1260] Terminal

[1261] The terminal functions as a user interface and has the following roles:

[1262] Question input: The user inputs a question about a movement or activity and sends it to the server.

[1263] Display guidance: Displays the guidance received from the server and provides it to the user in the form of audio or video.

[1264] Emotion transmission: The user's emotions are captured using the built-in camera and transmitted to the server.

[1265] Guidance Database

[1266] The guidance database stores specific guidance information related to the user's movements and activities, and also includes guidance information corresponding to the user's emotions.

[1267] Emotion Engine

[1268] The emotion engine processes image data and analyzes the user's emotions. This information is used by the server to customize guidance.

[1269] Data processing and calculation

[1270] This system uses the following hardware and software.

[1271] Camera: Used to take a picture of the user's face and recognize emotions.

[1272] Server (Python framework): Receives questions and emotion data, searches the guidance database, and generates appropriate guidance.

[1273] Emotion engine: A software module that acts as a user interface and analyzes the emotional state using image data as input.

[1274] Guidance database: Contains specific advice and guide data on movement and activities.

[1275] The server first receives the question sent by the terminal, converts it into an appropriate format, and searches the guidance database. Then, it generates suitable guidance in real time based on the obtained guidance information. At the same time, it analyzes the user's emotions through an emotion engine and customizes the content of the guidance based on the results. Finally, the server sends the customized guidance to the terminal, which then provides it to the user by display or voice.

[1276] Specific examples and generated AI model prompts

[1277] For example, if a user enters a question such as "I want to go to Shinjuku Station by the shortest route" in an autonomous vehicle, the server will search the guidance database and provide the optimal route. At the same time, if the user feels anxious about the displayed guidance, the emotion engine will detect this and provide additional guidance to relax.

[1278] Example prompt:

[1279] I am currently heading from my house to Shibuya Station, but I would like to know the shortest route. Also, I am feeling a bit anxious. What driving advice would you give me based on this feeling?

[1280] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1281] Step 1:

[1282] The user inputs a question about a movement or activity into the terminal. For example, the user inputs "I want to go to Shinjuku Station by the shortest route," and the question is sent to the terminal. This question becomes important data for generating guidance later.

[1283] Step 2:

[1284] The terminal sends the question entered by the user to the server. At this time, the terminal converts the question content into an appropriate format and transmits the data to the server using a secure communication method. Specifically, the terminal constructs the question in JSON format and transmits it via an HTTP request.

[1285] Step 3:

[1286] The server receives the question data sent from the terminal and searches the guidance database. The server generates a query to extract related information from the database based on the question content, and executes the query. For example, it extracts information related to "Shinjuku Station" and "shortest route".

[1287] Step 4:

[1288] The server generates guidance in real time based on information extracted from the guidance database. At this time, the server converts the extracted information into a form that is easy for the user to understand and creates a guidance message. For example, it creates specific instructions such as "The shortest route to Shinjuku Station is via XX Road and YY Street."

[1289] Step 5:

[1290] The terminal uses a camera to capture a face image of the user and transmits it to the server as emotion data. The emotion data is important information for recognizing the current emotion state of the user. For example, the image data is converted into an appropriate format before transmission.

[1291] Step 6:

[1292] The server uses an emotion engine to analyze the user's emotions. The received image data is input into the emotion engine, and the emotion state (e.g., anxiety, tension, relief, etc.) is output. Based on this output data, data is organized to adjust the guidance content.

[1293] Step 7:

[1294] The server customizes the guidance message based on the emotion data, for example adding an additional message to relax the user ("Take a deep breath and relax") to the guidance if the user feels anxious.

[1295] Step 8:

[1296] The server sends the customized guidance to the terminal. The server converts the tailored guidance message into a suitable format and transmits the data to the terminal using a secure communication means.

[1297] Step 9:

[1298] The terminal displays or provides the customized guidance received from the server to the user by voice. The user performs optimal movement or activity based on the customized guidance. For example, a voice synthesis system can be used to read out the guidance message to make it easier for the user to understand the instructions.

[1299] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1300] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1301] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1302] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1303] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1304] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1305] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1306] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1307] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1308] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1309] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[1310] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[1311] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1312] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1313] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1314] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1315] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1316] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1317] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1318] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[1319] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[1320] The following is further disclosed regarding the above embodiment.

[1321] (Appendix 1) A system for understanding movements or activities a user performs in daily life and providing real-time guidance regarding physical movements from a guidance database to enable safe movements or activities, the system including: a means for accepting a question regarding the movement or activity from the user; a means for extracting information related to the question from the guidance database and generating the guidance in real-time; and a means for providing the generated guidance to the user.

[1322] (Appendix 2) The means for accepting a question accepts the question regarding movement or activity from the user via a terminal. 2. The system of claim 1.

[1323] (Appendix 3) means for taking into account the characteristics and requirements of the user's movements or activities when extracting information relevant to the question from the guidance database; 10. The system of claim 1 or 2.

[1324] (Appendix 4) Further combining an emotion engine that recognizes the emotion of the user, A system according to any one of Supplementary Note 1 to Supplementary Note 3.

[1325] "Example 1"

[1326] (Appendix 1) A system for understanding movements or activities performed by a user in daily life and providing guidance on body movements in real time from a guidance database so as to enable a user to perform safe movements or activities, comprising: means for receiving queries from the user regarding movement or activity via a terminal; A server means for analyzing the content of the question, extracting information related to the question from the guidance database, and generating the guidance in real time using a generative AI model; a terminal means for providing the generated guidance to the user; A system including:

[1327] (Appendix 2) 2. The system of claim 1, wherein a prompt sentence is used to input to a generative AI model when generating the guidance.

[1328] (Appendix 3) 2. The system of claim 1, further comprising means for taking into account the user's movement or activity characteristics and requirements when extracting information relevant to the question from the guidance database.

[1329] "Application example 1"

[1330] (Appendix 1) A system for understanding movements or activities performed by a user in daily life and providing guidance on body movements in real time from a guidance database so as to enable a user to perform safe movements or activities, comprising: means for accepting queries from said user regarding travel or activity; means for extracting information relevant to said question from said guidance database and generating said guidance in real time; means for providing the generated guidance to the user; A means for transmitting the guidance to a robot control system and causing the robot to perform a task based on an appropriate path and operation procedure; A system including:

[1331] (Appendix 2) The system of claim 1, wherein the means for accepting a question accepts the question regarding movement or activity from the user via a terminal.

[1332] (Appendix 3) 2. The system of claim 1, further comprising means for taking into account the user's movement or activity characteristics and requirements when extracting information relevant to the question from the guidance database.

[1333] (Appendix 4) 2. The system of claim 1, wherein the control system of the robot includes means for selecting and executing an optimal path or motion procedure based on the received guidance.

[1334] (Appendix 5) The system of claim 1, further comprising a means for generating a prompt sentence using a generative AI model based on a question from the user, thereby providing more appropriate guidance.

[1335] "Example 2 of combining emotion engines"

[1336] (Appendix 1) A system for understanding movements or activities performed by a user in daily life and providing guidance on body movements in real time from a database so as to enable a user to perform safe movements or activities, comprising: means for accepting queries from said user regarding travel or activity; A means for analyzing the content of a question and recognizing a user's emotion; A means for extracting information related to the question from a database and generating guidance according to the user's emotion; means for providing the generated guidance to the user; A system including:

[1337] (Appendix 2) The means for accepting a question accepts the question regarding movement or activity from the user via an input / output device. 2. The system described in appendix 1.

[1338] (Appendix 3) means for taking into account the characteristics and requirements of the user's movements or activities when extracting information relevant to said query from said database; 2. The system described in appendix 1.

[1339] "Application example 2 when combining emotion engines"

[1340] (Appendix 1) A system for understanding movements or activities performed by a user in daily life and providing guidance on body movements in real time from a guidance database so as to enable a user to perform safe movements or activities, comprising: means for accepting queries from said user regarding travel or activity; means for extracting information relevant to said question from said guidance database and generating said guidance in real time; means for providing the generated guidance to the user; means for recognizing an emotion of the user and customizing the guidance based on the emotion; A system including:

[1341] (Appendix 2) The system of claim 1, wherein the means for accepting a question accepts the question regarding movement or activity from the user via a terminal.

[1342] (Appendix 3) 2. The system of claim 1, further comprising means for taking into account the user's movement or activity characteristics and requirements when extracting information relevant to the question from the guidance database. [Explanation of symbols]

[1343] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A system for understanding movements or activities performed by a user in daily life and providing guidance on body movements in real time from a guidance database so as to enable a user to perform safe movements or activities, comprising: means for accepting queries from said user regarding travel or activity; means for extracting information relevant to said question from said guidance database and generating said guidance in real time; means for providing the generated guidance to the user; A system including:

2. The means for accepting a question accepts the question regarding movement or activity from the user via a terminal. The system of claim 1 .

3. means for taking into account the characteristics and requirements of the user's movements or activities when extracting information relevant to the query from the guidance database; The system of claim 1 .

4. further combining an emotion engine that recognizes the emotion of the user; the means for generating guidance generates guidance according to the emotion of the user recognized using the emotion engine. The system of claim 1 .

5. means for transmitting the guidance to a control system of the robot and causing the robot to perform a task based on an appropriate path or operation procedure in accordance with the guidance; The system of claim 1 .

Images