System

Abstract

A system is provided.SOLUTION: A system comprising: a display device or an audio device worn by a user; a terminal that acquires and transfers voice data and motion data of the user in real time; and a server that analyzes the voice data and the motion data, generates an instruction for the user, and transmits the instruction to the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] To deliver a more effective and professional presentation, optimal adjustments to speaking speed, positioning, tone of voice, and other factors are required. However, it is often difficult to recognize and respond to these adjustments on one's own, which can result in a decline in the quality of the presentation. To solve this problem, a system is needed that provides users with appropriate instructions in real time and supports the progress of the presentation. [Means for solving the problem]

[0005] The present invention provides a system that includes a display or audio device worn by a user, a terminal that acquires and transfers the user's voice and motion data in real time, and a server that analyzes the voice and motion data, generates instructions for the user, and transmits them to the terminal. This allows the user to receive appropriate instructions in real time during a presentation, and to proceed with the presentation while optimally adjusting speaking speed, tone of voice, standing position, etc.

[0006] "User" refers to a person who uses the system to give a presentation.

[0007] A "display device" is a device worn by a user to visually display information, and an example of this is smart glasses.

[0008] An "audio device" is a device worn by a user to provide audio information, and an example of such a device is an earphone.

[0009] "Terminal" refers to a device that acquires user voice data and motion data in real time and transfers them to a server.

[0010] "Voice data" is digital data of the voice uttered by the user.

[0011] "Motion data" is digital data that represents the user's body movements and positions.

[0012] The term "server" refers to a computer system that analyzes the voice data and motion data received from the terminal, generates instructions for the user, and transmits them to the terminal.

[0013] "Analysis" refers to the process of evaluating collected voice and movement data using algorithms and machine learning models to extract necessary information.

[0014] "Instructions" are advice and guidance necessary for the user to proceed with the presentation more effectively.

[0015] "Display" refers to the act of visually presenting information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that allows a user to wear a display or audio device and receive instructions in real time in order to make a presentation more effective.

[0038] System Configuration

[0039] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[0040] 2. Display or audio devices:

[0041] Display device: A device worn by a user to visually display information, such as smart glasses.

[0042] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0043] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0044] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[0045] Program processing flow

[0046] System initialization and preparation

[0047] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[0048] Real-time monitoring during presentations

[0049] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns.

[0050] Instruction generation and notification

[0051] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, the server generates instructions such as "It would be better if you move around a bit more while you speak." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[0052] Specific examples

[0053] Example 1: Adjusting speaking rate

[0054] 1. The user puts on the smart glasses and begins their presentation.

[0055] 2. The device collects voice data in real time and sends it to the server.

[0056] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0057] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0058] 5. The device sends this instruction to the smart glasses and displays it as text.

[0059] 6. The user looks at the instructions and adjusts the speaking rate.

[0060] Example 2: Adjusting standing position

[0061] 1. The user puts on the earphones and begins their presentation.

[0062] 2. The device collects operational data in real time and sends it to the server.

[0063] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[0064] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0065] 5. The device sends this instruction to the earphones and plays it as audio.

[0066] 6. The user receives instructions and adjusts their position while speaking.

[0067] In this way, users can receive appropriate feedback in real time during their presentations, thereby improving the quality of their presentations and increasing their appeal to the audience.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The user puts on the smart glasses or earphones, launches the presentation software, and once the system is up and running, the user is ready to begin their presentation.

[0071] Step 2:

[0072] The device connects to smart glasses or earphones, initializes the sensors, checks network connectivity, and is ready to acquire data.

[0073] Step 3:

[0074] The device begins capturing the user's voice and movement data in real time. Voice data is collected through a microphone, and movement data is collected using an accelerometer and gyroscope.

[0075] Step 4:

[0076] The device transmits the acquired voice and movement data to the server. The data is processed in real time and transmitted without delay.

[0077] Step 5:

[0078] The server analyzes the received voice data and converts it into text using a speech recognition engine. This text data is then used to evaluate speaking speed and tone of voice.

[0079] Step 6:

[0080] The server analyzes the received motion data and evaluates the user's movement patterns, using combined data from the accelerometer and gyroscope.

[0081] Step 7:

[0082] The server generates appropriate instructions based on the analysis of the voice and movement data. For example, if the user is speaking too quickly, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[0083] Step 8:

[0084] The server sends the generated instructions to the terminal, which may be in text or voice format.

[0085] Step 9:

[0086] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, and in the case of earphones, the instructions are played back as audio.

[0087] Step 10:

[0088] Based on the received instructions, users can adjust their speaking speed, tone of voice, and position during their presentation, improving the quality of their presentation and making it more appealing to the audience.

[0089] Specific examples

[0090] Example 1: Speaking faster

[0091] Step 1:

[0092] The user puts on the smart glasses and begins their presentation.

[0093] Step 2:

[0094] The device collects voice data in real time and sends it to the server.

[0095] Step 3:

[0096] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[0097] Step 4:

[0098] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[0099] Step 5:

[0100] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0101] Step 6:

[0102] The user views the instructions and adjusts the speaking rate.

[0103] Example 2: When the standing position is fixed

[0104] Step 1:

[0105] The user puts on the earphones and begins the presentation.

[0106] Step 2:

[0107] The device collects operational data in real time and transmits it to a server.

[0108] Step 3:

[0109] The server analyzes the movement data and detects when the user has not moved from a fixed location.

[0110] Step 4:

[0111] The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0112] Step 5:

[0113] The device sends this instruction to the earphones, which then play back a voice message saying, "It would be better if you moved around a bit more while talking."

[0114] Step 6:

[0115] The user receives instructions and adjusts their position while speaking.

[0116] Example 1

[0117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] There is a need for a system that allows users to receive effective, real-time instructions while appropriately adjusting their speaking speed, tone of voice, and standing position during a presentation. However, current systems make it difficult for users to check and appropriately adjust their speaking speed and standing position. Furthermore, existing systems do not provide sufficient real-time feedback, making it difficult to improve the quality of presentations.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0120] In this invention, the server includes a data collection means for acquiring and transferring voice information and movement information in real time, an analysis means for analyzing the voice information and movement information, and an instruction generation means for generating instructions for the user and transmitting them to the data collection means, thereby enabling the user to receive appropriate feedback in real time during a presentation.

[0121] A "visual information display means" is a device that displays information visually when worn by a user, and specifically includes smart glasses.

[0122] The "audio output means" refers to a device that outputs audio information when worn by a user, and specifically includes earphones and a headset.

[0123] A "data collection means" is a device that acquires a user's voice information and movement information in real time and transfers it to a server, and specifically includes a smartphone or tablet.

[0124] The "analysis means" is a device or program that analyzes voice information and motion information within the server and generates feedback based on the user's actions.

[0125] The "instruction generating means" is a device or program that generates appropriate instructions for the user based on the analysis results obtained by the analysis means and transmits them to the data collecting means.

[0126] "Voice information" refers to voice data uttered by a user, including speaking speed, tone of voice, and the like.

[0127] "Motion information" is data relating to the user's movements and standing position, and is acquired using an acceleration sensor, a gyro sensor, or the like.

[0128] The present invention relates to a system in which a user can wear a visual information display means or an audio output means and receive instructions in real time in order to make a presentation more effective. The specific configuration for implementing this system is as follows.

[0129] System Configuration

[0130] 1. User:

[0131] The user is the person giving the presentation and wears a visual information display means such as smart glasses or an audio output means such as earphones, which allows the user to receive appropriate visual or audio feedback during the presentation.

[0132] 2. Visual or audio display means:

[0133] Examples of visual information display means include smart glasses, which are devices worn by users that display text information in real time. Examples of audio output means include earphones and headsets, which are worn by users and provide audio information in real time.

[0134] 3. Data collection methods:

[0135] Smartphones and tablets are used as data collection tools. These capture the user's voice and movement information in real time and transfer it to a server. Voice information is collected using the smartphone's microphone, and movement information is collected using the smartphone's built-in accelerometer and gyro sensor.

[0136] 4. Server:

[0137] The server analyzes the voice information and movement information sent from the data collection means and generates instructions for the user. It uses a voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice information, and data from an acceleration sensor and gyro sensor to analyze the movement information. Based on the analysis results, the instruction generation means generates appropriate feedback and sends it to the data collection means.

[0138] Specific examples

[0139] Example 1: Adjusting speaking rate

[0140] 1. The user puts on the smart glasses and begins their presentation.

[0141] 2. The data collection means collects voice data in real time and transmits it to the server.

[0142] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0143] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0144] 5. The data collection device sends this instruction to the smart glasses and displays it as text.

[0145] 6. The user looks at the instructions and adjusts the speaking rate.

[0146] Example 2: Adjusting standing position

[0147] 1. The user puts on the earphones and begins the presentation.

[0148] 2. The data collection means collects the operational data in real time and transmits it to the server.

[0149] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[0150] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0151] 5. The data collection means sends this instruction to the earphone and plays it back audibly.

[0152] 6. The user receives instructions and adjusts their position while speaking.

[0153] Prompt Sentence Examples

[0154] Sample prompt 1: Adjusting speaking speed

[0155] "Generate instructions to slow down speaking speed for users who are speaking too quickly."

[0156] Example output: "You're speaking too quickly. Please slow down."

[0157] Prompt example 2: Adjust your position

[0158] "If the user is standing still, generate prompts to encourage them to move and speak."

[0159] Example output: "You should move around a bit more while you speak."

[0160] Based on the system configuration and specific example described above, users can receive appropriate feedback in real time during a presentation, thereby improving the quality of the presentation.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1: Initialization and preparation

[0163] Before starting a presentation, a user puts on smart glasses or earphones, launches the application on their device, and connects to the system. The device connects to the smart glasses, initializes the microphone and accelerometer, and prepares to acquire voice and motion data. Meanwhile, the server generates a session ID to manage the user session and establishes a connection with the device.

[0164] Input: User operation (wearing smart glasses or earphones, launching applications on the device)

[0165] Output: Connection between smart glasses and device, generation of session ID

[0166] Specific working example:

[0167] The user puts on the smart glasses and launches the application on the device. The device automatically detects the smart glasses and establishes a connection. The server starts a session and sends feedback to the device confirming the connection.

[0168] Step 2: Data collection

[0169] The device uses the smartphone's microphone to collect voice data in real time when the user starts speaking, and also uses the device's accelerometer and gyro sensor to collect motion data as the user starts moving.

[0170] Input: User voice and movements

[0171] Output: Real-time collected audio and motion data

[0172] Specific working example:

[0173] When a user starts speaking during a presentation, the device's microphone begins capturing audio, while the accelerometer and gyroscope sensors pick up data every time the user walks or gestures.

[0174] Step 3: Data Transfer

[0175] The device periodically collects voice and motion data, compresses and encrypts it into packets, and sends them to a server. The data is transferred in real time via Wi-Fi or a mobile network.

[0176] Input: Real-time collected audio and motion data

[0177] Output: Encrypted data packet

[0178] Specific working example:

[0179] The device compresses, encrypts, and packetizes the voice and motion data, then transmits the data over the Internet to a server.

[0180] Step 4: Data analysis

[0181] The server unpacks the received data packets and analyzes the audio data using a speech recognition engine (e.g., Google Cloud Speech-to-Text) to assess speaking rate and tone, as well as motion data to assess the user's position and movement patterns.

[0182] Input: Encrypted data packet

[0183] Output: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[0184] Specific working example:

[0185] The server decompresses the received data and passes it to a speech recognition engine to convert the voice data into text and rating data. The accelerometer data is analyzed to visualize the user's movements.

[0186] Step 5: Instruction Generation

[0187] Based on the analysis results, the server generates instructions such as "You're speaking too fast. Please speak a little slower" if the user is speaking too fast, or "You should move around a bit more while you speak" if the user is not moving from a fixed location.

[0188] Input: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[0189] Output: Specific instruction message

[0190] Specific working example:

[0191] The server evaluates the analysis results and generates appropriate feedback. For example, if the evaluation is "You're speaking too fast," it generates a text message saying, "You're speaking too fast. Please speak a little slower."

[0192] Step 6: Instructions

[0193] The device receives instructions from the server and displays them as text on the smart glasses or plays them as audio through earphones, allowing users to receive instructions in real time and improve the quality of their presentations.

[0194] Input: Specific instruction message

[0195] Output: Text display on smart glasses or audio playback on earphones

[0196] Specific working example:

[0197] The server sends instruction messages to the device, which displays them as text on the smart glasses, or plays the same messages aloud if earphones are used.

[0198] Step 7: User interaction

[0199] The user receives real-time instructions from the device and can take actions to improve the quality of their presentation, such as adjusting their speaking speed or changing their position, allowing them to continue their presentation based on appropriate feedback.

[0200] Input: Text display on smart glasses or voice instructions on earphones

[0201] Output: User's adjusted behavior (adjusting speaking rate, changing standing position, etc.)

[0202] Specific working example:

[0203] The user may see instructions on the smart glasses and slow down their speaking speed, or hear instructions over the earphones such as "It would be good to move around a bit more while speaking" and change their position.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] In modern factory work, it is important for workers to carry out their work efficiently and safely, but the means for receiving real-time feedback and instructions are limited. To solve this problem, a system is needed that provides real-time feedback in a form that workers can intuitively understand.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes a means for analyzing manual movements and voice commands of workers in a factory and generating instructions related to work efficiency and safety, a means for acquiring and transferring user voice data and operation data in real time, and a means for wearing a display or audio device, which allows workers to receive appropriate feedback in real time and improve work efficiency and safety.

[0209] A "user" is a person who works in a factory and wears a display or audio device.

[0210] A "display device" is a device worn by a user that visually displays information, specifically referring to smart glasses.

[0211] An "audio device" is a device that provides audio information when worn by a user, and specifically refers to earphones and the like.

[0212] A "terminal" is a device that acquires a user's voice data and motion data in real time and transfers them to a server, and specifically includes smartphones, tablets, etc.

[0213] A "server" is a computer system that analyzes voice data and motion data received from a terminal, generates instructions for the user, and sends them to the terminal.

[0214] "Motion data" refers to data that indicates the user's manual movements and standing position.

[0215] "Voice data" refers to data that indicates the user's speaking rate and tone of voice.

[0216] "Real-time" refers to the temporal process in which data is acquired, analyzed, and instructions are generated without delay.

[0217] "Manual movements and voice commands by workers in the factory" means physical movements performed by workers in the factory and voice instructions given by workers.

[0218] "Work efficiency and safety instructions" means advice or instructions to optimize the speed and accuracy of work and maintain a safe work environment.

[0219] MODE FOR CARRYING OUT THE INVENTION

[0220] The present invention relates to a real-time feedback system for improving work efficiency and safety in a factory using a display device and an audio device worn by a worker. This system collects voice data and motion data of the user (worker) in real time, transfers the data to a server, analyzes the data, generates feedback, and transmits it to the display device and audio device.

[0221] System Configuration

[0222] 1. User: A person working in a factory who wears a device such as smart glasses or earphones.

[0223] 2. Display or audio devices:

[0224] Display device: A device worn by a user to visually display information, such as smart glasses.

[0225] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0226] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0227] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[0228] Program processing flow

[0229] 1. System initialization and preparation

[0230] Before starting work in the factory, users put on smart glasses or earphones and connect to the system using a terminal. The terminal initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[0231] 2. Real-time monitoring during work

[0232] The device continuously captures the user's voice and movements in real time and sends the data to a server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's manual movements, position, and tool usage.

[0233] 3. Instruction Generation and Notification

[0234] Based on the analysis results, the server generates instructions such as "You're working too fast. Please work a little slower for safety" if the user is working too quickly. If the user is using a tool incorrectly, the server generates instructions such as "You're using this tool incorrectly. Please check how to use it correctly." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time, improving work efficiency and safety.

[0235] Specific examples

[0236] 1. Adjusting the work speed

[0237] The user puts on the smart glasses and begins working.

[0238] The terminal collects voice data and movement data in real time and transmits it to a server.

[0239] The server analyzes the voice data and motion data to detect if the worker is working in a hurry.

[0240] The server generates an instruction saying, "You're working too fast. Let's work a little slower to be safe."

[0241] The device sends this instruction to the smart glasses and displays it as text.

[0242] The user looks at the instructions and adjusts the work speed.

[0243] 2. Proper use of tools

[0244] The user puts on the earphones and begins working.

[0245] The device collects operational data in real time and transmits it to a server.

[0246] The server analyzes the operation data and detects if the user is using the tool in an incorrect manner.

[0247] The server generates an instruction saying, "You're using this tool incorrectly. Let's check how to use it correctly."

[0248] The device sends this instruction to the earphones and plays it back as audio.

[0249] The user receives instructions and corrects the way the tool is used.

[0250] Prompt Sentence Examples

[0251] "If a worker is rushing, generate feedback to instruct them to work at an appropriate speed."

[0252] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0253] Step 1:

[0254] The user wears the smart glasses or earphones and connects to the system via the terminal. The terminal then performs an initialization process and calibrates the sensors. The input is the user's wearing behavior, and the output is a notification that the system has completed initialization.

[0255] Step 2:

[0256] The device begins to capture the user's voice and movement data in real time. It uses sensors to collect voice and movement data and temporarily stores it in a buffer within the device. The input is the user's voice and movement, and the output is the real-time data stored in the buffer.

[0257] Step 3:

[0258] The device sends the collected voice and motion data to the server. The data is pre-processed on the device, compressed, encrypted, and then transferred to the server. The input is the data stored in the buffer, and the output is the secure data transmission to the server.

[0259] Step 4:

[0260] The server analyzes the received voice data using a speech recognition engine to evaluate the user's speaking speed and tone of voice. The motion data is also analyzed using a motion analysis algorithm to evaluate the user's manual movements, standing position, and tool usage. The input is the data received from the device, and the output is the analysis result.

[0261] Step 5:

[0262] The server generates feedback instructions based on the analysis results. For example, if the working speed is too fast, the server generates an instruction such as "You are working too fast. Please work a little slower for safety." If the user is using a tool incorrectly, the server generates an instruction such as "You are using this tool incorrectly. Please check how to use it correctly." The input is the analysis results, and the output is the feedback instructions.

[0263] Step 6:

[0264] The server generates and sends instructions to the terminal. The instructions are generated in text or audio format, and are compressed and encrypted as needed before being transmitted to the terminal. The input is the feedback instruction, and the output is the transmission of the instruction to the terminal.

[0265] Step 7:

[0266] The device decompresses and decodes the received instructions and displays or plays them on the user's smart glasses or earphones. The smart glasses display text and the earphones play audio instructions. The input is the instructions received from the server, and the output is feedback provided to the user.

[0267] Through the above processing steps, the user receives feedback in real time, enabling improved work efficiency and safety.

[0268] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0269] The present invention relates to a system that allows users to wear a display or audio device and receive instructions in real time to make their presentations more effective. Furthermore, by combining it with an emotion engine, the system is able to recognize the user's emotions and provide more appropriate instructions.

[0270] System Configuration

[0271] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[0272] 2. Display or audio devices:

[0273] Display device: A device worn by a user to visually display information, such as smart glasses.

[0274] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0275] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0276] 4. Server: A computer system that analyzes the voice and motion data received from the device, generates instructions for the user, and sends them to the device. It also has an emotion engine that recognizes the user's emotions.

[0277] 5. Emotion engine: An algorithm that recognizes the user's emotions based on their voice and movement data, and generates appropriate instructions based on those emotions.

[0278] Program processing flow

[0279] System initialization and preparation

[0280] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis and emotion recognition.

[0281] Real-time monitoring during presentations

[0282] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a voice recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns. The emotion engine recognizes the user's emotions based on the voice and movement data.

[0283] Instruction generation and notification

[0284] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, it generates instructions such as "You should move around a bit more while you speak." Based on the recognized emotions, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[0285] Specific examples

[0286] Example 1: Adjusting speaking rate

[0287] 1. The user puts on the smart glasses and begins their presentation.

[0288] 2. The device collects voice data in real time and sends it to the server.

[0289] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0290] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0291] 5. The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0292] 6. The user looks at the instructions and adjusts the speaking rate.

[0293] Example 2: Emotion recognition and regulation instructions

[0294] 1. The user puts on the earphones and begins their presentation.

[0295] 2. The device collects voice and movement data in real time and sends it to the server.

[0296] 3. The server analyzes the voice and movement data and uses an emotion engine to determine that the user is nervous.

[0297] 4. The server generates the instruction, "You seem nervous. Take a deep breath and relax."

[0298] 5. The device sends this instruction to the earphones and plays a voice message saying, "You seem nervous. Take a deep breath and relax."

[0299] 6. The user is instructed to take a deep breath and relax.

[0300] In this way, users can receive appropriate feedback and emotion-based advice in real time, improving the quality of their presentations and making them more appealing to the audience.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] The user puts on the smart glasses or earphones and launches the presentation software, which puts the system into preparation mode.

[0304] Step 2:

[0305] The device connects to smart glasses or earphones, initializes sensors, and checks network connectivity. The sensors include a microphone for capturing voice data, an accelerometer for capturing motion data, and a gyroscope.

[0306] Step 3:

[0307] The device will begin capturing the user's voice and movement data in real time, with voice data collected through a microphone and movement data collected through an accelerometer and gyroscope.

[0308] Step 4:

[0309] The device transmits the collected voice and movement data to a server, which transmits the data continuously in real time.

[0310] Step 5:

[0311] The server analyzes the received voice data using a speech recognition engine, converts it into text, and evaluates the speaking speed and tone of voice based on the converted text.

[0312] Step 6:

[0313] The server analyzes the received motion data and evaluates the user's position and movement patterns, using combined accelerometer and gyroscope data.

[0314] Step 7:

[0315] The emotion engine analyzes the received voice and movement data and recognizes the user's emotions based on the tone of voice, speaking patterns, and rhythm of movements. For example, a high-pitched voice and few movements may indicate a state of tension.

[0316] Step 8:

[0317] The server generates appropriate instructions based on the analysis of the voice and movement data and the recognition results of the emotion engine. For example, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly." If the user is nervous, the server generates instructions such as "You seem nervous. Take a deep breath and relax."

[0318] Step 9:

[0319] The server sends the generated instructions to the terminal, which may be in text or voice format.

[0320] Step 10:

[0321] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, while in the case of earphones, the instructions are played back as audio.

[0322] Step 11:

[0323] The user can then use the received instructions to adjust their speaking rate, tone of voice, position, or emotional state during the presentation, such as slowing down, adjusting their tone of voice, moving their position, or taking a deep breath to relax.

[0324] Specific examples

[0325] Example 1: Adjusting speaking rate

[0326] Step 1:

[0327] The user puts on the smart glasses and begins their presentation.

[0328] Step 2:

[0329] The device collects voice data in real time and sends it to the server.

[0330] Step 3:

[0331] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[0332] Step 4:

[0333] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[0334] Step 5:

[0335] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0336] Step 6:

[0337] The user views the instructions and adjusts the speaking rate.

[0338] Example 2: Emotion recognition and regulation instructions

[0339] Step 1:

[0340] The user puts on the earphones and begins the presentation.

[0341] Step 2:

[0342] The terminal collects voice data and movement data in real time and transmits it to a server.

[0343] Step 3:

[0344] The emotion engine analyzes the voice data and movement data and recognizes that the user is nervous based on the high tone of the voice and the lack of movement.

[0345] Step 4:

[0346] The server generates the instruction "You seem nervous. Take a deep breath and relax."

[0347] Step 5:

[0348] The device sends this instruction to the earphones, which play a voice message saying, "You seem nervous. Take a deep breath and relax."

[0349] Step 6:

[0350] The user is instructed to take a deep breath and relax.

[0351] In this way, users receive real-time, appropriate instructions and emotional feedback, improving the quality of their presentations and increasing their appeal to the audience.

[0352] Example 2

[0353] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0354] Conventional presentation support systems are limited to analyzing the user's real-time voice and movement data, and do not provide appropriate feedback that takes into account the user's emotions. Therefore, there is a need for systems that can analyze not only the user's speaking rate and tone of voice, but also changes in their emotions during a presentation in real time, and provide appropriate instructions.

[0355] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0356] In this invention, the server includes means for analyzing voice data and motion data to generate instructions for the user, an emotion engine for recognizing the user's emotion based on the analysis results, and means for notifying the user of the instructions generated based on the recognized emotion. This makes it possible to provide appropriate feedback in real time based on the user's speaking speed, tone of voice, and position as well as changes in emotion.

[0357] A "user" is a person who uses the system to give a presentation.

[0358] A "display device" refers to a device worn by a user that visually displays information.

[0359] "Audio device" refers to a device worn by a user that provides audio information.

[0360] "Terminal" refers to a device that acquires user voice data and motion data in real time and transfers it to a server. Specifically, this applies to portable devices such as smartphones and tablets.

[0361] The term "server" refers to a computer system that analyzes the voice data and motion data received from the terminal, generates instructions for the user, and transmits them to the terminal.

[0362] "Voice data" refers to a voice signal that includes the content and tone of voice of the user.

[0363] "Motion data" refers to data relating to the user's physical movements and position, and is acquired using a gyro sensor, acceleration sensor, etc.

[0364] "Analysis" is the process of assessing the user's condition based on the acquired audio and motion data.

[0365] "Instructions" refer to feedback and advice to the user that is generated based on the analysis results.

[0366] The "emotion engine" is a system equipped with an algorithm that recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions.

[0367] "Display as text" means providing instructions as textual information on a display device.

[0368] "Play as audio" means to notify the user of the instruction by voice through an audio device.

[0369] The present invention relates to a system for effectively supporting users' presentations. The system uses a display or audio device to analyze the user's voice and movement data in real time and provide appropriate feedback. It also uses an emotion engine to recognize the user's emotions and generate instructions based on those emotions.

[0370] System Components

[0371] User

[0372] The user is the person giving the presentation and wears the following devices:

[0373] Display device: A device worn by a user that visually displays information. A specific example is smart glasses.

[0374] Audio device: A device worn by a user that provides audio information. An example is an earphone.

[0375] Terminal

[0376] The terminal is a device that acquires the user's voice data and movement data in real time and transfers them to a server. Specifically, this applies to smartphones and tablets. For example, the smartphone's gyro sensor and acceleration sensor are used to acquire movement data, and the microphone collects voice data.

[0377] server

[0378] The server is a computer system that analyzes the voice and motion data received from the device and generates instructions for the user. It also has an emotion engine that recognizes the user's emotions. The server uses the following software and algorithms:

[0379] Speech recognition engine: For example, using the Google Speech-to-Text API, converts voice data into text.

[0380] Emotion engine: For example, using IBM Watson's emotion analysis API, it recognizes the user's emotions from voice data and movement data.

[0381] Program processing

[0382] The program of this system performs processing in the following order.

[0383] 1. System initialization and preparation:

[0384] Before starting a presentation, users put on smart glasses and earphones and launch a dedicated app on their device.

[0385] The device initializes various sensors (microphone, gyro sensor, acceleration sensor) and prepares to start acquiring data.

[0386] The server prepares to receive data from the device and prepares for analysis and emotion recognition.

[0387] 2. Real-time monitoring during presentation:

[0388] The terminal captures the user's voice and movement data in real time and transmits it to the server.

[0389] The server analyzes the audio data using the Google Speech-to-Text API and evaluates speaking speed and tone of voice.

[0390] It also analyzes motion data to evaluate the user's position and movement patterns.

[0391] The emotion engine recognizes the user's emotions based on voice data and action data.

[0392] 3. Generate instructions:

[0393] Based on the analysis results, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[0394] If the standing position is fixed, the instruction "It would be better if you move around a bit more while talking" is generated.

[0395] Based on the recognized emotion, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax."

[0396] 4. Notice of Instructions:

[0397] The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones.

[0398] Specific examples

[0399] Adjust speaking rate

[0400] The user puts on the smart glasses and begins the presentation.

[0401] The device collects voice data in real time and sends it to the server.

[0402] The server analyzes the audio data and determines that the user is speaking faster.

[0403] The server generates instructions such as "You're speaking too fast. Please speak a little more slowly," and displays them on the smart glasses via the device.

[0404] The user views the instructions and adjusts the speaking rate.

[0405] Emotion recognition and regulation instructions

[0406] The user puts on the earphones and begins the presentation.

[0407] The terminal collects voice data and movement data in real time and transmits it to a server.

[0408] The server analyzes the voice data and movement data and uses an emotion engine to recognize that the user is nervous.

[0409] The server generates instructions such as "You seem nervous. Take a deep breath and relax," and sends a voice notification to the earphones via the device.

[0410] The user is instructed to take a deep breath and relax.

[0411] Prompt Sentence Examples

[0412] "How should I give feedback if someone is speaking too fast during a presentation?"

[0413] "Generate appropriate advice when the user is nervous during a presentation"

[0414] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0415] Step 1: Initialize and prepare the system

[0416] The user puts on the smart glasses and earphones and launches the dedicated app on the device. Specifically, the smart glasses are turned on and the earphones are connected to the device via Bluetooth. This allows the user's device to be integrated into the system.

[0417] The device will initialize the microphone, gyro sensor, accelerometer, etc., and prepare for data collection. Specifically, tap "Start" in the app to calibrate each sensor. This will prepare the environment for collecting voice and movement data.

[0418] (Input) Sensor initialization command for the device worn by the user and the terminal

[0419] (Output) Sensor initialization completion status, connected device information

[0420] Step 2: Real-time monitoring during presentation

[0421] The device captures the user's voice and movement data in real time and sends the data to the server. Specifically, it collects voice with a microphone and captures the user's movements with a gyro sensor and accelerometer. The time-series data is then transferred to the server in real time.

[0422] The server converts the received voice data into text using the Google Speech-to-Text API and analyzes the speaking speed and tone of voice. Specifically, the voice data is divided into frames at regular time intervals and each frame is analyzed.

[0423] The server analyzes the motion data and evaluates the user's position and movement patterns. For example, it extracts information such as "the user is moving to the left" or "the user's position is fixed."

[0424] (Input) Real-time audio and motion data

[0425] (Output) Analyzed speaking rate, tone of voice, and movement patterns

[0426] Step 3: Emotion Recognition

[0427] The server's emotion engine uses voice and motion data to recognize the user's emotions. Specifically, it uses IBM Watson's emotion analysis API to extract emotional data such as "the user is nervous" or "the user is relaxed."

[0428] (Input) Voice data, movement data

[0429] (Output) Recognized emotion data

[0430] Step 4: Generate instructions

[0431] The server generates instructions for the user based on the analysis results. Based on the results of analyzing the voice data, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly," and based on the results of analyzing the movement data, it generates instructions such as "You should move around a bit more while you speak." It also generates instructions based on the emotions recognized by the emotion engine, such as "You seem nervous. Take a deep breath and relax."

[0432] (Input) Speaking rate, tone of voice, movement patterns, emotional data

[0433] (Output) Generated instructions (text or audio)

[0434] Step 5: Notification of instructions

[0435] The device notifies the user of the instructions received from the server. Specifically, it displays text on the smart glasses or plays it as audio through earphones. For example, the instruction "You're speaking too quickly. Please speak a little more slowly" is displayed as text on the smart glasses.

[0436] (Input) Instruction data from the server

[0437] (Output) Text display on smart glasses, audio playback on earphones

[0438] In this way, the system analyzes the user's voice and movement data in real time and provides appropriate feedback to improve the quality of the presentation.

[0439] (Application example 2)

[0440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Conventional presentation support systems and customer service support systems do not provide appropriate feedback that takes into account the user's situation and emotions, making it difficult to maximize the user's response ability or the quality of their presentation. In particular, there was a need for a system that could provide appropriate real-time instructions to store clerks when serving customers, based on the customer's emotions and attitude. This required a system that could improve customer satisfaction and the response ability of store clerks.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0443] In this invention, the server includes means for analyzing voice data and motion data and generating instructions for the user, emotion recognition means for generating appropriate instructions based on the recognized emotions, and means for generating instructions using a generative AI model. This allows the user to receive appropriate feedback in real time through a display device or audio device worn by the user, thereby improving the user's ability to respond in presentations and customer interactions.

[0444] A "user" is someone who uses the system to give presentations or interact with customers.

[0445] A "display device" is a device that provides visual information, such as smart glasses.

[0446] An "audio device" is a device that provides audio information, such as an earphone.

[0447] A "terminal" is a device that acquires user voice data and motion data in real time and transfers them to a server.

[0448] "Voice data" refers to the voice uttered by the user recorded as digital data.

[0449] "Motion data" refers to the user's movements, position, and other data recorded using sensors.

[0450] A "server" is a computer system that analyzes the voice data and motion data sent from the terminal, generates instructions, and sends them to the terminal.

[0451] The "emotion recognition means" is an algorithm that recognizes the user's emotions based on voice data and action data, and generates appropriate instructions based on those emotions.

[0452] A "generative AI model" is an artificial intelligence model that generates instructions in natural language based on data provided to it.

[0453] A "prompt sentence" is an input sentence for a generative AI model, and is the text that serves as the basis for generating instructions.

[0454] "Real time" refers to a time unit in which analysis and feedback are performed simultaneously with the user's actions and speech.

[0455] The present invention relates to a system that allows users to receive real-time instructions to effectively give presentations or serve customers. This system comprises a display or audio device worn by the user, a terminal that acquires and transfers voice and motion data in real time, a server that analyzes this data and generates instructions, and an emotion recognition means and a generative AI model to provide more appropriate instructions.

[0456] Overall system configuration

[0457] 1. User:

[0458] This is a person who uses the system to give presentations or interact with customers. The user wears a display or audio device such as smart glasses or earphones.

[0459] 2. Display or audio devices:

[0460] Display device: A device that provides visual information, such as smart glasses, allowing users to receive real-time instructions as text.

[0461] Audio device: A device that provides audio information, such as an earphone, allowing you to receive real-time instructions by voice.

[0462] 3. Terminal:

[0463] A device that captures user voice and motion data in real time and transfers it to a server. Specifically, this applies to smartphones and tablets.

[0464] 4. Server:

[0465] It is a computer system that analyzes voice and motion data sent from the device, generates instructions for the user, and sends them to the device. It can utilize cloud environments such as Amazon Web Services (AWS) and Google Cloud.

[0466] 5. Emotion recognition means:

[0467] This algorithm recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions. It uses deep learning libraries such as TensorFlow.

[0468] 6. Generative AI Models:

[0469] It is an artificial intelligence model that generates natural language instructions based on the data provided, such as OpenAI's GPT-4.

[0470] Program processing

[0471] The server first receives the voice and movement data sent from the device. The voice data is converted into text using Google Speech API or Amazon Transcribe. The voice and movement data are then analyzed together by an emotion recognition system to determine the user's emotion.

[0472] The server then inputs instructions based on the emotion recognition results into the generative AI model as prompt sentences, such as, "The customer's emotion is dissatisfaction. They said, 'I'd like to know more about this product.' Please instruct them on the appropriate response."

[0473] The generative AI model generates appropriate instructions based on the prompt and sends them back to the server. The server then sends the instructions to the device, which then sends feedback to the display and audio devices. For example, an instruction such as "Raise your voice a little to make the customer feel at ease" is displayed on the user's smart glasses. Audio instructions are also delivered through the earphones.

[0474] Specific examples

[0475] Consider a scenario in which a user wears smart glasses and acts as a store clerk while serving customers. The device collects voice and movement data in real time and sends it to a server. The server analyzes this data and uses emotion recognition to identify a disinterested tone of voice when a customer is asking a question. Based on this, the generative AI model creates instructions such as, "The customer seems anxious. Please respond in a more friendly tone." These instructions are displayed as text on the smart glasses, allowing the user to learn how to respond in real time.

[0476] In this way, this system allows users to receive appropriate feedback in real time during presentations and customer interactions, thereby improving their response skills.

[0477] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0478] Step 1:

[0479] The user puts on smart glasses or earphones and starts up the device (such as a smartphone). The device initializes the sensors and starts collecting voice and movement data. The input is the user's movements and voice, and the output is the collected data. Specifically, the device's application starts up, and the sensors detect the user's movements and voice.

[0480] Step 2:

[0481] The device transmits collected voice and motion data to the server in real time. The input is the data collected in real time, and the output is the transfer of this data to the server. The device continues to send data to the server via a network connection.

[0482] Step 3:

[0483] The server receives the voice data and converts it to text using a speech recognition engine (such as Google Speech API or Amazon Transcribe). The input is voice data and the output is text data. Specifically, the server makes an API call to convert the voice data to text.

[0484] Step 4:

[0485] The server analyzes the motion data and evaluates the user's position and movements. The input is the motion data, and the output is the analysis result. Specific actions are identified by processing the motion data using libraries such as OpenCV.

[0486] Step 5:

[0487] The server uses emotion recognition means to recognize the user's emotions based on the analyzed voice data and movement data. The input is the analyzed voice data and movement data, and the output is emotion data. Specifically, this is done by an emotion recognition algorithm using TensorFlow.

[0488] Step 6:

[0489] Based on the emotion recognition results, the server inputs a prompt sentence into the generative AI model to generate instructions. The input is emotion data and a prompt sentence, and the output is the generated instruction. Specifically, the prompt sentence is processed using OpenAI's GPT-4 or similar software to generate instructions in natural language.

[0490] Example prompt sentence:

[0491] "The customer is unhappy. They stated, 'I'd like to know more about this product.' Please direct them to the appropriate response."

[0492] Step 7:

[0493] The server sends the generated instructions to the terminal. The input is the generated instructions, and the output is the transfer to the terminal. Specifically, the server uses WebSocket or REST API to send the instruction data to the terminal.

[0494] Step 8:

[0495] The device then feeds back the received instructions to the user's display device (smart glasses) and audio device (earphones). The input is the instruction sent from the server, and the output is a notification to the user. Specifically, the instructions are displayed in text on the smart glasses, and the instructions are played back in audio on the earphones.

[0496] In this way, users receive relevant feedback in real time, enabling them to improve their presentations and customer interactions.

[0497] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0499] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0500] [Second embodiment]

[0501] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0502] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0503] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0504] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0505] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0506] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0507] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0508] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0509] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0510] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0511] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0512] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0513] The present invention relates to a system that allows a user to wear a display or audio device and receive instructions in real time in order to make a presentation more effective.

[0514] System Configuration

[0515] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[0516] 2. Display or audio devices:

[0517] Display device: A device worn by a user to visually display information, such as smart glasses.

[0518] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0519] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0520] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[0521] Program processing flow

[0522] System initialization and preparation

[0523] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[0524] Real-time monitoring during presentations

[0525] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns.

[0526] Instruction generation and notification

[0527] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, the server generates instructions such as "It would be better if you move around a bit more while you speak." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[0528] Specific examples

[0529] Example 1: Adjusting speaking rate

[0530] 1. The user puts on the smart glasses and begins their presentation.

[0531] 2. The device collects voice data in real time and sends it to the server.

[0532] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0533] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0534] 5. The device sends this instruction to the smart glasses and displays it as text.

[0535] 6. The user looks at the instructions and adjusts the speaking rate.

[0536] Example 2: Adjusting standing position

[0537] 1. The user puts on the earphones and begins their presentation.

[0538] 2. The device collects operational data in real time and sends it to the server.

[0539] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[0540] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0541] 5. The device sends this instruction to the earphones and plays it as audio.

[0542] 6. The user receives instructions and adjusts their position while speaking.

[0543] In this way, users can receive appropriate feedback in real time during their presentations, thereby improving the quality of their presentations and increasing their appeal to the audience.

[0544] The processing flow will be explained below.

[0545] Step 1:

[0546] The user puts on the smart glasses or earphones, launches the presentation software, and once the system is up and running, the user is ready to begin their presentation.

[0547] Step 2:

[0548] The device connects to smart glasses or earphones, initializes the sensors, checks network connectivity, and is ready to acquire data.

[0549] Step 3:

[0550] The device begins capturing the user's voice and movement data in real time. Voice data is collected through a microphone, and movement data is collected using an accelerometer and gyroscope.

[0551] Step 4:

[0552] The device transmits the acquired voice and movement data to the server. The data is processed in real time and transmitted without delay.

[0553] Step 5:

[0554] The server analyzes the received voice data and converts it into text using a speech recognition engine. This text data is then used to evaluate speaking speed and tone of voice.

[0555] Step 6:

[0556] The server analyzes the received motion data and evaluates the user's movement patterns, using combined data from the accelerometer and gyroscope.

[0557] Step 7:

[0558] The server generates appropriate instructions based on the analysis of the voice and movement data. For example, if the user is speaking too quickly, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[0559] Step 8:

[0560] The server sends the generated instructions to the terminal, which may be in text or voice format.

[0561] Step 9:

[0562] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, and in the case of earphones, the instructions are played back as audio.

[0563] Step 10:

[0564] Based on the received instructions, users can adjust their speaking speed, tone of voice, and position during their presentation, improving the quality of their presentation and making it more appealing to the audience.

[0565] Specific examples

[0566] Example 1: Speaking faster

[0567] Step 1:

[0568] The user puts on the smart glasses and begins their presentation.

[0569] Step 2:

[0570] The device collects voice data in real time and sends it to the server.

[0571] Step 3:

[0572] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[0573] Step 4:

[0574] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[0575] Step 5:

[0576] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0577] Step 6:

[0578] The user views the instructions and adjusts the speaking rate.

[0579] Example 2: When the standing position is fixed

[0580] Step 1:

[0581] The user puts on the earphones and begins the presentation.

[0582] Step 2:

[0583] The device collects operational data in real time and transmits it to a server.

[0584] Step 3:

[0585] The server analyzes the movement data and detects when the user has not moved from a fixed location.

[0586] Step 4:

[0587] The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0588] Step 5:

[0589] The device sends this instruction to the earphones, which then play back a voice message saying, "It would be better if you moved around a bit more while talking."

[0590] Step 6:

[0591] The user receives instructions and adjusts their position while speaking.

[0592] Example 1

[0593] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] There is a need for a system that allows users to receive effective, real-time instructions while appropriately adjusting their speaking speed, tone of voice, and standing position during a presentation. However, current systems make it difficult for users to check and appropriately adjust their speaking speed and standing position. Furthermore, existing systems do not provide sufficient real-time feedback, making it difficult to improve the quality of presentations.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0596] In this invention, the server includes a data collection means for acquiring and transferring voice information and movement information in real time, an analysis means for analyzing the voice information and movement information, and an instruction generation means for generating instructions for the user and transmitting them to the data collection means, thereby enabling the user to receive appropriate feedback in real time during a presentation.

[0597] A "visual information display means" is a device that displays information visually when worn by a user, and specifically includes smart glasses.

[0598] The "audio output means" refers to a device that outputs audio information when worn by a user, and specifically includes earphones and a headset.

[0599] A "data collection means" is a device that acquires a user's voice information and movement information in real time and transfers it to a server, and specifically includes a smartphone or tablet.

[0600] The "analysis means" is a device or program that analyzes voice information and motion information within the server and generates feedback based on the user's actions.

[0601] The "instruction generating means" is a device or program that generates appropriate instructions for the user based on the analysis results obtained by the analysis means and transmits them to the data collecting means.

[0602] "Voice information" refers to voice data uttered by a user, including speaking speed, tone of voice, and the like.

[0603] "Motion information" is data relating to the user's movements and standing position, and is acquired using an acceleration sensor, a gyro sensor, or the like.

[0604] The present invention relates to a system in which a user can wear a visual information display means or an audio output means and receive instructions in real time in order to make a presentation more effective. The specific configuration for implementing this system is as follows.

[0605] System Configuration

[0606] 1. User:

[0607] The user is the person giving the presentation and wears a visual information display means such as smart glasses or an audio output means such as earphones, which allows the user to receive appropriate visual or audio feedback during the presentation.

[0608] 2. Visual or audio display means:

[0609] Examples of visual information display means include smart glasses, which are devices worn by users that display text information in real time. Examples of audio output means include earphones and headsets, which are worn by users and provide audio information in real time.

[0610] 3. Data collection methods:

[0611] Smartphones and tablets are used as data collection tools. These capture the user's voice and movement information in real time and transfer it to a server. Voice information is collected using the smartphone's microphone, and movement information is collected using the smartphone's built-in accelerometer and gyro sensor.

[0612] 4. Server:

[0613] The server analyzes the voice information and movement information sent from the data collection means and generates instructions for the user. It uses a voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice information, and data from an acceleration sensor and gyro sensor to analyze the movement information. Based on the analysis results, the instruction generation means generates appropriate feedback and sends it to the data collection means.

[0614] Specific examples

[0615] Example 1: Adjusting speaking rate

[0616] 1. The user puts on the smart glasses and begins their presentation.

[0617] 2. The data collection means collects voice data in real time and transmits it to the server.

[0618] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0619] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0620] 5. The data collection device sends this instruction to the smart glasses and displays it as text.

[0621] 6. The user looks at the instructions and adjusts the speaking rate.

[0622] Example 2: Adjusting standing position

[0623] 1. The user puts on the earphones and begins the presentation.

[0624] 2. The data collection means collects the operational data in real time and transmits it to the server.

[0625] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[0626] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[0627] 5. The data collection means sends this instruction to the earphone and plays it back audibly.

[0628] 6. The user receives instructions and adjusts their position while speaking.

[0629] Prompt Sentence Examples

[0630] Sample prompt 1: Adjusting speaking speed

[0631] "Generate instructions to slow down speaking speed for users who are speaking too quickly."

[0632] Example output: "You're speaking too quickly. Please slow down."

[0633] Prompt example 2: Adjust your position

[0634] "If the user is standing still, generate prompts to encourage them to move and speak."

[0635] Example output: "You should move around a bit more while you speak."

[0636] Based on the system configuration and specific example described above, users can receive appropriate feedback in real time during a presentation, thereby improving the quality of the presentation.

[0637] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0638] Step 1: Initialization and preparation

[0639] Before starting a presentation, a user puts on smart glasses or earphones, launches the application on their device, and connects to the system. The device connects to the smart glasses, initializes the microphone and accelerometer, and prepares to acquire voice and motion data. Meanwhile, the server generates a session ID to manage the user session and establishes a connection with the device.

[0640] Input: User operation (wearing smart glasses or earphones, launching applications on the device)

[0641] Output: Connection between smart glasses and device, generation of session ID

[0642] Specific working example:

[0643] The user puts on the smart glasses and launches the application on the device. The device automatically detects the smart glasses and establishes a connection. The server starts a session and sends feedback to the device confirming the connection.

[0644] Step 2: Data collection

[0645] The device uses the smartphone's microphone to collect voice data in real time when the user starts speaking, and also uses the device's accelerometer and gyro sensor to collect motion data as the user starts moving.

[0646] Input: User voice and movements

[0647] Output: Real-time collected audio and motion data

[0648] Specific working example:

[0649] When a user starts speaking during a presentation, the device's microphone begins capturing audio, while the accelerometer and gyroscope sensors pick up data every time the user walks or gestures.

[0650] Step 3: Data Transfer

[0651] The device periodically collects voice and motion data, compresses and encrypts it into packets, and sends them to a server. The data is transferred in real time via Wi-Fi or a mobile network.

[0652] Input: Real-time collected audio and motion data

[0653] Output: Encrypted data packet

[0654] Specific working example:

[0655] The device compresses, encrypts, and packetizes the voice and motion data, then transmits the data over the Internet to a server.

[0656] Step 4: Data analysis

[0657] The server unpacks the received data packets and analyzes the audio data using a speech recognition engine (e.g., Google Cloud Speech-to-Text) to assess speaking rate and tone, as well as motion data to assess the user's position and movement patterns.

[0658] Input: Encrypted data packet

[0659] Output: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[0660] Specific working example:

[0661] The server decompresses the received data and passes it to a speech recognition engine to convert the voice data into text and rating data. The accelerometer data is analyzed to visualize the user's movements.

[0662] Step 5: Instruction Generation

[0663] Based on the analysis results, the server generates instructions such as "You're speaking too fast. Please speak a little slower" if the user is speaking too fast, or "You should move around a bit more while you speak" if the user is not moving from a fixed location.

[0664] Input: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[0665] Output: Specific instruction message

[0666] Specific working example:

[0667] The server evaluates the analysis results and generates appropriate feedback. For example, if the evaluation is "You're speaking too fast," it generates a text message saying, "You're speaking too fast. Please speak a little slower."

[0668] Step 6: Instructions

[0669] The device receives instructions from the server and displays them as text on the smart glasses or plays them as audio through earphones, allowing users to receive instructions in real time and improve the quality of their presentations.

[0670] Input: Specific instruction message

[0671] Output: Text display on smart glasses or audio playback on earphones

[0672] Specific working example:

[0673] The server sends instruction messages to the device, which displays them as text on the smart glasses, or plays the same messages aloud if earphones are used.

[0674] Step 7: User interaction

[0675] The user receives real-time instructions from the device and can take actions to improve the quality of their presentation, such as adjusting their speaking speed or changing their position, allowing them to continue their presentation based on appropriate feedback.

[0676] Input: Text display on smart glasses or voice instructions on earphones

[0677] Output: User's adjusted behavior (adjusting speaking rate, changing standing position, etc.)

[0678] Specific working example:

[0679] The user may see instructions on the smart glasses and slow down their speaking speed, or hear instructions over the earphones such as "It would be good to move around a bit more while speaking" and change their position.

[0680] (Application example 1)

[0681] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0682] In modern factory work, it is important for workers to carry out their work efficiently and safely, but the means for receiving real-time feedback and instructions are limited. To solve this problem, a system is needed that provides real-time feedback in a form that workers can intuitively understand.

[0683] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0684] In this invention, the server includes a means for analyzing manual movements and voice commands of workers in a factory and generating instructions related to work efficiency and safety, a means for acquiring and transferring user voice data and operation data in real time, and a means for wearing a display or audio device, which allows workers to receive appropriate feedback in real time and improve work efficiency and safety.

[0685] A "user" is a person who works in a factory and wears a display or audio device.

[0686] A "display device" is a device worn by a user that visually displays information, specifically referring to smart glasses.

[0687] An "audio device" is a device that provides audio information when worn by a user, and specifically refers to earphones and the like.

[0688] A "terminal" is a device that acquires a user's voice data and motion data in real time and transfers them to a server, and specifically includes smartphones, tablets, etc.

[0689] A "server" is a computer system that analyzes voice data and motion data received from a terminal, generates instructions for the user, and sends them to the terminal.

[0690] "Motion data" refers to data that indicates the user's manual movements and standing position.

[0691] "Voice data" refers to data that indicates the user's speaking rate and tone of voice.

[0692] "Real-time" refers to the temporal process in which data is acquired, analyzed, and instructions are generated without delay.

[0693] "Manual movements and voice commands by workers in the factory" means physical movements performed by workers in the factory and voice instructions given by workers.

[0694] "Work efficiency and safety instructions" means advice or instructions to optimize the speed and accuracy of work and maintain a safe work environment.

[0695] MODE FOR CARRYING OUT THE INVENTION

[0696] The present invention relates to a real-time feedback system for improving work efficiency and safety in a factory using a display device and an audio device worn by a worker. This system collects voice data and motion data of the user (worker) in real time, transfers the data to a server, analyzes the data, generates feedback, and transmits it to the display device and audio device.

[0697] System Configuration

[0698] 1. User: A person working in a factory who wears a device such as smart glasses or earphones.

[0699] 2. Display or audio devices:

[0700] Display device: A device worn by a user to visually display information, such as smart glasses.

[0701] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0702] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0703] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[0704] Program processing flow

[0705] 1. System initialization and preparation

[0706] Before starting work in the factory, users put on smart glasses or earphones and connect to the system using a terminal. The terminal initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[0707] 2. Real-time monitoring during work

[0708] The device continuously captures the user's voice and movements in real time and sends the data to a server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's manual movements, position, and tool usage.

[0709] 3. Instruction Generation and Notification

[0710] Based on the analysis results, the server generates instructions such as "You're working too fast. Please work a little slower for safety" if the user is working too quickly. If the user is using a tool incorrectly, the server generates instructions such as "You're using this tool incorrectly. Please check how to use it correctly." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time, improving work efficiency and safety.

[0711] Specific examples

[0712] 1. Adjusting the work speed

[0713] The user puts on the smart glasses and begins working.

[0714] The terminal collects voice data and movement data in real time and transmits it to a server.

[0715] The server analyzes the voice data and motion data to detect if the worker is working in a hurry.

[0716] The server generates an instruction saying, "You're working too fast. Let's work a little slower to be safe."

[0717] The device sends this instruction to the smart glasses and displays it as text.

[0718] The user looks at the instructions and adjusts the work speed.

[0719] 2. Proper use of tools

[0720] The user puts on the earphones and begins working.

[0721] The device collects operational data in real time and transmits it to a server.

[0722] The server analyzes the operation data and detects if the user is using the tool in an incorrect manner.

[0723] The server generates an instruction saying, "You're using this tool incorrectly. Let's check how to use it correctly."

[0724] The device sends this instruction to the earphones and plays it back as audio.

[0725] The user receives instructions and corrects the way the tool is used.

[0726] Prompt Sentence Examples

[0727] "If a worker is rushing, generate feedback to instruct them to work at an appropriate speed."

[0728] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0729] Step 1:

[0730] The user wears the smart glasses or earphones and connects to the system via the terminal. The terminal then performs an initialization process and calibrates the sensors. The input is the user's wearing behavior, and the output is a notification that the system has completed initialization.

[0731] Step 2:

[0732] The device begins to capture the user's voice and movement data in real time. It uses sensors to collect voice and movement data and temporarily stores it in a buffer within the device. The input is the user's voice and movement, and the output is the real-time data stored in the buffer.

[0733] Step 3:

[0734] The device sends the collected voice and motion data to the server. The data is pre-processed on the device, compressed, encrypted, and then transferred to the server. The input is the data stored in the buffer, and the output is the secure data transmission to the server.

[0735] Step 4:

[0736] The server analyzes the received voice data using a speech recognition engine to evaluate the user's speaking speed and tone of voice. The motion data is also analyzed using a motion analysis algorithm to evaluate the user's manual movements, standing position, and tool usage. The input is the data received from the device, and the output is the analysis result.

[0737] Step 5:

[0738] The server generates feedback instructions based on the analysis results. For example, if the working speed is too fast, the server generates an instruction such as "You are working too fast. Please work a little slower for safety." If the user is using a tool incorrectly, the server generates an instruction such as "You are using this tool incorrectly. Please check how to use it correctly." The input is the analysis results, and the output is the feedback instructions.

[0739] Step 6:

[0740] The server generates and sends instructions to the terminal. The instructions are generated in text or audio format, and are compressed and encrypted as needed before being transmitted to the terminal. The input is the feedback instruction, and the output is the transmission of the instruction to the terminal.

[0741] Step 7:

[0742] The device decompresses and decodes the received instructions and displays or plays them on the user's smart glasses or earphones. The smart glasses display text and the earphones play audio instructions. The input is the instructions received from the server, and the output is feedback provided to the user.

[0743] Through the above processing steps, the user receives feedback in real time, enabling improved work efficiency and safety.

[0744] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0745] The present invention relates to a system that allows users to wear a display or audio device and receive instructions in real time to make their presentations more effective. Furthermore, by combining it with an emotion engine, the system is able to recognize the user's emotions and provide more appropriate instructions.

[0746] System Configuration

[0747] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[0748] 2. Display or audio devices:

[0749] Display device: A device worn by a user to visually display information, such as smart glasses.

[0750] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0751] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0752] 4. Server: A computer system that analyzes the voice and motion data received from the device, generates instructions for the user, and sends them to the device. It also has an emotion engine that recognizes the user's emotions.

[0753] 5. Emotion engine: An algorithm that recognizes the user's emotions based on their voice and movement data, and generates appropriate instructions based on those emotions.

[0754] Program processing flow

[0755] System initialization and preparation

[0756] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis and emotion recognition.

[0757] Real-time monitoring during presentations

[0758] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a voice recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns. The emotion engine recognizes the user's emotions based on the voice and movement data.

[0759] Instruction generation and notification

[0760] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, it generates instructions such as "You should move around a bit more while you speak." Based on the recognized emotions, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[0761] Specific examples

[0762] Example 1: Adjusting speaking rate

[0763] 1. The user puts on the smart glasses and begins their presentation.

[0764] 2. The device collects voice data in real time and sends it to the server.

[0765] 3. The server analyzes the audio data and determines that the user is speaking faster.

[0766] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[0767] 5. The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0768] 6. The user looks at the instructions and adjusts the speaking rate.

[0769] Example 2: Emotion recognition and regulation instructions

[0770] 1. The user puts on the earphones and begins their presentation.

[0771] 2. The device collects voice and movement data in real time and sends it to the server.

[0772] 3. The server analyzes the voice and movement data and uses an emotion engine to determine that the user is nervous.

[0773] 4. The server generates the instruction, "You seem nervous. Take a deep breath and relax."

[0774] 5. The device sends this instruction to the earphones and plays a voice message saying, "You seem nervous. Take a deep breath and relax."

[0775] 6. The user is instructed to take a deep breath and relax.

[0776] In this way, users can receive appropriate feedback and emotion-based advice in real time, improving the quality of their presentations and making them more appealing to the audience.

[0777] The processing flow will be explained below.

[0778] Step 1:

[0779] The user puts on the smart glasses or earphones and launches the presentation software, which puts the system into preparation mode.

[0780] Step 2:

[0781] The device connects to smart glasses or earphones, initializes sensors, and checks network connectivity. The sensors include a microphone for capturing voice data, an accelerometer for capturing motion data, and a gyroscope.

[0782] Step 3:

[0783] The device will begin capturing the user's voice and movement data in real time, with voice data collected through a microphone and movement data collected through an accelerometer and gyroscope.

[0784] Step 4:

[0785] The device transmits the collected voice and movement data to a server, which transmits the data continuously in real time.

[0786] Step 5:

[0787] The server analyzes the received voice data using a speech recognition engine, converts it into text, and evaluates the speaking speed and tone of voice based on the converted text.

[0788] Step 6:

[0789] The server analyzes the received motion data and evaluates the user's position and movement patterns, using combined accelerometer and gyroscope data.

[0790] Step 7:

[0791] The emotion engine analyzes the received voice and movement data and recognizes the user's emotions based on the tone of voice, speaking patterns, and rhythm of movements. For example, a high-pitched voice and few movements may indicate a state of tension.

[0792] Step 8:

[0793] The server generates appropriate instructions based on the analysis of the voice and movement data and the recognition results of the emotion engine. For example, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly." If the user is nervous, the server generates instructions such as "You seem nervous. Take a deep breath and relax."

[0794] Step 9:

[0795] The server sends the generated instructions to the terminal, which may be in text or voice format.

[0796] Step 10:

[0797] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, while in the case of earphones, the instructions are played back as audio.

[0798] Step 11:

[0799] The user can then use the received instructions to adjust their speaking rate, tone of voice, position, or emotional state during the presentation, such as slowing down, adjusting their tone of voice, moving their position, or taking a deep breath to relax.

[0800] Specific examples

[0801] Example 1: Adjusting speaking rate

[0802] Step 1:

[0803] The user puts on the smart glasses and begins their presentation.

[0804] Step 2:

[0805] The device collects voice data in real time and sends it to the server.

[0806] Step 3:

[0807] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[0808] Step 4:

[0809] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[0810] Step 5:

[0811] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[0812] Step 6:

[0813] The user views the instructions and adjusts the speaking rate.

[0814] Example 2: Emotion recognition and regulation instructions

[0815] Step 1:

[0816] The user puts on the earphones and begins the presentation.

[0817] Step 2:

[0818] The terminal collects voice data and movement data in real time and transmits it to a server.

[0819] Step 3:

[0820] The emotion engine analyzes the voice data and movement data and recognizes that the user is nervous based on the high tone of the voice and the lack of movement.

[0821] Step 4:

[0822] The server generates the instruction "You seem nervous. Take a deep breath and relax."

[0823] Step 5:

[0824] The device sends this instruction to the earphones, which play a voice message saying, "You seem nervous. Take a deep breath and relax."

[0825] Step 6:

[0826] The user is instructed to take a deep breath and relax.

[0827] In this way, users receive real-time, appropriate instructions and emotional feedback, improving the quality of their presentations and increasing their appeal to the audience.

[0828] Example 2

[0829] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] Conventional presentation support systems are limited to analyzing the user's real-time voice and movement data, and do not provide appropriate feedback that takes into account the user's emotions. Therefore, there is a need for systems that can analyze not only the user's speaking rate and tone of voice, but also changes in their emotions during a presentation in real time, and provide appropriate instructions.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0832] In this invention, the server includes means for analyzing voice data and motion data to generate instructions for the user, an emotion engine for recognizing the user's emotion based on the analysis results, and means for notifying the user of the instructions generated based on the recognized emotion. This makes it possible to provide appropriate feedback in real time based on the user's speaking speed, tone of voice, and position as well as changes in emotion.

[0833] A "user" is a person who uses the system to give a presentation.

[0834] A "display device" refers to a device worn by a user that visually displays information.

[0835] "Audio device" refers to a device worn by a user that provides audio information.

[0836] "Terminal" refers to a device that acquires user voice data and motion data in real time and transfers it to a server. Specifically, this applies to portable devices such as smartphones and tablets.

[0837] The term "server" refers to a computer system that analyzes the voice data and motion data received from the terminal, generates instructions for the user, and transmits them to the terminal.

[0838] "Voice data" refers to a voice signal that includes the content and tone of voice of the user.

[0839] "Motion data" refers to data relating to the user's physical movements and position, and is acquired using a gyro sensor, acceleration sensor, etc.

[0840] "Analysis" is the process of assessing the user's condition based on the acquired audio and motion data.

[0841] "Instructions" refer to feedback and advice to the user that is generated based on the analysis results.

[0842] The "emotion engine" is a system equipped with an algorithm that recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions.

[0843] "Display as text" means providing instructions as textual information on a display device.

[0844] "Play as audio" means to notify the user of the instruction by voice through an audio device.

[0845] The present invention relates to a system for effectively supporting users' presentations. The system uses a display or audio device to analyze the user's voice and movement data in real time and provide appropriate feedback. It also uses an emotion engine to recognize the user's emotions and generate instructions based on those emotions.

[0846] System Components

[0847] User

[0848] The user is the person giving the presentation and wears the following devices:

[0849] Display device: A device worn by a user that visually displays information. A specific example is smart glasses.

[0850] Audio device: A device worn by a user that provides audio information. An example is an earphone.

[0851] Terminal

[0852] The terminal is a device that acquires the user's voice data and movement data in real time and transfers them to a server. Specifically, this applies to smartphones and tablets. For example, the smartphone's gyro sensor and acceleration sensor are used to acquire movement data, and the microphone collects voice data.

[0853] server

[0854] The server is a computer system that analyzes the voice and motion data received from the device and generates instructions for the user. It also has an emotion engine that recognizes the user's emotions. The server uses the following software and algorithms:

[0855] Speech recognition engine: For example, using the Google Speech-to-Text API, converts voice data into text.

[0856] Emotion engine: For example, using IBM Watson's emotion analysis API, it recognizes the user's emotions from voice data and movement data.

[0857] Program processing

[0858] The program of this system performs processing in the following order.

[0859] 1. System initialization and preparation:

[0860] Before starting a presentation, users put on smart glasses and earphones and launch a dedicated app on their device.

[0861] The device initializes various sensors (microphone, gyro sensor, acceleration sensor) and prepares to start acquiring data.

[0862] The server prepares to receive data from the device and prepares for analysis and emotion recognition.

[0863] 2. Real-time monitoring during presentation:

[0864] The terminal captures the user's voice and movement data in real time and transmits it to the server.

[0865] The server analyzes the audio data using the Google Speech-to-Text API and evaluates speaking speed and tone of voice.

[0866] It also analyzes motion data to evaluate the user's position and movement patterns.

[0867] The emotion engine recognizes the user's emotions based on voice data and action data.

[0868] 3. Generate instructions:

[0869] Based on the analysis results, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[0870] If the standing position is fixed, the instruction "It would be better if you move around a bit more while talking" is generated.

[0871] Based on the recognized emotion, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax."

[0872] 4. Notice of Instructions:

[0873] The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones.

[0874] Specific examples

[0875] Adjust speaking rate

[0876] The user puts on the smart glasses and begins the presentation.

[0877] The device collects voice data in real time and sends it to the server.

[0878] The server analyzes the audio data and determines that the user is speaking faster.

[0879] The server generates instructions such as "You're speaking too fast. Please speak a little more slowly," and displays them on the smart glasses via the device.

[0880] The user views the instructions and adjusts the speaking rate.

[0881] Emotion recognition and regulation instructions

[0882] The user puts on the earphones and begins the presentation.

[0883] The terminal collects voice data and movement data in real time and transmits it to a server.

[0884] The server analyzes the voice data and movement data and uses an emotion engine to recognize that the user is nervous.

[0885] The server generates instructions such as "You seem nervous. Take a deep breath and relax," and sends a voice notification to the earphones via the device.

[0886] The user is instructed to take a deep breath and relax.

[0887] Prompt Sentence Examples

[0888] "How should I give feedback if someone is speaking too fast during a presentation?"

[0889] "Generate appropriate advice when the user is nervous during a presentation"

[0890] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0891] Step 1: Initialize and prepare the system

[0892] The user puts on the smart glasses and earphones and launches the dedicated app on the device. Specifically, the smart glasses are turned on and the earphones are connected to the device via Bluetooth. This allows the user's device to be integrated into the system.

[0893] The device will initialize the microphone, gyro sensor, accelerometer, etc., and prepare for data collection. Specifically, tap "Start" in the app to calibrate each sensor. This will prepare the environment for collecting voice and movement data.

[0894] (Input) Sensor initialization command for the device worn by the user and the terminal

[0895] (Output) Sensor initialization completion status, connected device information

[0896] Step 2: Real-time monitoring during presentation

[0897] The device captures the user's voice and movement data in real time and sends the data to the server. Specifically, it collects voice with a microphone and captures the user's movements with a gyro sensor and accelerometer. The time-series data is then transferred to the server in real time.

[0898] The server converts the received voice data into text using the Google Speech-to-Text API and analyzes the speaking speed and tone of voice. Specifically, the voice data is divided into frames at regular time intervals and each frame is analyzed.

[0899] The server analyzes the motion data and evaluates the user's position and movement patterns. For example, it extracts information such as "the user is moving to the left" or "the user's position is fixed."

[0900] (Input) Real-time audio and motion data

[0901] (Output) Analyzed speaking rate, tone of voice, and movement patterns

[0902] Step 3: Emotion Recognition

[0903] The server's emotion engine uses voice and motion data to recognize the user's emotions. Specifically, it uses IBM Watson's emotion analysis API to extract emotional data such as "the user is nervous" or "the user is relaxed."

[0904] (Input) Voice data, movement data

[0905] (Output) Recognized emotion data

[0906] Step 4: Generate instructions

[0907] The server generates instructions for the user based on the analysis results. Based on the results of analyzing the voice data, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly," and based on the results of analyzing the movement data, it generates instructions such as "You should move around a bit more while you speak." It also generates instructions based on the emotions recognized by the emotion engine, such as "You seem nervous. Take a deep breath and relax."

[0908] (Input) Speaking rate, tone of voice, movement patterns, emotional data

[0909] (Output) Generated instructions (text or audio)

[0910] Step 5: Notification of instructions

[0911] The device notifies the user of the instructions received from the server. Specifically, it displays text on the smart glasses or plays it as audio through earphones. For example, the instruction "You're speaking too quickly. Please speak a little more slowly" is displayed as text on the smart glasses.

[0912] (Input) Instruction data from the server

[0913] (Output) Text display on smart glasses, audio playback on earphones

[0914] In this way, the system analyzes the user's voice and movement data in real time and provides appropriate feedback to improve the quality of the presentation.

[0915] (Application example 2)

[0916] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0917] Conventional presentation support systems and customer service support systems do not provide appropriate feedback that takes into account the user's situation and emotions, making it difficult to maximize the user's response ability or the quality of their presentation. In particular, there was a need for a system that could provide appropriate real-time instructions to store clerks when serving customers, based on the customer's emotions and attitude. This required a system that could improve customer satisfaction and the response ability of store clerks.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0919] In this invention, the server includes means for analyzing voice data and motion data and generating instructions for the user, emotion recognition means for generating appropriate instructions based on the recognized emotions, and means for generating instructions using a generative AI model. This allows the user to receive appropriate feedback in real time through a display device or audio device worn by the user, thereby improving the user's ability to respond in presentations and customer interactions.

[0920] A "user" is someone who uses the system to give presentations or interact with customers.

[0921] A "display device" is a device that provides visual information, such as smart glasses.

[0922] An "audio device" is a device that provides audio information, such as an earphone.

[0923] A "terminal" is a device that acquires user voice data and motion data in real time and transfers them to a server.

[0924] "Voice data" refers to the voice uttered by the user recorded as digital data.

[0925] "Motion data" refers to the user's movements, position, and other data recorded using sensors.

[0926] A "server" is a computer system that analyzes the voice data and motion data sent from the terminal, generates instructions, and sends them to the terminal.

[0927] The "emotion recognition means" is an algorithm that recognizes the user's emotions based on voice data and action data, and generates appropriate instructions based on those emotions.

[0928] A "generative AI model" is an artificial intelligence model that generates instructions in natural language based on data provided to it.

[0929] A "prompt sentence" is an input sentence for a generative AI model, and is the text that serves as the basis for generating instructions.

[0930] "Real time" refers to a time unit in which analysis and feedback are performed simultaneously with the user's actions and speech.

[0931] The present invention relates to a system that allows users to receive real-time instructions to effectively give presentations or serve customers. This system comprises a display or audio device worn by the user, a terminal that acquires and transfers voice and motion data in real time, a server that analyzes this data and generates instructions, and an emotion recognition means and a generative AI model to provide more appropriate instructions.

[0932] Overall system configuration

[0933] 1. User:

[0934] This is a person who uses the system to give presentations or interact with customers. The user wears a display or audio device such as smart glasses or earphones.

[0935] 2. Display or audio devices:

[0936] Display device: A device that provides visual information, such as smart glasses, allowing users to receive real-time instructions as text.

[0937] Audio device: A device that provides audio information, such as an earphone, allowing you to receive real-time instructions by voice.

[0938] 3. Terminal:

[0939] A device that captures user voice and motion data in real time and transfers it to a server. Specifically, this applies to smartphones and tablets.

[0940] 4. Server:

[0941] It is a computer system that analyzes voice and motion data sent from the device, generates instructions for the user, and sends them to the device. It can utilize cloud environments such as Amazon Web Services (AWS) and Google Cloud.

[0942] 5. Emotion recognition means:

[0943] This algorithm recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions. It uses deep learning libraries such as TensorFlow.

[0944] 6. Generative AI Models:

[0945] It is an artificial intelligence model that generates natural language instructions based on the data provided, such as OpenAI's GPT-4.

[0946] Program processing

[0947] The server first receives the voice and movement data sent from the device. The voice data is converted into text using Google Speech API or Amazon Transcribe. The voice and movement data are then analyzed together by an emotion recognition system to determine the user's emotion.

[0948] The server then inputs instructions based on the emotion recognition results into the generative AI model as prompt sentences, such as, "The customer's emotion is dissatisfaction. They said, 'I'd like to know more about this product.' Please instruct them on the appropriate response."

[0949] The generative AI model generates appropriate instructions based on the prompt and sends them back to the server. The server then sends the instructions to the device, which then sends feedback to the display and audio devices. For example, an instruction such as "Raise your voice a little to make the customer feel at ease" is displayed on the user's smart glasses. Audio instructions are also delivered through the earphones.

[0950] Specific examples

[0951] Consider a scenario in which a user wears smart glasses and acts as a store clerk while serving customers. The device collects voice and movement data in real time and sends it to a server. The server analyzes this data and uses emotion recognition to identify a disinterested tone of voice when a customer is asking a question. Based on this, the generative AI model creates instructions such as, "The customer seems anxious. Please respond in a more friendly tone." These instructions are displayed as text on the smart glasses, allowing the user to learn how to respond in real time.

[0952] In this way, this system allows users to receive appropriate feedback in real time during presentations and customer interactions, thereby improving their response skills.

[0953] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0954] Step 1:

[0955] The user puts on smart glasses or earphones and starts up the device (such as a smartphone). The device initializes the sensors and starts collecting voice and movement data. The input is the user's movements and voice, and the output is the collected data. Specifically, the device's application starts up, and the sensors detect the user's movements and voice.

[0956] Step 2:

[0957] The device transmits collected voice and motion data to the server in real time. The input is the data collected in real time, and the output is the transfer of this data to the server. The device continues to send data to the server via a network connection.

[0958] Step 3:

[0959] The server receives the voice data and converts it to text using a speech recognition engine (such as Google Speech API or Amazon Transcribe). The input is voice data and the output is text data. Specifically, the server makes an API call to convert the voice data to text.

[0960] Step 4:

[0961] The server analyzes the motion data and evaluates the user's position and movements. The input is the motion data, and the output is the analysis result. Specific actions are identified by processing the motion data using libraries such as OpenCV.

[0962] Step 5:

[0963] The server uses emotion recognition means to recognize the user's emotions based on the analyzed voice data and movement data. The input is the analyzed voice data and movement data, and the output is emotion data. Specifically, this is done by an emotion recognition algorithm using TensorFlow.

[0964] Step 6:

[0965] Based on the emotion recognition results, the server inputs a prompt sentence into the generative AI model to generate instructions. The input is emotion data and a prompt sentence, and the output is the generated instruction. Specifically, the prompt sentence is processed using OpenAI's GPT-4 or similar software to generate instructions in natural language.

[0966] Example prompt sentence:

[0967] "The customer is unhappy. They stated, 'I'd like to know more about this product.' Please direct them to the appropriate response."

[0968] Step 7:

[0969] The server sends the generated instructions to the terminal. The input is the generated instructions, and the output is the transfer to the terminal. Specifically, the server uses WebSocket or REST API to send the instruction data to the terminal.

[0970] Step 8:

[0971] The device then feeds back the received instructions to the user's display device (smart glasses) and audio device (earphones). The input is the instruction sent from the server, and the output is a notification to the user. Specifically, the instructions are displayed in text on the smart glasses, and the instructions are played back in audio on the earphones.

[0972] In this way, users receive relevant feedback in real time, enabling them to improve their presentations and customer interactions.

[0973] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0974] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0975] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0976] [Third embodiment]

[0977] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0978] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0979] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0980] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0981] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0982] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0983] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0984] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0985] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0986] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0987] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0988] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0989] The present invention relates to a system that allows a user to wear a display or audio device and receive instructions in real time in order to make a presentation more effective.

[0990] System Configuration

[0991] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[0992] 2. Display or audio devices:

[0993] Display device: A device worn by a user to visually display information, such as smart glasses.

[0994] Audio device: A device worn by a user that provides audio information, such as an earphone.

[0995] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[0996] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[0997] Program processing flow

[0998] System initialization and preparation

[0999] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[1000] Real-time monitoring during presentations

[1001] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns.

[1002] Instruction generation and notification

[1003] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, the server generates instructions such as "It would be better if you move around a bit more while you speak." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[1004] Specific examples

[1005] Example 1: Adjusting speaking rate

[1006] 1. The user puts on the smart glasses and begins their presentation.

[1007] 2. The device collects voice data in real time and sends it to the server.

[1008] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1009] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1010] 5. The device sends this instruction to the smart glasses and displays it as text.

[1011] 6. The user looks at the instructions and adjusts the speaking rate.

[1012] Example 2: Adjusting standing position

[1013] 1. The user puts on the earphones and begins their presentation.

[1014] 2. The device collects operational data in real time and sends it to the server.

[1015] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[1016] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1017] 5. The device sends this instruction to the earphones and plays it as audio.

[1018] 6. The user receives instructions and adjusts their position while speaking.

[1019] In this way, users can receive appropriate feedback in real time during their presentations, thereby improving the quality of their presentations and increasing their appeal to the audience.

[1020] The processing flow will be explained below.

[1021] Step 1:

[1022] The user puts on the smart glasses or earphones, launches the presentation software, and once the system is up and running, the user is ready to begin their presentation.

[1023] Step 2:

[1024] The device connects to smart glasses or earphones, initializes the sensors, checks network connectivity, and is ready to acquire data.

[1025] Step 3:

[1026] The device begins capturing the user's voice and movement data in real time. Voice data is collected through a microphone, and movement data is collected using an accelerometer and gyroscope.

[1027] Step 4:

[1028] The device transmits the acquired voice and movement data to the server. The data is processed in real time and transmitted without delay.

[1029] Step 5:

[1030] The server analyzes the received voice data and converts it into text using a speech recognition engine. This text data is then used to evaluate speaking speed and tone of voice.

[1031] Step 6:

[1032] The server analyzes the received motion data and evaluates the user's movement patterns, using combined data from the accelerometer and gyroscope.

[1033] Step 7:

[1034] The server generates appropriate instructions based on the analysis of the voice and movement data. For example, if the user is speaking too quickly, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[1035] Step 8:

[1036] The server sends the generated instructions to the terminal, which may be in text or voice format.

[1037] Step 9:

[1038] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, and in the case of earphones, the instructions are played back as audio.

[1039] Step 10:

[1040] Based on the received instructions, users can adjust their speaking speed, tone of voice, and position during their presentation, improving the quality of their presentation and making it more appealing to the audience.

[1041] Specific examples

[1042] Example 1: Speaking faster

[1043] Step 1:

[1044] The user puts on the smart glasses and begins their presentation.

[1045] Step 2:

[1046] The device collects voice data in real time and sends it to the server.

[1047] Step 3:

[1048] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[1049] Step 4:

[1050] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[1051] Step 5:

[1052] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1053] Step 6:

[1054] The user views the instructions and adjusts the speaking rate.

[1055] Example 2: When the standing position is fixed

[1056] Step 1:

[1057] The user puts on the earphones and begins the presentation.

[1058] Step 2:

[1059] The device collects operational data in real time and transmits it to a server.

[1060] Step 3:

[1061] The server analyzes the movement data and detects when the user has not moved from a fixed location.

[1062] Step 4:

[1063] The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1064] Step 5:

[1065] The device sends this instruction to the earphones, which then play back a voice message saying, "It would be better if you moved around a bit more while talking."

[1066] Step 6:

[1067] The user receives instructions and adjusts their position while speaking.

[1068] Example 1

[1069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1070] There is a need for a system that allows users to receive effective, real-time instructions while appropriately adjusting their speaking speed, tone of voice, and standing position during a presentation. However, current systems make it difficult for users to check and appropriately adjust their speaking speed and standing position. Furthermore, existing systems do not provide sufficient real-time feedback, making it difficult to improve the quality of presentations.

[1071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1072] In this invention, the server includes a data collection means for acquiring and transferring voice information and movement information in real time, an analysis means for analyzing the voice information and movement information, and an instruction generation means for generating instructions for the user and transmitting them to the data collection means, thereby enabling the user to receive appropriate feedback in real time during a presentation.

[1073] A "visual information display means" is a device that displays information visually when worn by a user, and specifically includes smart glasses.

[1074] The "audio output means" refers to a device that outputs audio information when worn by a user, and specifically includes earphones and a headset.

[1075] A "data collection means" is a device that acquires a user's voice information and movement information in real time and transfers it to a server, and specifically includes a smartphone or tablet.

[1076] The "analysis means" is a device or program that analyzes voice information and motion information within the server and generates feedback based on the user's actions.

[1077] The "instruction generating means" is a device or program that generates appropriate instructions for the user based on the analysis results obtained by the analysis means and transmits them to the data collecting means.

[1078] "Voice information" refers to voice data uttered by a user, including speaking speed, tone of voice, and the like.

[1079] "Motion information" is data relating to the user's movements and standing position, and is acquired using an acceleration sensor, a gyro sensor, or the like.

[1080] The present invention relates to a system in which a user can wear a visual information display means or an audio output means and receive instructions in real time in order to make a presentation more effective. The specific configuration for implementing this system is as follows.

[1081] System Configuration

[1082] 1. User:

[1083] The user is the person giving the presentation and wears a visual information display means such as smart glasses or an audio output means such as earphones, which allows the user to receive appropriate visual or audio feedback during the presentation.

[1084] 2. Visual or audio display means:

[1085] Examples of visual information display means include smart glasses, which are devices worn by users that display text information in real time. Examples of audio output means include earphones and headsets, which are worn by users and provide audio information in real time.

[1086] 3. Data collection methods:

[1087] Smartphones and tablets are used as data collection tools. These capture the user's voice and movement information in real time and transfer it to a server. Voice information is collected using the smartphone's microphone, and movement information is collected using the smartphone's built-in accelerometer and gyro sensor.

[1088] 4. Server:

[1089] The server analyzes the voice information and movement information sent from the data collection means and generates instructions for the user. It uses a voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice information, and data from an acceleration sensor and gyro sensor to analyze the movement information. Based on the analysis results, the instruction generation means generates appropriate feedback and sends it to the data collection means.

[1090] Specific examples

[1091] Example 1: Adjusting speaking rate

[1092] 1. The user puts on the smart glasses and begins their presentation.

[1093] 2. The data collection means collects voice data in real time and transmits it to the server.

[1094] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1095] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1096] 5. The data collection device sends this instruction to the smart glasses and displays it as text.

[1097] 6. The user looks at the instructions and adjusts the speaking rate.

[1098] Example 2: Adjusting standing position

[1099] 1. The user puts on the earphones and begins the presentation.

[1100] 2. The data collection means collects the operational data in real time and transmits it to the server.

[1101] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[1102] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1103] 5. The data collection means sends this instruction to the earphone and plays it back audibly.

[1104] 6. The user receives instructions and adjusts their position while speaking.

[1105] Prompt Sentence Examples

[1106] Sample prompt 1: Adjusting speaking speed

[1107] "Generate instructions to slow down speaking speed for users who are speaking too quickly."

[1108] Example output: "You're speaking too quickly. Please slow down."

[1109] Prompt example 2: Adjust your position

[1110] "If the user is standing still, generate prompts to encourage them to move and speak."

[1111] Example output: "You should move around a bit more while you speak."

[1112] Based on the system configuration and specific example described above, users can receive appropriate feedback in real time during a presentation, thereby improving the quality of the presentation.

[1113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1114] Step 1: Initialization and preparation

[1115] Before starting a presentation, a user puts on smart glasses or earphones, launches the application on their device, and connects to the system. The device connects to the smart glasses, initializes the microphone and accelerometer, and prepares to acquire voice and motion data. Meanwhile, the server generates a session ID to manage the user session and establishes a connection with the device.

[1116] Input: User operation (wearing smart glasses or earphones, launching applications on the device)

[1117] Output: Connection between smart glasses and device, generation of session ID

[1118] Specific working example:

[1119] The user puts on the smart glasses and launches the application on the device. The device automatically detects the smart glasses and establishes a connection. The server starts a session and sends feedback to the device confirming the connection.

[1120] Step 2: Data collection

[1121] The device uses the smartphone's microphone to collect voice data in real time when the user starts speaking, and also uses the device's accelerometer and gyro sensor to collect motion data as the user starts moving.

[1122] Input: User voice and movements

[1123] Output: Real-time collected audio and motion data

[1124] Specific working example:

[1125] When a user starts speaking during a presentation, the device's microphone begins capturing audio, while the accelerometer and gyroscope sensors pick up data every time the user walks or gestures.

[1126] Step 3: Data Transfer

[1127] The device periodically collects voice and motion data, compresses and encrypts it into packets, and sends them to a server. The data is transferred in real time via Wi-Fi or a mobile network.

[1128] Input: Real-time collected audio and motion data

[1129] Output: Encrypted data packet

[1130] Specific working example:

[1131] The device compresses, encrypts, and packetizes the voice and motion data, then transmits the data over the Internet to a server.

[1132] Step 4: Data analysis

[1133] The server unpacks the received data packets and analyzes the audio data using a speech recognition engine (e.g., Google Cloud Speech-to-Text) to assess speaking rate and tone, as well as motion data to assess the user's position and movement patterns.

[1134] Input: Encrypted data packet

[1135] Output: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[1136] Specific working example:

[1137] The server decompresses the received data and passes it to a speech recognition engine to convert the voice data into text and rating data. The accelerometer data is analyzed to visualize the user's movements.

[1138] Step 5: Instruction Generation

[1139] Based on the analysis results, the server generates instructions such as "You're speaking too fast. Please speak a little slower" if the user is speaking too fast, or "You should move around a bit more while you speak" if the user is not moving from a fixed location.

[1140] Input: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[1141] Output: Specific instruction message

[1142] Specific working example:

[1143] The server evaluates the analysis results and generates appropriate feedback. For example, if the evaluation is "You're speaking too fast," it generates a text message saying, "You're speaking too fast. Please speak a little slower."

[1144] Step 6: Instructions

[1145] The device receives instructions from the server and displays them as text on the smart glasses or plays them as audio through earphones, allowing users to receive instructions in real time and improve the quality of their presentations.

[1146] Input: Specific instruction message

[1147] Output: Text display on smart glasses or audio playback on earphones

[1148] Specific working example:

[1149] The server sends instruction messages to the device, which displays them as text on the smart glasses, or plays the same messages aloud if earphones are used.

[1150] Step 7: User interaction

[1151] The user receives real-time instructions from the device and can take actions to improve the quality of their presentation, such as adjusting their speaking speed or changing their position, allowing them to continue their presentation based on appropriate feedback.

[1152] Input: Text display on smart glasses or voice instructions on earphones

[1153] Output: User's adjusted behavior (adjusting speaking rate, changing standing position, etc.)

[1154] Specific working example:

[1155] The user may see instructions on the smart glasses and slow down their speaking speed, or hear instructions over the earphones such as "It would be good to move around a bit more while speaking" and change their position.

[1156] (Application example 1)

[1157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1158] In modern factory work, it is important for workers to carry out their work efficiently and safely, but the means for receiving real-time feedback and instructions are limited. To solve this problem, a system is needed that provides real-time feedback in a form that workers can intuitively understand.

[1159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1160] In this invention, the server includes a means for analyzing manual movements and voice commands of workers in a factory and generating instructions related to work efficiency and safety, a means for acquiring and transferring user voice data and operation data in real time, and a means for wearing a display or audio device, which allows workers to receive appropriate feedback in real time and improve work efficiency and safety.

[1161] A "user" is a person who works in a factory and wears a display or audio device.

[1162] A "display device" is a device worn by a user that visually displays information, specifically referring to smart glasses.

[1163] An "audio device" is a device that provides audio information when worn by a user, and specifically refers to earphones and the like.

[1164] A "terminal" is a device that acquires a user's voice data and motion data in real time and transfers them to a server, and specifically includes smartphones, tablets, etc.

[1165] A "server" is a computer system that analyzes voice data and motion data received from a terminal, generates instructions for the user, and sends them to the terminal.

[1166] "Motion data" refers to data that indicates the user's manual movements and standing position.

[1167] "Voice data" refers to data that indicates the user's speaking rate and tone of voice.

[1168] "Real-time" refers to the temporal process in which data is acquired, analyzed, and instructions are generated without delay.

[1169] "Manual movements and voice commands by workers in the factory" means physical movements performed by workers in the factory and voice instructions given by workers.

[1170] "Work efficiency and safety instructions" means advice or instructions to optimize the speed and accuracy of work and maintain a safe work environment.

[1171] MODE FOR CARRYING OUT THE INVENTION

[1172] The present invention relates to a real-time feedback system for improving work efficiency and safety in a factory using a display device and an audio device worn by a worker. This system collects voice data and motion data of the user (worker) in real time, transfers the data to a server, analyzes the data, generates feedback, and transmits it to the display device and audio device.

[1173] System Configuration

[1174] 1. User: A person working in a factory who wears a device such as smart glasses or earphones.

[1175] 2. Display or audio devices:

[1176] Display device: A device worn by a user to visually display information, such as smart glasses.

[1177] Audio device: A device worn by a user that provides audio information, such as an earphone.

[1178] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[1179] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[1180] Program processing flow

[1181] 1. System initialization and preparation

[1182] Before starting work in the factory, users put on smart glasses or earphones and connect to the system using a terminal. The terminal initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[1183] 2. Real-time monitoring during work

[1184] The device continuously captures the user's voice and movements in real time and sends the data to a server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's manual movements, position, and tool usage.

[1185] 3. Instruction Generation and Notification

[1186] Based on the analysis results, the server generates instructions such as "You're working too fast. Please work a little slower for safety" if the user is working too quickly. If the user is using a tool incorrectly, the server generates instructions such as "You're using this tool incorrectly. Please check how to use it correctly." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time, improving work efficiency and safety.

[1187] Specific examples

[1188] 1. Adjusting the work speed

[1189] The user puts on the smart glasses and begins working.

[1190] The terminal collects voice data and movement data in real time and transmits it to a server.

[1191] The server analyzes the voice data and motion data to detect if the worker is working in a hurry.

[1192] The server generates an instruction saying, "You're working too fast. Let's work a little slower to be safe."

[1193] The device sends this instruction to the smart glasses and displays it as text.

[1194] The user looks at the instructions and adjusts the work speed.

[1195] 2. Proper use of tools

[1196] The user puts on the earphones and begins working.

[1197] The device collects operational data in real time and transmits it to a server.

[1198] The server analyzes the operation data and detects if the user is using the tool in an incorrect manner.

[1199] The server generates an instruction saying, "You're using this tool incorrectly. Let's check how to use it correctly."

[1200] The device sends this instruction to the earphones and plays it back as audio.

[1201] The user receives instructions and corrects the way the tool is used.

[1202] Prompt Sentence Examples

[1203] "If a worker is rushing, generate feedback to instruct them to work at an appropriate speed."

[1204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1205] Step 1:

[1206] The user wears the smart glasses or earphones and connects to the system via the terminal. The terminal then performs an initialization process and calibrates the sensors. The input is the user's wearing behavior, and the output is a notification that the system has completed initialization.

[1207] Step 2:

[1208] The device begins to capture the user's voice and movement data in real time. It uses sensors to collect voice and movement data and temporarily stores it in a buffer within the device. The input is the user's voice and movement, and the output is the real-time data stored in the buffer.

[1209] Step 3:

[1210] The device sends the collected voice and motion data to the server. The data is pre-processed on the device, compressed, encrypted, and then transferred to the server. The input is the data stored in the buffer, and the output is the secure data transmission to the server.

[1211] Step 4:

[1212] The server analyzes the received voice data using a speech recognition engine to evaluate the user's speaking speed and tone of voice. The motion data is also analyzed using a motion analysis algorithm to evaluate the user's manual movements, standing position, and tool usage. The input is the data received from the device, and the output is the analysis result.

[1213] Step 5:

[1214] The server generates feedback instructions based on the analysis results. For example, if the working speed is too fast, the server generates an instruction such as "You are working too fast. Please work a little slower for safety." If the user is using a tool incorrectly, the server generates an instruction such as "You are using this tool incorrectly. Please check how to use it correctly." The input is the analysis results, and the output is the feedback instructions.

[1215] Step 6:

[1216] The server generates and sends instructions to the terminal. The instructions are generated in text or audio format, and are compressed and encrypted as needed before being transmitted to the terminal. The input is the feedback instruction, and the output is the transmission of the instruction to the terminal.

[1217] Step 7:

[1218] The device decompresses and decodes the received instructions and displays or plays them on the user's smart glasses or earphones. The smart glasses display text and the earphones play audio instructions. The input is the instructions received from the server, and the output is feedback provided to the user.

[1219] Through the above processing steps, the user receives feedback in real time, enabling improved work efficiency and safety.

[1220] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1221] The present invention relates to a system that allows users to wear a display or audio device and receive instructions in real time to make their presentations more effective. Furthermore, by combining it with an emotion engine, the system is able to recognize the user's emotions and provide more appropriate instructions.

[1222] System Configuration

[1223] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[1224] 2. Display or audio devices:

[1225] Display device: A device worn by a user to visually display information, such as smart glasses.

[1226] Audio device: A device worn by a user that provides audio information, such as an earphone.

[1227] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[1228] 4. Server: A computer system that analyzes the voice and motion data received from the device, generates instructions for the user, and sends them to the device. It also has an emotion engine that recognizes the user's emotions.

[1229] 5. Emotion engine: An algorithm that recognizes the user's emotions based on their voice and movement data, and generates appropriate instructions based on those emotions.

[1230] Program processing flow

[1231] System initialization and preparation

[1232] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis and emotion recognition.

[1233] Real-time monitoring during presentations

[1234] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a voice recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns. The emotion engine recognizes the user's emotions based on the voice and movement data.

[1235] Instruction generation and notification

[1236] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, it generates instructions such as "You should move around a bit more while you speak." Based on the recognized emotions, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[1237] Specific examples

[1238] Example 1: Adjusting speaking rate

[1239] 1. The user puts on the smart glasses and begins their presentation.

[1240] 2. The device collects voice data in real time and sends it to the server.

[1241] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1242] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1243] 5. The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1244] 6. The user looks at the instructions and adjusts the speaking rate.

[1245] Example 2: Emotion recognition and regulation instructions

[1246] 1. The user puts on the earphones and begins their presentation.

[1247] 2. The device collects voice and movement data in real time and sends it to the server.

[1248] 3. The server analyzes the voice and movement data and uses an emotion engine to determine that the user is nervous.

[1249] 4. The server generates the instruction, "You seem nervous. Take a deep breath and relax."

[1250] 5. The device sends this instruction to the earphones and plays a voice message saying, "You seem nervous. Take a deep breath and relax."

[1251] 6. The user is instructed to take a deep breath and relax.

[1252] In this way, users can receive appropriate feedback and emotion-based advice in real time, improving the quality of their presentations and making them more appealing to the audience.

[1253] The processing flow will be explained below.

[1254] Step 1:

[1255] The user puts on the smart glasses or earphones and launches the presentation software, which puts the system into preparation mode.

[1256] Step 2:

[1257] The device connects to smart glasses or earphones, initializes sensors, and checks network connectivity. The sensors include a microphone for capturing voice data, an accelerometer for capturing motion data, and a gyroscope.

[1258] Step 3:

[1259] The device will begin capturing the user's voice and movement data in real time, with voice data collected through a microphone and movement data collected through an accelerometer and gyroscope.

[1260] Step 4:

[1261] The device transmits the collected voice and movement data to a server, which transmits the data continuously in real time.

[1262] Step 5:

[1263] The server analyzes the received voice data using a speech recognition engine, converts it into text, and evaluates the speaking speed and tone of voice based on the converted text.

[1264] Step 6:

[1265] The server analyzes the received motion data and evaluates the user's position and movement patterns, using combined accelerometer and gyroscope data.

[1266] Step 7:

[1267] The emotion engine analyzes the received voice and movement data and recognizes the user's emotions based on the tone of voice, speaking patterns, and rhythm of movements. For example, a high-pitched voice and few movements may indicate a state of tension.

[1268] Step 8:

[1269] The server generates appropriate instructions based on the analysis of the voice and movement data and the recognition results of the emotion engine. For example, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly." If the user is nervous, the server generates instructions such as "You seem nervous. Take a deep breath and relax."

[1270] Step 9:

[1271] The server sends the generated instructions to the terminal, which may be in text or voice format.

[1272] Step 10:

[1273] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, while in the case of earphones, the instructions are played back as audio.

[1274] Step 11:

[1275] The user can then use the received instructions to adjust their speaking rate, tone of voice, position, or emotional state during the presentation, such as slowing down, adjusting their tone of voice, moving their position, or taking a deep breath to relax.

[1276] Specific examples

[1277] Example 1: Adjusting speaking rate

[1278] Step 1:

[1279] The user puts on the smart glasses and begins their presentation.

[1280] Step 2:

[1281] The device collects voice data in real time and sends it to the server.

[1282] Step 3:

[1283] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[1284] Step 4:

[1285] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[1286] Step 5:

[1287] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1288] Step 6:

[1289] The user views the instructions and adjusts the speaking rate.

[1290] Example 2: Emotion recognition and regulation instructions

[1291] Step 1:

[1292] The user puts on the earphones and begins the presentation.

[1293] Step 2:

[1294] The terminal collects voice data and movement data in real time and transmits it to a server.

[1295] Step 3:

[1296] The emotion engine analyzes the voice data and movement data and recognizes that the user is nervous based on the high tone of the voice and the lack of movement.

[1297] Step 4:

[1298] The server generates the instruction "You seem nervous. Take a deep breath and relax."

[1299] Step 5:

[1300] The device sends this instruction to the earphones, which play a voice message saying, "You seem nervous. Take a deep breath and relax."

[1301] Step 6:

[1302] The user is instructed to take a deep breath and relax.

[1303] In this way, users receive real-time, appropriate instructions and emotional feedback, improving the quality of their presentations and increasing their appeal to the audience.

[1304] Example 2

[1305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1306] Conventional presentation support systems are limited to analyzing the user's real-time voice and movement data, and do not provide appropriate feedback that takes into account the user's emotions. Therefore, there is a need for systems that can analyze not only the user's speaking rate and tone of voice, but also changes in their emotions during a presentation in real time, and provide appropriate instructions.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1308] In this invention, the server includes means for analyzing voice data and motion data to generate instructions for the user, an emotion engine for recognizing the user's emotion based on the analysis results, and means for notifying the user of the instructions generated based on the recognized emotion. This makes it possible to provide appropriate feedback in real time based on the user's speaking speed, tone of voice, and position as well as changes in emotion.

[1309] A "user" is a person who uses the system to give a presentation.

[1310] A "display device" refers to a device worn by a user that visually displays information.

[1311] "Audio device" refers to a device worn by a user that provides audio information.

[1312] "Terminal" refers to a device that acquires user voice data and motion data in real time and transfers it to a server. Specifically, this applies to portable devices such as smartphones and tablets.

[1313] The term "server" refers to a computer system that analyzes the voice data and motion data received from the terminal, generates instructions for the user, and transmits them to the terminal.

[1314] "Voice data" refers to a voice signal that includes the content and tone of voice of the user.

[1315] "Motion data" refers to data relating to the user's physical movements and position, and is acquired using a gyro sensor, acceleration sensor, etc.

[1316] "Analysis" is the process of assessing the user's condition based on the acquired audio and motion data.

[1317] "Instructions" refer to feedback and advice to the user that is generated based on the analysis results.

[1318] The "emotion engine" is a system equipped with an algorithm that recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions.

[1319] "Display as text" means providing instructions as textual information on a display device.

[1320] "Play as audio" means to notify the user of the instruction by voice through an audio device.

[1321] The present invention relates to a system for effectively supporting users' presentations. The system uses a display or audio device to analyze the user's voice and movement data in real time and provide appropriate feedback. It also uses an emotion engine to recognize the user's emotions and generate instructions based on those emotions.

[1322] System Components

[1323] User

[1324] The user is the person giving the presentation and wears the following devices:

[1325] Display device: A device worn by a user that visually displays information. A specific example is smart glasses.

[1326] Audio device: A device worn by a user that provides audio information. An example is an earphone.

[1327] Terminal

[1328] The terminal is a device that acquires the user's voice data and movement data in real time and transfers them to a server. Specifically, this applies to smartphones and tablets. For example, the smartphone's gyro sensor and acceleration sensor are used to acquire movement data, and the microphone collects voice data.

[1329] server

[1330] The server is a computer system that analyzes the voice and motion data received from the device and generates instructions for the user. It also has an emotion engine that recognizes the user's emotions. The server uses the following software and algorithms:

[1331] Speech recognition engine: For example, using the Google Speech-to-Text API, converts voice data into text.

[1332] Emotion engine: For example, using IBM Watson's emotion analysis API, it recognizes the user's emotions from voice data and movement data.

[1333] Program processing

[1334] The program of this system performs processing in the following order.

[1335] 1. System initialization and preparation:

[1336] Before starting a presentation, users put on smart glasses and earphones and launch a dedicated app on their device.

[1337] The device initializes various sensors (microphone, gyro sensor, acceleration sensor) and prepares to start acquiring data.

[1338] The server prepares to receive data from the device and prepares for analysis and emotion recognition.

[1339] 2. Real-time monitoring during presentation:

[1340] The terminal captures the user's voice and movement data in real time and transmits it to the server.

[1341] The server analyzes the audio data using the Google Speech-to-Text API and evaluates speaking speed and tone of voice.

[1342] It also analyzes motion data to evaluate the user's position and movement patterns.

[1343] The emotion engine recognizes the user's emotions based on voice data and action data.

[1344] 3. Generate instructions:

[1345] Based on the analysis results, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[1346] If the standing position is fixed, the instruction "It would be better if you move around a bit more while talking" is generated.

[1347] Based on the recognized emotion, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax."

[1348] 4. Notice of Instructions:

[1349] The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones.

[1350] Specific examples

[1351] Adjust speaking rate

[1352] The user puts on the smart glasses and begins the presentation.

[1353] The device collects voice data in real time and sends it to the server.

[1354] The server analyzes the audio data and determines that the user is speaking faster.

[1355] The server generates instructions such as "You're speaking too fast. Please speak a little more slowly," and displays them on the smart glasses via the device.

[1356] The user views the instructions and adjusts the speaking rate.

[1357] Emotion recognition and regulation instructions

[1358] The user puts on the earphones and begins the presentation.

[1359] The terminal collects voice data and movement data in real time and transmits it to a server.

[1360] The server analyzes the voice data and movement data and uses an emotion engine to recognize that the user is nervous.

[1361] The server generates instructions such as "You seem nervous. Take a deep breath and relax," and sends a voice notification to the earphones via the device.

[1362] The user is instructed to take a deep breath and relax.

[1363] Prompt Sentence Examples

[1364] "How should I give feedback if someone is speaking too fast during a presentation?"

[1365] "Generate appropriate advice when the user is nervous during a presentation"

[1366] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1367] Step 1: Initialize and prepare the system

[1368] The user puts on the smart glasses and earphones and launches the dedicated app on the device. Specifically, the smart glasses are turned on and the earphones are connected to the device via Bluetooth. This allows the user's device to be integrated into the system.

[1369] The device will initialize the microphone, gyro sensor, accelerometer, etc., and prepare for data collection. Specifically, tap "Start" in the app to calibrate each sensor. This will prepare the environment for collecting voice and movement data.

[1370] (Input) Sensor initialization command for the device worn by the user and the terminal

[1371] (Output) Sensor initialization completion status, connected device information

[1372] Step 2: Real-time monitoring during presentation

[1373] The device captures the user's voice and movement data in real time and sends the data to the server. Specifically, it collects voice with a microphone and captures the user's movements with a gyro sensor and accelerometer. The time-series data is then transferred to the server in real time.

[1374] The server converts the received voice data into text using the Google Speech-to-Text API and analyzes the speaking speed and tone of voice. Specifically, the voice data is divided into frames at regular time intervals and each frame is analyzed.

[1375] The server analyzes the motion data and evaluates the user's position and movement patterns. For example, it extracts information such as "the user is moving to the left" or "the user's position is fixed."

[1376] (Input) Real-time audio and motion data

[1377] (Output) Analyzed speaking rate, tone of voice, and movement patterns

[1378] Step 3: Emotion Recognition

[1379] The server's emotion engine uses voice and motion data to recognize the user's emotions. Specifically, it uses IBM Watson's emotion analysis API to extract emotional data such as "the user is nervous" or "the user is relaxed."

[1380] (Input) Voice data, movement data

[1381] (Output) Recognized emotion data

[1382] Step 4: Generate instructions

[1383] The server generates instructions for the user based on the analysis results. Based on the results of analyzing the voice data, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly," and based on the results of analyzing the movement data, it generates instructions such as "You should move around a bit more while you speak." It also generates instructions based on the emotions recognized by the emotion engine, such as "You seem nervous. Take a deep breath and relax."

[1384] (Input) Speaking rate, tone of voice, movement patterns, emotional data

[1385] (Output) Generated instructions (text or audio)

[1386] Step 5: Notification of instructions

[1387] The device notifies the user of the instructions received from the server. Specifically, it displays text on the smart glasses or plays it as audio through earphones. For example, the instruction "You're speaking too quickly. Please speak a little more slowly" is displayed as text on the smart glasses.

[1388] (Input) Instruction data from the server

[1389] (Output) Text display on smart glasses, audio playback on earphones

[1390] In this way, the system analyzes the user's voice and movement data in real time and provides appropriate feedback to improve the quality of the presentation.

[1391] (Application example 2)

[1392] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1393] Conventional presentation support systems and customer service support systems do not provide appropriate feedback that takes into account the user's situation and emotions, making it difficult to maximize the user's response ability or the quality of their presentation. In particular, there was a need for a system that could provide appropriate real-time instructions to store clerks when serving customers, based on the customer's emotions and attitude. This required a system that could improve customer satisfaction and the response ability of store clerks.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1395] In this invention, the server includes means for analyzing voice data and motion data and generating instructions for the user, emotion recognition means for generating appropriate instructions based on the recognized emotions, and means for generating instructions using a generative AI model. This allows the user to receive appropriate feedback in real time through a display device or audio device worn by the user, thereby improving the user's ability to respond in presentations and customer interactions.

[1396] A "user" is someone who uses the system to give presentations or interact with customers.

[1397] A "display device" is a device that provides visual information, such as smart glasses.

[1398] An "audio device" is a device that provides audio information, such as an earphone.

[1399] A "terminal" is a device that acquires user voice data and motion data in real time and transfers them to a server.

[1400] "Voice data" refers to the voice uttered by the user recorded as digital data.

[1401] "Motion data" refers to the user's movements, position, and other data recorded using sensors.

[1402] A "server" is a computer system that analyzes the voice data and motion data sent from the terminal, generates instructions, and sends them to the terminal.

[1403] The "emotion recognition means" is an algorithm that recognizes the user's emotions based on voice data and action data, and generates appropriate instructions based on those emotions.

[1404] A "generative AI model" is an artificial intelligence model that generates instructions in natural language based on data provided to it.

[1405] A "prompt sentence" is an input sentence for a generative AI model, and is the text that serves as the basis for generating instructions.

[1406] "Real time" refers to a time unit in which analysis and feedback are performed simultaneously with the user's actions and speech.

[1407] The present invention relates to a system that allows users to receive real-time instructions to effectively give presentations or serve customers. This system comprises a display or audio device worn by the user, a terminal that acquires and transfers voice and motion data in real time, a server that analyzes this data and generates instructions, and an emotion recognition means and a generative AI model to provide more appropriate instructions.

[1408] Overall system configuration

[1409] 1. User:

[1410] This is a person who uses the system to give presentations or interact with customers. The user wears a display or audio device such as smart glasses or earphones.

[1411] 2. Display or audio devices:

[1412] Display device: A device that provides visual information, such as smart glasses, allowing users to receive real-time instructions as text.

[1413] Audio device: A device that provides audio information, such as an earphone, allowing you to receive real-time instructions by voice.

[1414] 3. Terminal:

[1415] A device that captures user voice and motion data in real time and transfers it to a server. Specifically, this applies to smartphones and tablets.

[1416] 4. Server:

[1417] It is a computer system that analyzes voice and motion data sent from the device, generates instructions for the user, and sends them to the device. It can utilize cloud environments such as Amazon Web Services (AWS) and Google Cloud.

[1418] 5. Emotion recognition means:

[1419] This algorithm recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions. It uses deep learning libraries such as TensorFlow.

[1420] 6. Generative AI Models:

[1421] It is an artificial intelligence model that generates natural language instructions based on the data provided, such as OpenAI's GPT-4.

[1422] Program processing

[1423] The server first receives the voice and movement data sent from the device. The voice data is converted into text using Google Speech API or Amazon Transcribe. The voice and movement data are then analyzed together by an emotion recognition system to determine the user's emotion.

[1424] The server then inputs instructions based on the emotion recognition results into the generative AI model as prompt sentences, such as, "The customer's emotion is dissatisfaction. They said, 'I'd like to know more about this product.' Please instruct them on the appropriate response."

[1425] The generative AI model generates appropriate instructions based on the prompt and sends them back to the server. The server then sends the instructions to the device, which then sends feedback to the display and audio devices. For example, an instruction such as "Raise your voice a little to make the customer feel at ease" is displayed on the user's smart glasses. Audio instructions are also delivered through the earphones.

[1426] Specific examples

[1427] Consider a scenario in which a user wears smart glasses and acts as a store clerk while serving customers. The device collects voice and movement data in real time and sends it to a server. The server analyzes this data and uses emotion recognition to identify a disinterested tone of voice when a customer is asking a question. Based on this, the generative AI model creates instructions such as, "The customer seems anxious. Please respond in a more friendly tone." These instructions are displayed as text on the smart glasses, allowing the user to learn how to respond in real time.

[1428] In this way, this system allows users to receive appropriate feedback in real time during presentations and customer interactions, thereby improving their response skills.

[1429] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1430] Step 1:

[1431] The user puts on smart glasses or earphones and starts up the device (such as a smartphone). The device initializes the sensors and starts collecting voice and movement data. The input is the user's movements and voice, and the output is the collected data. Specifically, the device's application starts up, and the sensors detect the user's movements and voice.

[1432] Step 2:

[1433] The device transmits collected voice and motion data to the server in real time. The input is the data collected in real time, and the output is the transfer of this data to the server. The device continues to send data to the server via a network connection.

[1434] Step 3:

[1435] The server receives the voice data and converts it to text using a speech recognition engine (such as Google Speech API or Amazon Transcribe). The input is voice data and the output is text data. Specifically, the server makes an API call to convert the voice data to text.

[1436] Step 4:

[1437] The server analyzes the motion data and evaluates the user's position and movements. The input is the motion data, and the output is the analysis result. Specific actions are identified by processing the motion data using libraries such as OpenCV.

[1438] Step 5:

[1439] The server uses emotion recognition means to recognize the user's emotions based on the analyzed voice data and movement data. The input is the analyzed voice data and movement data, and the output is emotion data. Specifically, this is done by an emotion recognition algorithm using TensorFlow.

[1440] Step 6:

[1441] Based on the emotion recognition results, the server inputs a prompt sentence into the generative AI model to generate instructions. The input is emotion data and a prompt sentence, and the output is the generated instruction. Specifically, the prompt sentence is processed using OpenAI's GPT-4 or similar software to generate instructions in natural language.

[1442] Example prompt sentence:

[1443] "The customer is unhappy. They stated, 'I'd like to know more about this product.' Please direct them to the appropriate response."

[1444] Step 7:

[1445] The server sends the generated instructions to the terminal. The input is the generated instructions, and the output is the transfer to the terminal. Specifically, the server uses WebSocket or REST API to send the instruction data to the terminal.

[1446] Step 8:

[1447] The device then feeds back the received instructions to the user's display device (smart glasses) and audio device (earphones). The input is the instruction sent from the server, and the output is a notification to the user. Specifically, the instructions are displayed in text on the smart glasses, and the instructions are played back in audio on the earphones.

[1448] In this way, users receive relevant feedback in real time, enabling them to improve their presentations and customer interactions.

[1449] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1450] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1451] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1452] [Fourth embodiment]

[1453] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1454] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1455] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1456] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1457] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1458] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1459] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1460] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1461] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1462] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1463] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1464] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1465] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1466] The present invention relates to a system that allows a user to wear a display or audio device and receive instructions in real time in order to make a presentation more effective.

[1467] System Configuration

[1468] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[1469] 2. Display or audio devices:

[1470] Display device: A device worn by a user to visually display information, such as smart glasses.

[1471] Audio device: A device worn by a user that provides audio information, such as an earphone.

[1472] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[1473] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[1474] Program processing flow

[1475] System initialization and preparation

[1476] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[1477] Real-time monitoring during presentations

[1478] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns.

[1479] Instruction generation and notification

[1480] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, the server generates instructions such as "It would be better if you move around a bit more while you speak." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[1481] Specific examples

[1482] Example 1: Adjusting speaking rate

[1483] 1. The user puts on the smart glasses and begins their presentation.

[1484] 2. The device collects voice data in real time and sends it to the server.

[1485] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1486] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1487] 5. The device sends this instruction to the smart glasses and displays it as text.

[1488] 6. The user looks at the instructions and adjusts the speaking rate.

[1489] Example 2: Adjusting standing position

[1490] 1. The user puts on the earphones and begins their presentation.

[1491] 2. The device collects operational data in real time and sends it to the server.

[1492] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[1493] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1494] 5. The device sends this instruction to the earphones and plays it as audio.

[1495] 6. The user receives instructions and adjusts their position while speaking.

[1496] In this way, users can receive appropriate feedback in real time during their presentations, thereby improving the quality of their presentations and increasing their appeal to the audience.

[1497] The processing flow will be explained below.

[1498] Step 1:

[1499] The user puts on the smart glasses or earphones, launches the presentation software, and once the system is up and running, the user is ready to begin their presentation.

[1500] Step 2:

[1501] The device connects to smart glasses or earphones, initializes the sensors, checks network connectivity, and is ready to acquire data.

[1502] Step 3:

[1503] The device begins capturing the user's voice and movement data in real time. Voice data is collected through a microphone, and movement data is collected using an accelerometer and gyroscope.

[1504] Step 4:

[1505] The device transmits the acquired voice and movement data to the server. The data is processed in real time and transmitted without delay.

[1506] Step 5:

[1507] The server analyzes the received voice data and converts it into text using a speech recognition engine. This text data is then used to evaluate speaking speed and tone of voice.

[1508] Step 6:

[1509] The server analyzes the received motion data and evaluates the user's movement patterns, using combined data from the accelerometer and gyroscope.

[1510] Step 7:

[1511] The server generates appropriate instructions based on the analysis of the voice and movement data. For example, if the user is speaking too quickly, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[1512] Step 8:

[1513] The server sends the generated instructions to the terminal, which may be in text or voice format.

[1514] Step 9:

[1515] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, and in the case of earphones, the instructions are played back as audio.

[1516] Step 10:

[1517] Based on the received instructions, users can adjust their speaking speed, tone of voice, and position during their presentation, improving the quality of their presentation and making it more appealing to the audience.

[1518] Specific examples

[1519] Example 1: Speaking faster

[1520] Step 1:

[1521] The user puts on the smart glasses and begins their presentation.

[1522] Step 2:

[1523] The device collects voice data in real time and sends it to the server.

[1524] Step 3:

[1525] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[1526] Step 4:

[1527] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[1528] Step 5:

[1529] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1530] Step 6:

[1531] The user views the instructions and adjusts the speaking rate.

[1532] Example 2: When the standing position is fixed

[1533] Step 1:

[1534] The user puts on the earphones and begins the presentation.

[1535] Step 2:

[1536] The device collects operational data in real time and transmits it to a server.

[1537] Step 3:

[1538] The server analyzes the movement data and detects when the user has not moved from a fixed location.

[1539] Step 4:

[1540] The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1541] Step 5:

[1542] The device sends this instruction to the earphones, which then play back a voice message saying, "It would be better if you moved around a bit more while talking."

[1543] Step 6:

[1544] The user receives instructions and adjusts their position while speaking.

[1545] Example 1

[1546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1547] There is a need for a system that allows users to receive effective, real-time instructions while appropriately adjusting their speaking speed, tone of voice, and standing position during a presentation. However, current systems make it difficult for users to check and appropriately adjust their speaking speed and standing position. Furthermore, existing systems do not provide sufficient real-time feedback, making it difficult to improve the quality of presentations.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1549] In this invention, the server includes a data collection means for acquiring and transferring voice information and movement information in real time, an analysis means for analyzing the voice information and movement information, and an instruction generation means for generating instructions for the user and transmitting them to the data collection means, thereby enabling the user to receive appropriate feedback in real time during a presentation.

[1550] A "visual information display means" is a device that displays information visually when worn by a user, and specifically includes smart glasses.

[1551] The "audio output means" refers to a device that outputs audio information when worn by a user, and specifically includes earphones and a headset.

[1552] A "data collection means" is a device that acquires a user's voice information and movement information in real time and transfers it to a server, and specifically includes a smartphone or tablet.

[1553] The "analysis means" is a device or program that analyzes voice information and motion information within the server and generates feedback based on the user's actions.

[1554] The "instruction generating means" is a device or program that generates appropriate instructions for the user based on the analysis results obtained by the analysis means and transmits them to the data collecting means.

[1555] "Voice information" refers to voice data uttered by a user, including speaking speed, tone of voice, and the like.

[1556] "Motion information" is data relating to the user's movements and standing position, and is acquired using an acceleration sensor, a gyro sensor, or the like.

[1557] The present invention relates to a system in which a user can wear a visual information display means or an audio output means and receive instructions in real time in order to make a presentation more effective. The specific configuration for implementing this system is as follows.

[1558] System Configuration

[1559] 1. User:

[1560] The user is the person giving the presentation and wears a visual information display means such as smart glasses or an audio output means such as earphones, which allows the user to receive appropriate visual or audio feedback during the presentation.

[1561] 2. Visual or audio display means:

[1562] Examples of visual information display means include smart glasses, which are devices worn by users that display text information in real time. Examples of audio output means include earphones and headsets, which are worn by users and provide audio information in real time.

[1563] 3. Data collection methods:

[1564] Smartphones and tablets are used as data collection tools. These capture the user's voice and movement information in real time and transfer it to a server. Voice information is collected using the smartphone's microphone, and movement information is collected using the smartphone's built-in accelerometer and gyro sensor.

[1565] 4. Server:

[1566] The server analyzes the voice information and movement information sent from the data collection means and generates instructions for the user. It uses a voice recognition engine (e.g., Google Cloud Speech-to-Text) to analyze the voice information, and data from an acceleration sensor and gyro sensor to analyze the movement information. Based on the analysis results, the instruction generation means generates appropriate feedback and sends it to the data collection means.

[1567] Specific examples

[1568] Example 1: Adjusting speaking rate

[1569] 1. The user puts on the smart glasses and begins their presentation.

[1570] 2. The data collection means collects voice data in real time and transmits it to the server.

[1571] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1572] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1573] 5. The data collection device sends this instruction to the smart glasses and displays it as text.

[1574] 6. The user looks at the instructions and adjusts the speaking rate.

[1575] Example 2: Adjusting standing position

[1576] 1. The user puts on the earphones and begins the presentation.

[1577] 2. The data collection means collects the operational data in real time and transmits it to the server.

[1578] 3. The server analyzes the movement data and detects that the user has not moved from a fixed location.

[1579] 4. The server generates an instruction saying, "It would be better if you moved around a bit more while speaking."

[1580] 5. The data collection means sends this instruction to the earphone and plays it back audibly.

[1581] 6. The user receives instructions and adjusts their position while speaking.

[1582] Prompt Sentence Examples

[1583] Sample prompt 1: Adjusting speaking speed

[1584] "Generate instructions to slow down speaking speed for users who are speaking too quickly."

[1585] Example output: "You're speaking too quickly. Please slow down."

[1586] Prompt example 2: Adjust your position

[1587] "If the user is standing still, generate prompts to encourage them to move and speak."

[1588] Example output: "You should move around a bit more while you speak."

[1589] Based on the system configuration and specific example described above, users can receive appropriate feedback in real time during a presentation, thereby improving the quality of the presentation.

[1590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1591] Step 1: Initialization and preparation

[1592] Before starting a presentation, a user puts on smart glasses or earphones, launches the application on their device, and connects to the system. The device connects to the smart glasses, initializes the microphone and accelerometer, and prepares to acquire voice and motion data. Meanwhile, the server generates a session ID to manage the user session and establishes a connection with the device.

[1593] Input: User operation (wearing smart glasses or earphones, launching applications on the device)

[1594] Output: Connection between smart glasses and device, generation of session ID

[1595] Specific working example:

[1596] The user puts on the smart glasses and launches the application on the device. The device automatically detects the smart glasses and establishes a connection. The server starts a session and sends feedback to the device confirming the connection.

[1597] Step 2: Data collection

[1598] The device uses the smartphone's microphone to collect voice data in real time when the user starts speaking, and also uses the device's accelerometer and gyro sensor to collect motion data as the user starts moving.

[1599] Input: User voice and movements

[1600] Output: Real-time collected audio and motion data

[1601] Specific working example:

[1602] When a user starts speaking during a presentation, the device's microphone begins capturing audio, while the accelerometer and gyroscope sensors pick up data every time the user walks or gestures.

[1603] Step 3: Data Transfer

[1604] The device periodically collects voice and motion data, compresses and encrypts it into packets, and sends them to a server. The data is transferred in real time via Wi-Fi or a mobile network.

[1605] Input: Real-time collected audio and motion data

[1606] Output: Encrypted data packet

[1607] Specific working example:

[1608] The device compresses, encrypts, and packetizes the voice and motion data, then transmits the data over the Internet to a server.

[1609] Step 4: Data analysis

[1610] The server unpacks the received data packets and analyzes the audio data using a speech recognition engine (e.g., Google Cloud Speech-to-Text) to assess speaking rate and tone, as well as motion data to assess the user's position and movement patterns.

[1611] Input: Encrypted data packet

[1612] Output: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[1613] Specific working example:

[1614] The server decompresses the received data and passes it to a speech recognition engine to convert the voice data into text and rating data. The accelerometer data is analyzed to visualize the user's movements.

[1615] Step 5: Instruction Generation

[1616] Based on the analysis results, the server generates instructions such as "You're speaking too fast. Please speak a little slower" if the user is speaking too fast, or "You should move around a bit more while you speak" if the user is not moving from a fixed location.

[1617] Input: Analysis results (speaking speed, tone of voice, user position and movement patterns)

[1618] Output: Specific instruction message

[1619] Specific working example:

[1620] The server evaluates the analysis results and generates appropriate feedback. For example, if the evaluation is "You're speaking too fast," it generates a text message saying, "You're speaking too fast. Please speak a little slower."

[1621] Step 6: Instructions

[1622] The device receives instructions from the server and displays them as text on the smart glasses or plays them as audio through earphones, allowing users to receive instructions in real time and improve the quality of their presentations.

[1623] Input: Specific instruction message

[1624] Output: Text display on smart glasses or audio playback on earphones

[1625] Specific working example:

[1626] The server sends instruction messages to the device, which displays them as text on the smart glasses, or plays the same messages aloud if earphones are used.

[1627] Step 7: User interaction

[1628] The user receives real-time instructions from the device and can take actions to improve the quality of their presentation, such as adjusting their speaking speed or changing their position, allowing them to continue their presentation based on appropriate feedback.

[1629] Input: Text display on smart glasses or voice instructions on earphones

[1630] Output: User's adjusted behavior (adjusting speaking rate, changing standing position, etc.)

[1631] Specific working example:

[1632] The user may see instructions on the smart glasses and slow down their speaking speed, or hear instructions over the earphones such as "It would be good to move around a bit more while speaking" and change their position.

[1633] (Application example 1)

[1634] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1635] In modern factory work, it is important for workers to carry out their work efficiently and safely, but the means for receiving real-time feedback and instructions are limited. To solve this problem, a system is needed that provides real-time feedback in a form that workers can intuitively understand.

[1636] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1637] In this invention, the server includes a means for analyzing manual movements and voice commands of workers in a factory and generating instructions related to work efficiency and safety, a means for acquiring and transferring user voice data and operation data in real time, and a means for wearing a display or audio device, which allows workers to receive appropriate feedback in real time and improve work efficiency and safety.

[1638] A "user" is a person who works in a factory and wears a display or audio device.

[1639] A "display device" is a device worn by a user that visually displays information, specifically referring to smart glasses.

[1640] An "audio device" is a device that provides audio information when worn by a user, and specifically refers to earphones and the like.

[1641] A "terminal" is a device that acquires a user's voice data and motion data in real time and transfers them to a server, and specifically includes smartphones, tablets, etc.

[1642] A "server" is a computer system that analyzes voice data and motion data received from a terminal, generates instructions for the user, and sends them to the terminal.

[1643] "Motion data" refers to data that indicates the user's manual movements and standing position.

[1644] "Voice data" refers to data that indicates the user's speaking rate and tone of voice.

[1645] "Real-time" refers to the temporal process in which data is acquired, analyzed, and instructions are generated without delay.

[1646] "Manual movements and voice commands by workers in the factory" means physical movements performed by workers in the factory and voice instructions given by workers.

[1647] "Work efficiency and safety instructions" means advice or instructions to optimize the speed and accuracy of work and maintain a safe work environment.

[1648] MODE FOR CARRYING OUT THE INVENTION

[1649] The present invention relates to a real-time feedback system for improving work efficiency and safety in a factory using a display device and an audio device worn by a worker. This system collects voice data and motion data of the user (worker) in real time, transfers the data to a server, analyzes the data, generates feedback, and transmits it to the display device and audio device.

[1650] System Configuration

[1651] 1. User: A person working in a factory who wears a device such as smart glasses or earphones.

[1652] 2. Display or audio devices:

[1653] Display device: A device worn by a user to visually display information, such as smart glasses.

[1654] Audio device: A device worn by a user that provides audio information, such as an earphone.

[1655] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[1656] 4. Server: A computer system that analyzes the voice data and movement data received from the terminal, generates instructions for the user, and sends them to the terminal.

[1657] Program processing flow

[1658] 1. System initialization and preparation

[1659] Before starting work in the factory, users put on smart glasses or earphones and connect to the system using a terminal. The terminal initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis.

[1660] 2. Real-time monitoring during work

[1661] The device continuously captures the user's voice and movements in real time and sends the data to a server. The server then analyzes the received voice data using a speech recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's manual movements, position, and tool usage.

[1662] 3. Instruction Generation and Notification

[1663] Based on the analysis results, the server generates instructions such as "You're working too fast. Please work a little slower for safety" if the user is working too quickly. If the user is using a tool incorrectly, the server generates instructions such as "You're using this tool incorrectly. Please check how to use it correctly." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time, improving work efficiency and safety.

[1664] Specific examples

[1665] 1. Adjusting the work speed

[1666] The user puts on the smart glasses and begins working.

[1667] The terminal collects voice data and movement data in real time and transmits it to a server.

[1668] The server analyzes the voice data and motion data to detect if the worker is working in a hurry.

[1669] The server generates an instruction saying, "You're working too fast. Let's work a little slower to be safe."

[1670] The device sends this instruction to the smart glasses and displays it as text.

[1671] The user looks at the instructions and adjusts the work speed.

[1672] 2. Proper use of tools

[1673] The user puts on the earphones and begins working.

[1674] The device collects operational data in real time and transmits it to a server.

[1675] The server analyzes the operation data and detects if the user is using the tool in an incorrect manner.

[1676] The server generates an instruction saying, "You're using this tool incorrectly. Let's check how to use it correctly."

[1677] The device sends this instruction to the earphones and plays it back as audio.

[1678] The user receives instructions and corrects the way the tool is used.

[1679] Prompt Sentence Examples

[1680] "If a worker is rushing, generate feedback to instruct them to work at an appropriate speed."

[1681] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1682] Step 1:

[1683] The user wears the smart glasses or earphones and connects to the system via the terminal. The terminal then performs an initialization process and calibrates the sensors. The input is the user's wearing behavior, and the output is a notification that the system has completed initialization.

[1684] Step 2:

[1685] The device begins to capture the user's voice and movement data in real time. It uses sensors to collect voice and movement data and temporarily stores it in a buffer within the device. The input is the user's voice and movement, and the output is the real-time data stored in the buffer.

[1686] Step 3:

[1687] The device sends the collected voice and motion data to the server. The data is pre-processed on the device, compressed, encrypted, and then transferred to the server. The input is the data stored in the buffer, and the output is the secure data transmission to the server.

[1688] Step 4:

[1689] The server analyzes the received voice data using a speech recognition engine to evaluate the user's speaking speed and tone of voice. The motion data is also analyzed using a motion analysis algorithm to evaluate the user's manual movements, standing position, and tool usage. The input is the data received from the device, and the output is the analysis result.

[1690] Step 5:

[1691] The server generates feedback instructions based on the analysis results. For example, if the working speed is too fast, the server generates an instruction such as "You are working too fast. Please work a little slower for safety." If the user is using a tool incorrectly, the server generates an instruction such as "You are using this tool incorrectly. Please check how to use it correctly." The input is the analysis results, and the output is the feedback instructions.

[1692] Step 6:

[1693] The server generates and sends instructions to the terminal. The instructions are generated in text or audio format, and are compressed and encrypted as needed before being transmitted to the terminal. The input is the feedback instruction, and the output is the transmission of the instruction to the terminal.

[1694] Step 7:

[1695] The device decompresses and decodes the received instructions and displays or plays them on the user's smart glasses or earphones. The smart glasses display text and the earphones play audio instructions. The input is the instructions received from the server, and the output is feedback provided to the user.

[1696] Through the above processing steps, the user receives feedback in real time, enabling improved work efficiency and safety.

[1697] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1698] The present invention relates to a system that allows users to wear a display or audio device and receive instructions in real time to make their presentations more effective. Furthermore, by combining it with an emotion engine, the system is able to recognize the user's emotions and provide more appropriate instructions.

[1699] System Configuration

[1700] 1. User: The person giving the presentation who wears a device such as smart glasses or earphones.

[1701] 2. Display or audio devices:

[1702] Display device: A device worn by a user to visually display information, such as smart glasses.

[1703] Audio device: A device worn by a user that provides audio information, such as an earphone.

[1704] 3. Terminal: A device that collects user voice and movement data in real time and transmits it to a server. For example, a smartphone or tablet.

[1705] 4. Server: A computer system that analyzes the voice and motion data received from the device, generates instructions for the user, and sends them to the device. It also has an emotion engine that recognizes the user's emotions.

[1706] 5. Emotion engine: An algorithm that recognizes the user's emotions based on their voice and movement data, and generates appropriate instructions based on those emotions.

[1707] Program processing flow

[1708] System initialization and preparation

[1709] Before starting a presentation, a user puts on smart glasses or earphones and connects to the system using a device. The device initializes the sensors and prepares to start acquiring voice and motion data. Meanwhile, the server establishes a connection to receive data and prepares for data analysis and emotion recognition.

[1710] Real-time monitoring during presentations

[1711] The device continuously captures the user's voice and movements in real time and sends the data to the server. The server then analyzes the received voice data using a voice recognition engine to evaluate speaking speed and tone of voice. It also analyzes the movement data to evaluate the user's position and movement patterns. The emotion engine recognizes the user's emotions based on the voice and movement data.

[1712] Instruction generation and notification

[1713] Based on the analysis results, the server generates instructions such as "You're speaking too quickly. Speak a little more slowly" if the user is speaking too quickly. If the user is standing in a fixed position, it generates instructions such as "You should move around a bit more while you speak." Based on the recognized emotions, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax." The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones. This allows the user to receive appropriate feedback in real time and improve the quality of their presentation.

[1714] Specific examples

[1715] Example 1: Adjusting speaking rate

[1716] 1. The user puts on the smart glasses and begins their presentation.

[1717] 2. The device collects voice data in real time and sends it to the server.

[1718] 3. The server analyzes the audio data and determines that the user is speaking faster.

[1719] 4. The server generates an instruction saying, "You're speaking too fast. Please speak a little slower."

[1720] 5. The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1721] 6. The user looks at the instructions and adjusts the speaking rate.

[1722] Example 2: Emotion recognition and regulation instructions

[1723] 1. The user puts on the earphones and begins their presentation.

[1724] 2. The device collects voice and movement data in real time and sends it to the server.

[1725] 3. The server analyzes the voice and movement data and uses an emotion engine to determine that the user is nervous.

[1726] 4. The server generates the instruction, "You seem nervous. Take a deep breath and relax."

[1727] 5. The device sends this instruction to the earphones and plays a voice message saying, "You seem nervous. Take a deep breath and relax."

[1728] 6. The user is instructed to take a deep breath and relax.

[1729] In this way, users can receive appropriate feedback and emotion-based advice in real time, improving the quality of their presentations and making them more appealing to the audience.

[1730] The processing flow will be explained below.

[1731] Step 1:

[1732] The user puts on the smart glasses or earphones and launches the presentation software, which puts the system into preparation mode.

[1733] Step 2:

[1734] The device connects to smart glasses or earphones, initializes sensors, and checks network connectivity. The sensors include a microphone for capturing voice data, an accelerometer for capturing motion data, and a gyroscope.

[1735] Step 3:

[1736] The device will begin capturing the user's voice and movement data in real time, with voice data collected through a microphone and movement data collected through an accelerometer and gyroscope.

[1737] Step 4:

[1738] The device transmits the collected voice and movement data to a server, which transmits the data continuously in real time.

[1739] Step 5:

[1740] The server analyzes the received voice data using a speech recognition engine, converts it into text, and evaluates the speaking speed and tone of voice based on the converted text.

[1741] Step 6:

[1742] The server analyzes the received motion data and evaluates the user's position and movement patterns, using combined accelerometer and gyroscope data.

[1743] Step 7:

[1744] The emotion engine analyzes the received voice and movement data and recognizes the user's emotions based on the tone of voice, speaking patterns, and rhythm of movements. For example, a high-pitched voice and few movements may indicate a state of tension.

[1745] Step 8:

[1746] The server generates appropriate instructions based on the analysis of the voice and movement data and the recognition results of the emotion engine. For example, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly." If the user is nervous, the server generates instructions such as "You seem nervous. Take a deep breath and relax."

[1747] Step 9:

[1748] The server sends the generated instructions to the terminal, which may be in text or voice format.

[1749] Step 10:

[1750] The device then notifies the user of the received instructions via smart glasses or earphones. In the case of smart glasses, the instructions are displayed as text on the screen, while in the case of earphones, the instructions are played back as audio.

[1751] Step 11:

[1752] The user can then use the received instructions to adjust their speaking rate, tone of voice, position, or emotional state during the presentation, such as slowing down, adjusting their tone of voice, moving their position, or taking a deep breath to relax.

[1753] Specific examples

[1754] Example 1: Adjusting speaking rate

[1755] Step 1:

[1756] The user puts on the smart glasses and begins their presentation.

[1757] Step 2:

[1758] The device collects voice data in real time and sends it to the server.

[1759] Step 3:

[1760] The server analyzes the audio data and determines that the speaking rate is faster than normal.

[1761] Step 4:

[1762] The server generates an instruction saying, "You're speaking too fast. Slow down a bit."

[1763] Step 5:

[1764] The device sends this instruction to the smart glasses, which then displays a text message saying, "You're speaking too fast. Please speak a little slower."

[1765] Step 6:

[1766] The user views the instructions and adjusts the speaking rate.

[1767] Example 2: Emotion recognition and regulation instructions

[1768] Step 1:

[1769] The user puts on the earphones and begins the presentation.

[1770] Step 2:

[1771] The terminal collects voice data and movement data in real time and transmits it to a server.

[1772] Step 3:

[1773] The emotion engine analyzes the voice data and movement data and recognizes that the user is nervous based on the high tone of the voice and the lack of movement.

[1774] Step 4:

[1775] The server generates the instruction "You seem nervous. Take a deep breath and relax."

[1776] Step 5:

[1777] The device sends this instruction to the earphones, which play a voice message saying, "You seem nervous. Take a deep breath and relax."

[1778] Step 6:

[1779] The user is instructed to take a deep breath and relax.

[1780] In this way, users receive real-time, appropriate instructions and emotional feedback, improving the quality of their presentations and increasing their appeal to the audience.

[1781] Example 2

[1782] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1783] Conventional presentation support systems are limited to analyzing the user's real-time voice and movement data, and do not provide appropriate feedback that takes into account the user's emotions. Therefore, there is a need for systems that can analyze not only the user's speaking rate and tone of voice, but also changes in their emotions during a presentation in real time, and provide appropriate instructions.

[1784] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1785] In this invention, the server includes means for analyzing voice data and motion data to generate instructions for the user, an emotion engine for recognizing the user's emotion based on the analysis results, and means for notifying the user of the instructions generated based on the recognized emotion. This makes it possible to provide appropriate feedback in real time based on the user's speaking speed, tone of voice, and position as well as changes in emotion.

[1786] A "user" is a person who uses the system to give a presentation.

[1787] A "display device" refers to a device worn by a user that visually displays information.

[1788] "Audio device" refers to a device worn by a user that provides audio information.

[1789] "Terminal" refers to a device that acquires user voice data and motion data in real time and transfers it to a server. Specifically, this applies to portable devices such as smartphones and tablets.

[1790] The term "server" refers to a computer system that analyzes the voice data and motion data received from the terminal, generates instructions for the user, and transmits them to the terminal.

[1791] "Voice data" refers to a voice signal that includes the content and tone of voice of the user.

[1792] "Motion data" refers to data relating to the user's physical movements and position, and is acquired using a gyro sensor, acceleration sensor, etc.

[1793] "Analysis" is the process of assessing the user's condition based on the acquired audio and motion data.

[1794] "Instructions" refer to feedback and advice to the user that is generated based on the analysis results.

[1795] The "emotion engine" is a system equipped with an algorithm that recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions.

[1796] "Display as text" means providing instructions as textual information on a display device.

[1797] "Play as audio" means to notify the user of the instruction by voice through an audio device.

[1798] The present invention relates to a system for effectively supporting users' presentations. The system uses a display or audio device to analyze the user's voice and movement data in real time and provide appropriate feedback. It also uses an emotion engine to recognize the user's emotions and generate instructions based on those emotions.

[1799] System Components

[1800] User

[1801] The user is the person giving the presentation and wears the following devices:

[1802] Display device: A device worn by a user that visually displays information. A specific example is smart glasses.

[1803] Audio device: A device worn by a user that provides audio information. An example is an earphone.

[1804] Terminal

[1805] The terminal is a device that acquires the user's voice data and movement data in real time and transfers them to a server. Specifically, this applies to smartphones and tablets. For example, the smartphone's gyro sensor and acceleration sensor are used to acquire movement data, and the microphone collects voice data.

[1806] server

[1807] The server is a computer system that analyzes the voice and motion data received from the device and generates instructions for the user. It also has an emotion engine that recognizes the user's emotions. The server uses the following software and algorithms:

[1808] Speech recognition engine: For example, using the Google Speech-to-Text API, converts voice data into text.

[1809] Emotion engine: For example, using IBM Watson's emotion analysis API, it recognizes the user's emotions from voice data and movement data.

[1810] Program processing

[1811] The program of this system performs processing in the following order.

[1812] 1. System initialization and preparation:

[1813] Before starting a presentation, users put on smart glasses and earphones and launch a dedicated app on their device.

[1814] The device initializes various sensors (microphone, gyro sensor, acceleration sensor) and prepares to start acquiring data.

[1815] The server prepares to receive data from the device and prepares for analysis and emotion recognition.

[1816] 2. Real-time monitoring during presentation:

[1817] The terminal captures the user's voice and movement data in real time and transmits it to the server.

[1818] The server analyzes the audio data using the Google Speech-to-Text API and evaluates speaking speed and tone of voice.

[1819] It also analyzes motion data to evaluate the user's position and movement patterns.

[1820] The emotion engine recognizes the user's emotions based on voice data and action data.

[1821] 3. Generate instructions:

[1822] Based on the analysis results, if the user is speaking too quickly, the server generates instructions such as "You're speaking too quickly. Please speak a little more slowly."

[1823] If the standing position is fixed, the instruction "It would be better if you move around a bit more while talking" is generated.

[1824] Based on the recognized emotion, the emotion engine generates instructions such as "You seem nervous. Take a deep breath and relax."

[1825] 4. Notice of Instructions:

[1826] The device receives the generated instructions and displays them as text on the user's smart glasses or plays them as audio through earphones.

[1827] Specific examples

[1828] Adjust speaking rate

[1829] The user puts on the smart glasses and begins the presentation.

[1830] The device collects voice data in real time and sends it to the server.

[1831] The server analyzes the audio data and determines that the user is speaking faster.

[1832] The server generates instructions such as "You're speaking too fast. Please speak a little more slowly," and displays them on the smart glasses via the device.

[1833] The user views the instructions and adjusts the speaking rate.

[1834] Emotion recognition and regulation instructions

[1835] The user puts on the earphones and begins the presentation.

[1836] The terminal collects voice data and movement data in real time and transmits it to a server.

[1837] The server analyzes the voice data and movement data and uses an emotion engine to recognize that the user is nervous.

[1838] The server generates instructions such as "You seem nervous. Take a deep breath and relax," and sends a voice notification to the earphones via the device.

[1839] The user is instructed to take a deep breath and relax.

[1840] Prompt Sentence Examples

[1841] "How should I give feedback if someone is speaking too fast during a presentation?"

[1842] "Generate appropriate advice when the user is nervous during a presentation"

[1843] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1844] Step 1: Initialize and prepare the system

[1845] The user puts on the smart glasses and earphones and launches the dedicated app on the device. Specifically, the smart glasses are turned on and the earphones are connected to the device via Bluetooth. This allows the user's device to be integrated into the system.

[1846] The device will initialize the microphone, gyro sensor, accelerometer, etc., and prepare for data collection. Specifically, tap "Start" in the app to calibrate each sensor. This will prepare the environment for collecting voice and movement data.

[1847] (Input) Sensor initialization command for the device worn by the user and the terminal

[1848] (Output) Sensor initialization completion status, connected device information

[1849] Step 2: Real-time monitoring during presentation

[1850] The device captures the user's voice and movement data in real time and sends the data to the server. Specifically, it collects voice with a microphone and captures the user's movements with a gyro sensor and accelerometer. The time-series data is then transferred to the server in real time.

[1851] The server converts the received voice data into text using the Google Speech-to-Text API and analyzes the speaking speed and tone of voice. Specifically, the voice data is divided into frames at regular time intervals and each frame is analyzed.

[1852] The server analyzes the motion data and evaluates the user's position and movement patterns. For example, it extracts information such as "the user is moving to the left" or "the user's position is fixed."

[1853] (Input) Real-time audio and motion data

[1854] (Output) Analyzed speaking rate, tone of voice, and movement patterns

[1855] Step 3: Emotion Recognition

[1856] The server's emotion engine uses voice and motion data to recognize the user's emotions. Specifically, it uses IBM Watson's emotion analysis API to extract emotional data such as "the user is nervous" or "the user is relaxed."

[1857] (Input) Voice data, movement data

[1858] (Output) Recognized emotion data

[1859] Step 4: Generate instructions

[1860] The server generates instructions for the user based on the analysis results. Based on the results of analyzing the voice data, it generates instructions such as "You're speaking too quickly. Please speak a little more slowly," and based on the results of analyzing the movement data, it generates instructions such as "You should move around a bit more while you speak." It also generates instructions based on the emotions recognized by the emotion engine, such as "You seem nervous. Take a deep breath and relax."

[1861] (Input) Speaking rate, tone of voice, movement patterns, emotional data

[1862] (Output) Generated instructions (text or audio)

[1863] Step 5: Notification of instructions

[1864] The device notifies the user of the instructions received from the server. Specifically, it displays text on the smart glasses or plays it as audio through earphones. For example, the instruction "You're speaking too quickly. Please speak a little more slowly" is displayed as text on the smart glasses.

[1865] (Input) Instruction data from the server

[1866] (Output) Text display on smart glasses, audio playback on earphones

[1867] In this way, the system analyzes the user's voice and movement data in real time and provides appropriate feedback to improve the quality of the presentation.

[1868] (Application example 2)

[1869] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1870] Conventional presentation support systems and customer service support systems do not provide appropriate feedback that takes into account the user's situation and emotions, making it difficult to maximize the user's response ability or the quality of their presentation. In particular, there was a need for a system that could provide appropriate real-time instructions to store clerks when serving customers, based on the customer's emotions and attitude. This required a system that could improve customer satisfaction and the response ability of store clerks.

[1871] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1872] In this invention, the server includes means for analyzing voice data and motion data and generating instructions for the user, emotion recognition means for generating appropriate instructions based on the recognized emotions, and means for generating instructions using a generative AI model. This allows the user to receive appropriate feedback in real time through a display device or audio device worn by the user, thereby improving the user's ability to respond in presentations and customer interactions.

[1873] A "user" is someone who uses the system to give presentations or interact with customers.

[1874] A "display device" is a device that provides visual information, such as smart glasses.

[1875] An "audio device" is a device that provides audio information, such as an earphone.

[1876] A "terminal" is a device that acquires user voice data and motion data in real time and transfers them to a server.

[1877] "Voice data" refers to the voice uttered by the user recorded as digital data.

[1878] "Motion data" refers to the user's movements, position, and other data recorded using sensors.

[1879] A "server" is a computer system that analyzes the voice data and motion data sent from the terminal, generates instructions, and sends them to the terminal.

[1880] The "emotion recognition means" is an algorithm that recognizes the user's emotions based on voice data and action data, and generates appropriate instructions based on those emotions.

[1881] A "generative AI model" is an artificial intelligence model that generates instructions in natural language based on data provided to it.

[1882] A "prompt sentence" is an input sentence for a generative AI model, and is the text that serves as the basis for generating instructions.

[1883] "Real time" refers to a time unit in which analysis and feedback are performed simultaneously with the user's actions and speech.

[1884] The present invention relates to a system that allows users to receive real-time instructions to effectively give presentations or serve customers. This system comprises a display or audio device worn by the user, a terminal that acquires and transfers voice and motion data in real time, a server that analyzes this data and generates instructions, and an emotion recognition means and a generative AI model to provide more appropriate instructions.

[1885] Overall system configuration

[1886] 1. User:

[1887] This is a person who uses the system to give presentations or interact with customers. The user wears a display or audio device such as smart glasses or earphones.

[1888] 2. Display or audio devices:

[1889] Display device: A device that provides visual information, such as smart glasses, allowing users to receive real-time instructions as text.

[1890] Audio device: A device that provides audio information, such as an earphone, allowing you to receive real-time instructions by voice.

[1891] 3. Terminal:

[1892] A device that captures user voice and motion data in real time and transfers it to a server. Specifically, this applies to smartphones and tablets.

[1893] 4. Server:

[1894] It is a computer system that analyzes voice and motion data sent from the device, generates instructions for the user, and sends them to the device. It can utilize cloud environments such as Amazon Web Services (AWS) and Google Cloud.

[1895] 5. Emotion recognition means:

[1896] This algorithm recognizes the user's emotions based on voice and movement data, and generates appropriate instructions based on those emotions. It uses deep learning libraries such as TensorFlow.

[1897] 6. Generative AI Models:

[1898] It is an artificial intelligence model that generates natural language instructions based on the data provided, such as OpenAI's GPT-4.

[1899] Program processing

[1900] The server first receives the voice and movement data sent from the device. The voice data is converted into text using Google Speech API or Amazon Transcribe. The voice and movement data are then analyzed together by an emotion recognition system to determine the user's emotion.

[1901] The server then inputs instructions based on the emotion recognition results into the generative AI model as prompt sentences, such as, "The customer's emotion is dissatisfaction. They said, 'I'd like to know more about this product.' Please instruct them on the appropriate response."

[1902] The generative AI model generates appropriate instructions based on the prompt and sends them back to the server. The server then sends the instructions to the device, which then sends feedback to the display and audio devices. For example, an instruction such as "Raise your voice a little to make the customer feel at ease" is displayed on the user's smart glasses. Audio instructions are also delivered through the earphones.

[1903] Specific examples

[1904] Consider a scenario in which a user wears smart glasses and acts as a store clerk while serving customers. The device collects voice and movement data in real time and sends it to a server. The server analyzes this data and uses emotion recognition to identify a disinterested tone of voice when a customer is asking a question. Based on this, the generative AI model creates instructions such as, "The customer seems anxious. Please respond in a more friendly tone." These instructions are displayed as text on the smart glasses, allowing the user to learn how to respond in real time.

[1905] In this way, this system allows users to receive appropriate feedback in real time during presentations and customer interactions, thereby improving their response skills.

[1906] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1907] Step 1:

[1908] The user puts on smart glasses or earphones and starts up the device (such as a smartphone). The device initializes the sensors and starts collecting voice and movement data. The input is the user's movements and voice, and the output is the collected data. Specifically, the device's application starts up, and the sensors detect the user's movements and voice.

[1909] Step 2:

[1910] The device transmits collected voice and motion data to the server in real time. The input is the data collected in real time, and the output is the transfer of this data to the server. The device continues to send data to the server via a network connection.

[1911] Step 3:

[1912] The server receives the voice data and converts it to text using a speech recognition engine (such as Google Speech API or Amazon Transcribe). The input is voice data and the output is text data. Specifically, the server makes an API call to convert the voice data to text.

[1913] Step 4:

[1914] The server analyzes the motion data and evaluates the user's position and movements. The input is the motion data, and the output is the analysis result. Specific actions are identified by processing the motion data using libraries such as OpenCV.

[1915] Step 5:

[1916] The server uses emotion recognition means to recognize the user's emotions based on the analyzed voice data and movement data. The input is the analyzed voice data and movement data, and the output is emotion data. Specifically, this is done by an emotion recognition algorithm using TensorFlow.

[1917] Step 6:

[1918] Based on the emotion recognition results, the server inputs a prompt sentence into the generative AI model to generate instructions. The input is emotion data and a prompt sentence, and the output is the generated instruction. Specifically, the prompt sentence is processed using OpenAI's GPT-4 or similar software to generate instructions in natural language.

[1919] Example prompt sentence:

[1920] "The customer is unhappy. They stated, 'I'd like to know more about this product.' Please direct them to the appropriate response."

[1921] Step 7:

[1922] The server sends the generated instructions to the terminal. The input is the generated instructions, and the output is the transfer to the terminal. Specifically, the server uses WebSocket or REST API to send the instruction data to the terminal.

[1923] Step 8:

[1924] The device then feeds back the received instructions to the user's display device (smart glasses) and audio device (earphones). The input is the instruction sent from the server, and the output is a notification to the user. Specifically, the instructions are displayed in text on the smart glasses, and the instructions are played back in audio on the earphones.

[1925] In this way, users receive relevant feedback in real time, enabling them to improve their presentations and customer interactions.

[1926] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1927] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1928] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1929] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1930] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1931] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1932] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1933] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1934] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1935] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1936] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1937] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1938] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1939] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1940] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1941] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1942] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1943] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1944] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1945] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1946] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1947] The following is further disclosed regarding the above embodiment.

[1948] (Claim 1)

[1949] a user-worn display or audio device;

[1950] a terminal that acquires and transfers user voice data and motion data in real time;

[1951] a server that analyzes the voice data and the motion data, generates instructions for the user, and transmits the instructions to the terminal;

[1952] A system including:

[1953] (Claim 2)

[1954] 10. The system of claim 1, wherein the instructions are based on the user's speaking rate, tone of voice, or standing position.

[1955] (Claim 3)

[1956] 10. The system of claim 1, wherein the display device is a smart glass and the instructions are displayed as text.

[1957] (Claim 4)

[1958] 10. The system of claim 1, wherein the audio device is an earphone and the instructions are played as audio.

[1959] (Claim 5)

[1960] 2. The system according to claim 1, wherein the terminal is equipped with a voice recognition engine and a motion sensor, and captures the user's voice and motion in real time and transmits them to the server.

[1961] "Example 1"

[1962] (Claim 1)

[1963] a visual information display means or a voice output means worn by a user;

[1964] a data collection means for acquiring and transferring user voice information and movement information in real time;

[1965] an analysis means for analyzing the voice information and the movement information, generating instructions for the user, and transmitting the instructions to the data collection means;

[1966] A system including:

[1967] (Claim 2)

[1968] 10. The system of claim 1, wherein the instructions are based on the user's speaking rate, tone of voice, or standing position.

[1969] (Claim 3)

[1970] 2. The system of claim 1, wherein the visual information display means is smart glasses and the instructions are displayed as text.

[1971] "Application Example 1"

[1972] (Claim 1)

[1973] a user-worn display or audio device;

[1974] a terminal that acquires and transfers user voice data and motion data in real time;

[1975] a server that analyzes the voice data and the motion data, generates instructions for the user, and transmits the instructions to the terminal;

[1976] means for analyzing manual movements and voice commands of workers in a factory and generating instructions regarding work efficiency and safety;

[1977] A system including:

[1978] (Claim 2)

[1979] 2. The system of claim 1, wherein the instructions are based on the user's speaking rate, tone of voice, or position, and further based on the speed of work and the method of using tools.

[1980] (Claim 3)

[1981] 10. The system of claim 1, wherein the display device is a smart glass and the instructions are displayed as text.

[1982] "Example 2: Combining Emotion Engines"

[1983] (Claim 1)

[1984] a user-worn display or audio device;

[1985] a terminal that acquires and transfers user voice data and motion data in real time;

[1986] a server that analyzes the voice data and the motion data, generates instructions for the user, and transmits the instructions to the terminal;

[1987] an emotion engine that recognizes the user's emotion based on the analysis results;

[1988] means for notifying a user of an instruction generated based on the emotion recognized by the emotion engine;

[1989] A system including:

[1990] (Claim 2)

[1991] 10. The system of claim 1, wherein the instructions are based on the user's speaking rate, tone of voice, position, or emotion.

[1992] (Claim 3)

[1993] 10. The system of claim 1, wherein the display device is a smart glass and the instructions are displayed as text.

[1994] (Claim 4)

[1995] 10. The system of claim 1, wherein the audio device is an earphone and the instructions are played as audio.

[1996] "Application example 2 when combining emotion engines"

[1997] (Claim 1)

[1998] a user-worn display or audio device;

[1999] a terminal that acquires and transfers user voice data and motion data in real time;

[2000] a server that analyzes the voice data and the motion data, generates instructions for the user, and transmits the instructions to the terminal;

[2001] emotion recognition means for generating appropriate instructions based on the recognized emotion;

[2002] A system including:

[2003] (Claim 2)

[2004] 10. The system of claim 1, wherein the instructions are based on the user's speaking rate, tone of voice, or standing position.

[2005] (Claim 3)

[2006] 10. The system of claim 1, wherein the display device is a smart glass and the instructions are displayed as text.

[2007] (Claim 4)

[2008] 10. The system of claim 1, wherein the server generates the instructions using a generative AI model.

[2009] (Claim 5)

[2010] 5. The system of claim 4, wherein the generative AI model generates instructions in natural language using prompt sentences. [Explanation of symbols]

[2011] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a user-worn display or audio device; a terminal that acquires and transfers user voice data and motion data in real time; a server that analyzes the voice data and the motion data, generates instructions for the user, and transmits the instructions to the terminal; A system including:

2. The system of claim 1 , wherein the prompts are based on the user's speaking rate, tone of voice, or standing position.

3. The system of claim 1 , wherein the display device is a smart glass and the instructions are displayed as text.

4. 10. The system of claim 1, wherein the audio device is an earphone and the instructions are reproduced as audio.

5. The system according to claim 1 , wherein the terminal is equipped with a voice recognition engine and a motion sensor, and captures the user's voice and motion in real time and transmits them to the server.

Images