system

Abstract

Provide a system. The device comprises: a terminal operated by a user, means for acquiring video, images, or audio; means for transmitting the acquired data to a server; means for receiving an inference result from the server; The system further includes means for displaying or audibly outputting the received inference results.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Children with communication difficulties, non-speaking people, and foreigners often have difficulty expressing themselves properly. This communication barrier often leads to misunderstandings and confusion in everyday life. Conventional technologies lack sufficient means to properly analyze non-verbal information such as gestures, foreign languages, drawings, and humming, and to infer their intentions. Therefore, there is a need for a method to achieve smoother and more accurate communication. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means.

[0006] A system is provided that includes a terminal operated by a user, a means for acquiring video, images, or audio, a means for transmitting the acquired data to a server, a means for receiving an inference result from the server, and a means for displaying or outputting the received inference result as audio. Furthermore, by providing a server that includes a means for analyzing the acquired data and inferring its contents, and a means for returning the inference result to the terminal, a system is realized that enables a user to easily understand inference results based on gestures, foreign language, drawings, or humming, and accurately grasp the other person's intentions.

[0007] "User" refers to an individual who uses the system to input non-verbal information and performs operations to obtain analysis results for that information.

[0008] "Terminal" refers to a device that acquires video, images, or audio through user operation and transmits the acquired data to a server.

[0009] "Video, images, or audio" refers to the medium of information expressed non-verbally in communication, and is the data to be analyzed.

[0010] "Means of acquisition" refers to the function of capturing video, images, or audio using sensors such as a camera or microphone on the device.

[0011] The term "server" refers to a central processing unit that analyzes the acquired data and returns the analysis results to the terminal.

[0012] "Inference results" refers to information containing intent or meaning derived from video, image, or audio data analyzed by the server.

[0013] "Means for transmitting" refers to the communication function for sending data from the terminal to the server.

[0014] "Means for receiving" refers to the communication function for receiving the analysis results returned from the server to the terminal.

[0015] "Means for displaying or outputting by voice" refers to a function on the terminal for visually or audibly informing the user of the inference results.

[0016] "Means of analysis" refers to the algorithms and AI models that the server uses to infer the intent and meaning of the data it acquires.

[0017] "Gesture" refers to a non-verbal means of communication expressed through bodily movements.

[0018] "Foreign language" refers to information presented in a language other than the user's native language.

[0019] "Drawing" refers to a picture or diagram made on paper or a digital device.

[0020] "Humming" refers to non-verbal audio information expressed by humming a melody. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0043] 1. User operations

[0044] A user launches a smartphone application. The application's main screen appears, and the user navigates through the menu to capture video, images, or audio. For example, the user taps the "Record" button to record video, and then taps the "Stop" button to save the data when finished.

[0045] 2. Terminal Processing

[0046] The device receives data entered by the user. It temporarily stores data acquired using sensors such as a camera or microphone, obtains the data path, and prepares it for subsequent processing. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request and sends it with the saved file attached.

[0047] 3. Server Processing

[0048] The server receives data sent from the device. It analyzes the received data and uses a pre-trained AI model to infer its contents. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. The analysis results are formatted and sent back to the device in JSON format.

[0049] 4. Terminal handling and output to the user

[0050] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0051] Specific examples

[0052] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and takes a video of the child's pointing movements. The device receives the video data, saves it, and sends it to a server. The server analyzes the received video, infers the child's intention, such as "I'm hungry," and sends the inference result back to the device. The device receives the inference result and displays it on the screen or tells the user "I'm hungry" by voice. In this way, the user can properly understand the child's intention and take appropriate action.

[0053] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, if the content is inferred as "Where is the toilet?", the result is sent back to the device, and the user is informed by displaying or audibly saying "Where is the toilet?", enabling smooth communication with the foreigner.

[0054] In this way, the system of the present invention can analyze non-verbal information and convey intentions to the user in an easily understandable form, thereby solving various communication problems.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] A user launches an app on their smartphone. The user taps an icon to launch the app, and the app's main screen appears. The user selects an option to capture video, images, or audio.

[0058] Step 2:

[0059] The user performs an operation to capture video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button at the desired time to save the data. The same operation is performed for images and audio.

[0060] Step 3:

[0061] The device receives data entered by the user. The acquired data is temporarily stored in the device, and the data path and metadata are prepared.

[0062] Step 4:

[0063] The device sends the acquired data to the server by creating an HTTP POST request to the server URL, attaching the acquired data file, and including any necessary authentication information.

[0064] Step 5:

[0065] The server receives the data sent from the device, saves it as a file, and inputs it into the AI ​​model for analysis.

[0066] Step 6:

[0067] The server analyzes the received data. It uses an AI model to analyze the data and understand non-verbal information such as gestures, foreign languages, drawings, and humming, and generates an analysis result. The generated result is formatted in JSON format.

[0068] Step 7:

[0069] The server returns the inference results to the device. The inference results are formatted in JSON and sent to the device as an HTTP response.

[0070] Step 8:

[0071] The device analyzes the inference results received from the server, parses the received JSON data to obtain the analysis results, and checks whether the inference results are in the correct format.

[0072] Step 9:

[0073] The device displays or outputs the inference results to the user. The analysis results are displayed as text on the screen or, if necessary, conveyed to the user using a text-to-speech function.

[0074] Step 10:

[0075] The user can check the displayed or spoken inference results and understand the other person's intention. For example, if a child's gesture indicates that they are hungry, the user can take appropriate action based on that information.

[0076] Through these steps, users can understand the meaning of videos, images, and audio that contain non-verbal information, enabling them to communicate appropriately.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] With the diversification of modern communication, there is a growing need for systems that can accurately analyze non-verbal information such as gestures, foreign languages, and voice, and convey it to users in an easy-to-understand manner. However, existing systems have difficulty efficiently analyzing non-verbal information in real time and providing appropriate feedback. The objective of this invention is to provide a system that analyzes video, image, and voice data acquired by a user on a server, quickly returns the results to a terminal, and displays or outputs the results to the user as voice.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for analyzing acquired data and inferring its contents, means for returning the inference results to the terminal, means for analyzing the data using a pre-trained AI model, and means for returning the analysis results to the terminal in JSON format, thereby enabling quick and accurate analysis of non-verbal information acquired from the user and providing appropriate feedback to the user.

[0082] A "user-operated terminal" is a device that is directly operated by a user to obtain video, images, or audio.

[0083] "Means of acquisition" refers to the mechanism for collecting video, image, or audio data using sensors such as cameras and microphones installed on the device.

[0084] "Means of sending to the server" refers to the protocols and procedures for transferring the acquired video, image, and audio data to the server via the network.

[0085] The "means for receiving inference results" is a function for receiving data on the analysis and inference results returned from the server.

[0086] "Means for displaying or outputting by voice" refers to the ability to display the received inference results on the terminal screen in text or graphical form or to play them back by voice.

[0087] "Temporary storage means" refers to a mechanism for temporarily storing acquired data in the device's memory or storage.

[0088] "Means for creating an HTTP POST request" is the process of generating a request to send the retrieved data to a server using the HTTP protocol.

[0089] "Means for analyzing data received from the server" refers to technology for interpreting data returned from the server and converting it into a form that can be used by the user.

[0090] A "pre-trained AI model" is an algorithm that has been trained for a specific task and performs analysis and inference based on input data.

[0091] The JSON format is a lightweight data exchange format for storing and transferring structured data.

[0092] MODE FOR CARRYING OUT THE INVENTION

[0093] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0094] Hardware and Software Configuration

[0095] Device:

[0096] The terminal is a mobile device such as a smartphone or tablet. This terminal is equipped with a camera, microphone, and internal storage, which allows it to capture and temporarily store video, image, and audio data. It also requires a screen to display feedback and a speaker for audio output. The terminal software includes libraries for creating HTTP requests and parsing JSON data.

[0097] server:

[0098] The server receives data using a web server (e.g., Nginx, Tomcat, etc.). To analyze the data, it uses Python scripts and runs pre-trained AI models (e.g., TENSORFLOW (registered trademark), PyTorch). The server also requires a library to send the analysis results back to the terminal in JSON format.

[0099] Operational Overview

[0100] 1. Data Acquisition and Transmission

[0101] A user launches a smartphone application and captures video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button to save the data. The device temporarily stores the captured data in its internal storage and creates an HTTP POST request to send to the server.

[0102] 2. Data Analysis

[0103] The server receives data sent from the device. It analyzes the received data using a Python script and uses a pre-trained AI model (TensorFlow or PyTorch) to infer the content of the data. The AI ​​model understands and analyzes non-verbal information such as gestures, foreign languages, drawings, and humming. The analysis results are formatted in JSON and sent back to the device.

[0104] 3. Receiving and displaying inference results

[0105] The device receives the inference results in JSON format returned from the server. It analyzes the received results and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0106] Specific operation example

[0107] Example 1: Child gesture analysis

[0108] To analyze the gestures of a child who is not good at speaking, the user (parent) launches an app on their smartphone and records a video of the child's pointing movements. The device temporarily saves the video data and sends it to the server using an HTTP POST request. The server receives the video data, analyzes it with a Python script, and uses a pre-trained TensorFlow model to infer the child's intent, such as "I'm hungry." The analysis results are returned in JSON format, and the device receives the results, displaying "I'm hungry" on the screen or outputting it as voice. The user can then appropriately understand the child's intent and respond accordingly.

[0109] Example 2: Communicating with foreigners

[0110] When a foreigner tries to communicate something using a combination of gestures and a foreign language, the user films the situation with their smartphone. The device then sends the filmed video data to a server. The server then analyzes the received data using a Python script and uses a pre-trained TensorFlow model to infer the intent, "Where is the toilet?" The inference result is sent back to the device in JSON format, and the device receives the result and displays "Where is the toilet?" on the screen or outputs it as voice. This allows users to communicate smoothly with foreigners.

[0111] Example prompt sentence:

[0112] "The system takes a video of the child's pointing action, sends the video data to a server, and returns the analysis results. The system helps understand the child's intentions by displaying the inference results on the screen and outputting them as audio."

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1:

[0115] The user launches the application on their smartphone. The main screen of the application is displayed, and the user operates the menu to acquire data.

[0116] Specific operation: Tap the "Record" button from the app's startup screen and select video mode.

[0117] Input: User actions

[0118] Output: Instruction to start data acquisition

[0119] Step 2:

[0120] The device accepts instructions from the user and acquires video, image, or audio data. It collects data using sensors such as a camera and microphone and temporarily stores it in its internal storage.

[0121] Specific operation: Record with the camera, and when finished, tap the "Stop" button to save the data.

[0122] Input: Data from the camera or microphone

[0123] Output: Stored media data

[0124] Step 3:

[0125] The device retrieves the path to the stored data and creates an HTTP POST request for sending it to the server, including the retrieved media data.

[0126] What it does: Creates an HTTP POST request and attaches a video file.

[0127] Input: Path to the saved media data

[0128] Output: Request sent to server

[0129] Step 4:

[0130] The server receives the data sent from the terminal. A web server (e.g., Nginx, Tomcat) receives the request and temporarily stores the data.

[0131] Specific operation: After receiving the data, save it in the specified folder.

[0132] Input: HTTP POST request from the terminal

[0133] Output: Data stored in the server

[0134] Step 5:

[0135] The server analyzes the received data, runs a Python script, and uses pre-trained AI models to analyze non-verbal information such as gestures, foreign words, drawings, and humming, and generates analysis results.

[0136] Specific operations: Run AI models using TensorFlow and PyTorch to analyze and infer data.

[0137] Input: Stored media data

[0138] Output: Analysis results (gesture recognition results as a 3D array, translation results as text, etc.)

[0139] Step 6:

[0140] The server formats the generated analysis results and returns them to the terminal in JSON format, sending JSON data containing the analysis results as an HTTP response.

[0141] Specific operation: Formats data into JSON format and creates an HTTP response.

[0142] Input: Parsed data

[0143] Output: Parsed results in JSON format

[0144] Step 7:

[0145] The device receives the JSON data returned from the server, parses the HTTP response, and extracts the data.

[0146] Specific operation: Parse the JSON data and obtain the required information.

[0147] Input: Parsed result in JSON format

[0148] Output: Text and audio data

[0149] Step 8:

[0150] The device displays or outputs the extracted analysis results to the user as text on the screen or as voice using a speech synthesis engine.

[0151] Specific action: Displaying text such as "I'm hungry" on the screen or reading it aloud.

[0152] Input: Text or audio data

[0153] Output: Feedback to the user (visual or auditory)

[0154] Examples:

[0155] Example of child gesture analysis:

[0156] The user launches the smartphone app and records the child's pointing behavior. The device temporarily saves the video data, generates an HTTP POST request, and sends it to the server. The server uses TensorFlow to analyze the gesture, infers that the child is hungry, and returns the result in JSON format. The device receives the JSON data and displays the text on the screen. The user can then look at the screen to understand the child's intention.

[0157] Examples of communication with foreigners:

[0158] The user launches the app and records the foreigner's gestures and speech in the foreign language. The device sends the data to the server, which analyzes it and returns the inference result, "Where is the toilet?" The device receives the result and displays it as text or outputs it as voice. The user can use this information to guide the foreigner.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] In industrial environments such as factories, there is a growing need for support systems to help workers efficiently receive work instructions. With conventional methods, it can take time for workers to understand work instructions, which can lead to reduced work efficiency. Additionally, when language barriers or non-verbal communication are required, it can be difficult to receive appropriate feedback. This can affect overall productivity and safety.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes a means for acquiring video, image, or audio, a means for transmitting the acquired data to the server, a means for receiving inference results from the server, a means for displaying or outputting the received inference results as audio, a means for analyzing hand gestures and voice commands in an industrial environment, and a means for generating work instructions based on the hand gestures and voice commands, thereby enabling effective provision of work instructions and feedback within a factory, improving work efficiency and overall productivity.

[0164] A "terminal" is an electronic device that is operated by a user to acquire video, images, or audio and transmit it to a server.

[0165] A "server" is a computer system that analyzes the acquired data and returns the inference results to the terminal.

[0166] "Inference results" are results based on the meaning and content of data obtained by the server through analysis.

[0167] "Display" refers to the act and means of visually conveying information to a user.

[0168] "Audio output" refers to the act and means of transmitting information to the user audibly.

[0169] "Industrial environment" refers to factories, manufacturing sites, and other workplaces where machinery and equipment are operated.

[0170] A "gesture" is an act of conveying intentions or instructions using hand or body movements.

[0171] A "voice command" is a voice signal used to give instructions to the system using voice.

[0172] "Work instructions" are information that instructs the procedures and actions for performing a specific task.

[0173] To implement this invention as a factory work support system, a server, a terminal operated by a user, and a network environment connecting these are required.

[0174] 1. Hardware and Software Requirements

[0175] Hardware:

[0176] Terminal: For example, a head-mounted display (HMD) or a smartphone. It is desirable that these have a built-in camera and microphone.

[0177] Server: A high-performance computer that must be capable of executing inference processing for AI models in real time.

[0178] software:

[0179] Flask: A Python®-based web framework for receiving and processing data on the server side.

[0180] PyTorch: A deep learning library that contains models for analyzing the acquired data.

[0181] HTTP request library: Used to send and receive data between the terminal and the server at high speed.

[0182] 2. System Overview

[0183] The user captures video, images, or audio on the device. The captured data is sent to the server via an HTTP request. The server receives the data and analyzes it using a generative AI model. The inference results are returned to the device in JSON format, which the device then displays or outputs as audio. Particularly in industrial environments, analyzing gestures and voice commands and generating appropriate work instructions can improve work efficiency.

[0184] 3. Specific Examples

[0185] Example 1: Providing work instructions

[0186] A factory worker puts on the HMD and issues a voice command, saying, "What do I do next?" The HMD's built-in microphone picks up the voice and sends the data to the server. The server analyzes the voice and generates a work instruction, such as "Install part A on the right side." This instruction is displayed on the HMD display and is also read aloud.

[0187] Example 2: Support for safety confirmation

[0188] When a worker is performing a dangerous task, he or she signals with a hand gesture, "Please check." The HMD camera captures the gesture and sends the data to the server. The server analyzes the gesture and generates a warning message, such as "Be careful, it's hot." This message is displayed on the HMD display and an audio warning is also issued.

[0189] 4. Examples of prompts

[0190] The following prompts can be used to make the necessary inferences for the generative AI model:

[0191] You are a factory manager robot. A worker is asking for instructions. He has part A in his hand. What should he do next?

[0192] In this way, the present invention provides the tools necessary to support practical tasks in the work environment and provide an efficient and safe environment.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] An application is launched on the terminal operated by the user to acquire video, images, or audio.

[0196] Input: Video, images, and audio data (e.g., worker makes hand gestures or voice commands)

[0197] Output: The captured data (e.g., worker gestures and voice are captured)

[0198] Step 2:

[0199] The device temporarily stores the acquired data and sends it to the server via an HTTP request.

[0200] Input: Data acquired in the previous step (e.g., data captured by the HMD camera or microphone)

[0201] Output: Data sent to the server (e.g., sent in an HTTP request)

[0202] Step 3:

[0203] The server passes the received data to the AI ​​model for analysis.

[0204] Input: Data sent from the device (e.g., captured gestures and voice data)

[0205] Output: Analyzed results (e.g., inference results from an AI model)

[0206] Step 4:

[0207] The server formats the inference results in JSON format and returns them to the terminal as an HTTP response.

[0208] Input: The inference result of the AI ​​model (e.g., "Install part A on the right side")

[0209] Output: HTTP response to the terminal (e.g., inference results in JSON format)

[0210] Step 5:

[0211] The terminal analyzes the received inference results and outputs them to the user by display or voice.

[0212] Input: Inference results returned from the server (e.g., JSON data)

[0213] Output: Display or audio output to the user interface (e.g., "Please install part A on the right side" is displayed on the HMD display and also read aloud)

[0214] Step 6:

[0215] The user performs the task by following the displayed information and audio guidance.

[0216] Input: Work instructions provided by the device (e.g., instructions displayed on the HMD display)

[0217] Output: The action performed by the user (e.g., attaching part A to the right side)

[0218] In this way, each step works together, allowing the user to perform the task efficiently and the system to provide feedback in real time.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, combines the inference results with an emotion engine, and returns the results to the terminal in a form that reflects the user's emotions and outputs them as display or audio.

[0221] 1. User operations

[0222] A user launches an application on their smartphone. The application's main screen appears, and the user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record video and then taps the "Stop" button at the desired time to save the data.

[0223] 2. Terminal Processing

[0224] The device receives data entered by the user. The acquired data is temporarily stored on the device, and the data path and metadata are prepared. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request, attaches the saved file, and sends it to the server's URL. The request also includes any necessary authentication information.

[0225] 3. Server analysis process

[0226] The server receives the data sent from the device. It receives the data and saves it as a file. The saved data is then input into the AI ​​model for analysis. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. In addition, the emotion engine analyzes elements such as the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0227] 4. Server emotion analysis and inference results generation

[0228] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference result. For example, if a child gestures to say "I'm hungry" while also making a dissatisfied expression, a detailed inference result such as "I'm hungry but in a bad mood" will be obtained. The generated results are formatted in JSON format.

[0229] 5. Terminal handling and output to the user

[0230] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can confirm the intention and emotion inferred by the server through text displayed on the screen or audible reading.

[0231] Specific examples

[0232] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and records a video of the child pointing. The device receives the video data, saves it, and sends it to the server. The server analyzes the received video and infers that the child is "hungry," while the emotion engine analyzes the child's dissatisfied facial expression and determines that the child is "in a bad mood." This result is formatted as "hungry but in a bad mood" and sent back to the device. The device receives the inference result and tells the user "hungry but in a bad mood" on the screen or by voice. The user can appropriately understand the child's intentions and emotions and respond quickly.

[0233] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, while the content is inferred as "Where is the toilet?", the emotion engine simultaneously detects the emotion of impatience and generates a detailed result such as "I need to go to the toilet urgently." This result is then sent back to the device, which then informs the user by displaying or audibly saying "I need to go to the toilet urgently," enabling smooth communication with the foreigner.

[0234] In this way, the system of the present invention can analyze non-verbal information and convey intentions and emotions to the user in an easy-to-understand manner, thereby solving various communication problems.

[0235] The processing flow will be explained below.

[0236] Step 1:

[0237] A user launches an app on their smartphone. The user taps the app icon to launch it, and the main screen appears. The user selects an option to capture video, images, or audio depending on the purpose of use.

[0238] Step 2:

[0239] The user captures video, image, or audio data. For example, the user taps the "record" button to record a video of the child's gestures, and then taps the "stop" button to save the data. Similarly, images and audio are captured appropriately.

[0240] Step 3:

[0241] The device receives the acquired data, temporarily stores the data acquired through sensors such as the camera and microphone, and records the file path and metadata of the data.

[0242] Step 4:

[0243] The device sends the acquired data to the server by creating an HTTP POST request and sending the acquired data file along with the data to the server's URL. The request also includes any necessary authentication information.

[0244] Step 5:

[0245] The server receives the data sent from the device, saves the received data as a file, and prepares it for data analysis.

[0246] Step 6:

[0247] The server begins the analysis process, using AI models such as image and voice recognition to analyze the data and infer intent from gestures, language, pictures, and humming.

[0248] Step 7:

[0249] The server uses an emotion engine to analyze the user's emotions, and estimates their emotional state by analyzing the acquired data, such as the user's facial expressions, tone of voice, and body movements.

[0250] Step 8:

[0251] The server combines the inference results and the emotion analysis results to generate the final inference result. For example, by combining "hungry," the gesture analysis result, and the emotion analysis result "bad mood," the detailed inference result "hungry but bad mood" is obtained.

[0252] Step 9:

[0253] The server returns the final inference results to the device, which then formats the integrated inference results in JSON format and sends them to the device as an HTTP response.

[0254] Step 10:

[0255] The device analyzes the inference results received from the server, parses the received JSON data, obtains the inference results, and prepares them to be displayed or output to the user in an appropriate format.

[0256] Step 11:

[0257] The device displays or outputs the inference result to the user. It can be displayed as text on the screen or read aloud to the user. For example, the inference result "I'm hungry, but I'm in a bad mood" can be displayed or output as audio.

[0258] Step 12:

[0259] The user can check the inference results displayed or output as voice and understand the other person's intentions and emotions. For example, based on the analysis result that a child is "hungry but in a bad mood," the user can take appropriate action.

[0260] In this way, the system provides multifunctional support to help users accurately understand their intentions and respond quickly.

[0261] Example 2

[0262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0263] Conventional systems lack the means to analyze non-verbal information and accurately convey the user's intentions and emotions. Furthermore, there are problems with time lags and reduced recognition accuracy in the process from sending the data acquired by the user to the server, receiving the analysis results, and outputting them as display or voice. Therefore, the present invention aims to solve these problems and provide a system that analyzes non-verbal information with high accuracy and provides rapid feedback to the user.

[0264] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0265] In this invention, the server includes means for analyzing the acquired data and inferring its contents, means for estimating the user's emotions using an emotion engine based on the inference results, and means for returning the inference results to the terminal, thereby improving the accuracy and speed of analyzing non-verbal information and enabling quick and accurate feedback to the user.

[0266] A "terminal" is a device that can be operated by a user, acquires video, images, or audio, transmits the acquired data to a server, receives the inference results returned from the server, and displays or outputs them as audio.

[0267] "Data" refers to information in the form of video, image or audio files acquired by the terminal.

[0268] A "server" is a computer system that receives data sent from a terminal, analyzes it, and returns the analysis results to the terminal.

[0269] "Inference results" refer to the estimated results obtained by the server when analyzing data, and include information about the user's intentions and actions.

[0270] An "HTTP POST request" is a request format that uses a web protocol for a terminal to send data to a server.

[0271] An "emotion engine" is a software system that analyzes information such as a user's facial expressions, tone of voice, and body movements based on the analysis results, and estimates the user's emotions.

[0272] The "JSON format" is a lightweight data description format for structuring, storing, and transferring data, and is a format that is mainly composed of key-value pairs.

[0273] The present invention is a system that acquires video, images, or audio using a terminal operated by a user, sends the data to a server for analysis, and combines the analysis results with an emotion engine to return the results to the terminal in a form that reflects the user's emotions, and outputs them as a display or audio.

[0274] To implement this system, the following hardware and software are used.

[0275] Hardware and software used

[0276] Device: Mobile device such as a smartphone or tablet

[0277] Server: A high-performance computer server

[0278] Software: Mobile applications, HTTP communication protocols, AI analysis models (e.g., OpenCV, TensorFlow), emotion engines (e.g., Emotion API), JSON-formatted user interfaces, and text-to-speech functions (e.g., Google® Text-to-Speech)

[0279] System Operation

[0280] A user launches a smartphone application and selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record a video. When finished, the user taps the "Stop" button to save the data.

[0281] The device temporarily stores the retrieved data in its internal storage, prepares the data path and metadata, and then makes an HTTP POST request to send the attached data file and authentication information to the server.

[0282] The server stores the received data in a specified directory and inputs it into the AI ​​model for analysis. The AI ​​model analyzes the transmitted video, image, and audio data, understanding non-verbal information such as gestures, foreign languages, drawings, and humming to generate analysis results. Furthermore, it uses an emotion engine to analyze the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0283] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference results, which are formatted in JSON format.

[0284] The server sends the generated inference results to the device via an HTTP POST request. The device analyzes the received JSON-formatted inference results and displays or outputs them to the user. The user can check the analysis results through text displayed on the screen or audio readout, and respond quickly.

[0285] Specific examples

[0286] For example, this system is used when a user (parent) uses a smartphone to record a pointing action by a child who is not good at speaking. The device receives the video data, temporarily stores it, and sends it to the server. The server analyzes the received video and generates the inference result "hungry," while also using the emotion engine to detect the child's dissatisfied facial expression. This results in a more detailed result: "hungry but in a bad mood." This result is sent back to the device, which tells the user "hungry but in a bad mood" by display or voice.

[0287] As another example, when a foreigner tries to communicate something using a combination of gestures and a foreign language, the user takes a photo of the situation with their smartphone. The device then sends the captured data to the server, which analyzes it to infer the intention of "Where is the toilet?" and uses an emotion engine to detect the emotion of urgency, generating the inference result of "I need to go to the toilet urgently." This result is then sent back to the device, which then displays or speaks to the user the message "I need to go to the toilet urgently." This enables smooth communication with foreigners.

[0288] Prompt Sentence Examples

[0289] "Explain how you can analyze a child's gestures and facial expressions to infer their intentions and emotions."

[0290] Based on this prompt, the generative AI model can explain the specific analysis method and how to communicate the results.

[0291] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0292] System processing steps

[0293] Step 1: User interaction

[0294] A user launches a smartphone application. The application's main screen appears. The user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to capture video, and the "Stop" button to save the data. The input is the video, image, or audio data captured by the user's operation. The output is the saved data file.

[0295] Step 2: Prepare the device to send data

[0296] The device temporarily stores the data acquired from the user in its internal storage. It prepares the data path and metadata and creates an HTTP POST request. The input is the acquired data file. The output is an HTTP POST request with the data file attached.

[0297] Step 3: Send data by device

[0298] The device includes the data file and authentication information in the prepared HTTP POST request and sends it to the specified URL on the server. Specifically, the request is sent using a request library (for example, the Python requests library). The input is the HTTP POST request and authentication information. The output is the result of sending the request to the server.

[0299] Step 4: Server receives data

[0300] The server receives an HTTP POST request sent from the device and saves the data file in the specified directory. The input is the HTTP POST request from the device. The output is the saved data file.

[0301] Step 5: Data analysis by the server

[0302] The server inputs the saved data files into the AI ​​model for analysis. For example, in the case of video data, analysis is performed using OpenCV or TensorFlow to identify gestures and foreign languages. The input is the saved data file. The output is text data (e.g., "I'm hungry") that is the result of the analysis.

[0303] Step 6: Sentiment analysis by server

[0304] The server uses an emotion engine based on the analysis results to estimate the user's emotion. For example, it uses Microsoft's (registered trademark) Emotion API to analyze facial expressions and detect emotions such as "unhappy expression." The input is the text data of the analysis results. The output is the estimated emotion (e.g., "unhappy").

[0305] Step 7: Server generates inference results

[0306] The server combines the analysis results and emotion estimation results to generate the final inference result, which is then formatted in JSON. The input is the text data of the analysis results and the emotion estimation result. The output is the inference result in JSON format (for example, {"intention": "hungry", "emotion": "unhappy"}).

[0307] Step 8: Server sends inference results

[0308] The server sends the inference results formatted in JSON to the terminal via an HTTP POST request. The input is the inference results in JSON format. The output is the result sent to the terminal.

[0309] Step 9: Receive and display the inference results on the device

[0310] The device receives an HTTP POST request sent from the server and parses the inference results in JSON format. It then displays the results to the user as text or audio. Specifically, it uses a text-to-speech conversion function such as Google Text-to-Speech. The input is the inference results in JSON format from the server. The output is the display or audio output for the user.

[0311] (Application example 2)

[0312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0313] In conventional security systems, surveillance cameras and sensors primarily rely on video data and specific behavioral patterns to detect abnormal behavior. However, this alone does not allow for an accurate understanding of the emotions of the monitored individual, limiting the accuracy of identifying suspicious individuals. For example, even if no suspicious behavior is observed, changes in emotions may be overlooked, potentially resulting in potential danger being overlooked. Another issue is that the inability to respond quickly based on emotional changes places a heavy burden on security personnel.

[0314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0315] In this invention, the server includes an emotion analysis means for analyzing acquired data and inferring emotions, a means for notifying security personnel based on the detection of suspicious behavior or emotions, and a means for returning the inference results to the terminal, thereby enabling the emotions of monitored persons to be accurately analyzed in real time and quickly notified to security personnel.

[0316] A "user-operated terminal" is an electronic device that is manually operated by a user to acquire video, images, or audio data and process that data.

[0317] "Means for acquiring video, images or audio" refers to a function for collecting video, still images and audio data using sensors such as cameras and microphones.

[0318] The "means for sending acquired data to a server" is a function for transferring acquired data to a server via the Internet using a communication protocol such as an HTTP POST request.

[0319] The "means for receiving inference results from the server" is a function for receiving analysis results and data returned from the server.

[0320] "Means for displaying or outputting received inference results as audio" refers to a function for displaying text or images on the terminal screen or transmitting information as audio from the speaker.

[0321] "Means of acquiring data in real time from surveillance cameras and sensors" refers to a function for acquiring video and audio in real time using installed surveillance cameras and various sensors.

[0322] "Emotion analysis means for analyzing acquired data and inferring emotions" is a function for inferring emotions by analyzing the facial expressions and tone of voice of the monitored person based on the acquired video and audio data.

[0323] "Means for notifying when suspicious behavior or emotions are detected" is a function that sends warnings and notifications to security personnel when an abnormality or suspicious situation is detected based on inferred emotions or behavior.

[0324] "Means of analyzing data and inferring its contents" refers to a function that uses certain rules and AI algorithms to understand the contents of acquired data and derive inferential results.

[0325] The "means for returning the inference results to the terminal" is a function for correcting the analysis results and sending them back to the original terminal.

[0326] "Means generated based on gestures, foreign language, drawings, or humming" refers to a function that analyzes a person's physical movements, speech in a specific language, drawings, humming, etc., and generates analysis results based on the content.

[0327] "Means generated based on real-time emotion analysis" refers to a function that instantly analyzes changes in emotions based on data obtained in real time and predicts the results.

[0328] This invention is a system that uses a terminal operated by a user to acquire video, images, or audio, transmits the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.In addition, it also has a means for acquiring data and analyzing emotions in real time, and quickly notifying security personnel of suspicious behavior or emotions.

[0329] Hardware and Software

[0330] Hardware:

[0331] User devices such as smartphones and tablets

[0332] Surveillance cameras and sensors

[0333] server

[0334] software:

[0335] Smartphone and tablet application software (compatible with iOS and ANDROID (registered trademark))

[0336] Server software (web frameworks such as Flask)

[0337] Sentiment Analysis Library

[0338] OpenCV (video processing library)

[0339] Acquiring and Sending Data

[0340] A user launches a surveillance system or smartphone application. The user device uses a camera or microphone to capture video, images, or audio. The captured data is temporarily stored on the device and then sent to the server using an HTTP POST request, including any necessary authentication information. Data is also captured in real time from surveillance cameras and sensors and sent to the server.

[0341] Data analysis

[0342] The server receives the transmitted data and saves it as a file. The saved data is then analyzed using an AI model. The AI ​​model understands non-verbal information from video, image, and audio data and generates analysis results. Furthermore, an emotion analysis tool estimates the user's emotions from facial expressions, tone of voice, body movements, etc.

[0343] Generating and notifying inference results

[0344] The emotions estimated by the emotion analysis method are added to the analysis results to generate the final inference results. For example, emotions such as "anxiety" or "impatience" are estimated from video footage captured by a surveillance camera, and if suspicious behavior is detected, the detailed inference results are formatted in JSON format. If an abnormality is detected, the analysis results are notified to security personnel in real time.

[0345] Output to the user

[0346] The user device receives the inference results returned from the server. The received results are analyzed and displayed or output as audio to the user. The user can confirm the intent and emotion inferred by the server through text displayed on the screen or audio readout. Security personnel can also respond quickly by checking the notification.

[0347] Examples of concrete examples and prompts

[0348] For example, if a suspicious person in a location where a surveillance camera is installed shows an anxious expression, the analysis result will be estimated as "anxiety / anxiety." This information will be sent to security personnel along with a notification that "There is a person with an anxious expression in this area."

[0349] Prompt Sentence Examples

[0350] Analyze the video data uploaded by users, understand non-verbal information (gestures and facial expressions), and analyze emotions using an emotion engine.

[0351] example:

[0352] 1. Video data: [Video file path]

[0353] 2. Estimated emotion: [Estimation result]

[0354] In this way, by linking the user terminal and the server, this invention enables real-time data acquisition and emotion analysis, thereby enhancing security.

[0355] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0356] Step 1:

[0357] A user launches a smartphone application and uses the camera or microphone to capture video, images, or audio. The input data can be video, image, or audio files. This data is temporarily stored on the device. Specifically, the user taps the record button in the app to start recording, and then taps the stop button to save the data.

[0358] Step 2:

[0359] The device sends the acquired data to the server using an HTTP POST request. The input is a saved video, image, or audio file, and the output is a request to the server. Authentication information is also added. Specifically, the device creates a request by specifying the path of the acquired file and sends it to the server's URL.

[0360] Step 3:

[0361] The server stores the received data and inputs it into the AI ​​model for analysis. The input is the submitted video, image, or audio file, and the output is the analysis result. Specifically, the server saves the file in a specified directory and processes the input data using the analysis library.

[0362] Step 4:

[0363] The server analyzes the data using an AI model and infers the user's emotions through an emotion analysis tool. The input is preprocessed data, and the output is the result of the emotion analysis. Specifically, the AI ​​model analyzes gestures and facial expressions, while the emotion engine analyzes voice tone and body movements.

[0364] Step 5:

[0365] The server adds the emotion analysis results to the inference results, generates the final inference results in JSON format, and returns them to the device. The inputs are the analysis results and emotion analysis results, and the output is the JSON data of the final inference results. Specifically, the server formats the data and creates a response to send back to the device.

[0366] Step 6:

[0367] The device analyzes the received inference results and displays or outputs them to the user. The input is inference result data in JSON format, and the output is displayed text or spoken audio. Specifically, the device parses the JSON data and displays it in a text view, or uses a speech synthesis engine to communicate it to the user.

[0368] Step 7:

[0369] Data is acquired in real time from surveillance cameras and sensors and sent to a server in the same way. The input is real-time video and audio data, and the output is a request to the server. Specifically, a surveillance camera system that operates continuously collects data and sends it to the server at regular intervals.

[0370] Step 8:

[0371] If the server detects suspicious behavior or emotions from the data analyzed in real time, it notifies security personnel. The input is the real-time analysis results, and the output is the notification content. Specifically, when an abnormality is detected, an SMS, email, or in-app push notification is sent.

[0372] In this way, each processing step repeats the process of data acquisition, transmission, analysis, and notification, building a system that achieves enhanced security.

[0373] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0374] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0375] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0376] [Second embodiment]

[0377] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0378] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0379] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0380] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0381] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0382] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0383] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0384] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0385] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0386] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0387] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0388] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0389] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0390] 1. User operations

[0391] A user launches a smartphone application. The application's main screen appears, and the user navigates through the menu to capture video, images, or audio. For example, the user taps the "Record" button to record video, and then taps the "Stop" button to save the data when finished.

[0392] 2. Terminal Processing

[0393] The device receives data entered by the user. It temporarily stores data acquired using sensors such as a camera or microphone, obtains the data path, and prepares it for subsequent processing. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request and sends it with the saved file attached.

[0394] 3. Server Processing

[0395] The server receives data sent from the device. It analyzes the received data and uses a pre-trained AI model to infer its contents. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. The analysis results are formatted and sent back to the device in JSON format.

[0396] 4. Terminal handling and output to the user

[0397] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0398] Specific examples

[0399] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and takes a video of the child's pointing movements. The device receives the video data, saves it, and sends it to a server. The server analyzes the received video, infers the child's intention, such as "I'm hungry," and sends the inference result back to the device. The device receives the inference result and displays it on the screen or tells the user "I'm hungry" by voice. In this way, the user can properly understand the child's intention and take appropriate action.

[0400] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, if the content is inferred as "Where is the toilet?", the result is sent back to the device, and the user is informed by displaying or audibly saying "Where is the toilet?", enabling smooth communication with the foreigner.

[0401] In this way, the system of the present invention can analyze non-verbal information and convey intentions to the user in an easily understandable form, thereby solving various communication problems.

[0402] The processing flow will be explained below.

[0403] Step 1:

[0404] A user launches an app on their smartphone. The user taps an icon to launch the app, and the app's main screen appears. The user selects an option to capture video, images, or audio.

[0405] Step 2:

[0406] The user performs an operation to capture video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button at the desired time to save the data. The same operation is performed for images and audio.

[0407] Step 3:

[0408] The device receives data entered by the user. The acquired data is temporarily stored in the device, and the data path and metadata are prepared.

[0409] Step 4:

[0410] The device sends the acquired data to the server by creating an HTTP POST request to the server URL, attaching the acquired data file, and including any necessary authentication information.

[0411] Step 5:

[0412] The server receives the data sent from the device, saves it as a file, and inputs it into the AI ​​model for analysis.

[0413] Step 6:

[0414] The server analyzes the received data. It uses an AI model to analyze the data and understand non-verbal information such as gestures, foreign languages, drawings, and humming, and generates an analysis result. The generated result is formatted in JSON format.

[0415] Step 7:

[0416] The server returns the inference results to the device. The inference results are formatted in JSON and sent to the device as an HTTP response.

[0417] Step 8:

[0418] The device analyzes the inference results received from the server, parses the received JSON data to obtain the analysis results, and checks whether the inference results are in the correct format.

[0419] Step 9:

[0420] The device displays or outputs the inference results to the user. The analysis results are displayed as text on the screen or, if necessary, conveyed to the user using a text-to-speech function.

[0421] Step 10:

[0422] The user can check the displayed or spoken inference results and understand the other person's intention. For example, if a child's gesture indicates that they are hungry, the user can take appropriate action based on that information.

[0423] Through these steps, users can understand the meaning of videos, images, and audio that contain non-verbal information, enabling them to communicate appropriately.

[0424] Example 1

[0425] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0426] With the diversification of modern communication, there is a growing need for systems that can accurately analyze non-verbal information such as gestures, foreign languages, and voice, and convey it to users in an easy-to-understand manner. However, existing systems have difficulty efficiently analyzing non-verbal information in real time and providing appropriate feedback. The objective of this invention is to provide a system that analyzes video, image, and voice data acquired by a user on a server, quickly returns the results to a terminal, and displays or outputs the results to the user as voice.

[0427] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0428] In this invention, the server includes means for analyzing acquired data and inferring its contents, means for returning the inference results to the terminal, means for analyzing the data using a pre-trained AI model, and means for returning the analysis results to the terminal in JSON format, thereby enabling quick and accurate analysis of non-verbal information acquired from the user and providing appropriate feedback to the user.

[0429] A "user-operated terminal" is a device that is directly operated by a user to obtain video, images, or audio.

[0430] "Means of acquisition" refers to the mechanism for collecting video, image, or audio data using sensors such as cameras and microphones installed on the device.

[0431] "Means of sending to the server" refers to the protocols and procedures for transferring the acquired video, image, and audio data to the server via the network.

[0432] The "means for receiving inference results" is a function for receiving data on the analysis and inference results returned from the server.

[0433] "Means for displaying or outputting by voice" refers to the ability to display the received inference results on the terminal screen in text or graphical form or to play them back by voice.

[0434] "Temporary storage means" refers to a mechanism for temporarily storing acquired data in the device's memory or storage.

[0435] "Means for creating an HTTP POST request" is the process of generating a request to send the retrieved data to a server using the HTTP protocol.

[0436] "Means for analyzing data received from the server" refers to technology for interpreting data returned from the server and converting it into a form that can be used by the user.

[0437] A "pre-trained AI model" is an algorithm that has been trained for a specific task and performs analysis and inference based on input data.

[0438] The JSON format is a lightweight data exchange format for storing and transferring structured data.

[0439] MODE FOR CARRYING OUT THE INVENTION

[0440] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0441] Hardware and Software Configuration

[0442] Device:

[0443] The terminal is a mobile device such as a smartphone or tablet. This terminal is equipped with a camera, microphone, and internal storage, which allows it to capture and temporarily store video, image, and audio data. It also requires a screen to display feedback and a speaker for audio output. The terminal software includes libraries for creating HTTP requests and parsing JSON data.

[0444] server:

[0445] The server receives data using a web server (e.g., Nginx, Tomcat, etc.), analyzes the data using Python scripts and runs pre-trained AI models (e.g., TensorFlow, PyTorch), and also requires a library to send the analysis results back to the device in JSON format.

[0446] Operational Overview

[0447] 1. Data Acquisition and Transmission

[0448] A user launches a smartphone application and captures video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button to save the data. The device temporarily stores the captured data in its internal storage and creates an HTTP POST request to send to the server.

[0449] 2. Data Analysis

[0450] The server receives data sent from the device. It analyzes the received data using a Python script and uses a pre-trained AI model (TensorFlow or PyTorch) to infer the content of the data. The AI ​​model understands and analyzes non-verbal information such as gestures, foreign languages, drawings, and humming. The analysis results are formatted in JSON and sent back to the device.

[0451] 3. Receiving and displaying inference results

[0452] The device receives the inference results in JSON format returned from the server. It analyzes the received results and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0453] Specific operation example

[0454] Example 1: Child gesture analysis

[0455] To analyze the gestures of a child who is not good at speaking, the user (parent) launches an app on their smartphone and records a video of the child's pointing movements. The device temporarily saves the video data and sends it to the server using an HTTP POST request. The server receives the video data, analyzes it with a Python script, and uses a pre-trained TensorFlow model to infer the child's intent, such as "I'm hungry." The analysis results are returned in JSON format, and the device receives the results, displaying "I'm hungry" on the screen or outputting it as voice. The user can then appropriately understand the child's intent and respond accordingly.

[0456] Example 2: Communicating with foreigners

[0457] When a foreigner tries to communicate something using a combination of gestures and a foreign language, the user films the situation with their smartphone. The device then sends the filmed video data to a server. The server then analyzes the received data using a Python script and uses a pre-trained TensorFlow model to infer the intent, "Where is the toilet?" The inference result is sent back to the device in JSON format, and the device receives the result and displays "Where is the toilet?" on the screen or outputs it as voice. This allows users to communicate smoothly with foreigners.

[0458] Example prompt sentence:

[0459] "The system takes a video of the child's pointing action, sends the video data to a server, and returns the analysis results. The system helps understand the child's intentions by displaying the inference results on the screen and outputting them as audio."

[0460] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0461] Step 1:

[0462] The user launches the application on their smartphone. The main screen of the application is displayed, and the user operates the menu to acquire data.

[0463] Specific operation: Tap the "Record" button from the app's startup screen and select video mode.

[0464] Input: User actions

[0465] Output: Instruction to start data acquisition

[0466] Step 2:

[0467] The device accepts instructions from the user and acquires video, image, or audio data. It collects data using sensors such as a camera and microphone and temporarily stores it in its internal storage.

[0468] Specific operation: Record with the camera, and when finished, tap the "Stop" button to save the data.

[0469] Input: Data from the camera or microphone

[0470] Output: Stored media data

[0471] Step 3:

[0472] The device retrieves the path to the stored data and creates an HTTP POST request for sending it to the server, including the retrieved media data.

[0473] What it does: Creates an HTTP POST request and attaches a video file.

[0474] Input: Path to the saved media data

[0475] Output: Request sent to server

[0476] Step 4:

[0477] The server receives the data sent from the terminal. A web server (e.g., Nginx, Tomcat) receives the request and temporarily stores the data.

[0478] Specific operation: After receiving the data, save it in the specified folder.

[0479] Input: HTTP POST request from the terminal

[0480] Output: Data stored in the server

[0481] Step 5:

[0482] The server analyzes the received data, runs a Python script, and uses pre-trained AI models to analyze non-verbal information such as gestures, foreign words, drawings, and humming, and generates analysis results.

[0483] Specific operations: Run AI models using TensorFlow and PyTorch to analyze and infer data.

[0484] Input: Stored media data

[0485] Output: Analysis results (gesture recognition results as a 3D array, translation results as text, etc.)

[0486] Step 6:

[0487] The server formats the generated analysis results and returns them to the terminal in JSON format, sending JSON data containing the analysis results as an HTTP response.

[0488] Specific operation: Formats data into JSON format and creates an HTTP response.

[0489] Input: Parsed data

[0490] Output: Parsed results in JSON format

[0491] Step 7:

[0492] The device receives the JSON data returned from the server, parses the HTTP response, and extracts the data.

[0493] Specific operation: Parse the JSON data and obtain the required information.

[0494] Input: Parsed result in JSON format

[0495] Output: Text and audio data

[0496] Step 8:

[0497] The device displays or outputs the extracted analysis results to the user as text on the screen or as voice using a speech synthesis engine.

[0498] Specific action: Displaying text such as "I'm hungry" on the screen or reading it aloud.

[0499] Input: Text or audio data

[0500] Output: Feedback to the user (visual or auditory)

[0501] Examples:

[0502] Example of child gesture analysis:

[0503] The user launches the smartphone app and records the child's pointing behavior. The device temporarily saves the video data, generates an HTTP POST request, and sends it to the server. The server uses TensorFlow to analyze the gesture, infers that the child is hungry, and returns the result in JSON format. The device receives the JSON data and displays the text on the screen. The user can then look at the screen to understand the child's intention.

[0504] Examples of communication with foreigners:

[0505] The user launches the app and records the foreigner's gestures and speech in the foreign language. The device sends the data to the server, which analyzes it and returns the inference result, "Where is the toilet?" The device receives the result and displays it as text or outputs it as voice. The user can use this information to guide the foreigner.

[0506] (Application example 1)

[0507] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0508] In industrial environments such as factories, there is a growing need for support systems to help workers efficiently receive work instructions. With conventional methods, it can take time for workers to understand work instructions, which can lead to reduced work efficiency. Additionally, when language barriers or non-verbal communication are required, it can be difficult to receive appropriate feedback. This can affect overall productivity and safety.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0510] In this invention, the server includes a means for acquiring video, image, or audio, a means for transmitting the acquired data to the server, a means for receiving inference results from the server, a means for displaying or outputting the received inference results as audio, a means for analyzing hand gestures and voice commands in an industrial environment, and a means for generating work instructions based on the hand gestures and voice commands, thereby enabling effective provision of work instructions and feedback within a factory, improving work efficiency and overall productivity.

[0511] A "terminal" is an electronic device that is operated by a user to acquire video, images, or audio and transmit it to a server.

[0512] A "server" is a computer system that analyzes the acquired data and returns the inference results to the terminal.

[0513] "Inference results" are results based on the meaning and content of data obtained by the server through analysis.

[0514] "Display" refers to the act and means of visually conveying information to a user.

[0515] "Audio output" refers to the act and means of transmitting information to the user audibly.

[0516] "Industrial environment" refers to factories, manufacturing sites, and other workplaces where machinery and equipment are operated.

[0517] A "gesture" is an act of conveying intentions or instructions using hand or body movements.

[0518] A "voice command" is a voice signal used to give instructions to the system using voice.

[0519] "Work instructions" are information that instructs the procedures and actions for performing a specific task.

[0520] To implement this invention as a factory work support system, a server, a terminal operated by a user, and a network environment connecting these are required.

[0521] 1. Hardware and Software Requirements

[0522] Hardware:

[0523] Terminal: For example, a head-mounted display (HMD) or a smartphone. It is desirable that these have a built-in camera and microphone.

[0524] Server: A high-performance computer that must be capable of executing inference processing for AI models in real time.

[0525] software:

[0526] Flask: A Python-based web framework for receiving and processing server-side data.

[0527] PyTorch: A deep learning library that contains models for analyzing the acquired data.

[0528] HTTP request library: Used to send and receive data between the terminal and the server at high speed.

[0529] 2. System Overview

[0530] The user captures video, images, or audio on the device. The captured data is sent to the server via an HTTP request. The server receives the data and analyzes it using a generative AI model. The inference results are returned to the device in JSON format, which the device then displays or outputs as audio. Particularly in industrial environments, analyzing gestures and voice commands and generating appropriate work instructions can improve work efficiency.

[0531] 3. Specific Examples

[0532] Example 1: Providing work instructions

[0533] A factory worker puts on the HMD and issues a voice command, saying, "What do I do next?" The HMD's built-in microphone picks up the voice and sends the data to the server. The server analyzes the voice and generates a work instruction, such as "Install part A on the right side." This instruction is displayed on the HMD display and is also read aloud.

[0534] Example 2: Support for safety confirmation

[0535] When a worker is performing a dangerous task, he or she signals with a hand gesture, "Please check." The HMD camera captures the gesture and sends the data to the server. The server analyzes the gesture and generates a warning message, such as "Be careful, it's hot." This message is displayed on the HMD display and an audio warning is also issued.

[0536] 4. Examples of prompts

[0537] The following prompts can be used to make the necessary inferences for the generative AI model:

[0538] You are a factory manager robot. A worker is asking for instructions. He has part A in his hand. What should he do next?

[0539] In this way, the present invention provides the tools necessary to support practical tasks in the work environment and provide an efficient and safe environment.

[0540] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0541] Step 1:

[0542] An application is launched on the terminal operated by the user to acquire video, images, or audio.

[0543] Input: Video, images, and audio data (e.g., worker makes hand gestures or voice commands)

[0544] Output: The captured data (e.g., worker gestures and voice are captured)

[0545] Step 2:

[0546] The device temporarily stores the acquired data and sends it to the server via an HTTP request.

[0547] Input: Data acquired in the previous step (e.g., data captured by the HMD camera or microphone)

[0548] Output: Data sent to the server (e.g., sent in an HTTP request)

[0549] Step 3:

[0550] The server passes the received data to the AI ​​model for analysis.

[0551] Input: Data sent from the device (e.g., captured gestures and voice data)

[0552] Output: Analyzed results (e.g., inference results from an AI model)

[0553] Step 4:

[0554] The server formats the inference results in JSON format and returns them to the terminal as an HTTP response.

[0555] Input: The inference result of the AI ​​model (e.g., "Install part A on the right side")

[0556] Output: HTTP response to the terminal (e.g., inference results in JSON format)

[0557] Step 5:

[0558] The terminal analyzes the received inference results and outputs them to the user by display or voice.

[0559] Input: Inference results returned from the server (e.g., JSON data)

[0560] Output: Display or audio output to the user interface (e.g., "Please install part A on the right side" is displayed on the HMD display and also read aloud)

[0561] Step 6:

[0562] The user performs the task by following the displayed information and audio guidance.

[0563] Input: Work instructions provided by the device (e.g., instructions displayed on the HMD display)

[0564] Output: The action performed by the user (e.g., attaching part A to the right side)

[0565] In this way, each step works together, allowing the user to perform the task efficiently and the system to provide feedback in real time.

[0566] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0567] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, combines the inference results with an emotion engine, and returns the results to the terminal in a form that reflects the user's emotions and outputs them as display or audio.

[0568] 1. User operations

[0569] A user launches an application on their smartphone. The application's main screen appears, and the user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record video and then taps the "Stop" button at the desired time to save the data.

[0570] 2. Terminal Processing

[0571] The device receives data entered by the user. The acquired data is temporarily stored on the device, and the data path and metadata are prepared. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request, attaches the saved file, and sends it to the server's URL. The request also includes any necessary authentication information.

[0572] 3. Server analysis process

[0573] The server receives the data sent from the device. It receives the data and saves it as a file. The saved data is then input into the AI ​​model for analysis. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. In addition, the emotion engine analyzes elements such as the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0574] 4. Server emotion analysis and inference results generation

[0575] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference result. For example, if a child gestures to say "I'm hungry" while also making a dissatisfied expression, a detailed inference result such as "I'm hungry but in a bad mood" will be obtained. The generated results are formatted in JSON format.

[0576] 5. Terminal handling and output to the user

[0577] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can confirm the intention and emotion inferred by the server through text displayed on the screen or audible reading.

[0578] Specific examples

[0579] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and records a video of the child pointing. The device receives the video data, saves it, and sends it to the server. The server analyzes the received video and infers that the child is "hungry," while the emotion engine analyzes the child's dissatisfied facial expression and determines that the child is "in a bad mood." This result is formatted as "hungry but in a bad mood" and sent back to the device. The device receives the inference result and tells the user "hungry but in a bad mood" on the screen or by voice. The user can appropriately understand the child's intentions and emotions and respond quickly.

[0580] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, while the content is inferred as "Where is the toilet?", the emotion engine simultaneously detects the emotion of impatience and generates a detailed result such as "I need to go to the toilet urgently." This result is then sent back to the device, which then informs the user by displaying or audibly saying "I need to go to the toilet urgently," enabling smooth communication with the foreigner.

[0581] In this way, the system of the present invention can analyze non-verbal information and convey intentions and emotions to the user in an easy-to-understand manner, thereby solving various communication problems.

[0582] The processing flow will be explained below.

[0583] Step 1:

[0584] A user launches an app on their smartphone. The user taps the app icon to launch it, and the main screen appears. The user selects an option to capture video, images, or audio depending on the purpose of use.

[0585] Step 2:

[0586] The user captures video, image, or audio data. For example, the user taps the "record" button to record a video of the child's gestures, and then taps the "stop" button to save the data. Similarly, images and audio are captured appropriately.

[0587] Step 3:

[0588] The device receives the acquired data, temporarily stores the data acquired through sensors such as the camera and microphone, and records the file path and metadata of the data.

[0589] Step 4:

[0590] The device sends the acquired data to the server by creating an HTTP POST request and sending the acquired data file along with the data to the server's URL. The request also includes any necessary authentication information.

[0591] Step 5:

[0592] The server receives the data sent from the device, saves the received data as a file, and prepares it for data analysis.

[0593] Step 6:

[0594] The server begins the analysis process, using AI models such as image and voice recognition to analyze the data and infer intent from gestures, language, pictures, and humming.

[0595] Step 7:

[0596] The server uses an emotion engine to analyze the user's emotions, and estimates their emotional state by analyzing the acquired data, such as the user's facial expressions, tone of voice, and body movements.

[0597] Step 8:

[0598] The server combines the inference results and the emotion analysis results to generate the final inference result. For example, by combining "hungry," the gesture analysis result, and the emotion analysis result "bad mood," the detailed inference result "hungry but bad mood" is obtained.

[0599] Step 9:

[0600] The server returns the final inference results to the device, which then formats the integrated inference results in JSON format and sends them to the device as an HTTP response.

[0601] Step 10:

[0602] The device analyzes the inference results received from the server, parses the received JSON data, obtains the inference results, and prepares them to be displayed or output to the user in an appropriate format.

[0603] Step 11:

[0604] The device displays or outputs the inference result to the user. It can be displayed as text on the screen or read aloud to the user. For example, the inference result "I'm hungry, but I'm in a bad mood" can be displayed or output as audio.

[0605] Step 12:

[0606] The user can check the inference results displayed or output as voice and understand the other person's intentions and emotions. For example, based on the analysis result that a child is "hungry but in a bad mood," the user can take appropriate action.

[0607] In this way, the system provides multifunctional support to help users accurately understand their intentions and respond quickly.

[0608] Example 2

[0609] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0610] Conventional systems lack the means to analyze non-verbal information and accurately convey the user's intentions and emotions. Furthermore, there are problems with time lags and reduced recognition accuracy in the process from sending the data acquired by the user to the server, receiving the analysis results, and outputting them as display or voice. Therefore, the present invention aims to solve these problems and provide a system that analyzes non-verbal information with high accuracy and provides rapid feedback to the user.

[0611] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0612] In this invention, the server includes means for analyzing the acquired data and inferring its contents, means for estimating the user's emotions using an emotion engine based on the inference results, and means for returning the inference results to the terminal, thereby improving the accuracy and speed of analyzing non-verbal information and enabling quick and accurate feedback to the user.

[0613] A "terminal" is a device that can be operated by a user, acquires video, images, or audio, transmits the acquired data to a server, receives the inference results returned from the server, and displays or outputs them as audio.

[0614] "Data" refers to information in the form of video, image or audio files acquired by the terminal.

[0615] A "server" is a computer system that receives data sent from a terminal, analyzes it, and returns the analysis results to the terminal.

[0616] "Inference results" refer to the estimated results obtained by the server when analyzing data, and include information about the user's intentions and actions.

[0617] An "HTTP POST request" is a request format that uses a web protocol for a terminal to send data to a server.

[0618] An "emotion engine" is a software system that analyzes information such as a user's facial expressions, tone of voice, and body movements based on the analysis results, and estimates the user's emotions.

[0619] The "JSON format" is a lightweight data description format for structuring, storing, and transferring data, and is a format that is mainly composed of key-value pairs.

[0620] The present invention is a system that acquires video, images, or audio using a terminal operated by a user, sends the data to a server for analysis, and combines the analysis results with an emotion engine to return the results to the terminal in a form that reflects the user's emotions, and outputs them as a display or audio.

[0621] To implement this system, the following hardware and software are used.

[0622] Hardware and software used

[0623] Device: Mobile device such as a smartphone or tablet

[0624] Server: A high-performance computer server

[0625] Software: Mobile application, HTTP communication protocol, AI analysis model (e.g., OpenCV, TensorFlow), emotion engine (e.g., Emotion API), JSON format user interface, and text-to-speech function (e.g., Google Text-to-Speech)

[0626] System Operation

[0627] A user launches a smartphone application and selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record a video. When finished, the user taps the "Stop" button to save the data.

[0628] The device temporarily stores the retrieved data in its internal storage, prepares the data path and metadata, and then makes an HTTP POST request to send the attached data file and authentication information to the server.

[0629] The server stores the received data in a specified directory and inputs it into the AI ​​model for analysis. The AI ​​model analyzes the transmitted video, image, and audio data, understanding non-verbal information such as gestures, foreign languages, drawings, and humming to generate analysis results. Furthermore, it uses an emotion engine to analyze the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0630] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference results, which are formatted in JSON format.

[0631] The server sends the generated inference results to the device via an HTTP POST request. The device analyzes the received JSON-formatted inference results and displays or outputs them to the user. The user can check the analysis results through text displayed on the screen or audio readout, and respond quickly.

[0632] Specific examples

[0633] For example, this system is used when a user (parent) uses a smartphone to record a pointing action by a child who is not good at speaking. The device receives the video data, temporarily stores it, and sends it to the server. The server analyzes the received video and generates the inference result "hungry," while also using the emotion engine to detect the child's dissatisfied facial expression. This results in a more detailed result: "hungry but in a bad mood." This result is sent back to the device, which tells the user "hungry but in a bad mood" by display or voice.

[0634] As another example, when a foreigner tries to communicate something using a combination of gestures and a foreign language, the user takes a photo of the situation with their smartphone. The device then sends the captured data to the server, which analyzes it to infer the intention of "Where is the toilet?" and uses an emotion engine to detect the emotion of urgency, generating the inference result of "I need to go to the toilet urgently." This result is then sent back to the device, which then displays or speaks to the user the message "I need to go to the toilet urgently." This enables smooth communication with foreigners.

[0635] Prompt Sentence Examples

[0636] "Explain how you can analyze a child's gestures and facial expressions to infer their intentions and emotions."

[0637] Based on this prompt, the generative AI model can explain the specific analysis method and how to communicate the results.

[0638] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0639] System processing steps

[0640] Step 1: User interaction

[0641] A user launches a smartphone application. The application's main screen appears. The user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to capture video, and the "Stop" button to save the data. The input is the video, image, or audio data captured by the user's operation. The output is the saved data file.

[0642] Step 2: Prepare the device to send data

[0643] The device temporarily stores the data acquired from the user in its internal storage. It prepares the data path and metadata and creates an HTTP POST request. The input is the acquired data file. The output is an HTTP POST request with the data file attached.

[0644] Step 3: Send data by device

[0645] The device includes the data file and authentication information in the prepared HTTP POST request and sends it to the specified URL on the server. Specifically, the request is sent using a request library (for example, the Python requests library). The input is the HTTP POST request and authentication information. The output is the result of sending the request to the server.

[0646] Step 4: Server receives data

[0647] The server receives an HTTP POST request sent from the device and saves the data file in the specified directory. The input is the HTTP POST request from the device. The output is the saved data file.

[0648] Step 5: Data analysis by the server

[0649] The server inputs the saved data files into the AI ​​model for analysis. For example, in the case of video data, analysis is performed using OpenCV or TensorFlow to identify gestures and foreign languages. The input is the saved data file. The output is text data (e.g., "I'm hungry") that is the result of the analysis.

[0650] Step 6: Sentiment analysis by server

[0651] The server uses an emotion engine based on the analysis results to estimate the user's emotion. For example, it uses Microsoft's Emotion API to analyze facial expressions and detect emotions such as "dissatisfied." The input is the text data of the analysis results. The output is the estimated emotion (for example, "dissatisfied").

[0652] Step 7: Server generates inference results

[0653] The server combines the analysis results and emotion estimation results to generate the final inference result, which is then formatted in JSON. The input is the text data of the analysis results and the emotion estimation result. The output is the inference result in JSON format (for example, {"intention": "hungry", "emotion": "unhappy"}).

[0654] Step 8: Server sends inference results

[0655] The server sends the inference results formatted in JSON to the terminal via an HTTP POST request. The input is the inference results in JSON format. The output is the result sent to the terminal.

[0656] Step 9: Receive and display the inference results on the device

[0657] The device receives an HTTP POST request sent from the server and parses the inference results in JSON format. It then displays the results to the user as text or audio. Specifically, it uses a text-to-speech conversion function such as Google Text-to-Speech. The input is the inference results in JSON format from the server. The output is the display or audio output for the user.

[0658] (Application example 2)

[0659] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] In conventional security systems, surveillance cameras and sensors primarily rely on video data and specific behavioral patterns to detect abnormal behavior. However, this alone does not allow for an accurate understanding of the emotions of the monitored individual, limiting the accuracy of identifying suspicious individuals. For example, even if no suspicious behavior is observed, changes in emotions may be overlooked, potentially resulting in potential danger being overlooked. Another issue is that the inability to respond quickly based on emotional changes places a heavy burden on security personnel.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0662] In this invention, the server includes an emotion analysis means for analyzing acquired data and inferring emotions, a means for notifying security personnel based on the detection of suspicious behavior or emotions, and a means for returning the inference results to the terminal, thereby enabling the emotions of monitored persons to be accurately analyzed in real time and quickly notified to security personnel.

[0663] A "user-operated terminal" is an electronic device that is manually operated by a user to acquire video, images, or audio data and process that data.

[0664] "Means for acquiring video, images or audio" refers to a function for collecting video, still images and audio data using sensors such as cameras and microphones.

[0665] The "means for sending acquired data to a server" is a function for transferring acquired data to a server via the Internet using a communication protocol such as an HTTP POST request.

[0666] The "means for receiving inference results from the server" is a function for receiving analysis results and data returned from the server.

[0667] "Means for displaying or outputting received inference results as audio" refers to a function for displaying text or images on the terminal screen or transmitting information as audio from the speaker.

[0668] "Means of acquiring data in real time from surveillance cameras and sensors" refers to a function for acquiring video and audio in real time using installed surveillance cameras and various sensors.

[0669] "Emotion analysis means for analyzing acquired data and inferring emotions" is a function for inferring emotions by analyzing the facial expressions and tone of voice of the monitored person based on the acquired video and audio data.

[0670] "Means for notifying when suspicious behavior or emotions are detected" is a function that sends warnings and notifications to security personnel when an abnormality or suspicious situation is detected based on inferred emotions or behavior.

[0671] "Means of analyzing data and inferring its contents" refers to a function that uses certain rules and AI algorithms to understand the contents of acquired data and derive inferential results.

[0672] The "means for returning the inference results to the terminal" is a function for correcting the analysis results and sending them back to the original terminal.

[0673] "Means generated based on gestures, foreign language, drawings, or humming" refers to a function that analyzes a person's physical movements, speech in a specific language, drawings, humming, etc., and generates analysis results based on the content.

[0674] "Means generated based on real-time emotion analysis" refers to a function that instantly analyzes changes in emotions based on data obtained in real time and predicts the results.

[0675] This invention is a system that uses a terminal operated by a user to acquire video, images, or audio, transmits the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.In addition, it also has a means for acquiring data and analyzing emotions in real time, and quickly notifying security personnel of suspicious behavior or emotions.

[0676] Hardware and Software

[0677] Hardware:

[0678] User devices such as smartphones and tablets

[0679] Surveillance cameras and sensors

[0680] server

[0681] software:

[0682] Smartphone and tablet application software (compatible with iOS and Android)

[0683] Server software (web frameworks such as Flask)

[0684] Sentiment Analysis Library

[0685] OpenCV (video processing library)

[0686] Acquiring and Sending Data

[0687] A user launches a surveillance system or smartphone application. The user device uses a camera or microphone to capture video, images, or audio. The captured data is temporarily stored on the device and then sent to the server using an HTTP POST request, including any necessary authentication information. Data is also captured in real time from surveillance cameras and sensors and sent to the server.

[0688] Data analysis

[0689] The server receives the transmitted data and saves it as a file. The saved data is then analyzed using an AI model. The AI ​​model understands non-verbal information from video, image, and audio data and generates analysis results. Furthermore, an emotion analysis tool estimates the user's emotions from facial expressions, tone of voice, body movements, etc.

[0690] Generating and notifying inference results

[0691] The emotions estimated by the emotion analysis method are added to the analysis results to generate the final inference results. For example, emotions such as "anxiety" or "impatience" are estimated from video footage captured by a surveillance camera, and if suspicious behavior is detected, the detailed inference results are formatted in JSON format. If an abnormality is detected, the analysis results are notified to security personnel in real time.

[0692] Output to the user

[0693] The user device receives the inference results returned from the server. The received results are analyzed and displayed or output as audio to the user. The user can confirm the intent and emotion inferred by the server through text displayed on the screen or audio readout. Security personnel can also respond quickly by checking the notification.

[0694] Examples of concrete examples and prompts

[0695] For example, if a suspicious person in a location where a surveillance camera is installed shows an anxious expression, the analysis result will be estimated as "anxiety / anxiety." This information will be sent to security personnel along with a notification that "There is a person with an anxious expression in this area."

[0696] Prompt Sentence Examples

[0697] Analyze the video data uploaded by users, understand non-verbal information (gestures and facial expressions), and analyze emotions using an emotion engine.

[0698] example:

[0699] 1. Video data: [Video file path]

[0700] 2. Estimated emotion: [Estimation result]

[0701] In this way, by linking the user terminal and the server, this invention enables real-time data acquisition and emotion analysis, thereby enhancing security.

[0702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0703] Step 1:

[0704] A user launches a smartphone application and uses the camera or microphone to capture video, images, or audio. The input data can be video, image, or audio files. This data is temporarily stored on the device. Specifically, the user taps the record button in the app to start recording, and then taps the stop button to save the data.

[0705] Step 2:

[0706] The device sends the acquired data to the server using an HTTP POST request. The input is a saved video, image, or audio file, and the output is a request to the server. Authentication information is also added. Specifically, the device creates a request by specifying the path of the acquired file and sends it to the server's URL.

[0707] Step 3:

[0708] The server stores the received data and inputs it into the AI ​​model for analysis. The input is the submitted video, image, or audio file, and the output is the analysis result. Specifically, the server saves the file in a specified directory and processes the input data using the analysis library.

[0709] Step 4:

[0710] The server analyzes the data using an AI model and infers the user's emotions through an emotion analysis tool. The input is preprocessed data, and the output is the result of the emotion analysis. Specifically, the AI ​​model analyzes gestures and facial expressions, while the emotion engine analyzes voice tone and body movements.

[0711] Step 5:

[0712] The server adds the emotion analysis results to the inference results, generates the final inference results in JSON format, and returns them to the device. The inputs are the analysis results and emotion analysis results, and the output is the JSON data of the final inference results. Specifically, the server formats the data and creates a response to send back to the device.

[0713] Step 6:

[0714] The device analyzes the received inference results and displays or outputs them to the user. The input is inference result data in JSON format, and the output is displayed text or spoken audio. Specifically, the device parses the JSON data and displays it in a text view, or uses a speech synthesis engine to communicate it to the user.

[0715] Step 7:

[0716] Data is acquired in real time from surveillance cameras and sensors and sent to a server in the same way. The input is real-time video and audio data, and the output is a request to the server. Specifically, a surveillance camera system that operates continuously collects data and sends it to the server at regular intervals.

[0717] Step 8:

[0718] If the server detects suspicious behavior or emotions from the data analyzed in real time, it notifies security personnel. The input is the real-time analysis results, and the output is the notification content. Specifically, when an abnormality is detected, an SMS, email, or in-app push notification is sent.

[0719] In this way, each processing step repeats the process of data acquisition, transmission, analysis, and notification, building a system that achieves enhanced security.

[0720] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0722] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0723] [Third embodiment]

[0724] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0725] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0726] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0727] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0728] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0729] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0730] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0731] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0732] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0733] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0734] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0735] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0736] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0737] 1. User operations

[0738] A user launches a smartphone application. The application's main screen appears, and the user navigates through the menu to capture video, images, or audio. For example, the user taps the "Record" button to record video, and then taps the "Stop" button to save the data when finished.

[0739] 2. Terminal Processing

[0740] The device receives data entered by the user. It temporarily stores data acquired using sensors such as a camera or microphone, obtains the data path, and prepares it for subsequent processing. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request and sends it with the saved file attached.

[0741] 3. Server Processing

[0742] The server receives data sent from the device. It analyzes the received data and uses a pre-trained AI model to infer its contents. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. The analysis results are formatted and sent back to the device in JSON format.

[0743] 4. Terminal handling and output to the user

[0744] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0745] Specific examples

[0746] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and takes a video of the child's pointing movements. The device receives the video data, saves it, and sends it to a server. The server analyzes the received video, infers the child's intention, such as "I'm hungry," and sends the inference result back to the device. The device receives the inference result and displays it on the screen or tells the user "I'm hungry" by voice. In this way, the user can properly understand the child's intention and take appropriate action.

[0747] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, if the content is inferred as "Where is the toilet?", the result is sent back to the device, and the user is informed by displaying or audibly saying "Where is the toilet?", enabling smooth communication with the foreigner.

[0748] In this way, the system of the present invention can analyze non-verbal information and convey intentions to the user in an easily understandable form, thereby solving various communication problems.

[0749] The processing flow will be explained below.

[0750] Step 1:

[0751] A user launches an app on their smartphone. The user taps an icon to launch the app, and the app's main screen appears. The user selects an option to capture video, images, or audio.

[0752] Step 2:

[0753] The user performs an operation to capture video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button at the desired time to save the data. The same operation is performed for images and audio.

[0754] Step 3:

[0755] The device receives data entered by the user. The acquired data is temporarily stored in the device, and the data path and metadata are prepared.

[0756] Step 4:

[0757] The device sends the acquired data to the server by creating an HTTP POST request to the server URL, attaching the acquired data file, and including any necessary authentication information.

[0758] Step 5:

[0759] The server receives the data sent from the device, saves it as a file, and inputs it into the AI ​​model for analysis.

[0760] Step 6:

[0761] The server analyzes the received data. It uses an AI model to analyze the data and understand non-verbal information such as gestures, foreign languages, drawings, and humming, and generates an analysis result. The generated result is formatted in JSON format.

[0762] Step 7:

[0763] The server returns the inference results to the device. The inference results are formatted in JSON and sent to the device as an HTTP response.

[0764] Step 8:

[0765] The device analyzes the inference results received from the server, parses the received JSON data to obtain the analysis results, and checks whether the inference results are in the correct format.

[0766] Step 9:

[0767] The device displays or outputs the inference results to the user. The analysis results are displayed as text on the screen or, if necessary, conveyed to the user using a text-to-speech function.

[0768] Step 10:

[0769] The user can check the displayed or spoken inference results and understand the other person's intention. For example, if a child's gesture indicates that they are hungry, the user can take appropriate action based on that information.

[0770] Through these steps, users can understand the meaning of videos, images, and audio that contain non-verbal information, enabling them to communicate appropriately.

[0771] Example 1

[0772] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0773] With the diversification of modern communication, there is a growing need for systems that can accurately analyze non-verbal information such as gestures, foreign languages, and voice, and convey it to users in an easy-to-understand manner. However, existing systems have difficulty efficiently analyzing non-verbal information in real time and providing appropriate feedback. The objective of this invention is to provide a system that analyzes video, image, and voice data acquired by a user on a server, quickly returns the results to a terminal, and displays or outputs the results to the user as voice.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0775] In this invention, the server includes means for analyzing acquired data and inferring its contents, means for returning the inference results to the terminal, means for analyzing the data using a pre-trained AI model, and means for returning the analysis results to the terminal in JSON format, thereby enabling quick and accurate analysis of non-verbal information acquired from the user and providing appropriate feedback to the user.

[0776] A "user-operated terminal" is a device that is directly operated by a user to obtain video, images, or audio.

[0777] "Means of acquisition" refers to the mechanism for collecting video, image, or audio data using sensors such as cameras and microphones installed on the device.

[0778] "Means of sending to the server" refers to the protocols and procedures for transferring the acquired video, image, and audio data to the server via the network.

[0779] The "means for receiving inference results" is a function for receiving data on the analysis and inference results returned from the server.

[0780] "Means for displaying or outputting by voice" refers to the ability to display the received inference results on the terminal screen in text or graphical form or to play them back by voice.

[0781] "Temporary storage means" refers to a mechanism for temporarily storing acquired data in the device's memory or storage.

[0782] "Means for creating an HTTP POST request" is the process of generating a request to send the retrieved data to a server using the HTTP protocol.

[0783] "Means for analyzing data received from the server" refers to technology for interpreting data returned from the server and converting it into a form that can be used by the user.

[0784] A "pre-trained AI model" is an algorithm that has been trained for a specific task and performs analysis and inference based on input data.

[0785] The JSON format is a lightweight data exchange format for storing and transferring structured data.

[0786] MODE FOR CARRYING OUT THE INVENTION

[0787] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[0788] Hardware and Software Configuration

[0789] Device:

[0790] The terminal is a mobile device such as a smartphone or tablet. This terminal is equipped with a camera, microphone, and internal storage, which allows it to capture and temporarily store video, image, and audio data. It also requires a screen to display feedback and a speaker for audio output. The terminal software includes libraries for creating HTTP requests and parsing JSON data.

[0791] server:

[0792] The server receives data using a web server (e.g., Nginx, Tomcat, etc.), analyzes the data using Python scripts and runs pre-trained AI models (e.g., TensorFlow, PyTorch), and also requires a library to send the analysis results back to the device in JSON format.

[0793] Operational Overview

[0794] 1. Data Acquisition and Transmission

[0795] A user launches a smartphone application and captures video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button to save the data. The device temporarily stores the captured data in its internal storage and creates an HTTP POST request to send to the server.

[0796] 2. Data Analysis

[0797] The server receives data sent from the device. It analyzes the received data using a Python script and uses a pre-trained AI model (TensorFlow or PyTorch) to infer the content of the data. The AI ​​model understands and analyzes non-verbal information such as gestures, foreign languages, drawings, and humming. The analysis results are formatted in JSON and sent back to the device.

[0798] 3. Receiving and displaying inference results

[0799] The device receives the inference results in JSON format returned from the server. It analyzes the received results and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[0800] Specific operation example

[0801] Example 1: Child gesture analysis

[0802] To analyze the gestures of a child who is not good at speaking, the user (parent) launches an app on their smartphone and records a video of the child's pointing movements. The device temporarily saves the video data and sends it to the server using an HTTP POST request. The server receives the video data, analyzes it with a Python script, and uses a pre-trained TensorFlow model to infer the child's intent, such as "I'm hungry." The analysis results are returned in JSON format, and the device receives the results, displaying "I'm hungry" on the screen or outputting it as voice. The user can then appropriately understand the child's intent and respond accordingly.

[0803] Example 2: Communicating with foreigners

[0804] When a foreigner tries to communicate something using a combination of gestures and a foreign language, the user films the situation with their smartphone. The device then sends the filmed video data to a server. The server then analyzes the received data using a Python script and uses a pre-trained TensorFlow model to infer the intent, "Where is the toilet?" The inference result is sent back to the device in JSON format, and the device receives the result and displays "Where is the toilet?" on the screen or outputs it as voice. This allows users to communicate smoothly with foreigners.

[0805] Example prompt sentence:

[0806] "The system takes a video of the child's pointing action, sends the video data to a server, and returns the analysis results. The system helps understand the child's intentions by displaying the inference results on the screen and outputting them as audio."

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] The user launches the application on their smartphone. The main screen of the application is displayed, and the user operates the menu to acquire data.

[0810] Specific operation: Tap the "Record" button from the app's startup screen and select video mode.

[0811] Input: User actions

[0812] Output: Instruction to start data acquisition

[0813] Step 2:

[0814] The device accepts instructions from the user and acquires video, image, or audio data. It collects data using sensors such as a camera and microphone and temporarily stores it in its internal storage.

[0815] Specific operation: Record with the camera, and when finished, tap the "Stop" button to save the data.

[0816] Input: Data from the camera or microphone

[0817] Output: Stored media data

[0818] Step 3:

[0819] The device retrieves the path to the stored data and creates an HTTP POST request for sending it to the server, including the retrieved media data.

[0820] What it does: Creates an HTTP POST request and attaches a video file.

[0821] Input: Path to the saved media data

[0822] Output: Request sent to server

[0823] Step 4:

[0824] The server receives the data sent from the terminal. A web server (e.g., Nginx, Tomcat) receives the request and temporarily stores the data.

[0825] Specific operation: After receiving the data, save it in the specified folder.

[0826] Input: HTTP POST request from the terminal

[0827] Output: Data stored in the server

[0828] Step 5:

[0829] The server analyzes the received data, runs a Python script, and uses pre-trained AI models to analyze non-verbal information such as gestures, foreign words, drawings, and humming, and generates analysis results.

[0830] Specific operations: Run AI models using TensorFlow and PyTorch to analyze and infer data.

[0831] Input: Stored media data

[0832] Output: Analysis results (gesture recognition results as a 3D array, translation results as text, etc.)

[0833] Step 6:

[0834] The server formats the generated analysis results and returns them to the terminal in JSON format, sending JSON data containing the analysis results as an HTTP response.

[0835] Specific operation: Formats data into JSON format and creates an HTTP response.

[0836] Input: Parsed data

[0837] Output: Parsed results in JSON format

[0838] Step 7:

[0839] The device receives the JSON data returned from the server, parses the HTTP response, and extracts the data.

[0840] Specific operation: Parse the JSON data and obtain the required information.

[0841] Input: Parsed result in JSON format

[0842] Output: Text and audio data

[0843] Step 8:

[0844] The device displays or outputs the extracted analysis results to the user as text on the screen or as voice using a speech synthesis engine.

[0845] Specific action: Displaying text such as "I'm hungry" on the screen or reading it aloud.

[0846] Input: Text or audio data

[0847] Output: Feedback to the user (visual or auditory)

[0848] Examples:

[0849] Example of child gesture analysis:

[0850] The user launches the smartphone app and records the child's pointing behavior. The device temporarily saves the video data, generates an HTTP POST request, and sends it to the server. The server uses TensorFlow to analyze the gesture, infers that the child is hungry, and returns the result in JSON format. The device receives the JSON data and displays the text on the screen. The user can then look at the screen to understand the child's intention.

[0851] Examples of communication with foreigners:

[0852] The user launches the app and records the foreigner's gestures and speech in the foreign language. The device sends the data to the server, which analyzes it and returns the inference result, "Where is the toilet?" The device receives the result and displays it as text or outputs it as voice. The user can use this information to guide the foreigner.

[0853] (Application example 1)

[0854] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0855] In industrial environments such as factories, there is a growing need for support systems to help workers efficiently receive work instructions. With conventional methods, it can take time for workers to understand work instructions, which can lead to reduced work efficiency. Additionally, when language barriers or non-verbal communication are required, it can be difficult to receive appropriate feedback. This can affect overall productivity and safety.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0857] In this invention, the server includes a means for acquiring video, image, or audio, a means for transmitting the acquired data to the server, a means for receiving inference results from the server, a means for displaying or outputting the received inference results as audio, a means for analyzing hand gestures and voice commands in an industrial environment, and a means for generating work instructions based on the hand gestures and voice commands, thereby enabling effective provision of work instructions and feedback within a factory, improving work efficiency and overall productivity.

[0858] A "terminal" is an electronic device that is operated by a user to acquire video, images, or audio and transmit it to a server.

[0859] A "server" is a computer system that analyzes the acquired data and returns the inference results to the terminal.

[0860] "Inference results" are results based on the meaning and content of data obtained by the server through analysis.

[0861] "Display" refers to the act and means of visually conveying information to a user.

[0862] "Audio output" refers to the act and means of transmitting information to the user audibly.

[0863] "Industrial environment" refers to factories, manufacturing sites, and other workplaces where machinery and equipment are operated.

[0864] A "gesture" is an act of conveying intentions or instructions using hand or body movements.

[0865] A "voice command" is a voice signal used to give instructions to the system using voice.

[0866] "Work instructions" are information that instructs the procedures and actions for performing a specific task.

[0867] To implement this invention as a factory work support system, a server, a terminal operated by a user, and a network environment connecting these are required.

[0868] 1. Hardware and Software Requirements

[0869] Hardware:

[0870] Terminal: For example, a head-mounted display (HMD) or a smartphone. It is desirable that these have a built-in camera and microphone.

[0871] Server: A high-performance computer that must be capable of executing inference processing for AI models in real time.

[0872] software:

[0873] Flask: A Python-based web framework for receiving and processing server-side data.

[0874] PyTorch: A deep learning library that contains models for analyzing the acquired data.

[0875] HTTP request library: Used to send and receive data between the terminal and the server at high speed.

[0876] 2. System Overview

[0877] The user captures video, images, or audio on the device. The captured data is sent to the server via an HTTP request. The server receives the data and analyzes it using a generative AI model. The inference results are returned to the device in JSON format, which the device then displays or outputs as audio. Particularly in industrial environments, analyzing gestures and voice commands and generating appropriate work instructions can improve work efficiency.

[0878] 3. Specific Examples

[0879] Example 1: Providing work instructions

[0880] A factory worker puts on the HMD and issues a voice command, saying, "What do I do next?" The HMD's built-in microphone picks up the voice and sends the data to the server. The server analyzes the voice and generates a work instruction, such as "Install part A on the right side." This instruction is displayed on the HMD display and is also read aloud.

[0881] Example 2: Support for safety confirmation

[0882] When a worker is performing a dangerous task, he or she signals with a hand gesture, "Please check." The HMD camera captures the gesture and sends the data to the server. The server analyzes the gesture and generates a warning message, such as "Be careful, it's hot." This message is displayed on the HMD display and an audio warning is also issued.

[0883] 4. Examples of prompts

[0884] The following prompts can be used to make the necessary inferences for the generative AI model:

[0885] You are a factory manager robot. A worker is asking for instructions. He has part A in his hand. What should he do next?

[0886] In this way, the present invention provides the tools necessary to support practical tasks in the work environment and provide an efficient and safe environment.

[0887] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0888] Step 1:

[0889] An application is launched on the terminal operated by the user to acquire video, images, or audio.

[0890] Input: Video, images, and audio data (e.g., worker makes hand gestures or voice commands)

[0891] Output: The captured data (e.g., worker gestures and voice are captured)

[0892] Step 2:

[0893] The device temporarily stores the acquired data and sends it to the server via an HTTP request.

[0894] Input: Data acquired in the previous step (e.g., data captured by the HMD camera or microphone)

[0895] Output: Data sent to the server (e.g., sent in an HTTP request)

[0896] Step 3:

[0897] The server passes the received data to the AI ​​model for analysis.

[0898] Input: Data sent from the device (e.g., captured gestures and voice data)

[0899] Output: Analyzed results (e.g., inference results from an AI model)

[0900] Step 4:

[0901] The server formats the inference results in JSON format and returns them to the terminal as an HTTP response.

[0902] Input: The inference result of the AI ​​model (e.g., "Install part A on the right side")

[0903] Output: HTTP response to the terminal (e.g., inference results in JSON format)

[0904] Step 5:

[0905] The terminal analyzes the received inference results and outputs them to the user by display or voice.

[0906] Input: Inference results returned from the server (e.g., JSON data)

[0907] Output: Display or audio output to the user interface (e.g., "Please install part A on the right side" is displayed on the HMD display and also read aloud)

[0908] Step 6:

[0909] The user performs the task by following the displayed information and audio guidance.

[0910] Input: Work instructions provided by the device (e.g., instructions displayed on the HMD display)

[0911] Output: The action performed by the user (e.g., attaching part A to the right side)

[0912] In this way, each step works together, allowing the user to perform the task efficiently and the system to provide feedback in real time.

[0913] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0914] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, combines the inference results with an emotion engine, and returns the results to the terminal in a form that reflects the user's emotions and outputs them as display or audio.

[0915] 1. User operations

[0916] A user launches an application on their smartphone. The application's main screen appears, and the user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record video and then taps the "Stop" button at the desired time to save the data.

[0917] 2. Terminal Processing

[0918] The device receives data entered by the user. The acquired data is temporarily stored on the device, and the data path and metadata are prepared. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request, attaches the saved file, and sends it to the server's URL. The request also includes any necessary authentication information.

[0919] 3. Server analysis process

[0920] The server receives the data sent from the device. It receives the data and saves it as a file. The saved data is then input into the AI ​​model for analysis. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. In addition, the emotion engine analyzes elements such as the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0921] 4. Server emotion analysis and inference results generation

[0922] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference result. For example, if a child gestures to say "I'm hungry" while also making a dissatisfied expression, a detailed inference result such as "I'm hungry but in a bad mood" will be obtained. The generated results are formatted in JSON format.

[0923] 5. Terminal handling and output to the user

[0924] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can confirm the intention and emotion inferred by the server through text displayed on the screen or audible reading.

[0925] Specific examples

[0926] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and records a video of the child pointing. The device receives the video data, saves it, and sends it to the server. The server analyzes the received video and infers that the child is "hungry," while the emotion engine analyzes the child's dissatisfied facial expression and determines that the child is "in a bad mood." This result is formatted as "hungry but in a bad mood" and sent back to the device. The device receives the inference result and tells the user "hungry but in a bad mood" on the screen or by voice. The user can appropriately understand the child's intentions and emotions and respond quickly.

[0927] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, while the content is inferred as "Where is the toilet?", the emotion engine simultaneously detects the emotion of impatience and generates a detailed result such as "I need to go to the toilet urgently." This result is then sent back to the device, which then informs the user by displaying or audibly saying "I need to go to the toilet urgently," enabling smooth communication with the foreigner.

[0928] In this way, the system of the present invention can analyze non-verbal information and convey intentions and emotions to the user in an easy-to-understand manner, thereby solving various communication problems.

[0929] The processing flow will be explained below.

[0930] Step 1:

[0931] A user launches an app on their smartphone. The user taps the app icon to launch it, and the main screen appears. The user selects an option to capture video, images, or audio depending on the purpose of use.

[0932] Step 2:

[0933] The user captures video, image, or audio data. For example, the user taps the "record" button to record a video of the child's gestures, and then taps the "stop" button to save the data. Similarly, images and audio are captured appropriately.

[0934] Step 3:

[0935] The device receives the acquired data, temporarily stores the data acquired through sensors such as the camera and microphone, and records the file path and metadata of the data.

[0936] Step 4:

[0937] The device sends the acquired data to the server by creating an HTTP POST request and sending the acquired data file along with the data to the server's URL. The request also includes any necessary authentication information.

[0938] Step 5:

[0939] The server receives the data sent from the device, saves the received data as a file, and prepares it for data analysis.

[0940] Step 6:

[0941] The server begins the analysis process, using AI models such as image and voice recognition to analyze the data and infer intent from gestures, language, pictures, and humming.

[0942] Step 7:

[0943] The server uses an emotion engine to analyze the user's emotions, and estimates their emotional state by analyzing the acquired data, such as the user's facial expressions, tone of voice, and body movements.

[0944] Step 8:

[0945] The server combines the inference results and the emotion analysis results to generate the final inference result. For example, by combining "hungry," the gesture analysis result, and the emotion analysis result "bad mood," the detailed inference result "hungry but bad mood" is obtained.

[0946] Step 9:

[0947] The server returns the final inference results to the device, which then formats the integrated inference results in JSON format and sends them to the device as an HTTP response.

[0948] Step 10:

[0949] The device analyzes the inference results received from the server, parses the received JSON data, obtains the inference results, and prepares them to be displayed or output to the user in an appropriate format.

[0950] Step 11:

[0951] The device displays or outputs the inference result to the user. It can be displayed as text on the screen or read aloud to the user. For example, the inference result "I'm hungry, but I'm in a bad mood" can be displayed or output as audio.

[0952] Step 12:

[0953] The user can check the inference results displayed or output as voice and understand the other person's intentions and emotions. For example, based on the analysis result that a child is "hungry but in a bad mood," the user can take appropriate action.

[0954] In this way, the system provides multifunctional support to help users accurately understand their intentions and respond quickly.

[0955] Example 2

[0956] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0957] Conventional systems lack the means to analyze non-verbal information and accurately convey the user's intentions and emotions. Furthermore, there are problems with time lags and reduced recognition accuracy in the process from sending the data acquired by the user to the server, receiving the analysis results, and outputting them as display or voice. Therefore, the present invention aims to solve these problems and provide a system that analyzes non-verbal information with high accuracy and provides rapid feedback to the user.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0959] In this invention, the server includes means for analyzing the acquired data and inferring its contents, means for estimating the user's emotions using an emotion engine based on the inference results, and means for returning the inference results to the terminal, thereby improving the accuracy and speed of analyzing non-verbal information and enabling quick and accurate feedback to the user.

[0960] A "terminal" is a device that can be operated by a user, acquires video, images, or audio, transmits the acquired data to a server, receives the inference results returned from the server, and displays or outputs them as audio.

[0961] "Data" refers to information in the form of video, image or audio files acquired by the terminal.

[0962] A "server" is a computer system that receives data sent from a terminal, analyzes it, and returns the analysis results to the terminal.

[0963] "Inference results" refer to the estimated results obtained by the server when analyzing data, and include information about the user's intentions and actions.

[0964] An "HTTP POST request" is a request format that uses a web protocol for a terminal to send data to a server.

[0965] An "emotion engine" is a software system that analyzes information such as a user's facial expressions, tone of voice, and body movements based on the analysis results, and estimates the user's emotions.

[0966] The "JSON format" is a lightweight data description format for structuring, storing, and transferring data, and is a format that is mainly composed of key-value pairs.

[0967] The present invention is a system that acquires video, images, or audio using a terminal operated by a user, sends the data to a server for analysis, and combines the analysis results with an emotion engine to return the results to the terminal in a form that reflects the user's emotions, and outputs them as a display or audio.

[0968] To implement this system, the following hardware and software are used.

[0969] Hardware and software used

[0970] Device: Mobile device such as a smartphone or tablet

[0971] Server: A high-performance computer server

[0972] Software: Mobile application, HTTP communication protocol, AI analysis model (e.g., OpenCV, TensorFlow), emotion engine (e.g., Emotion API), JSON format user interface, and text-to-speech function (e.g., Google Text-to-Speech)

[0973] System Operation

[0974] A user launches a smartphone application and selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record a video. When finished, the user taps the "Stop" button to save the data.

[0975] The device temporarily stores the retrieved data in its internal storage, prepares the data path and metadata, and then makes an HTTP POST request to send the attached data file and authentication information to the server.

[0976] The server stores the received data in a specified directory and inputs it into the AI ​​model for analysis. The AI ​​model analyzes the transmitted video, image, and audio data, understanding non-verbal information such as gestures, foreign languages, drawings, and humming to generate analysis results. Furthermore, it uses an emotion engine to analyze the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[0977] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference results, which are formatted in JSON format.

[0978] The server sends the generated inference results to the device via an HTTP POST request. The device analyzes the received JSON-formatted inference results and displays or outputs them to the user. The user can check the analysis results through text displayed on the screen or audio readout, and respond quickly.

[0979] Specific examples

[0980] For example, this system is used when a user (parent) uses a smartphone to record a pointing action by a child who is not good at speaking. The device receives the video data, temporarily stores it, and sends it to the server. The server analyzes the received video and generates the inference result "hungry," while also using the emotion engine to detect the child's dissatisfied facial expression. This results in a more detailed result: "hungry but in a bad mood." This result is sent back to the device, which tells the user "hungry but in a bad mood" by display or voice.

[0981] As another example, when a foreigner tries to communicate something using a combination of gestures and a foreign language, the user takes a photo of the situation with their smartphone. The device then sends the captured data to the server, which analyzes it to infer the intention of "Where is the toilet?" and uses an emotion engine to detect the emotion of urgency, generating the inference result of "I need to go to the toilet urgently." This result is then sent back to the device, which then displays or speaks to the user the message "I need to go to the toilet urgently." This enables smooth communication with foreigners.

[0982] Prompt Sentence Examples

[0983] "Explain how you can analyze a child's gestures and facial expressions to infer their intentions and emotions."

[0984] Based on this prompt, the generative AI model can explain the specific analysis method and how to communicate the results.

[0985] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0986] System processing steps

[0987] Step 1: User interaction

[0988] A user launches a smartphone application. The application's main screen appears. The user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to capture video, and the "Stop" button to save the data. The input is the video, image, or audio data captured by the user's operation. The output is the saved data file.

[0989] Step 2: Prepare the device to send data

[0990] The device temporarily stores the data acquired from the user in its internal storage. It prepares the data path and metadata and creates an HTTP POST request. The input is the acquired data file. The output is an HTTP POST request with the data file attached.

[0991] Step 3: Send data by device

[0992] The device includes the data file and authentication information in the prepared HTTP POST request and sends it to the specified URL on the server. Specifically, the request is sent using a request library (for example, the Python requests library). The input is the HTTP POST request and authentication information. The output is the result of sending the request to the server.

[0993] Step 4: Server receives data

[0994] The server receives an HTTP POST request sent from the device and saves the data file in the specified directory. The input is the HTTP POST request from the device. The output is the saved data file.

[0995] Step 5: Data analysis by the server

[0996] The server inputs the saved data files into the AI ​​model for analysis. For example, in the case of video data, analysis is performed using OpenCV or TensorFlow to identify gestures and foreign languages. The input is the saved data file. The output is text data (e.g., "I'm hungry") that is the result of the analysis.

[0997] Step 6: Sentiment analysis by server

[0998] The server uses an emotion engine based on the analysis results to estimate the user's emotion. For example, it uses Microsoft's Emotion API to analyze facial expressions and detect emotions such as "dissatisfied." The input is the text data of the analysis results. The output is the estimated emotion (for example, "dissatisfied").

[0999] Step 7: Server generates inference results

[1000] The server combines the analysis results and emotion estimation results to generate the final inference result, which is then formatted in JSON. The input is the text data of the analysis results and the emotion estimation result. The output is the inference result in JSON format (for example, {"intention": "hungry", "emotion": "unhappy"}).

[1001] Step 8: Server sends inference results

[1002] The server sends the inference results formatted in JSON to the terminal via an HTTP POST request. The input is the inference results in JSON format. The output is the result sent to the terminal.

[1003] Step 9: Receive and display the inference results on the device

[1004] The device receives an HTTP POST request sent from the server and parses the inference results in JSON format. It then displays the results to the user as text or audio. Specifically, it uses a text-to-speech conversion function such as Google Text-to-Speech. The input is the inference results in JSON format from the server. The output is the display or audio output for the user.

[1005] (Application example 2)

[1006] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1007] In conventional security systems, surveillance cameras and sensors primarily rely on video data and specific behavioral patterns to detect abnormal behavior. However, this alone does not allow for an accurate understanding of the emotions of the monitored individual, limiting the accuracy of identifying suspicious individuals. For example, even if no suspicious behavior is observed, changes in emotions may be overlooked, potentially resulting in potential danger being overlooked. Another issue is that the inability to respond quickly based on emotional changes places a heavy burden on security personnel.

[1008] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1009] In this invention, the server includes an emotion analysis means for analyzing acquired data and inferring emotions, a means for notifying security personnel based on the detection of suspicious behavior or emotions, and a means for returning the inference results to the terminal, thereby enabling the emotions of monitored persons to be accurately analyzed in real time and quickly notified to security personnel.

[1010] A "user-operated terminal" is an electronic device that is manually operated by a user to acquire video, images, or audio data and process that data.

[1011] "Means for acquiring video, images or audio" refers to a function for collecting video, still images and audio data using sensors such as cameras and microphones.

[1012] The "means for sending acquired data to a server" is a function for transferring acquired data to a server via the Internet using a communication protocol such as an HTTP POST request.

[1013] The "means for receiving inference results from the server" is a function for receiving analysis results and data returned from the server.

[1014] "Means for displaying or outputting received inference results as audio" refers to a function for displaying text or images on the terminal screen or transmitting information as audio from the speaker.

[1015] "Means of acquiring data in real time from surveillance cameras and sensors" refers to a function for acquiring video and audio in real time using installed surveillance cameras and various sensors.

[1016] "Emotion analysis means for analyzing acquired data and inferring emotions" is a function for inferring emotions by analyzing the facial expressions and tone of voice of the monitored person based on the acquired video and audio data.

[1017] "Means for notifying when suspicious behavior or emotions are detected" is a function that sends warnings and notifications to security personnel when an abnormality or suspicious situation is detected based on inferred emotions or behavior.

[1018] "Means of analyzing data and inferring its contents" refers to a function that uses certain rules and AI algorithms to understand the contents of acquired data and derive inferential results.

[1019] The "means for returning the inference results to the terminal" is a function for correcting the analysis results and sending them back to the original terminal.

[1020] "Means generated based on gestures, foreign language, drawings, or humming" refers to a function that analyzes a person's physical movements, speech in a specific language, drawings, humming, etc., and generates analysis results based on the content.

[1021] "Means generated based on real-time emotion analysis" refers to a function that instantly analyzes changes in emotions based on data obtained in real time and predicts the results.

[1022] This invention is a system that uses a terminal operated by a user to acquire video, images, or audio, transmits the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.In addition, it also has a means for acquiring data and analyzing emotions in real time, and quickly notifying security personnel of suspicious behavior or emotions.

[1023] Hardware and Software

[1024] Hardware:

[1025] User devices such as smartphones and tablets

[1026] Surveillance cameras and sensors

[1027] server

[1028] software:

[1029] Smartphone and tablet application software (compatible with iOS and Android)

[1030] Server software (web frameworks such as Flask)

[1031] Sentiment Analysis Library

[1032] OpenCV (video processing library)

[1033] Acquiring and Sending Data

[1034] A user launches a surveillance system or smartphone application. The user device uses a camera or microphone to capture video, images, or audio. The captured data is temporarily stored on the device and then sent to the server using an HTTP POST request, including any necessary authentication information. Data is also captured in real time from surveillance cameras and sensors and sent to the server.

[1035] Data analysis

[1036] The server receives the transmitted data and saves it as a file. The saved data is then analyzed using an AI model. The AI ​​model understands non-verbal information from video, image, and audio data and generates analysis results. Furthermore, an emotion analysis tool estimates the user's emotions from facial expressions, tone of voice, body movements, etc.

[1037] Generating and notifying inference results

[1038] The emotions estimated by the emotion analysis method are added to the analysis results to generate the final inference results. For example, emotions such as "anxiety" or "impatience" are estimated from video footage captured by a surveillance camera, and if suspicious behavior is detected, the detailed inference results are formatted in JSON format. If an abnormality is detected, the analysis results are notified to security personnel in real time.

[1039] Output to the user

[1040] The user device receives the inference results returned from the server. The received results are analyzed and displayed or output as audio to the user. The user can confirm the intent and emotion inferred by the server through text displayed on the screen or audio readout. Security personnel can also respond quickly by checking the notification.

[1041] Examples of concrete examples and prompts

[1042] For example, if a suspicious person in a location where a surveillance camera is installed shows an anxious expression, the analysis result will be estimated as "anxiety / anxiety." This information will be sent to security personnel along with a notification that "There is a person with an anxious expression in this area."

[1043] Prompt Sentence Examples

[1044] Analyze the video data uploaded by users, understand non-verbal information (gestures and facial expressions), and analyze emotions using an emotion engine.

[1045] example:

[1046] 1. Video data: [Video file path]

[1047] 2. Estimated emotion: [Estimation result]

[1048] In this way, by linking the user terminal and the server, this invention enables real-time data acquisition and emotion analysis, thereby enhancing security.

[1049] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1050] Step 1:

[1051] A user launches a smartphone application and uses the camera or microphone to capture video, images, or audio. The input data can be video, image, or audio files. This data is temporarily stored on the device. Specifically, the user taps the record button in the app to start recording, and then taps the stop button to save the data.

[1052] Step 2:

[1053] The device sends the acquired data to the server using an HTTP POST request. The input is a saved video, image, or audio file, and the output is a request to the server. Authentication information is also added. Specifically, the device creates a request by specifying the path of the acquired file and sends it to the server's URL.

[1054] Step 3:

[1055] The server stores the received data and inputs it into the AI ​​model for analysis. The input is the submitted video, image, or audio file, and the output is the analysis result. Specifically, the server saves the file in a specified directory and processes the input data using the analysis library.

[1056] Step 4:

[1057] The server analyzes the data using an AI model and infers the user's emotions through an emotion analysis tool. The input is preprocessed data, and the output is the result of the emotion analysis. Specifically, the AI ​​model analyzes gestures and facial expressions, while the emotion engine analyzes voice tone and body movements.

[1058] Step 5:

[1059] The server adds the emotion analysis results to the inference results, generates the final inference results in JSON format, and returns them to the device. The inputs are the analysis results and emotion analysis results, and the output is the JSON data of the final inference results. Specifically, the server formats the data and creates a response to send back to the device.

[1060] Step 6:

[1061] The device analyzes the received inference results and displays or outputs them to the user. The input is inference result data in JSON format, and the output is displayed text or spoken audio. Specifically, the device parses the JSON data and displays it in a text view, or uses a speech synthesis engine to communicate it to the user.

[1062] Step 7:

[1063] Data is acquired in real time from surveillance cameras and sensors and sent to a server in the same way. The input is real-time video and audio data, and the output is a request to the server. Specifically, a surveillance camera system that operates continuously collects data and sends it to the server at regular intervals.

[1064] Step 8:

[1065] If the server detects suspicious behavior or emotions from the data analyzed in real time, it notifies security personnel. The input is the real-time analysis results, and the output is the notification content. Specifically, when an abnormality is detected, an SMS, email, or in-app push notification is sent.

[1066] In this way, each processing step repeats the process of data acquisition, transmission, analysis, and notification, building a system that achieves enhanced security.

[1067] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1068] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1069] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1070] [Fourth embodiment]

[1071] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1072] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1073] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1074] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1075] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1076] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1077] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1078] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1079] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1080] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1081] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1082] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1083] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1084] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[1085] 1. User operations

[1086] A user launches a smartphone application. The application's main screen appears, and the user navigates through the menu to capture video, images, or audio. For example, the user taps the "Record" button to record video, and then taps the "Stop" button to save the data when finished.

[1087] 2. Terminal Processing

[1088] The device receives data entered by the user. It temporarily stores data acquired using sensors such as a camera or microphone, obtains the data path, and prepares it for subsequent processing. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request and sends it with the saved file attached.

[1089] 3. Server Processing

[1090] The server receives data sent from the device. It analyzes the received data and uses a pre-trained AI model to infer its contents. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. The analysis results are formatted and sent back to the device in JSON format.

[1091] 4. Terminal handling and output to the user

[1092] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[1093] Specific examples

[1094] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and takes a video of the child's pointing movements. The device receives the video data, saves it, and sends it to a server. The server analyzes the received video, infers the child's intention, such as "I'm hungry," and sends the inference result back to the device. The device receives the inference result and displays it on the screen or tells the user "I'm hungry" by voice. In this way, the user can properly understand the child's intention and take appropriate action.

[1095] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, if the content is inferred as "Where is the toilet?", the result is sent back to the device, and the user is informed by displaying or audibly saying "Where is the toilet?", enabling smooth communication with the foreigner.

[1096] In this way, the system of the present invention can analyze non-verbal information and convey intentions to the user in an easily understandable form, thereby solving various communication problems.

[1097] The processing flow will be explained below.

[1098] Step 1:

[1099] A user launches an app on their smartphone. The user taps an icon to launch the app, and the app's main screen appears. The user selects an option to capture video, images, or audio.

[1100] Step 2:

[1101] The user performs an operation to capture video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button at the desired time to save the data. The same operation is performed for images and audio.

[1102] Step 3:

[1103] The device receives data entered by the user. The acquired data is temporarily stored in the device, and the data path and metadata are prepared.

[1104] Step 4:

[1105] The device sends the acquired data to the server by creating an HTTP POST request to the server URL, attaching the acquired data file, and including any necessary authentication information.

[1106] Step 5:

[1107] The server receives the data sent from the device, saves it as a file, and inputs it into the AI ​​model for analysis.

[1108] Step 6:

[1109] The server analyzes the received data. It uses an AI model to analyze the data and understand non-verbal information such as gestures, foreign languages, drawings, and humming, and generates an analysis result. The generated result is formatted in JSON format.

[1110] Step 7:

[1111] The server returns the inference results to the device. The inference results are formatted in JSON and sent to the device as an HTTP response.

[1112] Step 8:

[1113] The device analyzes the inference results received from the server, parses the received JSON data to obtain the analysis results, and checks whether the inference results are in the correct format.

[1114] Step 9:

[1115] The device displays or outputs the inference results to the user. The analysis results are displayed as text on the screen or, if necessary, conveyed to the user using a text-to-speech function.

[1116] Step 10:

[1117] The user can check the displayed or spoken inference results and understand the other person's intention. For example, if a child's gesture indicates that they are hungry, the user can take appropriate action based on that information.

[1118] Through these steps, users can understand the meaning of videos, images, and audio that contain non-verbal information, enabling them to communicate appropriately.

[1119] Example 1

[1120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1121] With the diversification of modern communication, there is a growing need for systems that can accurately analyze non-verbal information such as gestures, foreign languages, and voice, and convey it to users in an easy-to-understand manner. However, existing systems have difficulty efficiently analyzing non-verbal information in real time and providing appropriate feedback. The objective of this invention is to provide a system that analyzes video, image, and voice data acquired by a user on a server, quickly returns the results to a terminal, and displays or outputs the results to the user as voice.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1123] In this invention, the server includes means for analyzing acquired data and inferring its contents, means for returning the inference results to the terminal, means for analyzing the data using a pre-trained AI model, and means for returning the analysis results to the terminal in JSON format, thereby enabling quick and accurate analysis of non-verbal information acquired from the user and providing appropriate feedback to the user.

[1124] A "user-operated terminal" is a device that is directly operated by a user to obtain video, images, or audio.

[1125] "Means of acquisition" refers to the mechanism for collecting video, image, or audio data using sensors such as cameras and microphones installed on the device.

[1126] "Means of sending to the server" refers to the protocols and procedures for transferring the acquired video, image, and audio data to the server via the network.

[1127] The "means for receiving inference results" is a function for receiving data on the analysis and inference results returned from the server.

[1128] "Means for displaying or outputting by voice" refers to the ability to display the received inference results on the terminal screen in text or graphical form or to play them back by voice.

[1129] "Temporary storage means" refers to a mechanism for temporarily storing acquired data in the device's memory or storage.

[1130] "Means for creating an HTTP POST request" is the process of generating a request to send the retrieved data to a server using the HTTP protocol.

[1131] "Means for analyzing data received from the server" refers to technology for interpreting data returned from the server and converting it into a form that can be used by the user.

[1132] A "pre-trained AI model" is an algorithm that has been trained for a specific task and performs analysis and inference based on input data.

[1133] The JSON format is a lightweight data exchange format for storing and transferring structured data.

[1134] MODE FOR CARRYING OUT THE INVENTION

[1135] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.

[1136] Hardware and Software Configuration

[1137] Device:

[1138] The terminal is a mobile device such as a smartphone or tablet. This terminal is equipped with a camera, microphone, and internal storage, which allows it to capture and temporarily store video, image, and audio data. It also requires a screen to display feedback and a speaker for audio output. The terminal software includes libraries for creating HTTP requests and parsing JSON data.

[1139] server:

[1140] The server receives data using a web server (e.g., Nginx, Tomcat, etc.), analyzes the data using Python scripts and runs pre-trained AI models (e.g., TensorFlow, PyTorch), and also requires a library to send the analysis results back to the device in JSON format.

[1141] Operational Overview

[1142] 1. Data Acquisition and Transmission

[1143] A user launches a smartphone application and captures video, images, or audio. For example, the user taps the "Record" button to record a video, and then taps the "Stop" button to save the data. The device temporarily stores the captured data in its internal storage and creates an HTTP POST request to send to the server.

[1144] 2. Data Analysis

[1145] The server receives data sent from the device. It analyzes the received data using a Python script and uses a pre-trained AI model (TensorFlow or PyTorch) to infer the content of the data. The AI ​​model understands and analyzes non-verbal information such as gestures, foreign languages, drawings, and humming. The analysis results are formatted in JSON and sent back to the device.

[1146] 3. Receiving and displaying inference results

[1147] The device receives the inference results in JSON format returned from the server. It analyzes the received results and displays or outputs them to the user. The user can understand the intent of the server's inferences through the text displayed on the screen or the audio reading.

[1148] Specific operation example

[1149] Example 1: Child gesture analysis

[1150] To analyze the gestures of a child who is not good at speaking, the user (parent) launches an app on their smartphone and records a video of the child's pointing movements. The device temporarily saves the video data and sends it to the server using an HTTP POST request. The server receives the video data, analyzes it with a Python script, and uses a pre-trained TensorFlow model to infer the child's intent, such as "I'm hungry." The analysis results are returned in JSON format, and the device receives the results, displaying "I'm hungry" on the screen or outputting it as voice. The user can then appropriately understand the child's intent and respond accordingly.

[1151] Example 2: Communicating with foreigners

[1152] When a foreigner tries to communicate something using a combination of gestures and a foreign language, the user films the situation with their smartphone. The device then sends the filmed video data to a server. The server then analyzes the received data using a Python script and uses a pre-trained TensorFlow model to infer the intent, "Where is the toilet?" The inference result is sent back to the device in JSON format, and the device receives the result and displays "Where is the toilet?" on the screen or outputs it as voice. This allows users to communicate smoothly with foreigners.

[1153] Example prompt sentence:

[1154] "The system takes a video of the child's pointing action, sends the video data to a server, and returns the analysis results. The system helps understand the child's intentions by displaying the inference results on the screen and outputting them as audio."

[1155] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1156] Step 1:

[1157] The user launches the application on their smartphone. The main screen of the application is displayed, and the user operates the menu to acquire data.

[1158] Specific operation: Tap the "Record" button from the app's startup screen and select video mode.

[1159] Input: User actions

[1160] Output: Instruction to start data acquisition

[1161] Step 2:

[1162] The device accepts instructions from the user and acquires video, image, or audio data. It collects data using sensors such as a camera and microphone and temporarily stores it in its internal storage.

[1163] Specific operation: Record with the camera, and when finished, tap the "Stop" button to save the data.

[1164] Input: Data from the camera or microphone

[1165] Output: Stored media data

[1166] Step 3:

[1167] The device retrieves the path to the stored data and creates an HTTP POST request for sending it to the server, including the retrieved media data.

[1168] What it does: Creates an HTTP POST request and attaches a video file.

[1169] Input: Path to the saved media data

[1170] Output: Request sent to server

[1171] Step 4:

[1172] The server receives the data sent from the terminal. A web server (e.g., Nginx, Tomcat) receives the request and temporarily stores the data.

[1173] Specific operation: After receiving the data, save it in the specified folder.

[1174] Input: HTTP POST request from the terminal

[1175] Output: Data stored in the server

[1176] Step 5:

[1177] The server analyzes the received data, runs a Python script, and uses pre-trained AI models to analyze non-verbal information such as gestures, foreign words, drawings, and humming, and generates analysis results.

[1178] Specific operations: Run AI models using TensorFlow and PyTorch to analyze and infer data.

[1179] Input: Stored media data

[1180] Output: Analysis results (gesture recognition results as a 3D array, translation results as text, etc.)

[1181] Step 6:

[1182] The server formats the generated analysis results and returns them to the terminal in JSON format, sending JSON data containing the analysis results as an HTTP response.

[1183] Specific operation: Formats data into JSON format and creates an HTTP response.

[1184] Input: Parsed data

[1185] Output: Parsed results in JSON format

[1186] Step 7:

[1187] The device receives the JSON data returned from the server, parses the HTTP response, and extracts the data.

[1188] Specific operation: Parse the JSON data and obtain the required information.

[1189] Input: Parsed result in JSON format

[1190] Output: Text and audio data

[1191] Step 8:

[1192] The device displays or outputs the extracted analysis results to the user as text on the screen or as voice using a speech synthesis engine.

[1193] Specific action: Displaying text such as "I'm hungry" on the screen or reading it aloud.

[1194] Input: Text or audio data

[1195] Output: Feedback to the user (visual or auditory)

[1196] Examples:

[1197] Example of child gesture analysis:

[1198] The user launches the smartphone app and records the child's pointing behavior. The device temporarily saves the video data, generates an HTTP POST request, and sends it to the server. The server uses TensorFlow to analyze the gesture, infers that the child is hungry, and returns the result in JSON format. The device receives the JSON data and displays the text on the screen. The user can then look at the screen to understand the child's intention.

[1199] Examples of communication with foreigners:

[1200] The user launches the app and records the foreigner's gestures and speech in the foreign language. The device sends the data to the server, which analyzes it and returns the inference result, "Where is the toilet?" The device receives the result and displays it as text or outputs it as voice. The user can use this information to guide the foreigner.

[1201] (Application example 1)

[1202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1203] In industrial environments such as factories, there is a growing need for support systems to help workers efficiently receive work instructions. With conventional methods, it can take time for workers to understand work instructions, which can lead to reduced work efficiency. Additionally, when language barriers or non-verbal communication are required, it can be difficult to receive appropriate feedback. This can affect overall productivity and safety.

[1204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1205] In this invention, the server includes a means for acquiring video, image, or audio, a means for transmitting the acquired data to the server, a means for receiving inference results from the server, a means for displaying or outputting the received inference results as audio, a means for analyzing hand gestures and voice commands in an industrial environment, and a means for generating work instructions based on the hand gestures and voice commands, thereby enabling effective provision of work instructions and feedback within a factory, improving work efficiency and overall productivity.

[1206] A "terminal" is an electronic device that is operated by a user to acquire video, images, or audio and transmit it to a server.

[1207] A "server" is a computer system that analyzes the acquired data and returns the inference results to the terminal.

[1208] "Inference results" are results based on the meaning and content of data obtained by the server through analysis.

[1209] "Display" refers to the act and means of visually conveying information to a user.

[1210] "Audio output" refers to the act and means of transmitting information to the user audibly.

[1211] "Industrial environment" refers to factories, manufacturing sites, and other workplaces where machinery and equipment are operated.

[1212] A "gesture" is an act of conveying intentions or instructions using hand or body movements.

[1213] A "voice command" is a voice signal used to give instructions to the system using voice.

[1214] "Work instructions" are information that instructs the procedures and actions for performing a specific task.

[1215] To implement this invention as a factory work support system, a server, a terminal operated by a user, and a network environment connecting these are required.

[1216] 1. Hardware and Software Requirements

[1217] Hardware:

[1218] Terminal: For example, a head-mounted display (HMD) or a smartphone. It is desirable that these have a built-in camera and microphone.

[1219] Server: A high-performance computer that must be capable of executing inference processing for AI models in real time.

[1220] software:

[1221] Flask: A Python-based web framework for receiving and processing server-side data.

[1222] PyTorch: A deep learning library that contains models for analyzing the acquired data.

[1223] HTTP request library: Used to send and receive data between the terminal and the server at high speed.

[1224] 2. System Overview

[1225] The user captures video, images, or audio on the device. The captured data is sent to the server via an HTTP request. The server receives the data and analyzes it using a generative AI model. The inference results are returned to the device in JSON format, which the device then displays or outputs as audio. Particularly in industrial environments, analyzing gestures and voice commands and generating appropriate work instructions can improve work efficiency.

[1226] 3. Specific Examples

[1227] Example 1: Providing work instructions

[1228] A factory worker puts on the HMD and issues a voice command, saying, "What do I do next?" The HMD's built-in microphone picks up the voice and sends the data to the server. The server analyzes the voice and generates a work instruction, such as "Install part A on the right side." This instruction is displayed on the HMD display and is also read aloud.

[1229] Example 2: Support for safety confirmation

[1230] When a worker is performing a dangerous task, he or she signals with a hand gesture, "Please check." The HMD camera captures the gesture and sends the data to the server. The server analyzes the gesture and generates a warning message, such as "Be careful, it's hot." This message is displayed on the HMD display and an audio warning is also issued.

[1231] 4. Examples of prompts

[1232] The following prompts can be used to make the necessary inferences for the generative AI model:

[1233] You are a factory manager robot. A worker is asking for instructions. He has part A in his hand. What should he do next?

[1234] In this way, the present invention provides the tools necessary to support practical tasks in the work environment and provide an efficient and safe environment.

[1235] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1236] Step 1:

[1237] An application is launched on the terminal operated by the user to acquire video, images, or audio.

[1238] Input: Video, images, and audio data (e.g., worker makes hand gestures or voice commands)

[1239] Output: The captured data (e.g., worker gestures and voice are captured)

[1240] Step 2:

[1241] The device temporarily stores the acquired data and sends it to the server via an HTTP request.

[1242] Input: Data acquired in the previous step (e.g., data captured by the HMD camera or microphone)

[1243] Output: Data sent to the server (e.g., sent in an HTTP request)

[1244] Step 3:

[1245] The server passes the received data to the AI ​​model for analysis.

[1246] Input: Data sent from the device (e.g., captured gestures and voice data)

[1247] Output: Analyzed results (e.g., inference results from an AI model)

[1248] Step 4:

[1249] The server formats the inference results in JSON format and returns them to the terminal as an HTTP response.

[1250] Input: The inference result of the AI ​​model (e.g., "Install part A on the right side")

[1251] Output: HTTP response to the terminal (e.g., inference results in JSON format)

[1252] Step 5:

[1253] The terminal analyzes the received inference results and outputs them to the user by display or voice.

[1254] Input: Inference results returned from the server (e.g., JSON data)

[1255] Output: Display or audio output to the user interface (e.g., "Please install part A on the right side" is displayed on the HMD display and also read aloud)

[1256] Step 6:

[1257] The user performs the task by following the displayed information and audio guidance.

[1258] Input: Work instructions provided by the device (e.g., instructions displayed on the HMD display)

[1259] Output: The action performed by the user (e.g., attaching part A to the right side)

[1260] In this way, each step works together, allowing the user to perform the task efficiently and the system to provide feedback in real time.

[1261] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1262] The present invention relates to a system that uses a terminal operated by a user to acquire video, images, or audio, sends the data to a server for analysis, combines the inference results with an emotion engine, and returns the results to the terminal in a form that reflects the user's emotions and outputs them as display or audio.

[1263] 1. User operations

[1264] A user launches an application on their smartphone. The application's main screen appears, and the user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record video and then taps the "Stop" button at the desired time to save the data.

[1265] 2. Terminal Processing

[1266] The device receives data entered by the user. The acquired data is temporarily stored on the device, and the data path and metadata are prepared. The device then sends the acquired data to the server. Specifically, it creates an HTTP POST request, attaches the saved file, and sends it to the server's URL. The request also includes any necessary authentication information.

[1267] 3. Server analysis process

[1268] The server receives the data sent from the device. It receives the data and saves it as a file. The saved data is then input into the AI ​​model for analysis. The AI ​​model understands non-verbal information such as gestures, foreign languages, drawings, and humming, and generates analysis results. In addition, the emotion engine analyzes elements such as the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[1269] 4. Server emotion analysis and inference results generation

[1270] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference result. For example, if a child gestures to say "I'm hungry" while also making a dissatisfied expression, a detailed inference result such as "I'm hungry but in a bad mood" will be obtained. The generated results are formatted in JSON format.

[1271] 5. Terminal handling and output to the user

[1272] The device receives the inference results returned from the server, analyzes them, and displays or outputs them to the user. The user can confirm the intention and emotion inferred by the server through text displayed on the screen or audible reading.

[1273] Specific examples

[1274] When analyzing the gestures of a child who is not good at conversation, the user (parent) launches an app on their smartphone and records a video of the child pointing. The device receives the video data, saves it, and sends it to the server. The server analyzes the received video and infers that the child is "hungry," while the emotion engine analyzes the child's dissatisfied facial expression and determines that the child is "in a bad mood." This result is formatted as "hungry but in a bad mood" and sent back to the device. The device receives the inference result and tells the user "hungry but in a bad mood" on the screen or by voice. The user can appropriately understand the child's intentions and emotions and respond quickly.

[1275] As another example, if a foreigner tries to communicate something using a combination of gestures and a foreign language, the user can take a photo of the situation with their smartphone. The device sends the data to a server, which analyzes it and infers the intention. For example, while the content is inferred as "Where is the toilet?", the emotion engine simultaneously detects the emotion of impatience and generates a detailed result such as "I need to go to the toilet urgently." This result is then sent back to the device, which then informs the user by displaying or audibly saying "I need to go to the toilet urgently," enabling smooth communication with the foreigner.

[1276] In this way, the system of the present invention can analyze non-verbal information and convey intentions and emotions to the user in an easy-to-understand manner, thereby solving various communication problems.

[1277] The processing flow will be explained below.

[1278] Step 1:

[1279] A user launches an app on their smartphone. The user taps the app icon to launch it, and the main screen appears. The user selects an option to capture video, images, or audio depending on the purpose of use.

[1280] Step 2:

[1281] The user captures video, image, or audio data. For example, the user taps the "record" button to record a video of the child's gestures, and then taps the "stop" button to save the data. Similarly, images and audio are captured appropriately.

[1282] Step 3:

[1283] The device receives the acquired data, temporarily stores the data acquired through sensors such as the camera and microphone, and records the file path and metadata of the data.

[1284] Step 4:

[1285] The device sends the acquired data to the server by creating an HTTP POST request and sending the acquired data file along with the data to the server's URL. The request also includes any necessary authentication information.

[1286] Step 5:

[1287] The server receives the data sent from the device, saves the received data as a file, and prepares it for data analysis.

[1288] Step 6:

[1289] The server begins the analysis process, using AI models such as image and voice recognition to analyze the data and infer intent from gestures, language, pictures, and humming.

[1290] Step 7:

[1291] The server uses an emotion engine to analyze the user's emotions, and estimates their emotional state by analyzing the acquired data, such as the user's facial expressions, tone of voice, and body movements.

[1292] Step 8:

[1293] The server combines the inference results and the emotion analysis results to generate the final inference result. For example, by combining "hungry," the gesture analysis result, and the emotion analysis result "bad mood," the detailed inference result "hungry but bad mood" is obtained.

[1294] Step 9:

[1295] The server returns the final inference results to the device, which then formats the integrated inference results in JSON format and sends them to the device as an HTTP response.

[1296] Step 10:

[1297] The device analyzes the inference results received from the server, parses the received JSON data, obtains the inference results, and prepares them to be displayed or output to the user in an appropriate format.

[1298] Step 11:

[1299] The device displays or outputs the inference result to the user. It can be displayed as text on the screen or read aloud to the user. For example, the inference result "I'm hungry, but I'm in a bad mood" can be displayed or output as audio.

[1300] Step 12:

[1301] The user can check the inference results displayed or output as voice and understand the other person's intentions and emotions. For example, based on the analysis result that a child is "hungry but in a bad mood," the user can take appropriate action.

[1302] In this way, the system provides multifunctional support to help users accurately understand their intentions and respond quickly.

[1303] Example 2

[1304] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1305] Conventional systems lack the means to analyze non-verbal information and accurately convey the user's intentions and emotions. Furthermore, there are problems with time lags and reduced recognition accuracy in the process from sending the data acquired by the user to the server, receiving the analysis results, and outputting them as display or voice. Therefore, the present invention aims to solve these problems and provide a system that analyzes non-verbal information with high accuracy and provides rapid feedback to the user.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1307] In this invention, the server includes means for analyzing the acquired data and inferring its contents, means for estimating the user's emotions using an emotion engine based on the inference results, and means for returning the inference results to the terminal, thereby improving the accuracy and speed of analyzing non-verbal information and enabling quick and accurate feedback to the user.

[1308] A "terminal" is a device that can be operated by a user, acquires video, images, or audio, transmits the acquired data to a server, receives the inference results returned from the server, and displays or outputs them as audio.

[1309] "Data" refers to information in the form of video, image or audio files acquired by the terminal.

[1310] A "server" is a computer system that receives data sent from a terminal, analyzes it, and returns the analysis results to the terminal.

[1311] "Inference results" refer to the estimated results obtained by the server when analyzing data, and include information about the user's intentions and actions.

[1312] An "HTTP POST request" is a request format that uses a web protocol for a terminal to send data to a server.

[1313] An "emotion engine" is a software system that analyzes information such as a user's facial expressions, tone of voice, and body movements based on the analysis results, and estimates the user's emotions.

[1314] The "JSON format" is a lightweight data description format for structuring, storing, and transferring data, and is a format that is mainly composed of key-value pairs.

[1315] The present invention is a system that acquires video, images, or audio using a terminal operated by a user, sends the data to a server for analysis, and combines the analysis results with an emotion engine to return the results to the terminal in a form that reflects the user's emotions, and outputs them as a display or audio.

[1316] To implement this system, the following hardware and software are used.

[1317] Hardware and software used

[1318] Device: Mobile device such as a smartphone or tablet

[1319] Server: A high-performance computer server

[1320] Software: Mobile application, HTTP communication protocol, AI analysis model (e.g., OpenCV, TensorFlow), emotion engine (e.g., Emotion API), JSON format user interface, and text-to-speech function (e.g., Google Text-to-Speech)

[1321] System Operation

[1322] A user launches a smartphone application and selects an option to capture video, images, or audio. For example, the user taps the "Record" button to record a video. When finished, the user taps the "Stop" button to save the data.

[1323] The device temporarily stores the retrieved data in its internal storage, prepares the data path and metadata, and then makes an HTTP POST request to send the attached data file and authentication information to the server.

[1324] The server stores the received data in a specified directory and inputs it into the AI ​​model for analysis. The AI ​​model analyzes the transmitted video, image, and audio data, understanding non-verbal information such as gestures, foreign languages, drawings, and humming to generate analysis results. Furthermore, it uses an emotion engine to analyze the user's facial expressions, tone of voice, and body movements to estimate the user's emotions.

[1325] The user's emotions estimated by the emotion engine are added to the analysis results to generate the final inference results, which are formatted in JSON format.

[1326] The server sends the generated inference results to the device via an HTTP POST request. The device analyzes the received JSON-formatted inference results and displays or outputs them to the user. The user can check the analysis results through text displayed on the screen or audio readout, and respond quickly.

[1327] Specific examples

[1328] For example, this system is used when a user (parent) uses a smartphone to record a pointing action by a child who is not good at speaking. The device receives the video data, temporarily stores it, and sends it to the server. The server analyzes the received video and generates the inference result "hungry," while also using the emotion engine to detect the child's dissatisfied facial expression. This results in a more detailed result: "hungry but in a bad mood." This result is sent back to the device, which tells the user "hungry but in a bad mood" by display or voice.

[1329] As another example, when a foreigner tries to communicate something using a combination of gestures and a foreign language, the user takes a photo of the situation with their smartphone. The device then sends the captured data to the server, which analyzes it to infer the intention of "Where is the toilet?" and uses an emotion engine to detect the emotion of urgency, generating the inference result of "I need to go to the toilet urgently." This result is then sent back to the device, which then displays or speaks to the user the message "I need to go to the toilet urgently." This enables smooth communication with foreigners.

[1330] Prompt Sentence Examples

[1331] "Explain how you can analyze a child's gestures and facial expressions to infer their intentions and emotions."

[1332] Based on this prompt, the generative AI model can explain the specific analysis method and how to communicate the results.

[1333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1334] System processing steps

[1335] Step 1: User interaction

[1336] A user launches a smartphone application. The application's main screen appears. The user selects an option to capture video, images, or audio. For example, the user taps the "Record" button to capture video, and the "Stop" button to save the data. The input is the video, image, or audio data captured by the user's operation. The output is the saved data file.

[1337] Step 2: Prepare the device to send data

[1338] The device temporarily stores the data acquired from the user in its internal storage. It prepares the data path and metadata and creates an HTTP POST request. The input is the acquired data file. The output is an HTTP POST request with the data file attached.

[1339] Step 3: Send data by device

[1340] The device includes the data file and authentication information in the prepared HTTP POST request and sends it to the specified URL on the server. Specifically, the request is sent using a request library (for example, the Python requests library). The input is the HTTP POST request and authentication information. The output is the result of sending the request to the server.

[1341] Step 4: Server receives data

[1342] The server receives an HTTP POST request sent from the device and saves the data file in the specified directory. The input is the HTTP POST request from the device. The output is the saved data file.

[1343] Step 5: Data analysis by the server

[1344] The server inputs the saved data files into the AI ​​model for analysis. For example, in the case of video data, analysis is performed using OpenCV or TensorFlow to identify gestures and foreign languages. The input is the saved data file. The output is text data (e.g., "I'm hungry") that is the result of the analysis.

[1345] Step 6: Sentiment analysis by server

[1346] The server uses an emotion engine based on the analysis results to estimate the user's emotion. For example, it uses Microsoft's Emotion API to analyze facial expressions and detect emotions such as "dissatisfied." The input is the text data of the analysis results. The output is the estimated emotion (for example, "dissatisfied").

[1347] Step 7: Server generates inference results

[1348] The server combines the analysis results and emotion estimation results to generate the final inference result, which is then formatted in JSON. The input is the text data of the analysis results and the emotion estimation result. The output is the inference result in JSON format (for example, {"intention": "hungry", "emotion": "unhappy"}).

[1349] Step 8: Server sends inference results

[1350] The server sends the inference results formatted in JSON to the terminal via an HTTP POST request. The input is the inference results in JSON format. The output is the result sent to the terminal.

[1351] Step 9: Receive and display the inference results on the device

[1352] The device receives an HTTP POST request sent from the server and parses the inference results in JSON format. It then displays the results to the user as text or audio. Specifically, it uses a text-to-speech conversion function such as Google Text-to-Speech. The input is the inference results in JSON format from the server. The output is the display or audio output for the user.

[1353] (Application example 2)

[1354] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1355] In conventional security systems, surveillance cameras and sensors primarily rely on video data and specific behavioral patterns to detect abnormal behavior. However, this alone does not allow for an accurate understanding of the emotions of the monitored individual, limiting the accuracy of identifying suspicious individuals. For example, even if no suspicious behavior is observed, changes in emotions may be overlooked, potentially resulting in potential danger being overlooked. Another issue is that the inability to respond quickly based on emotional changes places a heavy burden on security personnel.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1357] In this invention, the server includes an emotion analysis means for analyzing acquired data and inferring emotions, a means for notifying security personnel based on the detection of suspicious behavior or emotions, and a means for returning the inference results to the terminal, thereby enabling the emotions of monitored persons to be accurately analyzed in real time and quickly notified to security personnel.

[1358] A "user-operated terminal" is an electronic device that is manually operated by a user to acquire video, images, or audio data and process that data.

[1359] "Means for acquiring video, images or audio" refers to a function for collecting video, still images and audio data using sensors such as cameras and microphones.

[1360] The "means for sending acquired data to a server" is a function for transferring acquired data to a server via the Internet using a communication protocol such as an HTTP POST request.

[1361] The "means for receiving inference results from the server" is a function for receiving analysis results and data returned from the server.

[1362] "Means for displaying or outputting received inference results as audio" refers to a function for displaying text or images on the terminal screen or transmitting information as audio from the speaker.

[1363] "Means of acquiring data in real time from surveillance cameras and sensors" refers to a function for acquiring video and audio in real time using installed surveillance cameras and various sensors.

[1364] "Emotion analysis means for analyzing acquired data and inferring emotions" is a function for inferring emotions by analyzing the facial expressions and tone of voice of the monitored person based on the acquired video and audio data.

[1365] "Means for notifying when suspicious behavior or emotions are detected" is a function that sends warnings and notifications to security personnel when an abnormality or suspicious situation is detected based on inferred emotions or behavior.

[1366] "Means of analyzing data and inferring its contents" refers to a function that uses certain rules and AI algorithms to understand the contents of acquired data and derive inferential results.

[1367] The "means for returning the inference results to the terminal" is a function for correcting the analysis results and sending them back to the original terminal.

[1368] "Means generated based on gestures, foreign language, drawings, or humming" refers to a function that analyzes a person's physical movements, speech in a specific language, drawings, humming, etc., and generates analysis results based on the content.

[1369] "Means generated based on real-time emotion analysis" refers to a function that instantly analyzes changes in emotions based on data obtained in real time and predicts the results.

[1370] This invention is a system that uses a terminal operated by a user to acquire video, images, or audio, transmits the data to a server for analysis, and returns the inference results to the terminal for display or audio output to the user.In addition, it also has a means for acquiring data and analyzing emotions in real time, and quickly notifying security personnel of suspicious behavior or emotions.

[1371] Hardware and Software

[1372] Hardware:

[1373] User devices such as smartphones and tablets

[1374] Surveillance cameras and sensors

[1375] server

[1376] software:

[1377] Smartphone and tablet application software (compatible with iOS and Android)

[1378] Server software (web frameworks such as Flask)

[1379] Sentiment Analysis Library

[1380] OpenCV (video processing library)

[1381] Acquiring and Sending Data

[1382] A user launches a surveillance system or smartphone application. The user device uses a camera or microphone to capture video, images, or audio. The captured data is temporarily stored on the device and then sent to the server using an HTTP POST request, including any necessary authentication information. Data is also captured in real time from surveillance cameras and sensors and sent to the server.

[1383] Data analysis

[1384] The server receives the transmitted data and saves it as a file. The saved data is then analyzed using an AI model. The AI ​​model understands non-verbal information from video, image, and audio data and generates analysis results. Furthermore, an emotion analysis tool estimates the user's emotions from facial expressions, tone of voice, body movements, etc.

[1385] Generating and notifying inference results

[1386] The emotions estimated by the emotion analysis method are added to the analysis results to generate the final inference results. For example, emotions such as "anxiety" or "impatience" are estimated from video footage captured by a surveillance camera, and if suspicious behavior is detected, the detailed inference results are formatted in JSON format. If an abnormality is detected, the analysis results are notified to security personnel in real time.

[1387] Output to the user

[1388] The user device receives the inference results returned from the server. The received results are analyzed and displayed or output as audio to the user. The user can confirm the intent and emotion inferred by the server through text displayed on the screen or audio readout. Security personnel can also respond quickly by checking the notification.

[1389] Examples of concrete examples and prompts

[1390] For example, if a suspicious person in a location where a surveillance camera is installed shows an anxious expression, the analysis result will be estimated as "anxiety / anxiety." This information will be sent to security personnel along with a notification that "There is a person with an anxious expression in this area."

[1391] Prompt Sentence Examples

[1392] Analyze the video data uploaded by users, understand non-verbal information (gestures and facial expressions), and analyze emotions using an emotion engine.

[1393] example:

[1394] 1. Video data: [Video file path]

[1395] 2. Estimated emotion: [Estimation result]

[1396] In this way, by linking the user terminal and the server, this invention enables real-time data acquisition and emotion analysis, thereby enhancing security.

[1397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1398] Step 1:

[1399] A user launches a smartphone application and uses the camera or microphone to capture video, images, or audio. The input data can be video, image, or audio files. This data is temporarily stored on the device. Specifically, the user taps the record button in the app to start recording, and then taps the stop button to save the data.

[1400] Step 2:

[1401] The device sends the acquired data to the server using an HTTP POST request. The input is a saved video, image, or audio file, and the output is a request to the server. Authentication information is also added. Specifically, the device creates a request by specifying the path of the acquired file and sends it to the server's URL.

[1402] Step 3:

[1403] The server stores the received data and inputs it into the AI ​​model for analysis. The input is the submitted video, image, or audio file, and the output is the analysis result. Specifically, the server saves the file in a specified directory and processes the input data using the analysis library.

[1404] Step 4:

[1405] The server analyzes the data using an AI model and infers the user's emotions through an emotion analysis tool. The input is preprocessed data, and the output is the result of the emotion analysis. Specifically, the AI ​​model analyzes gestures and facial expressions, while the emotion engine analyzes voice tone and body movements.

[1406] Step 5:

[1407] The server adds the emotion analysis results to the inference results, generates the final inference results in JSON format, and returns them to the device. The inputs are the analysis results and emotion analysis results, and the output is the JSON data of the final inference results. Specifically, the server formats the data and creates a response to send back to the device.

[1408] Step 6:

[1409] The device analyzes the received inference results and displays or outputs them to the user. The input is inference result data in JSON format, and the output is displayed text or spoken audio. Specifically, the device parses the JSON data and displays it in a text view, or uses a speech synthesis engine to communicate it to the user.

[1410] Step 7:

[1411] Data is acquired in real time from surveillance cameras and sensors and sent to a server in the same way. The input is real-time video and audio data, and the output is a request to the server. Specifically, a surveillance camera system that operates continuously collects data and sends it to the server at regular intervals.

[1412] Step 8:

[1413] If the server detects suspicious behavior or emotions from the data analyzed in real time, it notifies security personnel. The input is the real-time analysis results, and the output is the notification content. Specifically, when an abnormality is detected, an SMS, email, or in-app push notification is sent.

[1414] In this way, each processing step repeats the process of data acquisition, transmission, analysis, and notification, building a system that achieves enhanced security.

[1415] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1417] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1418] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1419] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1420] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1421] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1422] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1423] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1424] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1425] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1426] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1427] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1428] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1429] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1430] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1431] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1432] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1433] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1434] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1435] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1436] The following is further disclosed regarding the above embodiment.

[1437] (Claim 1)

[1438] A terminal operated by a user, which is a means for acquiring video, images or audio;

[1439] means for transmitting the acquired data to a server;

[1440] means for receiving an inference result from the server;

[1441] The system further includes means for displaying or audibly outputting the received inference results.

[1442] (Claim 2)

[1443] means for analyzing the acquired data and inferring its contents;

[1444] means for returning the inference result to the terminal;

[1445] 10. The system of claim 1, further comprising a server comprising:

[1446] (Claim 3)

[1447] 10. The system of claim 1, further comprising means for generating the inference based on gestures, foreign language, drawing, or humming.

[1448] "Example 1"

[1449] (Claim 1)

[1450] A terminal operated by a user, which is a means for acquiring video, images or audio;

[1451] means for transmitting the acquired data to a server;

[1452] means for receiving an inference result from the server;

[1453] means for displaying or outputting the received inference result by voice;

[1454] means for temporarily storing said data;

[1455] means for creating an HTTP POST request to send said data to a server;

[1456] A means of analyzing data received from the server and using it as text or audio data;

[1457] A system including:

[1458] (Claim 2)

[1459] means for analyzing the acquired data and inferring its contents;

[1460] means for returning the inference result to the terminal;

[1461] A means of analyzing data using pre-trained AI models;

[1462] means for returning the analysis result to the terminal in JSON format;

[1463] 10. The system of claim 1, further comprising a server comprising:

[1464] (Claim 3)

[1465] 10. The system of claim 1, further comprising means for generating the inference based on gestures, foreign language, drawing, or humming.

[1466] "Application Example 1"

[1467] (Claim 1)

[1468] A terminal operated by a user, which is a means for acquiring video, images or audio;

[1469] means for transmitting the acquired data to a server;

[1470] means for receiving an inference result from the server;

[1471] means for displaying or outputting the received inference result by voice;

[1472] means for analyzing hand gestures and voice commands in an industrial environment;

[1473] means for generating work instructions based on the hand gestures and voice commands;

[1474] A system including:

[1475] (Claim 2)

[1476] means for analyzing the acquired data and inferring its contents;

[1477] means for returning the inference result to the terminal;

[1478] 10. The system of claim 1, further comprising a server comprising:

[1479] (Claim 3)

[1480] means for generating the inference result based on gestures, foreign language, drawing, or humming;

[1481] 10. The system of claim 1.

[1482] "Example 2: Combining Emotion Engines"

[1483] (Claim 1)

[1484] A terminal operated by a user, which is a means for acquiring video, images or audio;

[1485] means for transmitting the acquired data to a server;

[1486] means for receiving an inference result from the server;

[1487] means for displaying or outputting the received inference result by voice;

[1488] means for temporarily storing data within the terminal;

[1489] means for preparing paths and metadata for said data;

[1490] A means of making an HTTP POST request to send the data file and authentication information to the server;

[1491] A system including:

[1492] (Claim 2)

[1493] means for analyzing the acquired data and inferring its contents;

[1494] means for estimating a user's emotion using an emotion engine based on the inference result;

[1495] means for returning the inference result to the terminal;

[1496] 10. The system of claim 1, further comprising a server comprising:

[1497] (Claim 3)

[1498] means for generating the inference result based on gestures, foreign language, drawing, or humming;

[1499] A means to format the final inference results along with the estimated emotions in JSON format,

[1500] 10. The system of claim 1, comprising:

[1501] "Application example 2 when combining emotion engines"

[1502] (Claim 1)

[1503] A terminal operated by a user, which is a means for acquiring video, images or audio;

[1504] means for transmitting the acquired data to a server;

[1505] means for receiving an inference result from the server;

[1506] means for displaying or outputting the received inference result by voice;

[1507] A means of acquiring data in real time from surveillance cameras and sensors,

[1508] emotion analysis means for analyzing the acquired data and estimating emotions;

[1509] A means of notifying you if suspicious behavior or emotions are detected;

[1510] A system including:

[1511] (Claim 2)

[1512] means for analyzing the acquired data and inferring its contents;

[1513] means for returning the inference result to the terminal;

[1514] means for notifying security personnel based on the detection of said suspicious behavior or emotion;

[1515] 10. The system of claim 1, further comprising a server comprising:

[1516] (Claim 3)

[1517] The inference result is generated based on gestures, foreign language, drawing, or humming;

[1518] 10. The system of claim 1, further comprising means for generating based on real-time sentiment analysis. [Explanation of symbols]

[1519] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal operated by a user, which is a means for acquiring video, images or audio; means for transmitting the acquired data to a server; means for receiving an inference result from the server; The system further includes means for displaying or audibly outputting the received inference results.

2. means for analyzing the acquired data and inferring its contents; means for returning the inference result to the terminal; The system of claim 1 further comprising a server comprising:

3. The system of claim 1 , further comprising means for generating the inference based on a gesture, a foreign language, a drawing, or a humming.

Images