System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing audio data; means for converting the captured audio data to text data; means for natural language parsing the text data; means for generating a response based on the parsed text data; and means for converting the generated response to audio and outputting the audio.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional natural language processing systems require an internet connection, making them difficult to use in areas with poor communication environments or during disasters. This has led to delays in important decision-making and information provision in such areas and situations, reducing the efficiency of rescue operations and education and research. Furthermore, complex systems have high installation and operating costs, making it difficult to respond quickly on site. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a natural language processing system that operates without requiring an internet connection. This system includes a means for capturing voice data and converting it into text data using a voice recognition engine, a means for natural language analysis of the text data, a means for generating a response based on the analyzed text data, and a means for converting the generated response into voice and outputting it. In particular, by analyzing the context and meaning of the text data using a small-scale language model, advanced natural language processing is realized even in an offline environment. This enables rapid and accurate information provision and decision-making support even in areas with unstable communication infrastructure or in disaster situations. Furthermore, the overall system is simple and easy to use, allowing for rapid on-site implementation and operation.

[0006] "Voice data" is a digital representation of a user's vocalizations.

[0007] A "capturing means" is a device that includes a microphone or other input device for capturing audio data.

[0008] "Text data" is data in the form of a character string converted from voice data using a voice recognition engine.

[0009] A "means for natural language analysis" is a device or software that includes processes or algorithms that analyze the context and meaning of input text data.

[0010] A "small language model" is a natural language processing model that can operate with relatively small computing resources and is used to understand context and meaning.

[0011] A "means for generating a response" is a device or software that includes a process or algorithm that generates an appropriate response based on the parsed text data.

[0012] A "means for converting to speech and outputting" is a device that converts the generated text response into speech using a speech synthesis engine and plays it through a speaker or other output device. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention relates to an offline natural language processing system that can operate even in an unstable or non-existent communication environment. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, thereby providing users with fast and accurate information and decision-making support.

[0035] Program processing overview

[0036] Voice input

[0037] It starts with the user speaking into the device's microphone, asking a question like, "What's the weather forecast for tomorrow?" The device captures this audio data.

[0038] Voice Recognition

[0039] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0040] natural language analysis

[0041] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0042] Response Generation

[0043] The device generates an appropriate response based on the analyzed text data. The response is intended to answer the user's question in natural language. For example, if asked about the weather forecast, the device generates a response such as "Tomorrow's weather will be sunny."

[0044] Audio Output

[0045] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0046] Specific examples

[0047] Examples of use in educational settings

[0048] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0049] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0050] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0051] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0052] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0053] Examples of use in disaster relief situations

[0054] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0055] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0056] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0057] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0058] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0059] As described above, the present invention realizes advanced natural language processing and can provide users with fast and accurate information even in an offline environment that does not require an internet connection. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] Acquiring voice input

[0063] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[0064] The terminal captures the user's voice data through a microphone.

[0065] Step 2:

[0066] Audio data conversion

[0067] The terminal sends the captured voice data to a voice recognition engine.

[0068] The voice recognition engine analyzes the input voice data and converts it into text data.

[0069] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[0070] Step 3:

[0071] Text data analysis

[0072] The terminal inputs the converted text data into a small language model (SLM).

[0073] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[0074] Step 4:

[0075] Generating a response

[0076] The terminal generates an appropriate response based on the analysis results.

[0077] For example, it generates a response such as "The weather will be sunny tomorrow."

[0078] When generating a response, the SLM's knowledge base or internal database may be referenced.

[0079] Step 5:

[0080] Response transcription

[0081] The terminal sends the generated text response to a speech synthesis engine.

[0082] The speech synthesis engine converts text data into natural-sounding speech.

[0083] Step 6:

[0084] Audio Output

[0085] The device plays the voice generated by the speech synthesis engine through the speaker.

[0086] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[0087] By using the above steps, the present invention can realize advanced natural language processing even in an offline environment where an Internet connection is not required, and can provide users with information quickly and accurately.

[0088] Example 1

[0089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0090] Voice interfaces utilizing natural language processing technology are widespread these days, but few systems can achieve high-precision natural language processing even in situations where the communication environment is unstable or non-existent. Furthermore, in fields such as disaster relief, education, and research, there is a demand for reliable information provision even in offline environments. Therefore, a natural language processing system that can operate in offline environments and respond to diverse user information requests is needed.

[0091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0092] In this invention, the server includes means for acquiring voice data, means for analyzing the acquired voice data and converting it into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for generating an appropriate response based on the analyzed text data, and means for converting the generated response into voice and outputting it. This enables high-precision natural language processing to be achieved without being affected by the communication environment, enabling the rapid and accurate provision of information in a wide range of fields such as disaster relief, education, and research.

[0093] "Audio data" is data that represents an audio signal in digital form.

[0094] "Capturing means" refers to the hardware and software capabilities for capturing audio data.

[0095] The "means for analyzing and converting into text data" is a function that converts voice data into a character string format using a voice recognition engine.

[0096] "Text data" is digital data expressed in the form of a string of characters.

[0097] "Means for natural language analysis" refers to a function that uses natural language processing technology to analyze the context and meaning of text data.

[0098] A "small language model" is a lightweight and efficient language model that is primarily used for natural language processing tasks.

[0099] The "means for generating an appropriate response" is a function for generating a response corresponding to a user's request based on the results of natural language analysis.

[0100] The "means for converting the generated response into voice and outputting it" is a function that converts text data into voice data using a voice synthesis engine and outputs it through a speaker or the like.

[0101] "Disaster relief" refers to activities aimed at saving lives and reducing damage when a disaster occurs.

[0102] "Education" is an activity that transmits and acquires knowledge and skills.

[0103] "Research" is a scientific activity aimed at discovering and developing new knowledge and technologies.

[0104] The "short-time Fourier transform" is an analytical technique for converting a time-domain signal into the frequency domain.

[0105] A "feature" is an important value or pattern extracted from input data in data analysis or machine learning.

[0106] A "phoneme" is the smallest unit of speech in spoken languages.

[0107] "Tokenization" is the process of dividing text data into the smallest units such as words or characters.

[0108] A "self-attention mechanism" is a mechanism that calculates how each part of the input data is related to other parts.

[0109] This invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data from a user and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, enabling it to provide fast and accurate information and support decision-making.

[0110] To implement this system, the following hardware and software are used.

[0111] Hardware

[0112] Terminal: A device for performing the operations of this system, typically including a microphone, speaker, and processor.

[0113] Microphone: A device for capturing audio data

[0114] Speaker: A device that presents generated audio to the user.

[0115] processor: hardware for data analysis and processing

[0116] software

[0117] Speech recognition engine: Uses Google Speech-to-Text, DeepSpeech, etc. to convert voice data into text data.

[0118] Small Language Models (SLMs): Analyze the context and meaning of text data using models such as DistilBERT and TinyBERT.

[0119] Speech synthesis engine: Uses a tool such as Google Text-to-Speech or Amazon Polly to convert the generated text data into speech.

[0120] Specific processing flow

[0121] 1. The user speaks into the device's microphone, for example, asking a question such as "What's the weather forecast for tomorrow?"

[0122] 2. The device uses a microphone to capture audio data and saves it in linear PCM format.

[0123] 3. The device converts the captured voice data into text using a voice recognition engine. During this process, the voice signal is converted into features using a short-time Fourier transform (STFT) and segmented into phonemes.

[0124] 4. The device inputs the converted text data into a small-scale language model (such as DistilBERT or TinyBERT) for natural language analysis. Based on the analysis results, it understands the user's intent and generates an appropriate response.

[0125] 5. The device uses a rule-based system or a generative AI model (e.g., GPT-3) to generate a response, such as "Tomorrow's weather forecast is sunny."

[0126] 6. The device inputs the generated text data into a speech synthesis engine and converts it into voice data. Finally, the generated voice is provided to the user through the speaker.

[0127] Specific examples

[0128] Examples of use in educational settings

[0129] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[0130] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0131] 3. The device inputs this text into a small language model and analyzes it. The model recognizes the keyword "Pythagoras' theorem."

[0132] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0133] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0134] Examples of use in disaster relief situations

[0135] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0136] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0137] 3. The device inputs this text into a small language model for analysis, which recognizes the keywords "medical facility" and "nearest."

[0138] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0139] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0140] In this way, the present invention achieves highly accurate natural language processing even in an offline environment, enabling users to quickly and accurately provide information. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The user speaks into the device's microphone. For example, they might ask, "What's the weather forecast for tomorrow?" The device captures this voice data, which is then saved in linear PCM format.

[0144] Step 2:

[0145] The device inputs the acquired voice data into a voice recognition engine (e.g., Google Speech-to-Text or DeepSpeech). The voice recognition engine converts the voice signal into features using a short-time Fourier transform (STFT) and divides it into phonemes. This feature data is used to analyze the voice, and finally converts it into text data such as "What's the weather forecast for tomorrow?" The input is voice data, and the output is text data.

[0146] Step 3:

[0147] The device inputs the converted text data into a small-scale language model (SLM, e.g., DistilBERT or TinyBERT). The SLM tokenizes and analyzes the input text data to understand the context and meaning of the text. Specifically, it uses a self-attention mechanism to analyze the relationships between each token and understand the user's intent. The input for this step is the text data, and the output is the analysis results, which include the user's intent and contextual information.

[0148] Step 4:

[0149] The device generates a response based on the results of natural language analysis. It uses a generative AI model (e.g., GPT-3) or a rule-based system to derive an appropriate response based on the analysis results. In this case, it accesses a weather forecast API to obtain the necessary information and generates the response "Tomorrow's weather will be sunny." The input is the analysis result, and the output is the response text.

[0150] Step 5:

[0151] The device inputs the generated response text into a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The speech synthesis engine converts the input string into phonemes and uses an acoustic model to generate a smooth speech signal. Finally, this speech signal is played back through the speaker. The input is the response text, and the output is speech data.

[0152] Step 6:

[0153] The user receives an answer to the question by listening to the generated voice response from the speaker. For example, the voice response may say, "The weather will be sunny tomorrow." In this step, the voice data is provided to the user.

[0154] This series of steps enables fast and accurate information provision even in situations where communication is unstable or non-existent.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] In conventional factory environments, workers need complex operation interfaces and internet connections to give instructions to robots, resulting in problems of reduced efficiency and safety. Furthermore, these systems fail to function properly if the communication environment is unstable or non-existent. The present invention aims to solve these problems by enabling high-performance natural language processing even in offline environments, allowing workers to easily operate robots.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the system includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for receiving instructions from a worker in a factory environment by voice, understanding the instructions, and performing an appropriate operation, and control means for operating the above means in the robot. This enables factory robots to be operated by voice instructions using natural language processing without relying on the communication environment.

[0160] "Means for capturing voice data" refers to a technology that uses a voice input device to capture a user's speech as a digital signal and then stores or transfers that data in a form that can be processed.

[0161] The "means for converting captured voice data into text data" refers to a process that utilizes voice recognition technology to analyze captured voice data and convert it into corresponding character string data.

[0162] "Means for natural language analysis of text data" refers to programs and algorithms for understanding and analyzing the context and meaning of converted string data.

[0163] The "means for generating a response based on the analyzed text data" is a process for generating an appropriate response to a user's question or instruction using the results of natural language analysis.

[0164] The "means for converting the generated response into voice and outputting it" is a technology that converts the response generated as character string data into voice data using voice synthesis technology and plays it back from an output device such as a speaker.

[0165] "Means of receiving voice instructions from workers in a factory environment, understanding the content of the instructions, and executing appropriate actions" refers to a series of processes that enable a robot to recognize voice instructions given by workers in a factory, analyze the content of those instructions, and autonomously perform the required actions.

[0166] The "control means for operating the means in the robot" refers to hardware and software for the robot to receive instructions for executing each function and to control each operation based on those instructions.

[0167] This invention provides an offline natural language processing system that enables workers to give voice instructions to robots in a factory environment. This system realizes advanced natural language processing even when not connected to the Internet, thereby improving work efficiency and safety.

[0168] The following hardware and software are used to implement the invention.

[0169] Hardware / Software used

[0170] Device: Factory robot (with built-in system for voice input and output)

[0171] Speech recognition engine: Offline speech recognition engine (e.g. PocketSphinx)

[0172] Natural Language Analysis: Small language models (e.g., custom models in spaCy)

[0173] Speech synthesis engine: Open source speech synthesis engine (e.g. eSpeak)

[0174] Processing Overview

[0175] 1. Voice input:

[0176] The worker speaks commands, such as "take out the part," into the robot's microphone, and the device captures this audio.

[0177] 2. Speech Recognition:

[0178] The robot converts the captured voice data into text data using a speech recognition engine. For example, it converts "take out the parts" into the string "buhin wo toridashite."

[0179] 3. Natural language analysis:

[0180] The converted text data is then fed into a small language model for analysis. The software recognizes the keywords "parts" and "extract" and understands their context and meaning.

[0181] 4. Response Generation:

[0182] Based on the parsed text data, the robot generates an appropriate response, for example, "Pick up part A."

[0183] 5. Audio output:

[0184] The generated text response is converted into speech by a speech synthesis engine and played back through the robot's speaker.

[0185] Specific examples

[0186] Usage Scenario 1

[0187] 1. The worker tells the robot, "Please take out the next product."

[0188] 2. The robot records the voice and sends the data to the speech recognition engine.

[0189] 3. The speech recognition engine converts the speech into text.

[0190] 4. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0191] 5. The robot generates an appropriate response and converts it into speech using a speech synthesis engine.

[0192] 6. The robot responds, "I'll pick up the next product," and performs the instructed task.

[0193] Prompt Sentence Examples

[0194] A user says to a factory robot: "Pick up the next product."

[0195] The robot prepares a response in the following steps:

[0196] 1. The speech recognition engine converts speech into text.

[0197] 2. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0198] 3. Generate an appropriate response and convert it into speech using a text-to-speech engine.

[0199] As a result, the robot will respond with "Pick up next product."

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] The user issues voice commands into the robot's microphone. This voice includes specific instructions such as "Please take out the next product." The robot captures this voice data and records it as a digital signal. The input of this step is the voice signal, and the output is the captured digital voice data.

[0203] Step 2:

[0204] The device's speech recognition engine receives the captured voice data and converts the voice into text data. An offline speech recognition engine such as PocketSphinx is used to convert the voice signal into a string of characters. In this step, the voice signal is analyzed using techniques such as frequency analysis and pattern recognition, and mapped to a corresponding string of characters. The input is digital voice data, and the output is the corresponding text data.

[0205] Step 3:

[0206] The device's natural language analysis engine receives the converted text data and analyzes it using a small language model (for example, a custom model from spaCy). During the analysis process, it understands the context and meaning of the text and extracts keywords (e.g., "next product," "take out") The input is the text data, and the output is the keywords and their contextual information. Statistical and rule-based natural language processing techniques are used in this step.

[0207] Step 4:

[0208] The device generates an appropriate response based on the parsed text data. A generative AI model is used to form an answer based on the instruction. For example, the response to "Please pick up the next product" will be "Please pick up the next product." The input is contextual information and keywords, and the output is the response text. This step uses an algorithm to construct an appropriate response based on the existing context.

[0209] Step 5:

[0210] The device sends the generated text data to a speech synthesis engine (e.g., eSpeak) and converts it into voice data. The speech synthesis engine converts the text data into voice signals and generates an audio file to convey to the user. The input is the response text and the output is the voice data. In this step, text-to-voice conversion technology is used, and the generated voice is played from the speaker.

[0211] Step 6:

[0212] The robot executes a specific action based on the voice command. For example, to start picking up a specified product, it executes an operating program. The input is the voice command and the analysis result, and the output is a physical action. In this step, the control algorithm is executed, and the machine's operation is controlled.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] The present invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, speech output, and emotion recognition, enabling it to provide users with fast and accurate information and support for decision-making.

[0215] Program processing overview

[0216] Voice input

[0217] It starts with the user speaking a question or request into the device's microphone, such as "What's the weather forecast for tomorrow?", and the device captures this audio data.

[0218] Voice Recognition

[0219] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0220] natural language analysis

[0221] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0222] emotion recognition

[0223] The device uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice from the audio data to estimate the user's emotional state. For example, emotions such as "anger," "joy," and "sadness" can be identified.

[0224] Response Generation and Adjustment

[0225] The device generates appropriate responses based on the results of natural language analysis and emotion recognition. Responses are written in natural language as answers to the user's questions. Furthermore, the device adjusts the content and tone of the response based on the user's emotions as recognized by the emotion engine. For example, if the user is feeling stressed, the response will be more polite and reassuring.

[0226] Audio Output

[0227] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0228] Specific examples

[0229] Examples of use in educational settings

[0230] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0231] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0232] 3. The terminal inputs this text into the SLM for analysis. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0233] 4. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[0234] 5. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right triangle is equal to the sum of the squares of the other two sides." If the device recognizes the user's interest, it provides further examples or additional explanations.

[0235] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0236] Examples of use in disaster relief situations

[0237] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0238] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0239] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0240] 4. The device uses an emotion engine to recognize that rescue workers are in an emergency situation.

[0241] 5. The device generates a response: "The nearest medical facility is 3 kilometers east." If the device detects that the user is nervous, it adjusts the response to provide polite guidance in a calm voice.

[0242] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0243] As described above, the present invention realizes advanced natural language processing and emotion recognition in an offline environment that does not require a communication environment, and can provide users with information that is fast, accurate, and considers their emotions. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0244] The processing flow will be explained below.

[0245] Step 1:

[0246] Acquiring voice input

[0247] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[0248] The terminal captures the user's voice data through a microphone.

[0249] Step 2:

[0250] Audio data conversion

[0251] The terminal sends the captured voice data to a voice recognition engine.

[0252] The voice recognition engine analyzes the input voice data and converts it into text data.

[0253] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[0254] Step 3:

[0255] emotion recognition

[0256] The terminal transmits the voice data to the emotion engine.

[0257] The emotion engine analyzes the tone, pitch, and speed of the user's voice to estimate the user's emotional state.

[0258] The emotion engine identifies emotions such as "joy," "sadness," and "anger."

[0259] Step 4:

[0260] Text data analysis

[0261] The terminal inputs the converted text data into a small language model (SLM).

[0262] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[0263] Step 5:

[0264] Generating a response

[0265] The terminal generates an appropriate response based on the results of natural language analysis and emotion recognition.

[0266] Responses are formulated in natural language in response to the user's question.

[0267] For example, it generates a response such as "The weather will be sunny tomorrow."

[0268] Step 6:

[0269] Response adjustment

[0270] The terminal adjusts the content and tone of the response based on the user's emotions recognized by the emotion engine.

[0271] For example, if the user is feeling stressed, the response will be more polite and reassuring.

[0272] Step 7:

[0273] Response transcription

[0274] The terminal sends the generated text response to a speech synthesis engine.

[0275] The speech synthesis engine converts text data into natural-sounding speech.

[0276] Step 8:

[0277] Audio Output

[0278] The device plays the voice generated by the speech synthesis engine through the speaker.

[0279] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[0280] Example 2

[0281] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0282] There is a demand for natural language processing systems that can operate even in unstable or non-existent communication environments and provide accurate and prompt information from voice data. However, conventional systems have problems such as a poor user experience because they are highly dependent on the communication environment for speech recognition and response generation, and do not generate responses that take the user's emotional state into consideration.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0284] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for recognizing an emotion to estimate an emotional state from the voice data, means for generating a response based on the analyzed text data and the emotional state, and means for converting the generated response into voice and outputting it. This enables prompt and accurate information provision that is independent of the communication environment and takes into consideration the emotional state of the user.

[0285] "Voice data" is digital data that captures a user's speech using a voice input device such as a microphone.

[0286] "Text data" is data in the form of a character string converted from voice data by a voice recognition engine.

[0287] "Natural language analysis" is the process of analyzing the context and meaning of text data to understand the user's intent.

[0288] A "small language model" is a language model that is optimized for a specific task and is relatively lightweight and fast to operate.

[0289] "Emotion recognition" is the process of analyzing the tone, pitch, speed, etc. of a user's voice from audio data to estimate the user's emotional state.

[0290] "Response generation" is the process of creating an appropriate response based on the results of natural language analysis and emotion recognition.

[0291] "Speech synthesis" is a technique for converting generated text responses into speech and providing it to the user.

[0292] This invention is a system that combines speech-based natural language processing and emotion recognition to provide users with fast and accurate information even in offline environments. This has the advantage of being able to provide decision-making support without relying on the communication environment.

[0293] System configuration

[0294] The system consists of the following main components:

[0295] 1. Audio Input Device: Includes a microphone to capture audio data spoken by the user, which is then stored in a digital format.

[0296] 2. Speech recognition engine: Converts captured voice data into text data. Examples include Google Speech-to-Text and Microsoft Azure Speech Service.

[0297] 3. Small Language Models (SLMs): Used for natural language analysis of text data to understand user intent. Examples include a scaled-down version of GPT-2.

[0298] 4. Emotion Recognition Engine: Analyzes the tone, pitch, and rate of voice from audio data to estimate the user's emotional state. Open source libraries (e.g., OpenSMILE) are used.

[0299] 5. Response generator: Generates appropriate text responses based on the results obtained from natural language analysis and emotion recognition.

[0300] 6. Speech synthesis engine: Converts the generated text response into speech and provides it to the user through a speaker. Examples include Google Text-to-Speech and Amazon Polly.

[0301] System Operation

[0302] The system works as follows:

[0303] A user speaks a question or request into the device's microphone. For example, "What is the weather forecast for tomorrow?" The device captures this voice data and converts it into text using a speech recognition engine. The converted text data is then input into a small language model (SLM) for natural language analysis.

[0304] The device then uses an emotion recognition engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice data to infer the user's emotional state. An appropriate response text is generated based on the analysis and emotion recognition results.

[0305] The generated response text is converted into speech using a speech synthesis engine, and finally, this speech is played back through a speaker and conveyed to the user, thereby providing fast and accurate information.

[0306] Specific examples

[0307] Examples of use in educational settings

[0308] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[0309] 2. The device uses a microphone to capture the audio and stores it as digital audio data.

[0310] 3. The device uses a speech recognition engine to convert the text into "Please tell me about Pythagoras' theorem."

[0311] 4. The terminal inputs this text into the SLM for analysis. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0312] 5. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[0313] 6. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right-angled triangle is equal to the sum of the squares of the other two sides." If the device recognizes that the user is interested, it provides further examples and explanations, such as, "An example would be a right-angled triangle with sides measuring 3 cm, 4 cm, and 5 cm."

[0314] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to listen to the explanation and deepen their understanding.

[0315] Examples of use in disaster relief situations

[0316] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0317] 2. The device uses a microphone to capture the audio and stores it as digital audio data.

[0318] 3. The device uses a speech recognition engine to convert the text into "Please tell me where the nearest medical facility is."

[0319] 4. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0320] 5. The device uses an emotion engine to recognize when rescuers are in an emergency situation, gauge their level of tension from their tone of voice, and generate a quick and specific response.

[0321] 6. The device generates a response saying, "The nearest medical facility is three kilometers east." If the device detects that the user is nervous, it adjusts the response to be calm and polite, such as, "Please stay calm. It's three kilometers east."

[0322] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to act quickly based on this information.

[0323] This series of processes can be carried out offline as well, allowing users to obtain the information they need without relying on the communication environment.

[0324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0325] Step 1:

[0326] The user speaks a question or request into the device's microphone. The input is the user's voice data. A specific example of this is when the user speaks something like, "What's the weather forecast for tomorrow?" The output is captured as voice data on the device.

[0327] Step 2:

[0328] The device passes the captured voice data to a voice recognition engine. The voice recognition engine (for example, Google Speech-to-Text) is used to convert the voice data into text data. The input is the user's voice data. Specifically, the voice recognition engine analyzes the voice signal and outputs the text "What is the weather forecast for tomorrow?"

[0329] Step 3:

[0330] The device inputs the converted text data into a small language model (e.g., a miniature version of GPT-2) to perform natural language analysis. The input is the text "What is the weather forecast for tomorrow?". Specifically, the SLM divides the text into tokens and analyzes the context and meaning. The output is an analysis result that indicates the intent of the user's question.

[0331] Step 4:

[0332] The device passes the voice data to an emotion recognition engine (e.g., OpenSMILE) to estimate the user's emotional state. The input is the user's voice data. Specifically, the emotion recognition engine analyzes the tone, pitch, and speed of the voice and outputs the user's emotional state (e.g., "neutral," "excited," "anxious," etc.).

[0333] Step 5:

[0334] The device generates an appropriate response based on the results of natural language analysis and emotion recognition. The input is the analysis result and the emotional state. Specifically, the SLM generates a response text such as "Tomorrow's weather forecast is sunny." At the same time, it adjusts the content and tone of the response based on the emotion recognition results. The output is the adjusted response text.

[0335] Step 6:

[0336] The device passes the generated response text to a speech synthesis engine (e.g., Google Text-to-Speech) to convert it into speech. The input is the tailored response text. Specifically, the speech synthesis engine converts the text into speech and plays it through the speaker. The output is a spoken response that is communicated to the user.

[0337] Through these steps, the system will be able to achieve advanced natural language processing and emotion recognition even in offline environments, enabling it to provide users with fast and accurate information.

[0338] (Application example 2)

[0339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] There is a need to provide a means for workers and managers to smoothly communicate with factory robots even in situations where the communication environment is unstable or non-existent. In particular, there is a need for a system that can quickly and accurately process voice instructions and reports and generate responses that take into account the emotional state of the worker, thereby increasing productivity and improving the working environment.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0342] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for recognizing emotions, and means for adjusting the content and tone of the response based on the emotion. This enables workers and managers to have natural conversations with factory robots even in unstable communication environments, enabling efficient work execution through appropriate work instructions, reports, and questions. Furthermore, responses that take into account the emotional state of workers are expected to improve the working environment and reduce stress.

[0343] A "means for capturing audio data" is a means for collecting audio signals as digital data using a device such as a microphone.

[0344] The "means for converting captured voice data into text data" refers to a means for converting collected voice data into a string of characters using a voice recognition engine or software.

[0345] "Means for natural language analysis of text data" refers to means for analyzing the context and meaning of text data using a natural language processing engine.

[0346] The "means for generating a response based on the analyzed text data" is a means for creating an appropriate reply based on the analysis results.

[0347] "Means for converting the generated response into speech and outputting it" means means for converting text into speech using a text-to-speech (TTS) engine and outputting it via a speaker or the like.

[0348] The "means for recognizing emotions" is a means for analyzing emotional information from voice or text data and identifying the emotional state of the user.

[0349] The "means for adjusting content and tone based on emotion when generating a response" refers to a means for adjusting the content of a reply or tone of voice based on recognized emotional information to create a natural and considerate response.

[0350] A "small language model" is a lightweight natural language processing model specialized for a specific task or application, containing algorithms for quickly and efficiently analyzing the context and meaning of text.

[0351] MODE FOR CARRYING OUT THE INVENTION

[0352] The present invention is a system for seamless communication between workers and managers through factory robots. The system operates even in unstable or non-existent communication environments, processes voice instructions and reports from workers in real time, and generates appropriate responses.

[0353] Hardware and software used

[0354] Hardware: Factory robot with microphone and speaker

[0355] software:

[0356] Speech Recognition Engine: SpeechRecognition Library

[0357] Natural Language Processing Engine: Hugging Face's Transformer Library

[0358] Emotion recognition engine: A model embedded within a natural language analysis engine

[0359] Text-to-Speech (TTS) engine: gTTS library

[0360] System Overview

[0361] The system consists of the following main processing steps: First, the factory robot's microphone captures the worker's voice data. The voice data is converted into text data through a speech recognition engine. Next, a natural language processing engine analyzes the text data, and an emotion recognition engine identifies the worker's emotions from the text data. The system generates an appropriate response based on these analysis results. The generated response is converted into speech by a TTS engine and provided to the worker through a speaker.

[0362] Specific examples

[0363] Example 1: Receiving a work order

[0364] When a worker says to the robot, "Please bring me a new part," the robot captures the speech and converts it into text using a speech recognition engine: "Please bring me a new part." The natural language processing engine then analyzes the instruction, and the emotion recognition engine recognizes the worker's emotion as "neutral." The system generates a response: "Got it. I'll start working on your request," which is then converted into speech using a TTS engine and played back.

[0365] Example 2: Problem report

[0366] When a worker says "The machine is broken," the robot captures the speech and converts it into text using a speech recognition engine. A natural language processing engine then analyzes the report, and an emotion recognition engine recognizes that the worker is anxious. The system generates a response: "I will report this to the manager immediately. It's okay, please stay calm," which is then converted into audio using a TTS engine and played back.

[0367] Prompt Sentence Examples

[0368] If the user says "The machine is broken," the following prompt will be generated:

[0369] Text: "Machine is out of order"

[0370] Task: Report a problem

[0371] If the user says "Bring me a new part," the following prompt will be generated:

[0372] Text: "Bring me new parts"

[0373] Task: Work Order

[0374] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0375] Step 1:

[0376] Voice input

[0377] The user gives instructions or reports by voice into the microphone of the factory robot, and this voice data is captured by the microphone. The input of the processing is the user's voice data, and the output is a digital audio file.

[0378] Step 2:

[0379] Voice Recognition

[0380] The device converts the captured voice data into text data using the SpeechRecognition library. The input is voice data and the output is text data. Specifically, the device analyzes the voice signal using techniques such as the Fast Fourier Transform (FFT) and converts the content into a string of characters.

[0381] Step 3:

[0382] natural language analysis

[0383] The device then analyzes the converted text data using Hugging Face's Transformer library. The input is text data, and the output is an analysis of the text's context and meaning. Specifically, a zero-shot classification model is used to analyze the text's context. At this stage, the device classifies the text as an instruction, a report, or a question.

[0384] Step 4:

[0385] emotion recognition

[0386] The device uses an emotion recognition engine to identify the user's emotions based on text data obtained through natural language analysis. The input is text data, and the output is the identified emotional state. Specifically, the device analyzes the tone, pitch, and speed of the voice and text to classify the user's emotions.

[0387] Step 5:

[0388] Response Generation

[0389] The device generates an appropriate response based on the analyzed text data and emotional information. The input is the context analysis results and emotional information, and the output is the generated response text. Specifically, the response generation engine generates a response such as "Understood. We will begin working on your request" in the case of an instruction, or "We will report to the administrator immediately" in the case of a report. The content and tone are also adjusted depending on the emotion.

[0390] Step 6:

[0391] Audio Output

[0392] The device converts the generated response text into speech using the gTTS library and plays it from the speaker. The input is the generated response text, and the output is the audio to be played. Specifically, the text-to-speech (TTS) engine converts the text into a speech waveform and outputs it from the speaker.

[0393] These processing steps enable natural voice interaction with the user, allowing the factory robot to communicate smoothly with workers and managers.

[0394] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0395] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0396] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0397] [Second embodiment]

[0398] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0399] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0400] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0401] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0402] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0403] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0404] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0405] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0406] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0407] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0408] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0409] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0410] The present invention relates to an offline natural language processing system that can operate even in an unstable or non-existent communication environment. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, thereby providing users with fast and accurate information and decision-making support.

[0411] Program processing overview

[0412] Voice input

[0413] It starts with the user speaking into the device's microphone, asking a question like, "What's the weather forecast for tomorrow?" The device captures this audio data.

[0414] Voice Recognition

[0415] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0416] natural language analysis

[0417] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0418] Response Generation

[0419] The device generates an appropriate response based on the analyzed text data. The response is intended to answer the user's question in natural language. For example, if asked about the weather forecast, the device generates a response such as "Tomorrow's weather will be sunny."

[0420] Audio Output

[0421] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0422] Specific examples

[0423] Examples of use in educational settings

[0424] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0425] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0426] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0427] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0428] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0429] Examples of use in disaster relief situations

[0430] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0431] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0432] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0433] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0434] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0435] As described above, the present invention realizes advanced natural language processing and can provide users with fast and accurate information even in an offline environment that does not require an internet connection. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0436] The processing flow will be explained below.

[0437] Step 1:

[0438] Acquiring voice input

[0439] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[0440] The terminal captures the user's voice data through a microphone.

[0441] Step 2:

[0442] Audio data conversion

[0443] The terminal sends the captured voice data to a voice recognition engine.

[0444] The voice recognition engine analyzes the input voice data and converts it into text data.

[0445] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[0446] Step 3:

[0447] Text data analysis

[0448] The terminal inputs the converted text data into a small language model (SLM).

[0449] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[0450] Step 4:

[0451] Generating a response

[0452] The terminal generates an appropriate response based on the analysis results.

[0453] For example, it generates a response such as "The weather will be sunny tomorrow."

[0454] When generating a response, the SLM's knowledge base or internal database may be referenced.

[0455] Step 5:

[0456] Response transcription

[0457] The terminal sends the generated text response to a speech synthesis engine.

[0458] The speech synthesis engine converts text data into natural-sounding speech.

[0459] Step 6:

[0460] Audio Output

[0461] The device plays the voice generated by the speech synthesis engine through the speaker.

[0462] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[0463] By using the above steps, the present invention can realize advanced natural language processing even in an offline environment where an Internet connection is not required, and can provide users with information quickly and accurately.

[0464] Example 1

[0465] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] Voice interfaces utilizing natural language processing technology are widespread these days, but few systems can achieve high-precision natural language processing even in situations where the communication environment is unstable or non-existent. Furthermore, in fields such as disaster relief, education, and research, there is a demand for reliable information provision even in offline environments. Therefore, a natural language processing system that can operate in offline environments and respond to diverse user information requests is needed.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes means for acquiring voice data, means for analyzing the acquired voice data and converting it into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for generating an appropriate response based on the analyzed text data, and means for converting the generated response into voice and outputting it. This enables high-precision natural language processing to be achieved without being affected by the communication environment, enabling the rapid and accurate provision of information in a wide range of fields such as disaster relief, education, and research.

[0469] "Audio data" is data that represents an audio signal in digital form.

[0470] "Capturing means" refers to the hardware and software capabilities for capturing audio data.

[0471] The "means for analyzing and converting into text data" is a function that converts voice data into a character string format using a voice recognition engine.

[0472] "Text data" is digital data expressed in the form of a string of characters.

[0473] "Means for natural language analysis" refers to a function that uses natural language processing technology to analyze the context and meaning of text data.

[0474] A "small language model" is a lightweight and efficient language model that is primarily used for natural language processing tasks.

[0475] The "means for generating an appropriate response" is a function for generating a response corresponding to a user's request based on the results of natural language analysis.

[0476] The "means for converting the generated response into voice and outputting it" is a function that converts text data into voice data using a voice synthesis engine and outputs it through a speaker or the like.

[0477] "Disaster relief" refers to activities aimed at saving lives and reducing damage when a disaster occurs.

[0478] "Education" is an activity that transmits and acquires knowledge and skills.

[0479] "Research" is a scientific activity aimed at discovering and developing new knowledge and technologies.

[0480] The "short-time Fourier transform" is an analytical technique for converting a time-domain signal into the frequency domain.

[0481] A "feature" is an important value or pattern extracted from input data in data analysis or machine learning.

[0482] A "phoneme" is the smallest unit of speech in spoken languages.

[0483] "Tokenization" is the process of dividing text data into the smallest units such as words or characters.

[0484] A "self-attention mechanism" is a mechanism that calculates how each part of the input data is related to other parts.

[0485] This invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data from a user and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, enabling it to provide fast and accurate information and support decision-making.

[0486] To implement this system, the following hardware and software are used.

[0487] Hardware

[0488] Terminal: A device for performing the operations of this system, typically including a microphone, speaker, and processor.

[0489] Microphone: A device for capturing audio data

[0490] Speaker: A device that presents generated audio to the user.

[0491] processor: hardware for data analysis and processing

[0492] software

[0493] Speech recognition engine: Uses Google Speech-to-Text, DeepSpeech, etc. to convert voice data into text data.

[0494] Small Language Models (SLMs): Analyze the context and meaning of text data using models such as DistilBERT and TinyBERT.

[0495] Speech synthesis engine: Uses a tool such as Google Text-to-Speech or Amazon Polly to convert the generated text data into speech.

[0496] Specific processing flow

[0497] 1. The user speaks into the device's microphone, for example, asking a question such as "What's the weather forecast for tomorrow?"

[0498] 2. The device uses a microphone to capture audio data and saves it in linear PCM format.

[0499] 3. The device converts the captured voice data into text using a voice recognition engine. During this process, the voice signal is converted into features using a short-time Fourier transform (STFT) and segmented into phonemes.

[0500] 4. The device inputs the converted text data into a small-scale language model (such as DistilBERT or TinyBERT) for natural language analysis. Based on the analysis results, it understands the user's intent and generates an appropriate response.

[0501] 5. The device uses a rule-based system or a generative AI model (e.g., GPT-3) to generate a response, such as "Tomorrow's weather forecast is sunny."

[0502] 6. The device inputs the generated text data into a speech synthesis engine and converts it into voice data. Finally, the generated voice is provided to the user through the speaker.

[0503] Specific examples

[0504] Examples of use in educational settings

[0505] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[0506] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0507] 3. The device inputs this text into a small language model and analyzes it. The model recognizes the keyword "Pythagoras' theorem."

[0508] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0509] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0510] Examples of use in disaster relief situations

[0511] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0512] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0513] 3. The device inputs this text into a small language model for analysis, which recognizes the keywords "medical facility" and "nearest."

[0514] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0515] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0516] In this way, the present invention achieves highly accurate natural language processing even in an offline environment, enabling users to quickly and accurately provide information. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0517] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0518] Step 1:

[0519] The user speaks into the device's microphone. For example, they might ask, "What's the weather forecast for tomorrow?" The device captures this voice data, which is then saved in linear PCM format.

[0520] Step 2:

[0521] The device inputs the acquired voice data into a voice recognition engine (e.g., Google Speech-to-Text or DeepSpeech). The voice recognition engine converts the voice signal into features using a short-time Fourier transform (STFT) and divides it into phonemes. This feature data is used to analyze the voice, and finally converts it into text data such as "What's the weather forecast for tomorrow?" The input is voice data, and the output is text data.

[0522] Step 3:

[0523] The device inputs the converted text data into a small-scale language model (SLM, e.g., DistilBERT or TinyBERT). The SLM tokenizes and analyzes the input text data to understand the context and meaning of the text. Specifically, it uses a self-attention mechanism to analyze the relationships between each token and understand the user's intent. The input for this step is the text data, and the output is the analysis results, which include the user's intent and contextual information.

[0524] Step 4:

[0525] The device generates a response based on the results of natural language analysis. It uses a generative AI model (e.g., GPT-3) or a rule-based system to derive an appropriate response based on the analysis results. In this case, it accesses a weather forecast API to obtain the necessary information and generates the response "Tomorrow's weather will be sunny." The input is the analysis result, and the output is the response text.

[0526] Step 5:

[0527] The device inputs the generated response text into a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The speech synthesis engine converts the input string into phonemes and uses an acoustic model to generate a smooth speech signal. Finally, this speech signal is played back through the speaker. The input is the response text, and the output is speech data.

[0528] Step 6:

[0529] The user receives an answer to the question by listening to the generated voice response from the speaker. For example, the voice response may say, "The weather will be sunny tomorrow." In this step, the voice data is provided to the user.

[0530] This series of steps enables fast and accurate information provision even in situations where communication is unstable or non-existent.

[0531] (Application example 1)

[0532] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] In conventional factory environments, workers need complex operation interfaces and internet connections to give instructions to robots, resulting in problems of reduced efficiency and safety. Furthermore, these systems fail to function properly if the communication environment is unstable or non-existent. The present invention aims to solve these problems by enabling high-performance natural language processing even in offline environments, allowing workers to easily operate robots.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0535] In this invention, the system includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for receiving instructions from a worker in a factory environment by voice, understanding the instructions, and performing an appropriate operation, and control means for operating the above means in the robot. This enables factory robots to be operated by voice instructions using natural language processing without relying on the communication environment.

[0536] "Means for capturing voice data" refers to a technology that uses a voice input device to capture a user's speech as a digital signal and then stores or transfers that data in a form that can be processed.

[0537] The "means for converting captured voice data into text data" refers to a process that utilizes voice recognition technology to analyze captured voice data and convert it into corresponding character string data.

[0538] "Means for natural language analysis of text data" refers to programs and algorithms for understanding and analyzing the context and meaning of converted string data.

[0539] The "means for generating a response based on the analyzed text data" is a process for generating an appropriate response to a user's question or instruction using the results of natural language analysis.

[0540] The "means for converting the generated response into voice and outputting it" is a technology that converts the response generated as character string data into voice data using voice synthesis technology and plays it back from an output device such as a speaker.

[0541] "Means of receiving voice instructions from workers in a factory environment, understanding the content of the instructions, and executing appropriate actions" refers to a series of processes that enable a robot to recognize voice instructions given by workers in a factory, analyze the content of those instructions, and autonomously perform the required actions.

[0542] The "control means for operating the means in the robot" refers to hardware and software for the robot to receive instructions for executing each function and to control each operation based on those instructions.

[0543] This invention provides an offline natural language processing system that enables workers to give voice instructions to robots in a factory environment. This system realizes advanced natural language processing even when not connected to the Internet, thereby improving work efficiency and safety.

[0544] The following hardware and software are used to implement the invention.

[0545] Hardware / Software used

[0546] Device: Factory robot (with built-in system for voice input and output)

[0547] Speech recognition engine: Offline speech recognition engine (e.g. PocketSphinx)

[0548] Natural Language Analysis: Small language models (e.g., custom models in spaCy)

[0549] Speech synthesis engine: Open source speech synthesis engine (e.g. eSpeak)

[0550] Processing Overview

[0551] 1. Voice input:

[0552] The worker speaks commands, such as "take out the part," into the robot's microphone, and the device captures this audio.

[0553] 2. Speech Recognition:

[0554] The robot converts the captured voice data into text data using a speech recognition engine. For example, it converts "take out the parts" into the string "buhin wo toridashite."

[0555] 3. Natural language analysis:

[0556] The converted text data is then fed into a small language model for analysis. The software recognizes the keywords "parts" and "extract" and understands their context and meaning.

[0557] 4. Response Generation:

[0558] Based on the parsed text data, the robot generates an appropriate response, for example, "Pick up part A."

[0559] 5. Audio output:

[0560] The generated text response is converted into speech by a speech synthesis engine and played back through the robot's speaker.

[0561] Specific examples

[0562] Usage Scenario 1

[0563] 1. The worker tells the robot, "Please take out the next product."

[0564] 2. The robot records the voice and sends the data to the speech recognition engine.

[0565] 3. The speech recognition engine converts the speech into text.

[0566] 4. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0567] 5. The robot generates an appropriate response and converts it into speech using a speech synthesis engine.

[0568] 6. The robot responds, "I'll pick up the next product," and performs the instructed task.

[0569] Prompt Sentence Examples

[0570] A user says to a factory robot: "Pick up the next product."

[0571] The robot prepares a response in the following steps:

[0572] 1. The speech recognition engine converts speech into text.

[0573] 2. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0574] 3. Generate an appropriate response and convert it into speech using a text-to-speech engine.

[0575] As a result, the robot will respond with "Pick up next product."

[0576] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0577] Step 1:

[0578] The user issues voice commands into the robot's microphone. This voice includes specific instructions such as "Please take out the next product." The robot captures this voice data and records it as a digital signal. The input of this step is the voice signal, and the output is the captured digital voice data.

[0579] Step 2:

[0580] The device's speech recognition engine receives the captured voice data and converts the voice into text data. An offline speech recognition engine such as PocketSphinx is used to convert the voice signal into a string of characters. In this step, the voice signal is analyzed using techniques such as frequency analysis and pattern recognition, and mapped to a corresponding string of characters. The input is digital voice data, and the output is the corresponding text data.

[0581] Step 3:

[0582] The device's natural language analysis engine receives the converted text data and analyzes it using a small language model (for example, a custom model from spaCy). During the analysis process, it understands the context and meaning of the text and extracts keywords (e.g., "next product," "take out") The input is the text data, and the output is the keywords and their contextual information. Statistical and rule-based natural language processing techniques are used in this step.

[0583] Step 4:

[0584] The device generates an appropriate response based on the parsed text data. A generative AI model is used to form an answer based on the instruction. For example, the response to "Please pick up the next product" will be "Please pick up the next product." The input is contextual information and keywords, and the output is the response text. This step uses an algorithm to construct an appropriate response based on the existing context.

[0585] Step 5:

[0586] The device sends the generated text data to a speech synthesis engine (e.g., eSpeak) and converts it into voice data. The speech synthesis engine converts the text data into voice signals and generates an audio file to convey to the user. The input is the response text and the output is the voice data. In this step, text-to-voice conversion technology is used, and the generated voice is played from the speaker.

[0587] Step 6:

[0588] The robot executes specific actions based on voice instructions. For example, it executes an operating program to start picking up a specified product. The input is voice instructions and analysis results, and the output is physical actions. In this step, the control algorithm is executed, controlling the machine's operations.

[0589] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0590] The present invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, speech output, and emotion recognition, enabling it to provide users with fast and accurate information and support for decision-making.

[0591] Program processing overview

[0592] Voice input

[0593] It starts with the user speaking a question or request into the device's microphone, such as "What's the weather forecast for tomorrow?", and the device captures this audio data.

[0594] Voice Recognition

[0595] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0596] natural language analysis

[0597] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0598] emotion recognition

[0599] The device uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice from the audio data to estimate the user's emotional state. For example, emotions such as "anger," "joy," and "sadness" can be identified.

[0600] Response Generation and Adjustment

[0601] The device generates appropriate responses based on the results of natural language analysis and emotion recognition. Responses are written in natural language as answers to the user's questions. Furthermore, the device adjusts the content and tone of the response based on the user's emotions as recognized by the emotion engine. For example, if the user is feeling stressed, the response will be more polite and reassuring.

[0602] Audio Output

[0603] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0604] Specific examples

[0605] Examples of use in educational settings

[0606] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0607] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0608] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0609] 4. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[0610] 5. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right triangle is equal to the sum of the squares of the other two sides." If the device recognizes the user's interest, it provides further examples or additional explanations.

[0611] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0612] Examples of use in disaster relief situations

[0613] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0614] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0615] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0616] 4. The device uses its emotion engine to recognize that rescue workers are in an emergency situation.

[0617] 5. The device generates a response: "The nearest medical facility is 3 kilometers east." If the device detects that the user is nervous, it adjusts the response to provide polite guidance in a calm voice.

[0618] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0619] As described above, the present invention realizes advanced natural language processing and emotion recognition in an offline environment that does not require a communication environment, and can provide users with information that is fast, accurate, and considers their emotions. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0620] The processing flow will be explained below.

[0621] Step 1:

[0622] Acquiring voice input

[0623] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[0624] The terminal captures the user's voice data through a microphone.

[0625] Step 2:

[0626] Audio data conversion

[0627] The terminal sends the captured voice data to a voice recognition engine.

[0628] The voice recognition engine analyzes the input voice data and converts it into text data.

[0629] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[0630] Step 3:

[0631] emotion recognition

[0632] The terminal transmits the voice data to the emotion engine.

[0633] The emotion engine analyzes the tone, pitch, and speed of the user's voice to estimate the user's emotional state.

[0634] The emotion engine identifies emotions such as "joy," "sadness," and "anger."

[0635] Step 4:

[0636] Text data analysis

[0637] The terminal inputs the converted text data into a small language model (SLM).

[0638] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[0639] Step 5:

[0640] Generating a response

[0641] The terminal generates an appropriate response based on the results of natural language analysis and emotion recognition.

[0642] Responses are formulated in natural language in response to the user's question.

[0643] For example, it generates a response such as "The weather will be sunny tomorrow."

[0644] Step 6:

[0645] Response adjustment

[0646] The terminal adjusts the content and tone of the response based on the user's emotions recognized by the emotion engine.

[0647] For example, if the user is feeling stressed, the response will be more polite and reassuring.

[0648] Step 7:

[0649] Response transcription

[0650] The terminal sends the generated text response to a speech synthesis engine.

[0651] The speech synthesis engine converts text data into natural-sounding speech.

[0652] Step 8:

[0653] Audio Output

[0654] The device plays the voice generated by the speech synthesis engine through the speaker.

[0655] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[0656] Example 2

[0657] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0658] There is a demand for natural language processing systems that can operate even in unstable or non-existent communication environments and provide accurate and prompt information from voice data. However, conventional systems have problems such as a poor user experience because they are highly dependent on the communication environment for speech recognition and response generation, and do not generate responses that take the user's emotional state into consideration.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0660] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for recognizing an emotion to estimate an emotional state from the voice data, means for generating a response based on the analyzed text data and the emotional state, and means for converting the generated response into voice and outputting it. This enables prompt and accurate information provision that is independent of the communication environment and takes into consideration the emotional state of the user.

[0661] "Voice data" is digital data that captures a user's speech using a voice input device such as a microphone.

[0662] "Text data" is data in the form of a character string converted from voice data by a voice recognition engine.

[0663] "Natural language analysis" is the process of analyzing the context and meaning of text data to understand the user's intent.

[0664] A "small language model" is a language model that is optimized for a specific task and is relatively lightweight and fast to operate.

[0665] "Emotion recognition" is the process of analyzing the tone, pitch, speed, etc. of a user's voice from audio data to estimate the user's emotional state.

[0666] "Response generation" is the process of creating an appropriate response based on the results of natural language analysis and emotion recognition.

[0667] "Speech synthesis" is a technique for converting generated text responses into speech and providing it to the user.

[0668] This invention is a system that combines speech-based natural language processing and emotion recognition to provide users with fast and accurate information even in offline environments. This has the advantage of being able to provide decision-making support without relying on the communication environment.

[0669] System configuration

[0670] The system consists of the following main components:

[0671] 1. Audio Input Device: Includes a microphone to capture audio data spoken by the user, which is then stored in a digital format.

[0672] 2. Speech recognition engine: Converts captured voice data into text data. Examples include Google Speech-to-Text and Microsoft Azure Speech Service.

[0673] 3. Small Language Models (SLMs): Used for natural language analysis of text data to understand user intent. Examples include a smaller version of GPT-2.

[0674] 4. Emotion Recognition Engine: Analyzes the tone, pitch, and rate of voice from audio data to estimate the user's emotional state. Open source libraries (e.g., OpenSMILE) are used.

[0675] 5. Response generator: Generates appropriate text responses based on the results obtained from natural language analysis and emotion recognition.

[0676] 6. Speech synthesis engine: Converts the generated text response into speech and provides it to the user through a speaker. Examples include Google Text-to-Speech and Amazon Polly.

[0677] System Operation

[0678] The system works as follows:

[0679] A user speaks a question or request into the device's microphone. For example, "What is the weather forecast for tomorrow?" The device captures this voice data and converts it into text using a speech recognition engine. The converted text data is then input into a small language model (SLM) for natural language analysis.

[0680] The device then uses an emotion recognition engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice data to infer the user's emotional state. An appropriate response text is generated based on the analysis and emotion recognition results.

[0681] The generated response text is converted into speech using a speech synthesis engine, and finally, this speech is played back through a speaker and conveyed to the user, thereby providing fast and accurate information.

[0682] Specific examples

[0683] Examples of use in educational settings

[0684] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[0685] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[0686] 3. The device uses a speech recognition engine to convert the text into "Please tell me about Pythagoras' theorem."

[0687] 4. The terminal inputs this text into the SLM for analysis. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0688] 5. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[0689] 6. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right-angled triangle is equal to the sum of the squares of the other two sides." If the device recognizes that the user is interested, it provides further examples and explanations, such as, "An example would be a right-angled triangle with sides measuring 3 cm, 4 cm, and 5 cm."

[0690] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to listen to the explanation and deepen their understanding.

[0691] Examples of use in disaster relief situations

[0692] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0693] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[0694] 3. The device uses a speech recognition engine to convert the text into "Please tell me where the nearest medical facility is."

[0695] 4. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0696] 5. The device uses an emotion engine to recognize when rescuers are in an emergency situation, gauge their level of tension from their tone of voice, and generate a quick and specific response.

[0697] 6. The device generates a response saying, "The nearest medical facility is three kilometers east." If the device detects that the user is nervous, it adjusts the response to be calm and polite, such as, "Please stay calm. It's three kilometers east."

[0698] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to act quickly based on this information.

[0699] This series of processes can be carried out offline as well, allowing users to obtain the information they need without relying on the communication environment.

[0700] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0701] Step 1:

[0702] The user speaks a question or request into the device's microphone. The input is the user's voice data. A specific example of this is when the user speaks something like, "What's the weather forecast for tomorrow?" The output is captured as voice data on the device.

[0703] Step 2:

[0704] The device passes the captured voice data to a voice recognition engine. The voice recognition engine (for example, Google Speech-to-Text) is used to convert the voice data into text data. The input is the user's voice data. Specifically, the voice recognition engine analyzes the voice signal and outputs the text "What is the weather forecast for tomorrow?"

[0705] Step 3:

[0706] The device inputs the converted text data into a small language model (e.g., a miniature version of GPT-2) to perform natural language analysis. The input is the text "What is the weather forecast for tomorrow?". Specifically, the SLM divides the text into tokens and analyzes the context and meaning. The output is an analysis result that indicates the intent of the user's question.

[0707] Step 4:

[0708] The device passes the voice data to an emotion recognition engine (e.g., OpenSMILE) to estimate the user's emotional state. The input is the user's voice data. Specifically, the emotion recognition engine analyzes the tone, pitch, and speed of the voice and outputs the user's emotional state (e.g., "neutral," "excited," "anxious," etc.).

[0709] Step 5:

[0710] The device generates an appropriate response based on the results of natural language analysis and emotion recognition. The input is the analysis result and the emotional state. Specifically, the SLM generates a response text such as "Tomorrow's weather forecast is sunny." At the same time, it adjusts the content and tone of the response based on the emotion recognition results. The output is the adjusted response text.

[0711] Step 6:

[0712] The device passes the generated response text to a speech synthesis engine (e.g., Google Text-to-Speech) to convert it into speech. The input is the tailored response text. Specifically, the speech synthesis engine converts the text into speech and plays it through the speaker. The output is a spoken response that is communicated to the user.

[0713] Through these steps, the system will be able to achieve advanced natural language processing and emotion recognition even in offline environments, enabling it to provide users with fast and accurate information.

[0714] (Application example 2)

[0715] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] There is a need to provide a means for workers and managers to smoothly communicate with factory robots even in situations where the communication environment is unstable or non-existent. In particular, there is a need for a system that can quickly and accurately process voice instructions and reports and generate responses that take into account the emotional state of the worker, thereby increasing productivity and improving the working environment.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0718] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for recognizing emotions, and means for adjusting the content and tone of the response based on the emotion. This enables workers and managers to have natural conversations with factory robots even in unstable communication environments, enabling efficient work execution through appropriate work instructions, reports, and questions. Furthermore, responses that take into account the emotional state of workers are expected to improve the working environment and reduce stress.

[0719] A "means for capturing audio data" is a means for collecting audio signals as digital data using a device such as a microphone.

[0720] The "means for converting captured voice data into text data" refers to a means for converting collected voice data into a string of characters using a voice recognition engine or software.

[0721] "Means for natural language analysis of text data" refers to means for analyzing the context and meaning of text data using a natural language processing engine.

[0722] The "means for generating a response based on the analyzed text data" is a means for creating an appropriate reply based on the analysis results.

[0723] "Means for converting the generated response into speech and outputting it" means means for converting text into speech using a text-to-speech (TTS) engine and outputting it via a speaker or the like.

[0724] The "means for recognizing emotions" is a means for analyzing emotional information from voice or text data and identifying the emotional state of the user.

[0725] The "means for adjusting content and tone based on emotion when generating a response" refers to a means for adjusting the content of a reply or tone of voice based on recognized emotional information to create a natural and considerate response.

[0726] A "small language model" is a lightweight natural language processing model specialized for a specific task or application, containing algorithms for quickly and efficiently analyzing the context and meaning of text.

[0727] MODE FOR CARRYING OUT THE INVENTION

[0728] The present invention is a system for seamless communication between workers and managers through factory robots. The system operates even in unstable or non-existent communication environments, processes voice instructions and reports from workers in real time, and generates appropriate responses.

[0729] Hardware and software used

[0730] Hardware: Factory robot with microphone and speaker

[0731] software:

[0732] Speech Recognition Engine: SpeechRecognition Library

[0733] Natural Language Processing Engine: Hugging Face's Transformer Library

[0734] Emotion recognition engine: A model embedded within a natural language analysis engine

[0735] Text-to-Speech (TTS) engine: gTTS library

[0736] System Overview

[0737] The system consists of the following main processing steps: First, the factory robot's microphone captures the worker's voice data. The voice data is converted into text data through a speech recognition engine. Next, a natural language processing engine analyzes the text data, and an emotion recognition engine identifies the worker's emotions from the text data. The system generates an appropriate response based on these analysis results. The generated response is converted into speech by a TTS engine and provided to the worker through a speaker.

[0738] Specific examples

[0739] Example 1: Receiving a work order

[0740] When a worker says to the robot, "Please bring me a new part," the robot captures the speech and converts it into text using a speech recognition engine: "Please bring me a new part." The natural language processing engine then analyzes the instruction, and the emotion recognition engine recognizes the worker's emotion as "neutral." The system generates a response: "Got it. I'll start working on your request," which is then converted into speech using a TTS engine and played back.

[0741] Example 2: Problem report

[0742] When a worker says "The machine is broken," the robot captures the speech and converts it into text using a speech recognition engine. A natural language processing engine then analyzes the report, and an emotion recognition engine recognizes that the worker is anxious. The system generates a response: "I will report this to the manager immediately. It's okay, please stay calm," which is then converted into audio using a TTS engine and played back.

[0743] Prompt Sentence Examples

[0744] If the user says "The machine is broken," the following prompt will be generated:

[0745] Text: "Machine is out of order"

[0746] Task: Report a problem

[0747] If the user says "Bring me a new part," the following prompt will be generated:

[0748] Text: "Bring me new parts"

[0749] Task: Work Order

[0750] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0751] Step 1:

[0752] Voice input

[0753] The user gives instructions or reports by voice into the microphone of the factory robot, and this voice data is captured by the microphone. The input of the processing is the user's voice data, and the output is a digital audio file.

[0754] Step 2:

[0755] Voice Recognition

[0756] The device converts the captured voice data into text data using the SpeechRecognition library. The input is voice data and the output is text data. Specifically, the device analyzes the voice signal using techniques such as the Fast Fourier Transform (FFT) and converts the content into a string of characters.

[0757] Step 3:

[0758] natural language analysis

[0759] The device then analyzes the converted text data using Hugging Face's Transformer library. The input is text data, and the output is an analysis of the text's context and meaning. Specifically, a zero-shot classification model is used to analyze the text's context. At this stage, the device classifies the text as an instruction, a report, or a question.

[0760] Step 4:

[0761] emotion recognition

[0762] The device uses an emotion recognition engine to identify the user's emotions based on text data obtained through natural language analysis. The input is text data, and the output is the identified emotional state. Specifically, the device analyzes the tone, pitch, and speed of the voice and text to classify the user's emotions.

[0763] Step 5:

[0764] Response Generation

[0765] The device generates an appropriate response based on the analyzed text data and emotional information. The input is the context analysis results and emotional information, and the output is the generated response text. Specifically, the response generation engine generates a response such as "Understood. We will begin working on your request" in the case of an instruction, or "We will report to the administrator immediately" in the case of a report. The content and tone are also adjusted depending on the emotion.

[0766] Step 6:

[0767] Audio Output

[0768] The device converts the generated response text into speech using the gTTS library and plays it from the speaker. The input is the generated response text, and the output is the audio to be played. Specifically, the text-to-speech (TTS) engine converts the text into a speech waveform and outputs it from the speaker.

[0769] These processing steps enable natural voice interaction with the user, allowing the factory robot to communicate smoothly with workers and managers.

[0770] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0771] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0773] [Third embodiment]

[0774] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0775] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0776] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0777] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0778] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0779] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0780] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0781] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0782] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0783] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0784] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0785] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0786] The present invention relates to an offline natural language processing system that can operate even in an unstable or non-existent communication environment. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, thereby providing users with fast and accurate information and decision-making support.

[0787] Program processing overview

[0788] Voice input

[0789] It starts with the user speaking into the device's microphone, asking a question like, "What's the weather forecast for tomorrow?" The device captures this audio data.

[0790] Voice Recognition

[0791] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0792] natural language analysis

[0793] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0794] Response Generation

[0795] The device generates an appropriate response based on the analyzed text data. The response is intended to answer the user's question in natural language. For example, if asked about the weather forecast, the device generates a response such as "Tomorrow's weather will be sunny."

[0796] Audio Output

[0797] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0798] Specific examples

[0799] Examples of use in educational settings

[0800] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0801] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0802] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0803] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0804] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0805] Examples of use in disaster relief situations

[0806] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0807] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0808] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0809] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0810] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0811] As described above, the present invention realizes advanced natural language processing and can provide users with fast and accurate information even in an offline environment that does not require an internet connection. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0812] The processing flow will be explained below.

[0813] Step 1:

[0814] Acquiring voice input

[0815] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[0816] The terminal captures the user's voice data through a microphone.

[0817] Step 2:

[0818] Audio data conversion

[0819] The terminal sends the captured voice data to a voice recognition engine.

[0820] The voice recognition engine analyzes the input voice data and converts it into text data.

[0821] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[0822] Step 3:

[0823] Text data analysis

[0824] The terminal inputs the converted text data into a small language model (SLM).

[0825] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[0826] Step 4:

[0827] Generating a response

[0828] The terminal generates an appropriate response based on the analysis results.

[0829] For example, it generates a response such as "The weather will be sunny tomorrow."

[0830] When generating a response, the SLM's knowledge base or internal database may be referenced.

[0831] Step 5:

[0832] Response transcription

[0833] The terminal sends the generated text response to a speech synthesis engine.

[0834] The speech synthesis engine converts text data into natural-sounding speech.

[0835] Step 6:

[0836] Audio Output

[0837] The device plays the voice generated by the speech synthesis engine through the speaker.

[0838] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[0839] By using the above steps, the present invention can realize advanced natural language processing even in an offline environment where an Internet connection is not required, and can provide users with information quickly and accurately.

[0840] Example 1

[0841] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0842] Voice interfaces utilizing natural language processing technology are widespread these days, but few systems can achieve high-precision natural language processing even in situations where the communication environment is unstable or non-existent. Furthermore, in fields such as disaster relief, education, and research, there is a demand for reliable information provision even in offline environments. Therefore, a natural language processing system that can operate in offline environments and respond to diverse user information requests is needed.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0844] In this invention, the server includes means for acquiring voice data, means for analyzing the acquired voice data and converting it into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for generating an appropriate response based on the analyzed text data, and means for converting the generated response into voice and outputting it. This enables high-precision natural language processing to be achieved without being affected by the communication environment, enabling the rapid and accurate provision of information in a wide range of fields such as disaster relief, education, and research.

[0845] "Audio data" is data that represents an audio signal in digital form.

[0846] "Capturing means" refers to the hardware and software capabilities for capturing audio data.

[0847] The "means for analyzing and converting into text data" is a function that converts voice data into a character string format using a voice recognition engine.

[0848] "Text data" is digital data expressed in the form of a string of characters.

[0849] "Means for natural language analysis" refers to a function that uses natural language processing technology to analyze the context and meaning of text data.

[0850] A "small language model" is a lightweight and efficient language model that is primarily used for natural language processing tasks.

[0851] The "means for generating an appropriate response" is a function for generating a response corresponding to a user's request based on the results of natural language analysis.

[0852] The "means for converting the generated response into voice and outputting it" is a function that converts text data into voice data using a voice synthesis engine and outputs it through a speaker or the like.

[0853] "Disaster relief" refers to activities aimed at saving lives and reducing damage when a disaster occurs.

[0854] "Education" is an activity that transmits and acquires knowledge and skills.

[0855] "Research" is a scientific activity aimed at discovering and developing new knowledge and technologies.

[0856] The "short-time Fourier transform" is an analytical technique for converting a time-domain signal into the frequency domain.

[0857] A "feature" is an important value or pattern extracted from input data in data analysis or machine learning.

[0858] A "phoneme" is the smallest unit of speech in spoken languages.

[0859] "Tokenization" is the process of dividing text data into the smallest units such as words or characters.

[0860] A "self-attention mechanism" is a mechanism that calculates how each part of the input data is related to other parts.

[0861] This invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data from a user and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, enabling it to provide fast and accurate information and support decision-making.

[0862] To implement this system, the following hardware and software are used.

[0863] Hardware

[0864] Terminal: A device for performing the operations of this system, typically including a microphone, speaker, and processor.

[0865] Microphone: A device for capturing audio data

[0866] Speaker: A device that presents generated audio to the user.

[0867] processor: hardware for data analysis and processing

[0868] software

[0869] Speech recognition engine: Uses Google Speech-to-Text, DeepSpeech, etc. to convert voice data into text data.

[0870] Small Language Models (SLMs): Analyze the context and meaning of text data using models such as DistilBERT and TinyBERT.

[0871] Speech synthesis engine: Uses a tool such as Google Text-to-Speech or Amazon Polly to convert the generated text data into speech.

[0872] Specific processing flow

[0873] 1. The user speaks into the device's microphone, for example, asking a question such as "What's the weather forecast for tomorrow?"

[0874] 2. The device uses a microphone to capture audio data and saves it in linear PCM format.

[0875] 3. The device converts the captured voice data into text using a voice recognition engine. During this process, the voice signal is converted into features using a short-time Fourier transform (STFT) and segmented into phonemes.

[0876] 4. The device inputs the converted text data into a small-scale language model (such as DistilBERT or TinyBERT) for natural language analysis. Based on the analysis results, it understands the user's intent and generates an appropriate response.

[0877] 5. The device uses a rule-based system or a generative AI model (e.g., GPT-3) to generate a response, such as "Tomorrow's weather forecast is sunny."

[0878] 6. The device inputs the generated text data into a speech synthesis engine and converts it into voice data. Finally, the generated voice is provided to the user through the speaker.

[0879] Specific examples

[0880] Examples of use in educational settings

[0881] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[0882] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0883] 3. The device inputs this text into a small language model and analyzes it. The model recognizes the keyword "Pythagoras' theorem."

[0884] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[0885] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0886] Examples of use in disaster relief situations

[0887] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0888] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0889] 3. The device inputs this text into a small language model for analysis, which recognizes the keywords "medical facility" and "nearest."

[0890] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[0891] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[0892] In this way, the present invention achieves highly accurate natural language processing even in an offline environment, enabling users to quickly and accurately provide information. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0893] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0894] Step 1:

[0895] The user speaks into the device's microphone. For example, they might ask, "What's the weather forecast for tomorrow?" The device captures this voice data, which is then saved in linear PCM format.

[0896] Step 2:

[0897] The device inputs the acquired voice data into a voice recognition engine (e.g., Google Speech-to-Text or DeepSpeech). The voice recognition engine converts the voice signal into features using a short-time Fourier transform (STFT) and divides it into phonemes. This feature data is used to analyze the voice, and finally converts it into text data such as "What's the weather forecast for tomorrow?" The input is voice data, and the output is text data.

[0898] Step 3:

[0899] The device inputs the converted text data into a small-scale language model (SLM, e.g., DistilBERT or TinyBERT). The SLM tokenizes and analyzes the input text data to understand the context and meaning of the text. Specifically, it uses a self-attention mechanism to analyze the relationships between each token and understand the user's intent. The input for this step is the text data, and the output is the analysis results, which include the user's intent and contextual information.

[0900] Step 4:

[0901] The device generates a response based on the results of natural language analysis. It uses a generative AI model (e.g., GPT-3) or a rule-based system to derive an appropriate response based on the analysis results. In this case, it accesses a weather forecast API to obtain the necessary information and generates the response "Tomorrow's weather will be sunny." The input is the analysis result, and the output is the response text.

[0902] Step 5:

[0903] The device inputs the generated response text into a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The speech synthesis engine converts the input string into phonemes and uses an acoustic model to generate a smooth speech signal. Finally, this speech signal is played back through the speaker. The input is the response text, and the output is speech data.

[0904] Step 6:

[0905] The user receives an answer to the question by listening to the generated voice response from the speaker. For example, the voice response may say, "The weather will be sunny tomorrow." In this step, the voice data is provided to the user.

[0906] This series of steps enables fast and accurate information provision even in situations where communication is unstable or non-existent.

[0907] (Application example 1)

[0908] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0909] In conventional factory environments, workers need complex operation interfaces and internet connections to give instructions to robots, resulting in problems of reduced efficiency and safety. Furthermore, these systems fail to function properly if the communication environment is unstable or non-existent. The present invention aims to solve these problems by enabling high-performance natural language processing even in offline environments, allowing workers to easily operate robots.

[0910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0911] In this invention, the system includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for receiving instructions from a worker in a factory environment by voice, understanding the instructions, and performing an appropriate operation, and control means for operating the above means in the robot. This enables factory robots to be operated by voice instructions using natural language processing without relying on the communication environment.

[0912] "Means for capturing voice data" refers to a technology that uses a voice input device to capture a user's speech as a digital signal and then stores or transfers that data in a form that can be processed.

[0913] The "means for converting captured voice data into text data" refers to a process that utilizes voice recognition technology to analyze captured voice data and convert it into corresponding character string data.

[0914] "Means for natural language analysis of text data" refers to programs and algorithms for understanding and analyzing the context and meaning of converted string data.

[0915] The "means for generating a response based on the analyzed text data" is a process for generating an appropriate response to a user's question or instruction using the results of natural language analysis.

[0916] The "means for converting the generated response into voice and outputting it" is a technology that converts the response generated as character string data into voice data using voice synthesis technology and plays it back from an output device such as a speaker.

[0917] "Means of receiving voice instructions from workers in a factory environment, understanding the content of the instructions, and executing appropriate actions" refers to a series of processes that enable a robot to recognize voice instructions given by workers in a factory, analyze the content of those instructions, and autonomously perform the required actions.

[0918] The "control means for operating the means in the robot" refers to hardware and software for the robot to receive instructions for executing each function and to control each operation based on those instructions.

[0919] This invention provides an offline natural language processing system that enables workers to give voice instructions to robots in a factory environment. This system realizes advanced natural language processing even when not connected to the Internet, thereby improving work efficiency and safety.

[0920] The following hardware and software are used to implement the invention.

[0921] Hardware / Software used

[0922] Device: Factory robot (with built-in system for voice input and output)

[0923] Speech recognition engine: Offline speech recognition engine (e.g. PocketSphinx)

[0924] Natural Language Analysis: Small language models (e.g., custom models in spaCy)

[0925] Speech synthesis engine: Open source speech synthesis engine (e.g. eSpeak)

[0926] Processing Overview

[0927] 1. Voice input:

[0928] The worker speaks commands, such as "take out the part," into the robot's microphone, and the device captures this audio.

[0929] 2. Speech Recognition:

[0930] The robot converts the captured voice data into text data using a speech recognition engine. For example, it converts "take out the parts" into the string "buhin wo toridashite."

[0931] 3. Natural language analysis:

[0932] The converted text data is then fed into a small language model for analysis. The software recognizes the keywords "parts" and "extract" and understands their context and meaning.

[0933] 4. Response Generation:

[0934] Based on the parsed text data, the robot generates an appropriate response, for example, "Pick up part A."

[0935] 5. Audio output:

[0936] The generated text response is converted into speech by a speech synthesis engine and played back through the robot's speaker.

[0937] Specific examples

[0938] Usage Scenario 1

[0939] 1. The worker tells the robot, "Please take out the next product."

[0940] 2. The robot records the voice and sends the data to the speech recognition engine.

[0941] 3. The speech recognition engine converts the speech into text.

[0942] 4. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0943] 5. The robot generates an appropriate response and converts it into speech using a speech synthesis engine.

[0944] 6. The robot responds, "I'll pick up the next product," and performs the instructed task.

[0945] Prompt Sentence Examples

[0946] A user says to a factory robot: "Pick up the next product."

[0947] The robot prepares a response in the following steps:

[0948] 1. The speech recognition engine converts speech into text.

[0949] 2. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[0950] 3. Generate an appropriate response and convert it into speech using a text-to-speech engine.

[0951] As a result, the robot will respond with "Pick up next product."

[0952] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0953] Step 1:

[0954] The user issues voice commands into the robot's microphone. This voice includes specific instructions such as "Please take out the next product." The robot captures this voice data and records it as a digital signal. The input of this step is the voice signal, and the output is the captured digital voice data.

[0955] Step 2:

[0956] The device's speech recognition engine receives the captured voice data and converts the voice into text data. An offline speech recognition engine such as PocketSphinx is used to convert the voice signal into a string of characters. In this step, the voice signal is analyzed using techniques such as frequency analysis and pattern recognition, and mapped to a corresponding string of characters. The input is digital voice data, and the output is the corresponding text data.

[0957] Step 3:

[0958] The device's natural language analysis engine receives the converted text data and analyzes it using a small language model (for example, a custom model from spaCy). During the analysis process, it understands the context and meaning of the text and extracts keywords (e.g., "next product," "take out") The input is the text data, and the output is the keywords and their contextual information. Statistical and rule-based natural language processing techniques are used in this step.

[0959] Step 4:

[0960] The device generates an appropriate response based on the parsed text data. A generative AI model is used to form an answer based on the instruction. For example, the response to "Please pick up the next product" will be "Please pick up the next product." The input is contextual information and keywords, and the output is the response text. This step uses an algorithm to construct an appropriate response based on the existing context.

[0961] Step 5:

[0962] The device sends the generated text data to a speech synthesis engine (e.g., eSpeak) and converts it into voice data. The speech synthesis engine converts the text data into voice signals and generates an audio file to convey to the user. The input is the response text and the output is the voice data. In this step, text-to-voice conversion technology is used, and the generated voice is played from the speaker.

[0963] Step 6:

[0964] The robot executes specific actions based on voice instructions. For example, it executes an operating program to start picking up a specified product. The input is voice instructions and analysis results, and the output is physical actions. In this step, the control algorithm is executed, controlling the machine's operations.

[0965] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0966] The present invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, speech output, and emotion recognition, enabling it to provide users with fast and accurate information and support for decision-making.

[0967] Program processing overview

[0968] Voice input

[0969] It starts with the user speaking a question or request into the device's microphone, such as "What's the weather forecast for tomorrow?", and the device captures this audio data.

[0970] Voice Recognition

[0971] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[0972] natural language analysis

[0973] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[0974] emotion recognition

[0975] The device uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice from the audio data to estimate the user's emotional state. For example, emotions such as "anger," "joy," and "sadness" can be identified.

[0976] Response Generation and Adjustment

[0977] The device generates appropriate responses based on the results of natural language analysis and emotion recognition. Responses are written in natural language as answers to the user's questions. Furthermore, the device adjusts the content and tone of the response based on the user's emotions as recognized by the emotion engine. For example, if the user is feeling stressed, the response will be more polite and reassuring.

[0978] Audio Output

[0979] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[0980] Specific examples

[0981] Examples of use in educational settings

[0982] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[0983] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[0984] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[0985] 4. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[0986] 5. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right triangle is equal to the sum of the squares of the other two sides." If the device recognizes the user's interest, it provides further examples or additional explanations.

[0987] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0988] Examples of use in disaster relief situations

[0989] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[0990] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[0991] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[0992] 4. The device uses its emotion engine to recognize that rescue workers are in an emergency situation.

[0993] 5. The device generates a response: "The nearest medical facility is 3 kilometers east." If the device detects that the user is nervous, it adjusts the response to provide polite guidance in a calm voice.

[0994] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[0995] As described above, the present invention realizes advanced natural language processing and emotion recognition in an offline environment that does not require a communication environment, and can provide users with information that is fast, accurate, and considers their emotions. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[0996] The processing flow will be explained below.

[0997] Step 1:

[0998] Acquiring voice input

[0999] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[1000] The terminal captures the user's voice data through a microphone.

[1001] Step 2:

[1002] Audio data conversion

[1003] The terminal sends the captured voice data to a voice recognition engine.

[1004] The voice recognition engine analyzes the input voice data and converts it into text data.

[1005] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[1006] Step 3:

[1007] emotion recognition

[1008] The terminal transmits the voice data to the emotion engine.

[1009] The emotion engine analyzes the tone, pitch, and speed of the user's voice to estimate the user's emotional state.

[1010] The emotion engine identifies emotions such as "joy," "sadness," and "anger."

[1011] Step 4:

[1012] Text data analysis

[1013] The terminal inputs the converted text data into a small language model (SLM).

[1014] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[1015] Step 5:

[1016] Generating a response

[1017] The terminal generates an appropriate response based on the results of natural language analysis and emotion recognition.

[1018] Responses are formulated in natural language in response to the user's question.

[1019] For example, it generates a response such as "The weather will be sunny tomorrow."

[1020] Step 6:

[1021] Response adjustment

[1022] The terminal adjusts the content and tone of the response based on the user's emotions recognized by the emotion engine.

[1023] For example, if the user is feeling stressed, the response will be more polite and reassuring.

[1024] Step 7:

[1025] Response transcription

[1026] The terminal sends the generated text response to a speech synthesis engine.

[1027] The speech synthesis engine converts text data into natural-sounding speech.

[1028] Step 8:

[1029] Audio Output

[1030] The device plays the voice generated by the speech synthesis engine through the speaker.

[1031] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[1032] Example 2

[1033] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1034] There is a demand for natural language processing systems that can operate even in unstable or non-existent communication environments and provide accurate and prompt information from voice data. However, conventional systems have problems such as a poor user experience because they are highly dependent on the communication environment for speech recognition and response generation, and do not generate responses that take the user's emotional state into consideration.

[1035] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1036] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for recognizing an emotion to estimate an emotional state from the voice data, means for generating a response based on the analyzed text data and the emotional state, and means for converting the generated response into voice and outputting it. This enables prompt and accurate information provision that is independent of the communication environment and takes into consideration the emotional state of the user.

[1037] "Voice data" is digital data that captures a user's speech using a voice input device such as a microphone.

[1038] "Text data" is data in the form of a character string converted from voice data by a voice recognition engine.

[1039] "Natural language analysis" is the process of analyzing the context and meaning of text data to understand the user's intent.

[1040] A "small language model" is a language model that is optimized for a specific task and is relatively lightweight and fast to operate.

[1041] "Emotion recognition" is the process of analyzing the tone, pitch, speed, etc. of a user's voice from audio data to estimate the user's emotional state.

[1042] "Response generation" is the process of creating an appropriate response based on the results of natural language analysis and emotion recognition.

[1043] "Speech synthesis" is a technique for converting generated text responses into speech and providing it to the user.

[1044] This invention is a system that combines speech-based natural language processing and emotion recognition to provide users with fast and accurate information even in offline environments. This has the advantage of being able to provide decision-making support without relying on the communication environment.

[1045] System configuration

[1046] The system consists of the following main components:

[1047] 1. Audio Input Device: Includes a microphone to capture audio data spoken by the user, which is then stored in a digital format.

[1048] 2. Speech recognition engine: Converts captured voice data into text data. Examples include Google Speech-to-Text and Microsoft Azure Speech Service.

[1049] 3. Small Language Models (SLMs): Used for natural language analysis of text data to understand user intent. Examples include a smaller version of GPT-2.

[1050] 4. Emotion Recognition Engine: Analyzes the tone, pitch, and rate of voice from audio data to estimate the user's emotional state. Open source libraries (e.g., OpenSMILE) are used.

[1051] 5. Response generator: Generates appropriate text responses based on the results obtained from natural language analysis and emotion recognition.

[1052] 6. Speech synthesis engine: Converts the generated text response into speech and provides it to the user through a speaker. Examples include Google Text-to-Speech and Amazon Polly.

[1053] System Operation

[1054] The system works as follows:

[1055] A user speaks a question or request into the device's microphone. For example, "What is the weather forecast for tomorrow?" The device captures this voice data and converts it into text using a speech recognition engine. The converted text data is then input into a small language model (SLM) for natural language analysis.

[1056] The device then uses an emotion recognition engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice data to infer the user's emotional state. An appropriate response text is generated based on the analysis and emotion recognition results.

[1057] The generated response text is converted into speech using a speech synthesis engine, and finally, this speech is played back through a speaker and conveyed to the user, thereby providing fast and accurate information.

[1058] Specific examples

[1059] Examples of use in educational settings

[1060] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[1061] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[1062] 3. The device uses a speech recognition engine to convert the text into "Please tell me about Pythagoras' theorem."

[1063] 4. The terminal inputs this text into the SLM for analysis. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[1064] 5. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[1065] 6. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right-angled triangle is equal to the sum of the squares of the other two sides." If the device recognizes that the user is interested, it provides further examples and explanations, such as, "An example would be a right-angled triangle with sides measuring 3 cm, 4 cm, and 5 cm."

[1066] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to listen to the explanation and deepen their understanding.

[1067] Examples of use in disaster relief situations

[1068] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[1069] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[1070] 3. The device uses a speech recognition engine to convert the text into "Please tell me where the nearest medical facility is."

[1071] 4. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[1072] 5. The device uses an emotion engine to recognize when rescuers are in an emergency situation, gauge their level of tension from their tone of voice, and generate a quick and specific response.

[1073] 6. The device generates a response saying, "The nearest medical facility is three kilometers east." If the device detects that the user is nervous, it adjusts the response to be calm and polite, such as, "Please stay calm. It's three kilometers east."

[1074] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to act quickly based on this information.

[1075] This series of processes can be carried out offline as well, allowing users to obtain the information they need without relying on the communication environment.

[1076] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1077] Step 1:

[1078] The user speaks a question or request into the device's microphone. The input is the user's voice data. A specific example of this is when the user speaks something like, "What's the weather forecast for tomorrow?" The output is captured as voice data on the device.

[1079] Step 2:

[1080] The device passes the captured voice data to a voice recognition engine. The voice recognition engine (for example, Google Speech-to-Text) is used to convert the voice data into text data. The input is the user's voice data. Specifically, the voice recognition engine analyzes the voice signal and outputs the text "What is the weather forecast for tomorrow?"

[1081] Step 3:

[1082] The device inputs the converted text data into a small language model (e.g., a miniature version of GPT-2) to perform natural language analysis. The input is the text "What is the weather forecast for tomorrow?". Specifically, the SLM divides the text into tokens and analyzes the context and meaning. The output is an analysis result that indicates the intent of the user's question.

[1083] Step 4:

[1084] The device passes the voice data to an emotion recognition engine (e.g., OpenSMILE) to estimate the user's emotional state. The input is the user's voice data. Specifically, the emotion recognition engine analyzes the tone, pitch, and speed of the voice and outputs the user's emotional state (e.g., "neutral," "excited," "anxious," etc.).

[1085] Step 5:

[1086] The device generates an appropriate response based on the results of natural language analysis and emotion recognition. The input is the analysis result and the emotional state. Specifically, the SLM generates a response text such as "Tomorrow's weather forecast is sunny." At the same time, it adjusts the content and tone of the response based on the emotion recognition results. The output is the adjusted response text.

[1087] Step 6:

[1088] The device passes the generated response text to a speech synthesis engine (e.g., Google Text-to-Speech) to convert it into speech. The input is the tailored response text. Specifically, the speech synthesis engine converts the text into speech and plays it through the speaker. The output is a spoken response that is communicated to the user.

[1089] Through these steps, the system will be able to achieve advanced natural language processing and emotion recognition even in offline environments, enabling it to provide users with fast and accurate information.

[1090] (Application example 2)

[1091] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1092] There is a need to provide a means for workers and managers to smoothly communicate with factory robots even in situations where the communication environment is unstable or non-existent. In particular, there is a need for a system that can quickly and accurately process voice instructions and reports and generate responses that take into account the emotional state of the worker, thereby increasing productivity and improving the working environment.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1094] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for recognizing emotions, and means for adjusting the content and tone of the response based on the emotion. This enables workers and managers to have natural conversations with factory robots even in unstable communication environments, enabling efficient work execution through appropriate work instructions, reports, and questions. Furthermore, responses that take into account the emotional state of workers are expected to improve the working environment and reduce stress.

[1095] A "means for capturing audio data" is a means for collecting audio signals as digital data using a device such as a microphone.

[1096] The "means for converting captured voice data into text data" refers to a means for converting collected voice data into a string of characters using a voice recognition engine or software.

[1097] "Means for natural language analysis of text data" refers to means for analyzing the context and meaning of text data using a natural language processing engine.

[1098] The "means for generating a response based on the analyzed text data" is a means for creating an appropriate reply based on the analysis results.

[1099] "Means for converting the generated response into speech and outputting it" means means for converting text into speech using a text-to-speech (TTS) engine and outputting it via a speaker or the like.

[1100] The "means for recognizing emotions" is a means for analyzing emotional information from voice or text data and identifying the emotional state of the user.

[1101] The "means for adjusting content and tone based on emotion when generating a response" refers to a means for adjusting the content of a reply or tone of voice based on recognized emotional information to create a natural and considerate response.

[1102] A "small language model" is a lightweight natural language processing model specialized for a specific task or application, containing algorithms for quickly and efficiently analyzing the context and meaning of text.

[1103] MODE FOR CARRYING OUT THE INVENTION

[1104] The present invention is a system for seamless communication between workers and managers through factory robots. The system operates even in unstable or non-existent communication environments, processes voice instructions and reports from workers in real time, and generates appropriate responses.

[1105] Hardware and software used

[1106] Hardware: Factory robot with microphone and speaker

[1107] software:

[1108] Speech Recognition Engine: SpeechRecognition Library

[1109] Natural Language Processing Engine: Hugging Face's Transformer Library

[1110] Emotion recognition engine: A model embedded within a natural language analysis engine

[1111] Text-to-Speech (TTS) engine: gTTS library

[1112] System Overview

[1113] The system consists of the following main processing steps: First, the factory robot's microphone captures the worker's voice data. The voice data is converted into text data through a speech recognition engine. Next, a natural language processing engine analyzes the text data, and an emotion recognition engine identifies the worker's emotions from the text data. The system generates an appropriate response based on these analysis results. The generated response is converted into speech by a TTS engine and provided to the worker through a speaker.

[1114] Specific examples

[1115] Example 1: Receiving a work order

[1116] When a worker says to the robot, "Please bring me a new part," the robot captures the speech and converts it into text using a speech recognition engine: "Please bring me a new part." The natural language processing engine then analyzes the instruction, and the emotion recognition engine recognizes the worker's emotion as "neutral." The system generates a response: "Got it. I'll start working on your request," which is then converted into speech using a TTS engine and played back.

[1117] Example 2: Problem report

[1118] When a worker says "The machine is broken," the robot captures the speech and converts it into text using a speech recognition engine. A natural language processing engine then analyzes the report, and an emotion recognition engine recognizes that the worker is anxious. The system generates a response: "I will report this to the manager immediately. It's okay, please stay calm," which is then converted into audio using a TTS engine and played back.

[1119] Prompt Sentence Examples

[1120] If the user says "The machine is broken," the following prompt will be generated:

[1121] Text: "Machine is out of order"

[1122] Task: Report a problem

[1123] If the user says "Bring me a new part," the following prompt will be generated:

[1124] Text: "Bring me new parts"

[1125] Task: Work Order

[1126] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1127] Step 1:

[1128] Voice input

[1129] The user gives instructions or reports by voice into the microphone of the factory robot, and this voice data is captured by the microphone. The input of the processing is the user's voice data, and the output is a digital audio file.

[1130] Step 2:

[1131] Voice Recognition

[1132] The device converts the captured voice data into text data using the SpeechRecognition library. The input is voice data and the output is text data. Specifically, the device analyzes the voice signal using techniques such as the Fast Fourier Transform (FFT) and converts the content into a string of characters.

[1133] Step 3:

[1134] natural language analysis

[1135] The device then analyzes the converted text data using Hugging Face's Transformer library. The input is text data, and the output is an analysis of the text's context and meaning. Specifically, a zero-shot classification model is used to analyze the text's context. At this stage, the device classifies the text as an instruction, a report, or a question.

[1136] Step 4:

[1137] emotion recognition

[1138] The device uses an emotion recognition engine to identify the user's emotions based on text data obtained through natural language analysis. The input is text data, and the output is the identified emotional state. Specifically, the device analyzes the tone, pitch, and speed of the voice and text to classify the user's emotions.

[1139] Step 5:

[1140] Response Generation

[1141] The device generates an appropriate response based on the analyzed text data and emotional information. The input is the context analysis results and emotional information, and the output is the generated response text. Specifically, the response generation engine generates a response such as "Understood. We will begin working on your request" in the case of an instruction, or "We will report to the administrator immediately" in the case of a report. The content and tone are also adjusted depending on the emotion.

[1142] Step 6:

[1143] Audio Output

[1144] The device converts the generated response text into speech using the gTTS library and plays it from the speaker. The input is the generated response text, and the output is the audio to be played. Specifically, the text-to-speech (TTS) engine converts the text into a speech waveform and outputs it from the speaker.

[1145] These processing steps enable natural voice interaction with the user, allowing the factory robot to communicate smoothly with workers and managers.

[1146] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1147] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1148] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1149] [Fourth embodiment]

[1150] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1151] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1152] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1153] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1154] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1155] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1156] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1157] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1158] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1159] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1160] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1161] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1162] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1163] The present invention relates to an offline natural language processing system that can operate even in an unstable or non-existent communication environment. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, thereby providing users with fast and accurate information and decision-making support.

[1164] Program processing overview

[1165] Voice input

[1166] It starts with the user speaking into the device's microphone, asking a question like, "What's the weather forecast for tomorrow?" The device captures this audio data.

[1167] Voice Recognition

[1168] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[1169] natural language analysis

[1170] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[1171] Response Generation

[1172] The device generates an appropriate response based on the analyzed text data. The response is intended to answer the user's question in natural language. For example, if asked about the weather forecast, the device generates a response such as "Tomorrow's weather will be sunny."

[1173] Audio Output

[1174] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[1175] Specific examples

[1176] Examples of use in educational settings

[1177] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[1178] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[1179] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[1180] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[1181] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[1182] Examples of use in disaster relief situations

[1183] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[1184] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[1185] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[1186] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[1187] 5. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[1188] As described above, the present invention realizes advanced natural language processing and can provide users with fast and accurate information even in an offline environment that does not require an internet connection. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[1189] The processing flow will be explained below.

[1190] Step 1:

[1191] Acquiring voice input

[1192] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[1193] The terminal captures the user's voice data through a microphone.

[1194] Step 2:

[1195] Audio data conversion

[1196] The terminal sends the captured voice data to a voice recognition engine.

[1197] The voice recognition engine analyzes the input voice data and converts it into text data.

[1198] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[1199] Step 3:

[1200] Text data analysis

[1201] The terminal inputs the converted text data into a small language model (SLM).

[1202] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[1203] Step 4:

[1204] Generating a response

[1205] The terminal generates an appropriate response based on the analysis results.

[1206] For example, it generates a response such as "The weather will be sunny tomorrow."

[1207] When generating a response, the SLM's knowledge base or internal database may be referenced.

[1208] Step 5:

[1209] Response transcription

[1210] The terminal sends the generated text response to a speech synthesis engine.

[1211] The speech synthesis engine converts text data into natural-sounding speech.

[1212] Step 6:

[1213] Audio Output

[1214] The device plays the voice generated by the speech synthesis engine through the speaker.

[1215] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[1216] By using the above steps, the present invention can realize advanced natural language processing even in an offline environment where an Internet connection is not required, and can provide users with information quickly and accurately.

[1217] Example 1

[1218] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1219] Voice interfaces utilizing natural language processing technology are widespread these days, but few systems can achieve high-precision natural language processing even in situations where the communication environment is unstable or non-existent. Furthermore, in fields such as disaster relief, education, and research, there is a demand for reliable information provision even in offline environments. Therefore, a natural language processing system that can operate in offline environments and respond to diverse user information requests is needed.

[1220] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1221] In this invention, the server includes means for acquiring voice data, means for analyzing the acquired voice data and converting it into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for generating an appropriate response based on the analyzed text data, and means for converting the generated response into voice and outputting it. This enables high-precision natural language processing to be achieved without being affected by the communication environment, enabling the rapid and accurate provision of information in a wide range of fields such as disaster relief, education, and research.

[1222] "Audio data" is data that represents an audio signal in digital form.

[1223] "Capturing means" refers to the hardware and software capabilities for capturing audio data.

[1224] The "means for analyzing and converting into text data" is a function that converts voice data into a character string format using a voice recognition engine.

[1225] "Text data" is digital data expressed in the form of a string of characters.

[1226] "Means for natural language analysis" refers to a function that uses natural language processing technology to analyze the context and meaning of text data.

[1227] A "small language model" is a lightweight and efficient language model that is primarily used for natural language processing tasks.

[1228] The "means for generating an appropriate response" is a function for generating a response corresponding to a user's request based on the results of natural language analysis.

[1229] The "means for converting the generated response into voice and outputting it" is a function that converts text data into voice data using a voice synthesis engine and outputs it through a speaker or the like.

[1230] "Disaster relief" refers to activities aimed at saving lives and reducing damage when a disaster occurs.

[1231] "Education" is an activity that transmits and acquires knowledge and skills.

[1232] "Research" is a scientific activity aimed at discovering and developing new knowledge and technologies.

[1233] The "short-time Fourier transform" is an analytical technique for converting a time-domain signal into the frequency domain.

[1234] A "feature" is an important value or pattern extracted from input data in data analysis or machine learning.

[1235] A "phoneme" is the smallest unit of speech in spoken languages.

[1236] "Tokenization" is the process of dividing text data into the smallest units such as words or characters.

[1237] A "self-attention mechanism" is a mechanism that calculates how each part of the input data is related to other parts.

[1238] This invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data from a user and performs a series of processes including speech recognition, natural language analysis, response generation, and speech output, enabling it to provide fast and accurate information and support decision-making.

[1239] To implement this system, the following hardware and software are used.

[1240] Hardware

[1241] Terminal: A device for performing the operations of this system, typically including a microphone, speaker, and processor.

[1242] Microphone: A device for capturing audio data

[1243] Speaker: A device that presents generated audio to the user.

[1244] processor: hardware for data analysis and processing

[1245] software

[1246] Speech recognition engine: Uses Google Speech-to-Text, DeepSpeech, etc. to convert voice data into text data.

[1247] Small Language Models (SLMs): Analyze the context and meaning of text data using models such as DistilBERT and TinyBERT.

[1248] Speech synthesis engine: Uses a tool such as Google Text-to-Speech or Amazon Polly to convert the generated text data into speech.

[1249] Specific processing flow

[1250] 1. The user speaks into the device's microphone, for example, asking a question such as "What's the weather forecast for tomorrow?"

[1251] 2. The device uses a microphone to capture audio data and saves it in linear PCM format.

[1252] 3. The device converts the captured voice data into text using a voice recognition engine. During this process, the voice signal is converted into features using a short-time Fourier transform (STFT) and segmented into phonemes.

[1253] 4. The device inputs the converted text data into a small-scale language model (such as DistilBERT or TinyBERT) for natural language analysis. Based on the analysis results, it understands the user's intent and generates an appropriate response.

[1254] 5. The device uses a rule-based system or a generative AI model (e.g., GPT-3) to generate a response, such as "Tomorrow's weather forecast is sunny."

[1255] 6. The device inputs the generated text data into a speech synthesis engine and converts it into voice data. Finally, the generated voice is provided to the user through the speaker.

[1256] Specific examples

[1257] Examples of use in educational settings

[1258] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[1259] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[1260] 3. The device inputs this text into a small language model and analyzes it. The model recognizes the keyword "Pythagoras' theorem."

[1261] 4. The device generates the response "The Pythagorean theorem states that the square of the length of the hypotenuse of a right triangle is equal to the sum of the squares of the lengths of the other two sides."

[1262] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[1263] Examples of use in disaster relief situations

[1264] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[1265] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[1266] 3. The device inputs this text into a small language model for analysis, which recognizes the keywords "medical facility" and "nearest."

[1267] 4. The device generates the response, "The nearest medical facility is 3 kilometers east."

[1268] 5. The device inputs the generated response into a speech synthesis engine, converts it into speech, and plays it through the speaker.

[1269] In this way, the present invention achieves highly accurate natural language processing even in an offline environment, enabling users to quickly and accurately provide information. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[1270] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1271] Step 1:

[1272] The user speaks into the device's microphone. For example, they might ask, "What's the weather forecast for tomorrow?" The device captures this voice data, which is then saved in linear PCM format.

[1273] Step 2:

[1274] The device inputs the acquired voice data into a voice recognition engine (e.g., Google Speech-to-Text or DeepSpeech). The voice recognition engine converts the voice signal into features using a short-time Fourier transform (STFT) and divides it into phonemes. This feature data is used to analyze the voice, and finally converts it into text data such as "What's the weather forecast for tomorrow?" The input is voice data, and the output is text data.

[1275] Step 3:

[1276] The device inputs the converted text data into a small-scale language model (SLM, e.g., DistilBERT or TinyBERT). The SLM tokenizes and analyzes the input text data to understand the context and meaning of the text. Specifically, it uses a self-attention mechanism to analyze the relationships between each token and understand the user's intent. The input for this step is the text data, and the output is the analysis results, which include the user's intent and contextual information.

[1277] Step 4:

[1278] The device generates a response based on the results of natural language analysis. It uses a generative AI model (e.g., GPT-3) or a rule-based system to derive an appropriate response based on the analysis results. In this case, it accesses a weather forecast API to obtain the necessary information and generates the response "Tomorrow's weather will be sunny." The input is the analysis result, and the output is the response text.

[1279] Step 5:

[1280] The device inputs the generated response text into a speech synthesis engine (e.g., Google Text-to-Speech or Amazon Polly). The speech synthesis engine converts the input string into phonemes and uses an acoustic model to generate a smooth speech signal. Finally, this speech signal is played back through the speaker. The input is the response text, and the output is speech data.

[1281] Step 6:

[1282] The user receives an answer to the question by listening to the generated voice response from the speaker. For example, the voice response may say, "The weather will be sunny tomorrow." In this step, the voice data is provided to the user.

[1283] This series of steps enables fast and accurate information provision even in situations where communication is unstable or non-existent.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] In conventional factory environments, workers need complex operation interfaces and internet connections to give instructions to robots, resulting in problems of reduced efficiency and safety. Furthermore, these systems fail to function properly if the communication environment is unstable or non-existent. The present invention aims to solve these problems by enabling high-performance natural language processing even in offline environments, allowing workers to easily operate robots.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the system includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for receiving instructions from a worker in a factory environment by voice, understanding the instructions, and performing an appropriate operation, and control means for operating the above means in the robot. This enables factory robots to be operated by voice instructions using natural language processing without relying on the communication environment.

[1289] "Means for capturing voice data" refers to a technology that uses a voice input device to capture a user's speech as a digital signal and then stores or transfers that data in a form that can be processed.

[1290] The "means for converting captured voice data into text data" refers to a process that utilizes voice recognition technology to analyze captured voice data and convert it into corresponding character string data.

[1291] "Means for natural language analysis of text data" refers to programs and algorithms for understanding and analyzing the context and meaning of converted string data.

[1292] The "means for generating a response based on the analyzed text data" is a process for generating an appropriate response to a user's question or instruction using the results of natural language analysis.

[1293] The "means for converting the generated response into voice and outputting it" is a technology that converts the response generated as character string data into voice data using voice synthesis technology and plays it back from an output device such as a speaker.

[1294] "Means of receiving voice instructions from workers in a factory environment, understanding the content of the instructions, and executing appropriate actions" refers to a series of processes that enable a robot to recognize voice instructions given by workers in a factory, analyze the content of those instructions, and autonomously perform the required actions.

[1295] The "control means for operating the means in the robot" refers to hardware and software for the robot to receive instructions for executing each function and to control each operation based on those instructions.

[1296] This invention provides an offline natural language processing system that enables workers to give voice instructions to robots in a factory environment. This system realizes advanced natural language processing even when not connected to the Internet, thereby improving work efficiency and safety.

[1297] The following hardware and software are used to implement the invention.

[1298] Hardware / Software used

[1299] Device: Factory robot (with built-in system for voice input and output)

[1300] Speech recognition engine: Offline speech recognition engine (e.g. PocketSphinx)

[1301] Natural Language Analysis: Small language models (e.g., custom models in spaCy)

[1302] Speech synthesis engine: Open source speech synthesis engine (e.g. eSpeak)

[1303] Processing Overview

[1304] 1. Voice input:

[1305] The worker speaks commands, such as "take out the part," into the robot's microphone, and the device captures this audio.

[1306] 2. Speech Recognition:

[1307] The robot converts the captured voice data into text data using a speech recognition engine. For example, it converts "take out the parts" into the string "buhin wo toridashite."

[1308] 3. Natural language analysis:

[1309] The converted text data is then fed into a small language model for analysis. The software recognizes the keywords "parts" and "extract" and understands their context and meaning.

[1310] 4. Response Generation:

[1311] Based on the parsed text data, the robot generates an appropriate response, for example, "Pick up part A."

[1312] 5. Audio output:

[1313] The generated text response is converted into speech by a speech synthesis engine and played back through the robot's speaker.

[1314] Specific examples

[1315] Usage Scenario 1

[1316] 1. The worker tells the robot, "Please take out the next product."

[1317] 2. The robot records the voice and sends the data to the speech recognition engine.

[1318] 3. The speech recognition engine converts the speech into text.

[1319] 4. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[1320] 5. The robot generates an appropriate response and converts it into speech using a speech synthesis engine.

[1321] 6. The robot responds, "I'll pick up the next product," and performs the instructed task.

[1322] Prompt Sentence Examples

[1323] A user says to a factory robot: "Pick up the next product."

[1324] The robot prepares a response in the following steps:

[1325] 1. The speech recognition engine converts speech into text.

[1326] 2. A natural language processing engine analyzes the text and recognizes keywords (e.g., "take out," "product").

[1327] 3. Generate an appropriate response and convert it into speech using a text-to-speech engine.

[1328] As a result, the robot will respond with "Pick up next product."

[1329] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1330] Step 1:

[1331] The user issues voice commands into the robot's microphone. This voice includes specific instructions such as "Please take out the next product." The robot captures this voice data and records it as a digital signal. The input of this step is the voice signal, and the output is the captured digital voice data.

[1332] Step 2:

[1333] The device's speech recognition engine receives the captured voice data and converts the voice into text data. An offline speech recognition engine such as PocketSphinx is used to convert the voice signal into a string of characters. In this step, the voice signal is analyzed using techniques such as frequency analysis and pattern recognition, and mapped to a corresponding string of characters. The input is digital voice data, and the output is the corresponding text data.

[1334] Step 3:

[1335] The device's natural language analysis engine receives the converted text data and analyzes it using a small language model (for example, a custom model from spaCy). During the analysis process, it understands the context and meaning of the text and extracts keywords (e.g., "next product," "take out") The input is the text data, and the output is the keywords and their contextual information. Statistical and rule-based natural language processing techniques are used in this step.

[1336] Step 4:

[1337] The device generates an appropriate response based on the parsed text data. A generative AI model is used to form an answer based on the instruction. For example, the response to "Please pick up the next product" will be "Please pick up the next product." The input is contextual information and keywords, and the output is the response text. This step uses an algorithm to construct an appropriate response based on the existing context.

[1338] Step 5:

[1339] The device sends the generated text data to a speech synthesis engine (e.g., eSpeak) and converts it into voice data. The speech synthesis engine converts the text data into voice signals and generates an audio file to convey to the user. The input is the response text and the output is the voice data. In this step, text-to-voice conversion technology is used, and the generated voice is played from the speaker.

[1340] Step 6:

[1341] The robot executes specific actions based on voice instructions. For example, it executes an operating program to start picking up a specified product. The input is voice instructions and analysis results, and the output is physical actions. In this step, the control algorithm is executed, controlling the machine's operations.

[1342] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1343] The present invention relates to an offline natural language processing system that operates even in unstable or non-existent communication environments. This system acquires speech data and performs a series of processes including speech recognition, natural language analysis, response generation, speech output, and emotion recognition, enabling it to provide users with fast and accurate information and support for decision-making.

[1344] Program processing overview

[1345] Voice input

[1346] It starts with the user speaking a question or request into the device's microphone, such as "What's the weather forecast for tomorrow?", and the device captures this audio data.

[1347] Voice Recognition

[1348] The device converts the captured voice data into text data using a voice recognition engine, which analyzes the voice signal and represents its contents in a string format.

[1349] natural language analysis

[1350] The terminal uses a small language model (SLM) to analyze the converted text data in natural language. The SLM understands the context and meaning of the text and analyzes the user's intent. This step identifies keywords and context in the question or request.

[1351] emotion recognition

[1352] The device uses an emotion engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice from the audio data to estimate the user's emotional state. For example, emotions such as "anger," "joy," and "sadness" can be identified.

[1353] Response Generation and Adjustment

[1354] The device generates appropriate responses based on the results of natural language analysis and emotion recognition. Responses are written in natural language as answers to the user's questions. Furthermore, the device adjusts the content and tone of the response based on the user's emotions as recognized by the emotion engine. For example, if the user is feeling stressed, the response will be more polite and reassuring.

[1355] Audio Output

[1356] The device uses a speech synthesis engine to convert the generated text response into speech and provide it to the user, which is then played back through a speaker and communicated to the user.

[1357] Specific examples

[1358] Examples of use in educational settings

[1359] 1. The user asks the device, "Tell me about the Pythagorean theorem."

[1360] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Tell me about Pythagoras' theorem."

[1361] 3. The terminal inputs this text into the SLM and analyzes it. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[1362] 4. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[1363] 5. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right triangle is equal to the sum of the squares of the other two sides." If the device recognizes the user's interest, it provides further examples or additional explanations.

[1364] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[1365] Examples of use in disaster relief situations

[1366] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[1367] 2. The device captures the voice and uses a speech recognition engine to convert it into text: "Where is the nearest medical facility?"

[1368] 3. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[1369] 4. The device uses its emotion engine to recognize that rescue workers are in an emergency situation.

[1370] 5. The device generates a response: "The nearest medical facility is 3 kilometers east." If the device detects that the user is nervous, it adjusts the response to provide polite guidance in a calm voice.

[1371] 6. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker.

[1372] As described above, the present invention realizes advanced natural language processing and emotion recognition in an offline environment that does not require a communication environment, and can provide users with information that is fast, accurate, and considers their emotions. This enables the realization of a system that provides important decision-making and support in rescue operations after disasters, educational settings, and research activities in areas with limited communication environments.

[1373] The processing flow will be explained below.

[1374] Step 1:

[1375] Acquiring voice input

[1376] The user speaks a question or request into the device's microphone, for example, "What's the weather forecast for tomorrow?"

[1377] The terminal captures the user's voice data through a microphone.

[1378] Step 2:

[1379] Audio data conversion

[1380] The terminal sends the captured voice data to a voice recognition engine.

[1381] The voice recognition engine analyzes the input voice data and converts it into text data.

[1382] The converted text data becomes, for example, a character string such as "Please tell me the weather forecast for tomorrow."

[1383] Step 3:

[1384] emotion recognition

[1385] The terminal transmits the voice data to the emotion engine.

[1386] The emotion engine analyzes the tone, pitch, and speed of the user's voice to estimate the user's emotional state.

[1387] The emotion engine identifies emotions such as "joy," "sadness," and "anger."

[1388] Step 4:

[1389] Text data analysis

[1390] The terminal inputs the converted text data into a small language model (SLM).

[1391] SLM analyzes text data to understand its context and meaning, for example recognizing keywords such as "weather forecast" and "tomorrow."

[1392] Step 5:

[1393] Generating a response

[1394] The terminal generates an appropriate response based on the results of natural language analysis and emotion recognition.

[1395] Responses are formulated in natural language in response to the user's question.

[1396] For example, it generates a response such as "The weather will be sunny tomorrow."

[1397] Step 6:

[1398] Response adjustment

[1399] The terminal adjusts the content and tone of the response based on the user's emotions recognized by the emotion engine.

[1400] For example, if the user is feeling stressed, the response will be more polite and reassuring.

[1401] Step 7:

[1402] Response transcription

[1403] The terminal sends the generated text response to a speech synthesis engine.

[1404] The speech synthesis engine converts text data into natural-sounding speech.

[1405] Step 8:

[1406] Audio Output

[1407] The device plays the voice generated by the speech synthesis engine through the speaker.

[1408] For example, a voice message saying "Tomorrow's weather will be sunny" is played back to inform the user.

[1409] Example 2

[1410] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1411] There is a demand for natural language processing systems that can operate even in unstable or non-existent communication environments and provide accurate and prompt information from voice data. However, conventional systems have problems such as a poor user experience because they are highly dependent on the communication environment for speech recognition and response generation, and do not generate responses that take the user's emotional state into consideration.

[1412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1413] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for using a small-scale language model to perform natural language analysis on the text data, means for recognizing an emotion to estimate an emotional state from the voice data, means for generating a response based on the analyzed text data and the emotional state, and means for converting the generated response into voice and outputting it. This enables prompt and accurate information provision that is independent of the communication environment and takes into consideration the emotional state of the user.

[1414] "Voice data" is digital data that captures a user's speech using a voice input device such as a microphone.

[1415] "Text data" is data in the form of a character string converted from voice data by a voice recognition engine.

[1416] "Natural language analysis" is the process of analyzing the context and meaning of text data to understand the user's intent.

[1417] A "small language model" is a language model that is optimized for a specific task and is relatively lightweight and fast to operate.

[1418] "Emotion recognition" is the process of analyzing the tone, pitch, speed, etc. of a user's voice from audio data to estimate the user's emotional state.

[1419] "Response generation" is the process of creating an appropriate response based on the results of natural language analysis and emotion recognition.

[1420] "Speech synthesis" is a technique for converting generated text responses into speech and providing it to the user.

[1421] This invention is a system that combines speech-based natural language processing and emotion recognition to provide users with fast and accurate information even in offline environments. This has the advantage of being able to provide decision-making support without relying on the communication environment.

[1422] System configuration

[1423] The system consists of the following main components:

[1424] 1. Audio Input Device: Includes a microphone to capture audio data spoken by the user, which is then stored in a digital format.

[1425] 2. Speech recognition engine: Converts captured voice data into text data. Examples include Google Speech-to-Text and Microsoft Azure Speech Service.

[1426] 3. Small Language Models (SLMs): Used for natural language analysis of text data to understand user intent. Examples include a smaller version of GPT-2.

[1427] 4. Emotion Recognition Engine: Analyzes the tone, pitch, and rate of voice from audio data to estimate the user's emotional state. Open source libraries (e.g., OpenSMILE) are used.

[1428] 5. Response generator: Generates appropriate text responses based on the results obtained from natural language analysis and emotion recognition.

[1429] 6. Speech synthesis engine: Converts the generated text response into speech and provides it to the user through a speaker. Examples include Google Text-to-Speech and Amazon Polly.

[1430] System Operation

[1431] The system works as follows:

[1432] A user speaks a question or request into the device's microphone. For example, "What is the weather forecast for tomorrow?" The device captures this voice data and converts it into text using a speech recognition engine. The converted text data is then input into a small language model (SLM) for natural language analysis.

[1433] The device then uses an emotion recognition engine to recognize the user's emotions. The emotion engine analyzes the tone, pitch, and speed of the voice data to infer the user's emotional state. An appropriate response text is generated based on the analysis and emotion recognition results.

[1434] The generated response text is converted into speech using a speech synthesis engine, and finally, this speech is played back through a speaker and conveyed to the user, thereby providing fast and accurate information.

[1435] Specific examples

[1436] Examples of use in educational settings

[1437] 1. A user asks the device, "Tell me about the Pythagorean theorem."

[1438] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[1439] 3. The device uses a speech recognition engine to convert the text into "Please tell me about Pythagoras' theorem."

[1440] 4. The terminal inputs this text into the SLM for analysis. The SLM recognizes the keyword "Pythagoras' theorem" and understands its meaning.

[1441] 5. The device uses an emotion engine to recognize the user's emotions, such as whether they are interested or confused.

[1442] 6. The device generates a response saying, "The Pythagorean theorem states that the square of the hypotenuse of a right-angled triangle is equal to the sum of the squares of the other two sides." If the device recognizes that the user is interested, it provides further examples and explanations, such as, "An example would be a right-angled triangle with sides measuring 3 cm, 4 cm, and 5 cm."

[1443] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to listen to the explanation and deepen their understanding.

[1444] Examples of use in disaster relief situations

[1445] 1. The user (rescuer) speaks to the device, "Please tell me where the nearest medical facility is."

[1446] 2. The device uses a microphone to capture audio and stores it as digital audio data.

[1447] 3. The device uses a speech recognition engine to convert the text into "Please tell me where the nearest medical facility is."

[1448] 4. The device inputs this text into the SLM for analysis. The SLM recognizes the keywords "medical facility" and "nearest" and understands their meaning.

[1449] 5. The device uses an emotion engine to recognize when rescuers are in an emergency situation, gauge their level of tension from their tone of voice, and generate a quick and specific response.

[1450] 6. The device generates a response saying, "The nearest medical facility is three kilometers east." If the device detects that the user is nervous, it adjusts the response to be calm and polite, such as, "Please stay calm. It's three kilometers east."

[1451] 7. The device converts this response text into speech using a speech synthesis engine and plays it back through the speaker, allowing the user to act quickly based on this information.

[1452] This series of processes can be carried out offline as well, allowing users to obtain the information they need without relying on the communication environment.

[1453] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1454] Step 1:

[1455] The user speaks a question or request into the device's microphone. The input is the user's voice data. A specific example of this is when the user speaks something like, "What's the weather forecast for tomorrow?" The output is captured as voice data on the device.

[1456] Step 2:

[1457] The device passes the captured voice data to a voice recognition engine. The voice recognition engine (for example, Google Speech-to-Text) is used to convert the voice data into text data. The input is the user's voice data. Specifically, the voice recognition engine analyzes the voice signal and outputs the text "What is the weather forecast for tomorrow?"

[1458] Step 3:

[1459] The device inputs the converted text data into a small language model (e.g., a miniature version of GPT-2) to perform natural language analysis. The input is the text "What is the weather forecast for tomorrow?". Specifically, the SLM divides the text into tokens and analyzes the context and meaning. The output is an analysis result that indicates the intent of the user's question.

[1460] Step 4:

[1461] The device passes the voice data to an emotion recognition engine (e.g., OpenSMILE) to estimate the user's emotional state. The input is the user's voice data. Specifically, the emotion recognition engine analyzes the tone, pitch, and speed of the voice and outputs the user's emotional state (e.g., "neutral," "excited," "anxious," etc.).

[1462] Step 5:

[1463] The device generates an appropriate response based on the results of natural language analysis and emotion recognition. The input is the analysis result and the emotional state. Specifically, the SLM generates a response text such as "Tomorrow's weather forecast is sunny." At the same time, it adjusts the content and tone of the response based on the emotion recognition results. The output is the adjusted response text.

[1464] Step 6:

[1465] The device passes the generated response text to a speech synthesis engine (e.g., Google Text-to-Speech) to convert it into speech. The input is the tailored response text. Specifically, the speech synthesis engine converts the text into speech and plays it through the speaker. The output is a spoken response that is communicated to the user.

[1466] Through these steps, the system will be able to achieve advanced natural language processing and emotion recognition even in offline environments, enabling it to provide users with fast and accurate information.

[1467] (Application example 2)

[1468] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1469] There is a need to provide a means for workers and managers to smoothly communicate with factory robots even in situations where the communication environment is unstable or non-existent. In particular, there is a need for a system that can quickly and accurately process voice instructions and reports and generate responses that take into account the emotional state of the worker, thereby increasing productivity and improving the working environment.

[1470] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1471] In this invention, the server includes means for capturing voice data, means for converting the captured voice data into text data, means for natural language analysis of the text data, means for generating a response based on the analyzed text data, means for converting the generated response into voice and outputting it, means for recognizing emotions, and means for adjusting the content and tone of the response based on the emotion. This enables workers and managers to have natural conversations with factory robots even in unstable communication environments, enabling efficient work execution through appropriate work instructions, reports, and questions. Furthermore, responses that take into account the emotional state of workers are expected to improve the working environment and reduce stress.

[1472] A "means for capturing audio data" is a means for collecting audio signals as digital data using a device such as a microphone.

[1473] The "means for converting captured voice data into text data" refers to a means for converting collected voice data into a string of characters using a voice recognition engine or software.

[1474] "Means for natural language analysis of text data" refers to means for analyzing the context and meaning of text data using a natural language processing engine.

[1475] The "means for generating a response based on the analyzed text data" is a means for creating an appropriate reply based on the analysis results.

[1476] "Means for converting the generated response into speech and outputting it" means means for converting text into speech using a text-to-speech (TTS) engine and outputting it via a speaker or the like.

[1477] The "means for recognizing emotions" is a means for analyzing emotional information from voice or text data and identifying the emotional state of the user.

[1478] The "means for adjusting content and tone based on emotion when generating a response" refers to a means for adjusting the content of a reply or tone of voice based on recognized emotional information to create a natural and considerate response.

[1479] A "small language model" is a lightweight natural language processing model specialized for a specific task or application, containing algorithms for quickly and efficiently analyzing the context and meaning of text.

[1480] MODE FOR CARRYING OUT THE INVENTION

[1481] The present invention is a system for seamless communication between workers and managers through factory robots. The system operates even in unstable or non-existent communication environments, processes voice instructions and reports from workers in real time, and generates appropriate responses.

[1482] Hardware and software used

[1483] Hardware: Factory robot with microphone and speaker

[1484] software:

[1485] Speech Recognition Engine: SpeechRecognition Library

[1486] Natural Language Processing Engine: Hugging Face's Transformer Library

[1487] Emotion recognition engine: A model embedded within a natural language analysis engine

[1488] Text-to-Speech (TTS) engine: gTTS library

[1489] System Overview

[1490] The system consists of the following main processing steps: First, the factory robot's microphone captures the worker's voice data. The voice data is converted into text data through a speech recognition engine. Next, a natural language processing engine analyzes the text data, and an emotion recognition engine identifies the worker's emotions from the text data. The system generates an appropriate response based on these analysis results. The generated response is converted into speech by a TTS engine and provided to the worker through a speaker.

[1491] Specific examples

[1492] Example 1: Receiving a work order

[1493] When a worker says to the robot, "Please bring me a new part," the robot captures the speech and converts it into text using a speech recognition engine: "Please bring me a new part." The natural language processing engine then analyzes the instruction, and the emotion recognition engine recognizes the worker's emotion as "neutral." The system generates a response: "Got it. I'll start working on your request," which is then converted into speech using a TTS engine and played back.

[1494] Example 2: Problem report

[1495] When a worker says "The machine is broken," the robot captures the speech and converts it into text using a speech recognition engine. A natural language processing engine then analyzes the report, and an emotion recognition engine recognizes that the worker is anxious. The system generates a response: "I will report this to the manager immediately. It's okay, please stay calm," which is then converted into audio using a TTS engine and played back.

[1496] Prompt Sentence Examples

[1497] If the user says "The machine is broken," the following prompt will be generated:

[1498] Text: "Machine is out of order"

[1499] Task: Report a problem

[1500] If the user says "Bring me a new part," the following prompt will be generated:

[1501] Text: "Bring me new parts"

[1502] Task: Work Order

[1503] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1504] Step 1:

[1505] Voice input

[1506] The user gives instructions or reports by voice into the microphone of the factory robot, and this voice data is captured by the microphone. The input of the processing is the user's voice data, and the output is a digital audio file.

[1507] Step 2:

[1508] Voice Recognition

[1509] The device converts the captured voice data into text data using the SpeechRecognition library. The input is voice data and the output is text data. Specifically, the device analyzes the voice signal using techniques such as the Fast Fourier Transform (FFT) and converts the content into a string of characters.

[1510] Step 3:

[1511] natural language analysis

[1512] The device then analyzes the converted text data using Hugging Face's Transformer library. The input is text data, and the output is an analysis of the text's context and meaning. Specifically, a zero-shot classification model is used to analyze the text's context. At this stage, the device classifies the text as an instruction, a report, or a question.

[1513] Step 4:

[1514] emotion recognition

[1515] The device uses an emotion recognition engine to identify the user's emotions based on text data obtained through natural language analysis. The input is text data, and the output is the identified emotional state. Specifically, the device analyzes the tone, pitch, and speed of the voice and text to classify the user's emotions.

[1516] Step 5:

[1517] Response Generation

[1518] The device generates an appropriate response based on the analyzed text data and emotional information. The input is the context analysis results and emotional information, and the output is the generated response text. Specifically, the response generation engine generates a response such as "Understood. We will begin working on your request" in the case of an instruction, or "We will report to the administrator immediately" in the case of a report. The content and tone are also adjusted depending on the emotion.

[1519] Step 6:

[1520] Audio Output

[1521] The device converts the generated response text into speech using the gTTS library and plays it from the speaker. The input is the generated response text, and the output is the audio to be played. Specifically, the text-to-speech (TTS) engine converts the text into a speech waveform and outputs it from the speaker.

[1522] These processing steps enable natural voice interaction with the user, allowing the factory robot to communicate smoothly with workers and managers.

[1523] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1524] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1525] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1526] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1527] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1528] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1529] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1530] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1531] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1532] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1533] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1534] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1535] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1536] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1537] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1538] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1539] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1540] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1541] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1542] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1543] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1544] The following is further disclosed regarding the above embodiment.

[1545] (Claim 1)

[1546] means for capturing audio data;

[1547] means for converting the captured audio data into text data;

[1548] a means for natural language analysis of text data;

[1549] means for generating a response based on the parsed text data;

[1550] means for converting the generated response into speech and outputting the speech;

[1551] A system including:

[1552] (Claim 2)

[1553] 10. The system of claim 1, wherein the natural language analysis means uses a small language model to analyze the context and meaning of the text data.

[1554] (Claim 3)

[1555] 10. The system of claim 1, wherein the responses generated based on the text data provide relevant information to questions in disaster relief, education, and research.

[1556] "Example 1"

[1557] (Claim 1)

[1558] means for acquiring audio data;

[1559] A means for analyzing the acquired voice data and converting it into text data;

[1560] A method using a small language model to perform natural language analysis on text data;

[1561] means for generating an appropriate response based on the parsed text data;

[1562] means for converting the generated response into speech and outputting the speech;

[1563] A system including:

[1564] (Claim 2)

[1565] 2. The system according to claim 1, further comprising a speech recognition means for converting a speech signal into features by short-time Fourier transform and converting the features into text data.

[1566] (Claim 3)

[1567] 10. The system of claim 1, wherein the natural language analysis means uses a pre-trained small model to tokenize input text data and employs a self-attention mechanism for contextual understanding.

[1568] (Claim 4)

[1569] 10. The system of claim 1, wherein the responses generated based on the text data provide relevant information to questions in disaster relief, education, and research.

[1570] "Application Example 1"

[1571] (Claim 1)

[1572] means for capturing audio data;

[1573] means for converting the captured audio data into text data;

[1574] a means for natural language analysis of text data;

[1575] means for generating a response based on the parsed text data;

[1576] means for converting the generated response into speech and outputting the speech;

[1577] a means for receiving voice instructions from an operator in a factory environment, understanding the instructions, and performing appropriate actions;

[1578] control means for operating the means in the robot;

[1579] A system including:

[1580] (Claim 2)

[1581] 10. The system of claim 1, wherein the natural language analysis means uses a small language model to analyze the context and meaning of the text data.

[1582] (Claim 3)

[1583] 10. The system of claim 1, wherein the response generated based on the text data provides information appropriate for work instructions in a factory environment.

[1584] "Example 2: Combining Emotion Engines"

[1585] (Claim 1)

[1586] means for capturing audio data;

[1587] means for converting the captured audio data into text data;

[1588] a means for using a small language model for natural language analysis of text data;

[1589] emotion recognition means for estimating an emotional state from speech data;

[1590] means for generating a response based on the analyzed text data and the emotional state;

[1591] means for converting the generated response into speech and outputting the speech;

[1592] A system including:

[1593] (Claim 2)

[1594] 10. The system of claim 1, wherein natural language analysis means and emotion recognition means are used to generate responses to user questions that take into account the user's emotional state.

[1595] (Claim 3)

[1596] 10. The system of claim 1, wherein the responses generated based on the text data provide relevant information to questions in disaster relief, education, and research.

[1597] "Application example 2 when combining emotion engines"

[1598] (Claim 1)

[1599] means for capturing audio data;

[1600] means for converting the captured audio data into text data;

[1601] a means for natural language analysis of text data;

[1602] means for generating a response based on the parsed text data;

[1603] means for converting the generated response into speech and outputting the speech;

[1604] A means of recognizing emotions;

[1605] a means of adjusting content and tone based on emotion when generating responses;

[1606] A system including:

[1607] (Claim 2)

[1608] 10. The system of claim 1, wherein the natural language analysis means uses a small language model to analyze the context and meaning of the text data.

[1609] (Claim 3)

[1610] 10. The system of claim 1, wherein the responses generated based on the text data provide relevant information for questions in the fields of disaster relief, education, research, and industry. [Explanation of symbols]

[1611] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing audio data; means for converting the captured audio data into text data; a means for natural language analysis of text data; means for generating a response based on the parsed text data; means for converting the generated response into speech and outputting the speech; A system including:

2. 2. The system according to claim 1, wherein the natural language analysis means uses a small language model to analyze the context and meaning of the text data.

3. The system of claim 1 , wherein the responses generated based on the text data provide relevant information for questions in disaster relief, education, and research.

Images