system

Abstract

To provide a system for supporting the learning of children with mild physical or intellectual disabilities. [Solution] A system including a means for capturing voice data input by a user, a means for transmitting the voice data to a server, a voice recognition means for converting the voice data to text data at the server, a means for analyzing the text data and identifying the user's intentions, a means for selecting appropriate feedback and learning materials, a means for transmitting text data of the selected feedback and learning materials to a terminal, a voice synthesis means for converting the text data into voice data, and a means for playing the voice data to the user.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Children with mild physical or intellectual disabilities are highly motivated to learn, but the support system is insufficient, making it difficult for them to progress in their studies. Furthermore, the workload of parents and welfare workers is excessive, and there is a shortage of manpower to provide appropriate support. This can make it difficult for children to express their opinions meaningfully, and may result in a decline in their understanding of the learning content. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides a system for supporting the learning of children with mild physical or intellectual disabilities. The system includes: means for capturing voice data input by a user; means for transmitting the voice data to a server; speech recognition means for converting the voice data into text data in the server; means for analyzing the text data and identifying the user's intention; means for selecting appropriate feedback and learning materials; means for transmitting the text data of the selected feedback and learning materials to a terminal; speech synthesis means for converting the text data into voice data; and means for playing the voice data to the user. This system allows children to smoothly communicate their intentions and progress in their studies while receiving feedback. This reduces the burden on parents and welfare workers and enables the creation of an effective support system.

[0006] "Mild physical or intellectual disability" refers to a disability that requires some support in daily life and learning, but allows a person to act and understand independently to a certain extent.

[0007] "User" refers primarily to children who receive learning support using this system, and may also include their parents and welfare workers as necessary.

[0008] "Voice data" refers to a digital recording of the user's voice spoken into the terminal.

[0009] "Means for capturing" refers to the process by which the terminal detects the voice uttered by the user and records it as voice data.

[0010] A "server" is a computer system that processes and stores voice data, and performs various calculations such as voice recognition and voice synthesis using generation AI.

[0011] "Speech recognition means" refers to algorithms or programs that convert voice data into text data, and may use generative AI technology.

[0012] "Text data" refers to data of character information converted by voice recognition.

[0013] "Means of analysis" refers to the process of identifying the user's intent and learning needs based on text data, and may involve the use of natural language processing technology.

[0014] "Means for selecting feedback and learning materials" refers to the process of selecting appropriate learning support content and learning materials for users based on the analysis results.

[0015] "Terminal" refers to a computer or information device that is directly operated by a user to input voice data, play back voice feedback, and so on.

[0016] A "voice synthesis means" is an algorithm or program that converts text data into voice data, and may use generative AI technology.

[0017] "Means for playing" refers to the process by which the terminal outputs the generated audio data through a device such as a speaker or headphones. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system for supporting the learning of children with mild physical or intellectual disabilities, and its form is described in detail below. The system described below allows users (children) to smoothly communicate their intentions and receive necessary feedback and educational materials.

[0040] System Overview

[0041] This system captures the user's speech into the device and sends the speech data to a server. The server converts the speech data into text data and analyzes the user's intentions and learning needs. It then selects appropriate feedback and learning materials and sends them back to the device. The device then synthesizes the selected feedback and learning materials into speech and plays them back to the user.

[0042] Program processing details

[0043] Receiving and sending audio input

[0044] The user says to the terminal, "I want to study math today."

[0045] The device uses a microphone to capture the user's voice, and the captured voice data is stored in digital form.

[0046] The device generates and sends an API request to send the captured audio data to the server.

[0047] Speech Recognition and Analysis

[0048] The server receives the voice data sent from the device and inputs it into a voice recognition model (e.g., a model using generative AI technology).

[0049] The speech recognition model converts the speech data into text data, for example, in the form "I want to study math today."

[0050] The server analyzes the acquired text data and understands the user's intent using natural language processing (NLP) technology.

[0051] Based on the analysis results, the server selects appropriate feedback and learning materials according to the user's learning needs. For example, if a user requests to "study math," it will select addition problems appropriate to their level.

[0052] Sending feedback and text-to-speech

[0053] The server generates an API response to send the selected feedback and content of the teaching materials to the terminal as text data.

[0054] The device inputs the text data received from the server into a speech synthesis model (e.g., a model using generative AI technology).

[0055] The speech synthesis model converts text data into speech data, generating speech data such as "Let's try an addition problem today."

[0056] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0057] Specific examples

[0058] 1. Receiving voice input

[0059] A user says, "Today I want to study math."

[0060] The device captures the audio and stores it as audio data.

[0061] The terminal transmits the voice data to the server.

[0062] 2. Speech Recognition and Analysis

[0063] The server inputs the voice data into a voice recognition model and converts it into text data.

[0064] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[0065] The server selects appropriate teaching materials and generates feedback such as "Let's try some addition problems" as text data.

[0066] 3. Sending feedback and speech synthesis

[0067] The server sends the feedback to the device.

[0068] The device inputs the text data into a speech synthesis model and converts it into speech data.

[0069] The terminal plays back the generated voice data and provides feedback to the user.

[0070] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they progress through their studies. It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[0074] Step 2:

[0075] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[0076] Step 3:

[0077] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[0078] Step 4:

[0079] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[0080] Step 5:

[0081] The server inputs the captured voice data into a speech recognition model, which uses generative AI to perform advanced speech recognition.

[0082] Step 6:

[0083] The speech recognition model converts the speech data into text data, for example, text data in the format "I want to study math today."

[0084] Step 7:

[0085] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning needs.

[0086] Step 8:

[0087] The server selects appropriate feedback and learning materials based on the analysis results. For example, if a user requests "I want to study math," it will select "addition problems."

[0088] Step 9:

[0089] The server generates text data based on the selected feedback and teaching materials, such as "Let's try some addition problems today."

[0090] Step 10:

[0091] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[0092] Step 11:

[0093] The device receives the API response from the server and obtains the feedback text data.

[0094] Step 12:

[0095] The device inputs the captured text data into a speech synthesis model, which uses generative AI to convert the text data into speech data.

[0096] Step 13:

[0097] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's try an addition problem today."

[0098] Step 14:

[0099] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] With conventional learning support systems, children with mild physical or intellectual disabilities have difficulty expressing themselves smoothly, making it difficult for them to receive appropriate feedback and learning materials. Furthermore, the burden on parents and welfare workers is heavy, making it difficult to establish an effective support system.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means for converting the voice data into text data in the server, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to the terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This allows the user to smoothly communicate their intentions and progress in their studies while receiving appropriate feedback. It also reduces the burden on parents and welfare workers and enables the creation of an effective support system.

[0105] "User" refers to a child with a mild physical or intellectual disability who uses the system.

[0106] "Voice data" refers to digital data of a user's speech captured by a voice input device such as a microphone.

[0107] "Server" refers to a central processing unit that receives voice data, converts it into text data, and then analyzes the text data to identify the user's intent and select appropriate feedback and educational materials.

[0108] "Speech Recognition Method" refers to the technology or model used to convert voice data received by the server into text data, including, for example, generative AI technology.

[0109] "Text data" refers to character string data converted from voice data by a voice recognition means.

[0110] "Natural language processing technology" refers to technology for analyzing text data to understand its meaning and identify the user's intent.

[0111] "Feedback" refers to instructions, advice, or learnings provided based on user input.

[0112] "Instructional Materials" refers to educational content and materials provided to assist users in their learning.

[0113] "Speech synthesis method" refers to the technology or model used to convert text data into speech data, including, for example, generative AI technologies.

[0114] "Terminal" refers to a device that captures what a user says and communicates with a server. It may include a playback device.

[0115] The present invention is a system for supporting learning for children with mild physical or intellectual disabilities, and is described in detail below. This system allows users (children) to smoothly communicate their intentions and receive necessary feedback and learning materials.

[0116] System Overview

[0117] The system operates on the basis of voice data processing and communication between terminals, servers and users.

[0118] Hardware and Software Configuration

[0119] Device:

[0120] The built-in microphone is used to capture the user's voice.

[0121] It includes an API client to communicate with the server, for example using the HTTPS protocol.

[0122] Play audio using the built-in speaker or headphones.

[0123] Use a speech synthesis model (e.g., Google® Text-to-Speech API or Amazon Polly, which use generative AI techniques).

[0124] server:

[0125] The voice data is input into a speech recognition model (e.g., Google Speech-to-Text API or IBM Watson® Speech to Text, which utilize generative AI technology).

[0126] We use natural language processing (NLP) techniques (e.g., NLTK, spaCy) to analyze the text data obtained by speech recognition.

[0127] Select appropriate feedback and teaching materials from the database.

[0128] Text data of feedback and teaching materials is generated as an API response and sent to the device.

[0129] Capture and transmit audio data

[0130] When a user speaks into the device, such as "I want to study math today," the device's microphone captures the voice, which is then stored digitally and used to generate an API request and send it to the server.

[0131] Speech Recognition and Analysis

[0132] The server receives the voice data sent from the device and inputs it into a voice recognition model. The model converts the voice data into text data, for example, in the format "I want to study math today." The server analyzes this text data and uses natural language processing technology to understand the user's intention. Through this analysis, the server understands that the user's intention is "I want to study math."

[0133] Selecting and sending appropriate feedback and educational materials

[0134] The server selects appropriate feedback and learning materials from the database based on the user's wishes. For example, if the user says, "I want to study math," the server selects addition problems appropriate to the user's level. It then generates and sends an API response to send the text data of the selected feedback and learning materials to the device.

[0135] Speech synthesis and feedback playback

[0136] The device inputs the text data received from the server into a speech synthesis model and converts it into voice data. For example, voice data such as "Let's try an addition problem today" is generated. Finally, the device plays this voice data to the user through a playback device.

[0137] Specific examples

[0138] When a user says, "I want to study math today," the device captures the voice and sends the voice data to the server. The server inputs this voice data into a voice recognition model, converts it into text data, and then analyzes it using natural language processing technology to determine that the user's intention is "I want to study math." The server then selects an appropriate addition problem and sends the content of that problem to the device as text data. The device then inputs this text data into a voice synthesis model, generates voice data saying, "Let's try an addition problem today," and plays it back to the user.

[0139] Prompt Sentence Examples

[0140] "When a user says, 'I want to study math today,' select learning materials and feedback appropriate to that intent and generate them as text data."

[0141] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they learn. Furthermore, by utilizing a generative AI model, highly accurate speech recognition and speech synthesis are achieved.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] The user speaks to the terminal.

[0145] Input: User utterance (e.g., "I want to study math today.")

[0146] Output: The user's speech is input to the device's microphone as an analog audio signal.

[0147] Specific action: The user speaks into the built-in microphone.

[0148] Step 2:

[0149] The device captures the audio data and stores it digitally.

[0150] Input: Analog audio signal

[0151] Output: Digital audio data

[0152] Specific operation: The device's microphone driver converts the analog audio signal into digital audio data and temporarily stores it in internal memory.

[0153] Step 3:

[0154] The device generates and sends an API request to send the captured audio data to the server.

[0155] Input: Digital audio data

[0156] Output: API request to the server (including audio data)

[0157] Specific operation: The device sends audio data to the server using the HTTPS protocol.

[0158] Step 4:

[0159] The server receives the voice data transmitted from the terminal.

[0160] Input: API request (including audio data)

[0161] Output: Audio data stored on the server

[0162] Specific operation: The server's API endpoint receives the request, extracts the audio data, and saves it in the server's storage.

[0163] Step 5:

[0164] The server inputs the voice data into a voice recognition model and converts it into text data.

[0165] Input: Digital audio data

[0166] Output: Text data (e.g., "I want to study math today")

[0167] Specific operation: The server inputs voice data into a voice recognition model (which utilizes generative AI technology), and the model analyzes the voice data and converts it into text data.

[0168] Step 6:

[0169] The server analyzes the text data and identifies the user's intent.

[0170] Input: Text data

[0171] Output: User's intention (e.g., "I want to study math")

[0172] Specific operation: The server uses natural language processing (NLP) techniques to analyze the text data and identify the user's intent.

[0173] Step 7:

[0174] The server selects appropriate feedback and educational materials.

[0175] Input: User intent

[0176] Output: Feedback and text data for teaching materials (e.g., "Let's do the addition problem")

[0177] Specific operation: The server selects feedback and learning materials from the database according to the user's wishes.

[0178] Step 8:

[0179] The server generates and sends an API response to send the selected feedback and text data of the teaching materials to the terminal.

[0180] Input: Text data of feedback and teaching materials

[0181] Output: API response (including text data)

[0182] Specific operation: The server sends text data to the terminal using the HTTPS protocol.

[0183] Step 9:

[0184] The text data received by the terminal from the server is input into a speech synthesis model and converted into voice data.

[0185] Input: Text data

[0186] Output: Audio data (e.g., "Let's try some addition problems today.")

[0187] Specific operation: The device inputs text data into a speech synthesis model (using generative AI technology), which then converts the text data into speech data.

[0188] Step 10:

[0189] The terminal plays the generated audio data to the user using a playback device.

[0190] Input: Audio data

[0191] Output: The audio the user hears

[0192] What it does: The device's audio driver sends audio signals to speakers or headphones so the user can hear the sound.

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] When working in a factory, workers with mild physical or intellectual disabilities may find it difficult to receive appropriate work instructions and feedback. This reduces work efficiency and makes it difficult to ensure worker safety. Furthermore, workers with disabilities have difficulty communicating their intentions smoothly, which can lead to problems such as not receiving appropriate support.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means in the server for converting the voice data into text data, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, means for playing the voice data to the user, and means for capturing the voice of workers in a factory and providing appropriate feedback on work procedures and points to note. This enables workers to smoothly communicate their intentions and receive appropriate feedback on work procedures and points to note.

[0198] "Capture" means taking in data such as audio or images and saving it as digital data.

[0199] "Voice data" refers to data that is recorded using a microphone or other device and expressed in digital format.

[0200] A "server" is a computer system that receives and responds to requests from clients over a network.

[0201] "Text data" refers to data that represents character information in digital form.

[0202] "Speech recognition" is a technology that analyzes input voice data and converts it into corresponding text data.

[0203] "Analysis" is the process of examining data in detail to understand its content and trends.

[0204] "Feedback" refers to information or actions that provide a response or reaction to a particular action or request.

[0205] "Instructional Materials" means materials or content used for educational or training purposes.

[0206] "Speech synthesis" is a technology that converts text data into voice data.

[0207] "Playback" means outputting captured or generated audio or video data through an audio or video device.

[0208] "Factory" means the facility or workspace where manufacturing or production takes place.

[0209] "Worker" means a person who performs work in a factory.

[0210] A "work procedure" is the steps or process for accomplishing a particular task.

[0211] "Caution" refers to matters that require special attention in a particular situation or task.

[0212] System Overview

[0213] This invention relates to a system that supports workers with mild physical or intellectual disabilities who work in factories. This system captures the worker's voice input and sends the voice data to a server. The server converts the voice data into text data and analyzes the worker's intentions and work needs. It then selects appropriate work procedures and points to note and sends them back to the terminal. The terminal also supports the worker by synthesizing the selected feedback into voice and playing it back to the worker.

[0214] Hardware and software used

[0215] 1. Hardware:

[0216] Microphone: Used to capture the voice of the worker.

[0217] Terminal (e.g., tablet, smartphone): A device that allows workers to input voice and receive feedback.

[0218] Speaker: Used to play feedback audio.

[0219] 2. Software:

[0220] Python: The programming language used to implement the entire program.

[0221] speech_recognition library: A library for capturing and recognizing speech.

[0222] OpenAI® GPT-3® API: A generative AI model that analyzes voice data and generates appropriate feedback.

[0223] gTTS library: A library for synthesizing text data into voice data.

[0224] playsound library: A library for playing generated sound data.

[0225] System processing details

[0226] 1. Receiving audio input:

[0227] The worker talks to the terminal about the work he is doing and the problems he is having.

[0228] The device uses a microphone to capture audio data and store it in digital form.

[0229] The captured audio data is sent to a server.

[0230] 2. Speech Recognition and Analysis:

[0231] The server receives the transmitted voice data and converts it into text data using the speech_recognition library.

[0232] The converted text data is then analyzed using the OpenAI GPT-3 API.

[0233] Based on the analysis results, the server understands the worker's intentions and work needs, and generates appropriate work procedures and points to note.

[0234] 3. Send feedback and speech synthesis:

[0235] The server transmits the generated feedback text data to the terminal.

[0236] The device uses the gTTS library to convert the feedback text data into voice data.

[0237] The converted voice data is played back to the worker through a speaker.

[0238] Specific examples

[0239] Example 1:

[0240] The worker speaks into the terminal, "What should I do next?"

[0241] A microphone captures the audio and sends it to a server.

[0242] The server converts the voice data into text data and analyzes it using the OpenAI GPT-3 API.

[0243] Feedback is generated such as "Next, you need to assemble part A."

[0244] The terminal synthesizes the generated feedback into voice and plays it back to the worker.

[0245] Prompt Sentence Examples

[0246] User: What should I do next?

[0247] A basic import, a mutually exclusive reaction or a step-by-step process.

[0248] Using this system, workers can smoothly communicate their intentions and receive feedback on appropriate work procedures and points to note. It is also expected to contribute to improving work efficiency and ensuring safety.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] The user speaks instructions or questions into the device. The device's microphone captures this voice and stores it as digital audio data. The input is the user's voice, and the output is digital audio data. Specifically, the user speaks, "What should I do next?", and the device records the voice and converts it into digital format.

[0252] Step 2:

[0253] The device sends the captured audio data to the server. The input is digital audio data, and the output is the audio data sent to the server. Specifically, the device generates an HTTP request over the network and sends the audio data to the server.

[0254] Step 3:

[0255] The voice data received by the server is input into a voice recognition model (e.g., speech_recognition library) and converted into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "What should I do next?"

[0256] Step 4:

[0257] The server analyzes the text data and uses a generative AI model (e.g., OpenAI GPT-3) to understand the user's intentions. The input is text data, and the output is feedback text based on the analysis results. Specifically, the server inputs the text data into the generative AI model and creates feedback text such as, "Next, you need to assemble part A."

[0258] Step 5:

[0259] The server sends the generated feedback text to the terminal. The input is the feedback text data, and the output is the feedback text sent to the terminal. Specifically, the server generates an HTTP response and sends the feedback text data to the terminal.

[0260] Step 6:

[0261] The feedback text received by the device is input into a speech synthesis model (e.g., gTTS library) and converted into voice data. The input is the feedback text and the output is voice data. Specifically, the device inputs the text data into the gTTS library and generates voice data saying, "Next, you need to assemble part A."

[0262] Step 7:

[0263] The device plays the generated audio data through a speaker. The input is audio data, and the output is the actual audio heard by the worker. Specifically, the device plays the audio data through the playsound library, and the worker receives feedback via audio.

[0264] Through these steps, workers can smoothly communicate their intentions and receive audio feedback on appropriate work procedures and points to note.

[0265] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0266] The present invention relates to a system for supporting the learning of children with mild physical or intellectual disabilities, and in particular to a system that provides more appropriate feedback and educational materials based on the user's emotional state by combining an emotion engine.

[0267] System Overview

[0268] This system captures the user's voice as they speak into their device and sends the voice data to a server. The server converts the voice data into text data and analyzes the user's intentions and learning needs. It also uses an emotion engine to analyze the user's emotions and selects appropriate feedback and learning materials based on the analysis results. The selected feedback and learning materials are sent to the device and played back to the user using voice synthesis.

[0269] Program processing details

[0270] Receiving and sending audio input

[0271] The user says to the terminal, "I want to study math today."

[0272] The device captures the user's voice using a microphone and temporarily stores it in digital form.

[0273] The device generates and sends an API request to send the captured audio data to the server.

[0274] Speech Recognition and Analysis

[0275] The server receives the voice data sent from the device and inputs it into a voice recognition model, which uses generative AI to convert the voice data into text data.

[0276] For example, text data such as "I want to study math today" is generated.

[0277] The server analyzes the text data to understand the user's intent, using natural language processing (NLP) techniques.

[0278] Emotion analysis

[0279] The server inputs the voice or text data into an emotion engine to analyze the user's emotions. Generative AI technology is used to perform emotion analysis and identify the user's emotional state, such as whether they are happy, sad, or distressed.

[0280] The feedback and educational content is adjusted based on the results of sentiment analysis. For example, if the analysis shows that the user is in trouble, the feedback will be gentle and responsive.

[0281] Selecting and submitting feedback

[0282] The server selects appropriate feedback and learning materials based on the results of the analysis and sentiment analysis. For example, if the server determines that the user is having trouble with a request such as "I want to study math," it generates gentle feedback such as "Let's start with the basics."

[0283] The server sends the generated feedback and text data of the teaching materials to the terminal.

[0284] Speech synthesis and playback

[0285] The device inputs the text data received from the server into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[0286] For example, voice data saying "Let's start with the basics first" is generated.

[0287] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0288] Specific examples

[0289] 1. Receiving voice input

[0290] A user says, "Today I want to study math."

[0291] The device captures the audio and stores it as audio data.

[0292] The terminal transmits the voice data to the server.

[0293] 2. Speech Recognition and Analysis

[0294] The server inputs the voice data into a voice recognition model and converts it into text data.

[0295] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[0296] 3. Emotion analysis

[0297] The server uses an emotion engine to analyze the user's emotion from the voice data or text data.

[0298] Based on the analysis results, it is recognized that, for example, "the user is feeling anxious."

[0299] 4. Selecting and Submitting Feedback

[0300] Based on the results of the analysis and emotion analysis, the server generates gentle feedback such as, "Let's try an addition problem."

[0301] The generated feedback is sent to the terminal as text data.

[0302] 5. Speech Synthesis and Playback

[0303] The text data received by the terminal is input into a speech synthesis model and converted into voice data.

[0304] The device plays the generated audio and tells the user, "Let's start with the basics."

[0305] This system allows users to smoothly communicate their intentions, receive appropriate feedback, and progress in learning that is tailored to their emotional state.It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[0306] The processing flow will be explained below.

[0307] Step 1:

[0308] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[0309] Step 2:

[0310] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[0311] Step 3:

[0312] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[0313] Step 4:

[0314] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[0315] Step 5:

[0316] The server inputs the acquired voice data into a voice recognition model, which uses generative AI to perform advanced voice recognition.

[0317] Step 6:

[0318] The speech recognition model converts the speech data into text data, for example, "I want to study math today."

[0319] Step 7:

[0320] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning goals.

[0321] Step 8:

[0322] The server inputs the voice data into an emotion engine to analyze the user's emotions. Generative AI technology is used to identify emotional states (such as joy, sadness, or anxiety) from the voice.

[0323] Step 9:

[0324] The server selects feedback and teaching materials based on the emotional data analyzed by the emotion engine. For example, if the server determines that the user is feeling anxious, it will select gentle instructional content.

[0325] Step 10:

[0326] The server generates text data based on the selected feedback and learning materials, such as "Let's start with the basics."

[0327] Step 11:

[0328] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[0329] Step 12:

[0330] The device receives the API response from the server and obtains the feedback text data.

[0331] Step 13:

[0332] The device inputs the acquired text data into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[0333] Step 14:

[0334] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's start with the basics."

[0335] Step 15:

[0336] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0337] Example 2

[0338] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] This invention relates to a system for supporting the learning of children with mild disabilities. Conventional learning support systems often provide one-sided feedback and learning materials without considering the user's emotional state, limiting the effectiveness of learning. Furthermore, there are issues with the difficulty of selecting appropriate feedback and learning materials, making it difficult to maintain the user's motivation to learn. The objective of this invention is to solve these issues and provide more effective learning support.

[0340] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0341] In this invention,

[0342] The server includes means for capturing voice data input by a user, means for transmitting the voice data to a relay device, voice recognition means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for analyzing the voice data or the text data to identify the user's emotional state, means for selecting appropriate feedback and learning materials based on the user's intention and emotional state, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This makes it possible to provide appropriate feedback and learning materials that take the user's emotional state into consideration, thereby improving learning effectiveness and maintaining learning motivation.

[0343] "Mild disability" refers to a minor limitation in physical or intellectual function.

[0344] "Supporting learning" refers to helping users acquire knowledge and skills efficiently.

[0345] "User" refers to individuals who use the system, particularly children with mild disabilities.

[0346] "Voice data" refers to data that has been saved in digital form as the voice of a user speaking into a terminal.

[0347] "Capturing" refers to using a microphone to collect sound and convert it into a digital format.

[0348] "Relay device" refers to a device or software for transmitting data from a terminal to a server.

[0349] "Speech recognition" refers to the technology of converting voice data into text data.

[0350] "Text data" refers to text information converted from voice data using voice recognition technology.

[0351] "Analysis" refers to evaluating text data to understand the user's intentions and emotions.

[0352] "Emotional state" refers to the emotion the user is feeling at that time.

[0353] "Feedback" refers to a response or information to a user's input or state.

[0354] "Instructional Materials" refers to educational resources and materials intended to aid users in their learning.

[0355] "Speech synthesis" refers to the technology of converting text data into voice data.

[0356] "Playback" refers to listening to audio data through speakers or headphones.

[0357] "Generative AI" refers to models and techniques that use artificial intelligence techniques to generate or transform data.

[0358] This invention is a system for supporting the learning of children with mild disabilities. In particular, it improves learning effectiveness by analyzing the user's emotional state and providing appropriate feedback and learning materials based on that. The system has the function of capturing voice data input by the user and transmitting the voice data to a server.

[0359] The device captures the user's speech using a microphone and stores it in digital format. The stored speech data is then sent to a server via a relay device. The server uses speech recognition technology to convert the speech data into text data. For speech recognition, generative AI technology (e.g., a general generative AI speech recognition model) is used. For example, if a user says, "I want to study math today," the server converts this speech into text and generates the text data, "I want to study math today."

[0360] The server then analyzes this text data to identify the user's intent. This analysis uses natural language processing (NLP) techniques (e.g., widely used natural language processing models). After the user's intent is identified, this text data or the original voice data is input into a sentiment analysis engine. Sentiment analysis uses generative AI techniques (e.g., common sentiment analysis models) to identify the user's emotional state. For example, if the analysis finds that the user is in trouble, the system generates appropriate feedback based on this.

[0361] The server selects appropriate feedback and learning materials based on the user's intentions and emotional state. For example, if a user says, "I want to study math," and is also analyzed as "having trouble," the server generates gentle feedback such as, "Let's start with the basics." The generated feedback and learning materials are then sent to the device as text data.

[0362] The device converts the received text data into voice data using speech synthesis technology (for example, a common speech synthesis model). The converted voice data is played back to the user through speakers or headphones. For example, feedback such as "Let's start with the basics" may be played back as voice.

[0363] This system allows users to smoothly communicate their intentions and receive appropriate feedback and learning materials in response to their requests, while also taking their emotional state into consideration when studying. This improves learning effectiveness and maintains motivation to learn.

[0364] Specific examples

[0365] When a user says, "I want to study math today," the device captures this voice and saves it digitally. The saved voice data is sent to a server. The server converts this voice data into text data using generative AI technology, generating the text data "I want to study math today." Natural language processing technology is then used on this text data to identify the user's intention as "I want to study math."

[0366] The server then inputs this voice or text data into an emotion analysis engine to analyze the user's emotional state. If generative AI technology determines that the user is having trouble, it generates gentle feedback such as, "Let's start with the basics." The text data of this feedback is sent to the device, which then converts it into voice data and plays it back to the user.

[0367] Example prompt sentence:

[0368] I want to study math today.

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1:

[0371] The device captures the user's speech with a microphone and stores it in digital form. The input is the user's speech, and the device uses the microphone to convert analog speech into digital audio data. The output is the captured digital audio data.

[0372] Step 2:

[0373] The device generates an API request to send the stored digital audio data to the server, and sends the audio data to the server. The input is the digital audio data generated in step 1, and the output is the audio data sent to the server. Specifically, the device sends the audio data over the network.

[0374] Step 3:

[0375] The server receives the voice data sent from the device and inputs it into a voice recognition model. The input is the voice data sent from the device, and the voice data is converted into text data using a voice recognition model (for example, a generative AI voice recognition model). The output is the converted text data. For example, the text data generated is "I want to study math today."

[0376] Step 4:

[0377] The server analyzes the generated text data and identifies the user's intention. The input is the text data generated in step 3, and the server analyzes the data using natural language processing (NLP) techniques (e.g., a widely used NLP model). The output is data indicating the user's intention. For example, the intention "I want to study math" is identified.

[0378] Step 5:

[0379] The server inputs the voice or text data into the emotion engine and analyzes the user's emotional state. The input is the data generated in step 3 or step 4, and generative AI technology is used to identify the user's emotion. The output is data that indicates the user's emotional state. For example, the analysis may determine that the user is in trouble.

[0380] Step 6:

[0381] The server selects appropriate feedback and learning materials based on the user's intentions and the results of emotion analysis. The input is the data generated in steps 4 and 5, and an algorithm is used to select appropriate feedback. The output is the text data of the selected feedback and learning materials. For example, the generated feedback is "Let's start with the basics."

[0382] Step 7:

[0383] The server sends the selected feedback and text data of the teaching materials to the terminal. The input is the text data generated in step 6, and the output is the feedback text data sent to the terminal. Specifically, the server sends the text data over the network.

[0384] Step 8:

[0385] The device inputs the received text data into a speech synthesis model and converts it into speech data. The input is the text data received in step 7, and the generative AI speech synthesis model is used to convert the text data into speech data. The output is the converted speech data. For example, speech data such as "Let's start with the basics" is generated.

[0386] Step 9:

[0387] The device plays the generated audio data to the user through speakers or headphones. The input is the audio data generated in step 8, and the specific operation is to deliver the audio to the user using a playback device. The output is the audio the user hears. For example, the audio played to the user is, "Let's start with the basics."

[0388] (Application example 2)

[0389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] Conventional factory robots are unable to provide feedback or instructions based on the worker's emotions or situation, making it difficult to reduce worker stress or support efficient work. Furthermore, when a worker feels confused or anxious, delays in appropriate responses can lead to reduced work efficiency and an increased risk of mistakes. Therefore, there is a need for a system that can provide feedback and instructions based on the worker's emotional state, improving work efficiency and reducing worker stress.

[0391] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion analysis means for analyzing voice data or text data to identify the user's emotional state, means for adjusting feedback and teaching materials based on the emotional state, and means for capturing voice data. This allows appropriate feedback and instructions to be provided according to the worker's emotional state, improving work efficiency and reducing worker stress.

[0392] "Audio data" refers to data that represents audio in digital form.

[0393] A "server" is a high-performance computer that processes and manages data over a network.

[0394] "Speech recognition means" refers to technology that analyzes voice data and converts it into text data.

[0395] "Text data" refers to data that represents character information in a digital format.

[0396] "Emotion analysis means" refers to a technique for analyzing voice data or text data to identify the user's emotional state.

[0397] "Means for selecting feedback and educational materials" refers to techniques for selecting appropriate information and educational materials based on the user's intentions and emotional state.

[0398] "Speech synthesis means" refers to technology that converts text data into voice data.

[0399] "Playback means" refers to technology that allows users to listen to audio data through devices such as speakers or headphones.

[0400] "Means for analyzing data" refers to the technology for analyzing input data and extracting necessary information.

[0401] "Emotional state" refers to the user's psychological emotional state, and includes, for example, happiness, sadness, confusion, anxiety, and the like.

[0402] System configuration

[0403] The system of the present invention includes a voice input means, a means for transmitting voice data to a server, a voice recognition means, a sentiment analysis means, a means for selecting feedback and teaching materials, a voice synthesis means, and a playback means. Each of these means will be described in detail below.

[0404] Voice input means

[0405] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. The microphone in the smart glasses or HMD is used to capture the voice data. This voice data is then sent to the server in the next step.

[0406] A means of sending audio data to the server

[0407] The captured audio data is sent to the server via the device's communication module, using HTTP or HTTPS as the communication protocol.

[0408] Voice recognition means

[0409] The server uses a generative AI model, such as the Google Cloud Speech API, to convert the audio data into text. This is the step where speech recognition takes place and converts the user's input speech into text form.

[0410] Emotion analysis means

[0411] The generated text data is passed to a sentiment analysis engine, which analyzes the voice or text data to identify the user's emotions. Generative AI technology is used to identify the user's emotional state, such as whether they are happy, sad, or troubled.

[0412] Feedback and materials selection

[0413] Based on the results of the sentiment analysis, the server selects appropriate feedback and educational materials. For example, if the analysis indicates that the user is confused, the server selects the message "Please stay calm and continue working. Do you need further instructions?" as the corresponding feedback.

[0414] Voice synthesis means

[0415] The selected feedback and learning materials are then converted into audio data using generative AI models such as the Google Cloud Text-to-Speech API, which converts the text data into audio data and generates audio to provide appropriate instructions to the user.

[0416] Regeneration means

[0417] The generated audio data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive feedback and instructions in real time.

[0418] Specific examples

[0419] For example, if a user says, "I feel like I'm not making much progress on my work today. I'm having trouble," the voice input means captures this speech. The voice data is sent to the server and converted into text data by the voice recognition means. The generated text is passed to the "emotion analysis engine," which identifies the emotion "confused." In response, the server selects feedback such as "Please stay calm and continue working. Do you need further instructions?" and converts it into voice data by the voice synthesis means. Finally, the generated voice data is played back through the speakers of the smart glasses or HMD.

[0420] Prompt Sentence Examples

[0421] Analyze the following text to identify the user's sentiment: "I feel like I'm not getting anything done today. I'm frustrated."

[0422] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0423] Step 1:

[0424] Receiving and sending audio input

[0425] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. Specifically, the user utters, "I feel like I'm not making much progress on my work today. I'm in trouble." This captures the voice data. The captured voice data is sent to the server through the device's communication module. The input is the user's voice data, and the output is the voice data sent to the server.

[0426] Step 2:

[0427] Voice Recognition

[0428] The server analyzes the received voice data using a generative AI model such as Google Cloud Speech API. The voice data is converted into text data using speech recognition. Specifically, the generated text data is, "I feel like I'm not making much progress on my work today. I'm in trouble." The input is the voice data sent to the server, and the output is the generated text data.

[0429] Step 3:

[0430] Emotion analysis

[0431] The generated text data is passed to an emotion analysis engine. The server uses emotion analysis means to analyze the text data and identify the user's emotional state. Specifically, the emotion "confused" is identified from the text data "I feel like I can't get my work done well today. I'm in trouble." The input is the generated text data, and the output is the identified emotional state.

[0432] Step 4:

[0433] Feedback Selection

[0434] Based on the results of the emotion analysis, the server selects appropriate feedback. For example, if the emotion "confused" is identified, the server selects the feedback "Please stay calm. Do you need further instructions?" The input is the identified emotional state, and the output is the selected feedback message.

[0435] Step 5:

[0436] Speech synthesis

[0437] The selected feedback message is converted into audio data using a generative AI model such as the Google Cloud Text-to-Speech API. Specifically, the audio data generated is "Please stay calm and do your job. Do you need further instructions?" The input is the selected feedback message, and the output is the generated audio data.

[0438] Step 6:

[0439] Audio playback

[0440] The generated voice data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive appropriate feedback and instructions in real time. The input is the generated voice data, and the output is the played voice.

[0441] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0442] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0443] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0444] [Second embodiment]

[0445] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0446] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0447] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0448] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0449] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0450] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0451] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0452] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0453] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0454] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0455] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0456] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0457] The present invention is a system for supporting the learning of children with mild physical or intellectual disabilities, and its form is described in detail below. The system described below allows users (children) to smoothly communicate their intentions and receive necessary feedback and educational materials.

[0458] System Overview

[0459] This system captures the user's speech into the device and sends the speech data to a server. The server converts the speech data into text data and analyzes the user's intentions and learning needs. It then selects appropriate feedback and learning materials and sends them back to the device. The device then synthesizes the selected feedback and learning materials into speech and plays them back to the user.

[0460] Program processing details

[0461] Receiving and sending audio input

[0462] The user says to the terminal, "I want to study math today."

[0463] The device uses a microphone to capture the user's voice, and the captured voice data is stored in digital form.

[0464] The device generates and sends an API request to send the captured audio data to the server.

[0465] Speech Recognition and Analysis

[0466] The server receives the voice data sent from the device and inputs it into a voice recognition model (e.g., a model using generative AI technology).

[0467] The speech recognition model converts the speech data into text data, for example, in the form "I want to study math today."

[0468] The server analyzes the acquired text data and understands the user's intent using natural language processing (NLP) technology.

[0469] Based on the analysis results, the server selects appropriate feedback and learning materials according to the user's learning needs. For example, if a user requests to "study math," it will select addition problems appropriate to their level.

[0470] Sending feedback and text-to-speech

[0471] The server generates an API response to send the selected feedback and content of the teaching materials to the terminal as text data.

[0472] The device inputs the text data received from the server into a speech synthesis model (e.g., a model using generative AI technology).

[0473] The speech synthesis model converts text data into speech data, generating speech data such as "Let's try an addition problem today."

[0474] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0475] Specific examples

[0476] 1. Receiving voice input

[0477] A user says, "Today I want to study math."

[0478] The device captures the audio and stores it as audio data.

[0479] The terminal transmits the voice data to the server.

[0480] 2. Speech Recognition and Analysis

[0481] The server inputs the voice data into a voice recognition model and converts it into text data.

[0482] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[0483] The server selects appropriate teaching materials and generates feedback such as "Let's try some addition problems" as text data.

[0484] 3. Sending feedback and speech synthesis

[0485] The server sends the feedback to the device.

[0486] The device inputs the text data into a speech synthesis model and converts it into speech data.

[0487] The terminal plays back the generated voice data and provides feedback to the user.

[0488] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they progress through their studies. It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[0489] The processing flow will be explained below.

[0490] Step 1:

[0491] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[0492] Step 2:

[0493] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[0494] Step 3:

[0495] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[0496] Step 4:

[0497] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[0498] Step 5:

[0499] The server inputs the captured voice data into a speech recognition model, which uses generative AI to perform advanced speech recognition.

[0500] Step 6:

[0501] The speech recognition model converts the speech data into text data, for example, text data in the format "I want to study math today."

[0502] Step 7:

[0503] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intent and learning needs.

[0504] Step 8:

[0505] The server selects appropriate feedback and learning materials based on the analysis results. For example, if a user requests "I want to study math," it will select "addition problems."

[0506] Step 9:

[0507] The server generates text data based on the selected feedback and teaching materials, such as "Let's try some addition problems today."

[0508] Step 10:

[0509] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[0510] Step 11:

[0511] The device receives the API response from the server and obtains the feedback text data.

[0512] Step 12:

[0513] The device inputs the captured text data into a speech synthesis model, which uses generative AI to convert the text data into speech data.

[0514] Step 13:

[0515] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's try an addition problem today."

[0516] Step 14:

[0517] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0518] Example 1

[0519] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] With conventional learning support systems, children with mild physical or intellectual disabilities have difficulty expressing themselves smoothly, making it difficult for them to receive appropriate feedback and learning materials. Furthermore, the burden on parents and welfare workers is heavy, making it difficult to establish an effective support system.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0522] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means for converting the voice data into text data in the server, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to the terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This allows the user to smoothly communicate their intentions and progress in their studies while receiving appropriate feedback. It also reduces the burden on parents and welfare workers and enables the creation of an effective support system.

[0523] "User" refers to a child with a mild physical or intellectual disability who uses the system.

[0524] "Voice data" refers to digital data of a user's speech captured by a voice input device such as a microphone.

[0525] "Server" refers to a central processing unit that receives voice data, converts it into text data, and then analyzes the text data to identify the user's intent and select appropriate feedback and educational materials.

[0526] "Speech Recognition Method" refers to the technology or model used to convert voice data received by the server into text data, including, for example, generative AI technology.

[0527] "Text data" refers to character string data converted from voice data by a voice recognition means.

[0528] "Natural language processing technology" refers to technology for analyzing text data to understand its meaning and identify the user's intent.

[0529] "Feedback" refers to instructions, advice, or learnings provided based on user input.

[0530] "Instructional Materials" refers to educational content and materials provided to assist users in their learning.

[0531] "Speech synthesis method" refers to the technology or model used to convert text data into speech data, including, for example, generative AI technologies.

[0532] "Terminal" refers to a device that captures what a user says and communicates with a server. It may include a playback device.

[0533] The present invention is a system for supporting learning for children with mild physical or intellectual disabilities, and is described in detail below. This system allows users (children) to smoothly communicate their intentions and receive necessary feedback and learning materials.

[0534] System Overview

[0535] The system operates on the basis of voice data processing and communication between terminals, servers and users.

[0536] Hardware and Software Configuration

[0537] Device:

[0538] The built-in microphone is used to capture the user's voice.

[0539] It includes an API client to communicate with the server, for example using the HTTPS protocol.

[0540] Play audio using the built-in speaker or headphones.

[0541] Use a speech synthesis model (e.g., Google Text-to-Speech API or Amazon Polly, which use generative AI techniques).

[0542] server:

[0543] The voice data is input into a speech recognition model (e.g., Google Speech-to-Text API or IBM Watson Speech to Text, which utilize generative AI technology).

[0544] We use natural language processing (NLP) techniques (e.g., NLTK, spaCy) to analyze the text data obtained by speech recognition.

[0545] Select appropriate feedback and teaching materials from the database.

[0546] Text data of feedback and teaching materials is generated as an API response and sent to the device.

[0547] Capture and transmit audio data

[0548] When a user speaks into the device, such as "I want to study math today," the device's microphone captures the voice, which is then stored digitally and used to generate an API request and send it to the server.

[0549] Speech Recognition and Analysis

[0550] The server receives the voice data sent from the device and inputs it into a voice recognition model. The model converts the voice data into text data, for example, in the format "I want to study math today." The server analyzes this text data and uses natural language processing technology to understand the user's intention. Through this analysis, the server understands that the user's intention is "I want to study math."

[0551] Selecting and sending appropriate feedback and educational materials

[0552] The server selects appropriate feedback and learning materials from the database based on the user's wishes. For example, if the user says, "I want to study math," the server selects addition problems appropriate to the user's level. It then generates and sends an API response to send the text data of the selected feedback and learning materials to the device.

[0553] Speech synthesis and feedback playback

[0554] The device inputs the text data received from the server into a speech synthesis model and converts it into voice data. For example, voice data such as "Let's try an addition problem today" is generated. Finally, the device plays this voice data to the user through a playback device.

[0555] Specific examples

[0556] When a user says, "I want to study math today," the device captures the voice and sends the voice data to the server. The server inputs this voice data into a voice recognition model, converts it into text data, and then analyzes it using natural language processing technology to determine that the user's intention is "I want to study math." The server then selects an appropriate addition problem and sends the content of that problem to the device as text data. The device then inputs this text data into a voice synthesis model, generates voice data saying, "Let's try an addition problem today," and plays it back to the user.

[0557] Prompt Sentence Examples

[0558] "When a user says, 'I want to study math today,' select learning materials and feedback appropriate to that intent and generate them as text data."

[0559] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they learn. Furthermore, by utilizing a generative AI model, highly accurate speech recognition and speech synthesis are achieved.

[0560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0561] Step 1:

[0562] The user speaks to the terminal.

[0563] Input: User utterance (e.g., "I want to study math today.")

[0564] Output: The user's speech is input to the device's microphone as an analog audio signal.

[0565] Specific action: The user speaks into the built-in microphone.

[0566] Step 2:

[0567] The device captures the audio data and stores it digitally.

[0568] Input: Analog audio signal

[0569] Output: Digital audio data

[0570] Specific operation: The device's microphone driver converts the analog audio signal into digital audio data and temporarily stores it in internal memory.

[0571] Step 3:

[0572] The device generates and sends an API request to send the captured audio data to the server.

[0573] Input: Digital audio data

[0574] Output: API request to the server (including audio data)

[0575] Specific operation: The device sends audio data to the server using the HTTPS protocol.

[0576] Step 4:

[0577] The server receives the voice data transmitted from the terminal.

[0578] Input: API request (including audio data)

[0579] Output: Audio data stored on the server

[0580] Specific operation: The server's API endpoint receives the request, extracts the audio data, and saves it in the server's storage.

[0581] Step 5:

[0582] The server inputs the voice data into a voice recognition model and converts it into text data.

[0583] Input: Digital audio data

[0584] Output: Text data (e.g., "I want to study math today")

[0585] Specific operation: The server inputs voice data into a voice recognition model (which utilizes generative AI technology), and the model analyzes the voice data and converts it into text data.

[0586] Step 6:

[0587] The server analyzes the text data and identifies the user's intent.

[0588] Input: Text data

[0589] Output: User's intention (e.g., "I want to study math")

[0590] Specific operation: The server uses natural language processing (NLP) techniques to analyze the text data and identify the user's intent.

[0591] Step 7:

[0592] The server selects appropriate feedback and educational materials.

[0593] Input: User intent

[0594] Output: Feedback and text data for teaching materials (e.g., "Let's do the addition problem")

[0595] Specific operation: The server selects feedback and learning materials from the database according to the user's wishes.

[0596] Step 8:

[0597] The server generates and sends an API response to send the selected feedback and text data of the teaching materials to the terminal.

[0598] Input: Text data of feedback and teaching materials

[0599] Output: API response (including text data)

[0600] Specific operation: The server sends text data to the terminal using the HTTPS protocol.

[0601] Step 9:

[0602] The text data received by the terminal from the server is input into a speech synthesis model and converted into voice data.

[0603] Input: Text data

[0604] Output: Audio data (e.g., "Let's try some addition problems today.")

[0605] Specific operation: The device inputs text data into a speech synthesis model (using generative AI technology), which then converts the text data into speech data.

[0606] Step 10:

[0607] The terminal plays the generated audio data to the user using a playback device.

[0608] Input: Audio data

[0609] Output: The audio the user hears

[0610] What it does: The device's audio driver sends audio signals to speakers or headphones so the user can hear the sound.

[0611] (Application example 1)

[0612] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] When working in a factory, workers with mild physical or intellectual disabilities may find it difficult to receive appropriate work instructions and feedback. This reduces work efficiency and makes it difficult to ensure worker safety. Furthermore, workers with disabilities have difficulty communicating their intentions smoothly, which can lead to problems such as not receiving appropriate support.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0615] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means in the server for converting the voice data into text data, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, means for playing the voice data to the user, and means for capturing the voice of workers in a factory and providing appropriate feedback on work procedures and points to note. This enables workers to smoothly communicate their intentions and receive appropriate feedback on work procedures and points to note.

[0616] "Capture" means taking in data such as audio or images and saving it as digital data.

[0617] "Voice data" refers to data that is recorded using a microphone or other device and expressed in digital format.

[0618] A "server" is a computer system that receives and responds to requests from clients over a network.

[0619] "Text data" refers to data that represents character information in digital form.

[0620] "Speech recognition" is a technology that analyzes input voice data and converts it into corresponding text data.

[0621] "Analysis" is the process of examining data in detail to understand its content and trends.

[0622] "Feedback" refers to information or actions that provide a response or reaction to a particular action or request.

[0623] "Instructional Materials" means materials or content used for educational or training purposes.

[0624] "Speech synthesis" is a technology that converts text data into voice data.

[0625] "Playback" means outputting captured or generated audio or video data through an audio or video device.

[0626] "Factory" means the facility or workspace where manufacturing or production takes place.

[0627] "Worker" means a person who performs work in a factory.

[0628] A "work procedure" is the steps or process for accomplishing a particular task.

[0629] "Caution" refers to matters that require special attention in a particular situation or task.

[0630] System Overview

[0631] This invention relates to a system that supports workers with mild physical or intellectual disabilities who work in factories. This system captures the worker's voice input and sends the voice data to a server. The server converts the voice data into text data and analyzes the worker's intentions and work needs. It then selects appropriate work procedures and points to note and sends them back to the terminal. The terminal also supports the worker by synthesizing the selected feedback into voice and playing it back to the worker.

[0632] Hardware and software used

[0633] 1. Hardware:

[0634] Microphone: Used to capture the voice of the worker.

[0635] Terminal (e.g., tablet, smartphone): A device that allows workers to input voice and receive feedback.

[0636] Speaker: Used to play feedback audio.

[0637] 2. Software:

[0638] Python: The programming language used to implement the entire program.

[0639] speech_recognition library: A library for capturing and recognizing speech.

[0640] OpenAI GPT-3 API: A generative AI model that analyzes voice data and generates appropriate feedback.

[0641] gTTS library: A library for synthesizing text data into voice data.

[0642] playsound library: A library for playing generated sound data.

[0643] System processing details

[0644] 1. Receiving audio input:

[0645] The worker talks to the terminal about the work he is doing and the problems he is having.

[0646] The device uses a microphone to capture audio data and store it in digital form.

[0647] The captured audio data is sent to a server.

[0648] 2. Speech Recognition and Analysis:

[0649] The server receives the transmitted voice data and converts it into text data using the speech_recognition library.

[0650] The converted text data is then analyzed using the OpenAI GPT-3 API.

[0651] Based on the analysis results, the server understands the worker's intentions and work needs, and generates appropriate work procedures and points to note.

[0652] 3. Send feedback and speech synthesis:

[0653] The server transmits the generated feedback text data to the terminal.

[0654] The device uses the gTTS library to convert the feedback text data into voice data.

[0655] The converted voice data is played back to the worker through a speaker.

[0656] Specific examples

[0657] Example 1:

[0658] The worker speaks into the terminal, "What should I do next?"

[0659] A microphone captures the audio and sends it to a server.

[0660] The server converts the voice data into text data and analyzes it using the OpenAI GPT-3 API.

[0661] Feedback is generated such as "Next, you need to assemble part A."

[0662] The terminal synthesizes the generated feedback into voice and plays it back to the worker.

[0663] Prompt Sentence Examples

[0664] User: What should I do next?

[0665] A basic import, a mutually exclusive reaction or a step-by-step process.

[0666] Using this system, workers can smoothly communicate their intentions and receive feedback on appropriate work procedures and points to note. It is also expected to contribute to improving work efficiency and ensuring safety.

[0667] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0668] Step 1:

[0669] The user speaks instructions or questions into the device. The device's microphone captures this voice and stores it as digital audio data. The input is the user's voice, and the output is digital audio data. Specifically, the user speaks, "What should I do next?", and the device records the voice and converts it into digital format.

[0670] Step 2:

[0671] The device sends the captured audio data to the server. The input is digital audio data, and the output is the audio data sent to the server. Specifically, the device generates an HTTP request over the network and sends the audio data to the server.

[0672] Step 3:

[0673] The voice data received by the server is input into a voice recognition model (e.g., speech_recognition library) and converted into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "What should I do next?"

[0674] Step 4:

[0675] The server analyzes the text data and uses a generative AI model (e.g., OpenAI GPT-3) to understand the user's intentions. The input is text data, and the output is feedback text based on the analysis results. Specifically, the server inputs the text data into the generative AI model and creates feedback text such as, "Next, you need to assemble part A."

[0676] Step 5:

[0677] The server sends the generated feedback text to the terminal. The input is the feedback text data, and the output is the feedback text sent to the terminal. Specifically, the server generates an HTTP response and sends the feedback text data to the terminal.

[0678] Step 6:

[0679] The feedback text received by the device is input into a speech synthesis model (e.g., gTTS library) and converted into voice data. The input is the feedback text and the output is voice data. Specifically, the device inputs the text data into the gTTS library and generates voice data saying, "Next, you need to assemble part A."

[0680] Step 7:

[0681] The device plays the generated audio data through a speaker. The input is audio data, and the output is the actual audio heard by the worker. Specifically, the device plays the audio data through the playsound library, and the worker receives feedback via audio.

[0682] Through these steps, workers can smoothly communicate their intentions and receive audio feedback on appropriate work procedures and points to note.

[0683] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0684] The present invention relates to a system for supporting the learning of children with mild physical or intellectual disabilities, and in particular to a system that provides more appropriate feedback and educational materials based on the user's emotional state by combining an emotion engine.

[0685] System Overview

[0686] This system captures the user's voice as they speak into their device and sends the voice data to a server. The server converts the voice data into text data and analyzes the user's intentions and learning needs. It also uses an emotion engine to analyze the user's emotions and selects appropriate feedback and learning materials based on the analysis results. The selected feedback and learning materials are sent to the device and played back to the user using voice synthesis.

[0687] Program processing details

[0688] Receiving and sending audio input

[0689] The user says to the terminal, "I want to study math today."

[0690] The device captures the user's voice using a microphone and temporarily stores it in digital form.

[0691] The device generates and sends an API request to send the captured audio data to the server.

[0692] Speech Recognition and Analysis

[0693] The server receives the voice data sent from the device and inputs it into a voice recognition model, which uses generative AI to convert the voice data into text data.

[0694] For example, text data such as "I want to study math today" is generated.

[0695] The server analyzes the text data to understand the user's intent, using natural language processing (NLP) techniques.

[0696] Emotion analysis

[0697] The server inputs the voice or text data into an emotion engine to analyze the user's emotions. Generative AI technology is used to perform emotion analysis and identify the user's emotional state, such as whether they are happy, sad, or distressed.

[0698] The feedback and educational content is adjusted based on the results of sentiment analysis. For example, if the analysis shows that the user is in trouble, the feedback will be gentle and responsive.

[0699] Selecting and submitting feedback

[0700] The server selects appropriate feedback and learning materials based on the results of the analysis and sentiment analysis. For example, if the server determines that the user is having trouble with a request such as "I want to study math," it generates gentle feedback such as "Let's start with the basics."

[0701] The server sends the generated feedback and text data of the teaching materials to the terminal.

[0702] Speech synthesis and playback

[0703] The device inputs the text data received from the server into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[0704] For example, voice data saying "Let's start with the basics first" is generated.

[0705] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0706] Specific examples

[0707] 1. Receiving voice input

[0708] A user says, "Today I want to study math."

[0709] The device captures the audio and stores it as audio data.

[0710] The terminal transmits the voice data to the server.

[0711] 2. Speech Recognition and Analysis

[0712] The server inputs the voice data into a voice recognition model and converts it into text data.

[0713] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[0714] 3. Emotion analysis

[0715] The server uses an emotion engine to analyze the user's emotion from the voice data or text data.

[0716] Based on the analysis results, it is recognized that, for example, "the user is feeling anxious."

[0717] 4. Selecting and Submitting Feedback

[0718] Based on the results of the analysis and emotion analysis, the server generates gentle feedback such as, "Let's try an addition problem."

[0719] The generated feedback is sent to the terminal as text data.

[0720] 5. Speech Synthesis and Playback

[0721] The text data received by the terminal is input into a speech synthesis model and converted into voice data.

[0722] The device plays the generated audio and tells the user, "Let's start with the basics."

[0723] This system allows users to smoothly communicate their intentions, receive appropriate feedback, and progress in learning that is tailored to their emotional state.It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[0724] The processing flow will be explained below.

[0725] Step 1:

[0726] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[0727] Step 2:

[0728] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[0729] Step 3:

[0730] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[0731] Step 4:

[0732] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[0733] Step 5:

[0734] The server inputs the acquired voice data into a voice recognition model, which uses generative AI to perform advanced voice recognition.

[0735] Step 6:

[0736] The speech recognition model converts the speech data into text data, for example, "I want to study math today."

[0737] Step 7:

[0738] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning goals.

[0739] Step 8:

[0740] The server inputs the voice data into an emotion engine to analyze the user's emotions. Generative AI technology is used to identify emotional states (such as joy, sadness, or anxiety) from the voice.

[0741] Step 9:

[0742] The server selects feedback and teaching materials based on the emotional data analyzed by the emotion engine. For example, if the server determines that the user is feeling anxious, it will select gentle instructional content.

[0743] Step 10:

[0744] The server generates text data based on the selected feedback and learning materials, such as "Let's start with the basics."

[0745] Step 11:

[0746] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[0747] Step 12:

[0748] The device receives the API response from the server and obtains the feedback text data.

[0749] Step 13:

[0750] The device inputs the acquired text data into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[0751] Step 14:

[0752] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's start with the basics."

[0753] Step 15:

[0754] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0755] Example 2

[0756] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] This invention relates to a system for supporting the learning of children with mild disabilities. Conventional learning support systems often provide one-sided feedback and learning materials without considering the user's emotional state, limiting the effectiveness of learning. Furthermore, there are issues with the difficulty of selecting appropriate feedback and learning materials, making it difficult to maintain the user's motivation to learn. The objective of this invention is to solve these issues and provide more effective learning support.

[0758] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0759] In this invention,

[0760] The server includes means for capturing voice data input by a user, means for transmitting the voice data to a relay device, voice recognition means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for analyzing the voice data or the text data to identify the user's emotional state, means for selecting appropriate feedback and learning materials based on the user's intention and emotional state, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This makes it possible to provide appropriate feedback and learning materials that take the user's emotional state into consideration, thereby improving learning effectiveness and maintaining learning motivation.

[0761] "Mild disability" refers to a minor limitation in physical or intellectual function.

[0762] "Supporting learning" refers to helping users acquire knowledge and skills efficiently.

[0763] "User" refers to individuals who use the system, particularly children with mild disabilities.

[0764] "Voice data" refers to data that has been saved in digital form as the voice of a user speaking into a terminal.

[0765] "Capturing" refers to using a microphone to collect sound and convert it into a digital format.

[0766] "Relay device" refers to a device or software for transmitting data from a terminal to a server.

[0767] "Speech recognition" refers to the technology of converting voice data into text data.

[0768] "Text data" refers to text information converted from voice data using voice recognition technology.

[0769] "Analysis" refers to evaluating text data to understand the user's intentions and emotions.

[0770] "Emotional state" refers to the emotion the user is feeling at that time.

[0771] "Feedback" refers to a response or information to a user's input or state.

[0772] "Instructional Materials" refers to educational resources and materials intended to aid users in their learning.

[0773] "Speech synthesis" refers to the technology of converting text data into voice data.

[0774] "Playback" refers to listening to audio data through speakers or headphones.

[0775] "Generative AI" refers to models and techniques that use artificial intelligence techniques to generate or transform data.

[0776] This invention is a system for supporting the learning of children with mild disabilities. In particular, it improves learning effectiveness by analyzing the user's emotional state and providing appropriate feedback and learning materials based on that. The system has the function of capturing voice data input by the user and transmitting the voice data to a server.

[0777] The device captures the user's speech using a microphone and stores it in digital format. The stored speech data is then sent to a server via a relay device. The server uses speech recognition technology to convert the speech data into text data. For speech recognition, generative AI technology (e.g., a general generative AI speech recognition model) is used. For example, if a user says, "I want to study math today," the server converts this speech into text and generates the text data, "I want to study math today."

[0778] The server then analyzes this text data to identify the user's intent. This analysis uses natural language processing (NLP) techniques (e.g., widely used natural language processing models). After the user's intent is identified, this text data or the original voice data is input into a sentiment analysis engine. Sentiment analysis uses generative AI techniques (e.g., common sentiment analysis models) to identify the user's emotional state. For example, if the analysis finds that the user is in trouble, the system generates appropriate feedback based on this.

[0779] The server selects appropriate feedback and learning materials based on the user's intentions and emotional state. For example, if a user says, "I want to study math," and is also analyzed as "having trouble," the server generates gentle feedback such as, "Let's start with the basics." The generated feedback and learning materials are then sent to the device as text data.

[0780] The device converts the received text data into voice data using speech synthesis technology (for example, a common speech synthesis model). The converted voice data is played back to the user through speakers or headphones. For example, feedback such as "Let's start with the basics" may be played back as voice.

[0781] This system allows users to smoothly communicate their intentions and receive appropriate feedback and learning materials in response to their requests, while also taking their emotional state into consideration when studying. This improves learning effectiveness and maintains motivation to learn.

[0782] Specific examples

[0783] When a user says, "I want to study math today," the device captures this voice and saves it digitally. The saved voice data is sent to a server. The server converts this voice data into text data using generative AI technology, generating the text data "I want to study math today." Natural language processing technology is then used on this text data to identify the user's intention as "I want to study math."

[0784] The server then inputs this voice or text data into an emotion analysis engine to analyze the user's emotional state. If generative AI technology determines that the user is having trouble, it generates gentle feedback such as, "Let's start with the basics." The text data of this feedback is sent to the device, which then converts it into voice data and plays it back to the user.

[0785] Example prompt sentence:

[0786] I want to study math today.

[0787] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0788] Step 1:

[0789] The device captures the user's speech with a microphone and stores it in digital form. The input is the user's speech, and the device uses the microphone to convert analog speech into digital audio data. The output is the captured digital audio data.

[0790] Step 2:

[0791] The device generates an API request to send the stored digital audio data to the server, and sends the audio data to the server. The input is the digital audio data generated in step 1, and the output is the audio data sent to the server. Specifically, the device sends the audio data over the network.

[0792] Step 3:

[0793] The server receives the voice data sent from the device and inputs it into a voice recognition model. The input is the voice data sent from the device, and the voice data is converted into text data using a voice recognition model (for example, a generative AI voice recognition model). The output is the converted text data. For example, the text data generated is "I want to study math today."

[0794] Step 4:

[0795] The server analyzes the generated text data and identifies the user's intention. The input is the text data generated in step 3, and the server analyzes the data using natural language processing (NLP) techniques (e.g., a widely used NLP model). The output is data indicating the user's intention. For example, the intention "I want to study math" is identified.

[0796] Step 5:

[0797] The server inputs the voice or text data into the emotion engine and analyzes the user's emotional state. The input is the data generated in step 3 or step 4, and generative AI technology is used to identify the user's emotion. The output is data that indicates the user's emotional state. For example, the analysis may determine that the user is in trouble.

[0798] Step 6:

[0799] The server selects appropriate feedback and learning materials based on the user's intentions and the results of emotion analysis. The input is the data generated in steps 4 and 5, and an algorithm is used to select appropriate feedback. The output is the text data of the selected feedback and learning materials. For example, the generated feedback is "Let's start with the basics."

[0800] Step 7:

[0801] The server sends the selected feedback and text data of the teaching materials to the terminal. The input is the text data generated in step 6, and the output is the feedback text data sent to the terminal. Specifically, the server sends the text data over the network.

[0802] Step 8:

[0803] The device inputs the received text data into a speech synthesis model and converts it into speech data. The input is the text data received in step 7, and the generative AI speech synthesis model is used to convert the text data into speech data. The output is the converted speech data. For example, speech data such as "Let's start with the basics" is generated.

[0804] Step 9:

[0805] The device plays the generated audio data to the user through speakers or headphones. The input is the audio data generated in step 8, and the specific operation is to deliver the audio to the user using a playback device. The output is the audio the user hears. For example, the audio played to the user is, "Let's start with the basics."

[0806] (Application example 2)

[0807] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0808] Conventional factory robots are unable to provide feedback or instructions based on the worker's emotions or situation, making it difficult to reduce worker stress or support efficient work. Furthermore, when a worker feels confused or anxious, delays in appropriate responses can lead to reduced work efficiency and an increased risk of mistakes. Therefore, there is a need for a system that can provide feedback and instructions based on the worker's emotional state, improving work efficiency and reducing worker stress.

[0809] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion analysis means for analyzing voice data or text data to identify the user's emotional state, means for adjusting feedback and teaching materials based on the emotional state, and means for capturing voice data. This allows appropriate feedback and instructions to be provided according to the worker's emotional state, improving work efficiency and reducing worker stress.

[0810] "Audio data" refers to data that represents audio in digital form.

[0811] A "server" is a high-performance computer that processes and manages data over a network.

[0812] "Speech recognition means" refers to technology that analyzes voice data and converts it into text data.

[0813] "Text data" refers to data that represents character information in a digital format.

[0814] "Emotion analysis means" refers to a technique for analyzing voice data or text data to identify the user's emotional state.

[0815] "Means for selecting feedback and educational materials" refers to techniques for selecting appropriate information and educational materials based on the user's intentions and emotional state.

[0816] "Speech synthesis means" refers to technology that converts text data into voice data.

[0817] "Playback means" refers to technology that allows users to listen to audio data through devices such as speakers or headphones.

[0818] "Means for analyzing data" refers to the technology for analyzing input data and extracting necessary information.

[0819] "Emotional state" refers to the user's psychological emotional state, and includes, for example, happiness, sadness, confusion, anxiety, and the like.

[0820] System configuration

[0821] The system of the present invention includes a voice input means, a means for transmitting voice data to a server, a voice recognition means, a sentiment analysis means, a means for selecting feedback and teaching materials, a voice synthesis means, and a playback means. Each of these means will be described in detail below.

[0822] Voice input means

[0823] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. The microphone in the smart glasses or HMD is used to capture the voice data. This voice data is then sent to the server in the next step.

[0824] A means of sending audio data to the server

[0825] The captured audio data is sent to the server via the device's communication module, using HTTP or HTTPS as the communication protocol.

[0826] Voice recognition means

[0827] The server uses a generative AI model, such as the Google Cloud Speech API, to convert the audio data into text. This is the step where speech recognition takes place and converts the user's input speech into text form.

[0828] Emotion analysis means

[0829] The generated text data is passed to a sentiment analysis engine, which analyzes the voice or text data to identify the user's emotions. Generative AI technology is used to identify the user's emotional state, such as whether they are happy, sad, or troubled.

[0830] Feedback and materials selection

[0831] Based on the results of the sentiment analysis, the server selects appropriate feedback and educational materials. For example, if the analysis indicates that the user is confused, the server selects the message "Please stay calm and continue working. Do you need further instructions?" as the corresponding feedback.

[0832] Voice synthesis means

[0833] The selected feedback and learning materials are then converted into audio data using generative AI models such as the Google Cloud Text-to-Speech API, which converts the text data into audio data and generates audio to provide appropriate instructions to the user.

[0834] Regeneration means

[0835] The generated audio data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive feedback and instructions in real time.

[0836] Specific examples

[0837] For example, if a user says, "I feel like I'm not making much progress on my work today. I'm having trouble," the voice input means captures this speech. The voice data is sent to the server and converted into text data by the voice recognition means. The generated text is passed to the "emotion analysis engine," which identifies the emotion "confused." In response, the server selects feedback such as "Please stay calm and continue working. Do you need further instructions?" and converts it into voice data by the voice synthesis means. Finally, the generated voice data is played back through the speakers of the smart glasses or HMD.

[0838] Prompt Sentence Examples

[0839] Analyze the following text to identify the user's sentiment: "I feel like I'm not getting anything done today. I'm frustrated."

[0840] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0841] Step 1:

[0842] Receiving and sending audio input

[0843] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. Specifically, the user utters, "I feel like I'm not making much progress on my work today. I'm in trouble." This captures the voice data. The captured voice data is sent to the server through the device's communication module. The input is the user's voice data, and the output is the voice data sent to the server.

[0844] Step 2:

[0845] Voice Recognition

[0846] The server analyzes the received voice data using a generative AI model such as Google Cloud Speech API. The voice data is converted into text data using speech recognition. Specifically, the generated text data is, "I feel like I'm not making much progress on my work today. I'm in trouble." The input is the voice data sent to the server, and the output is the generated text data.

[0847] Step 3:

[0848] Emotion analysis

[0849] The generated text data is passed to an emotion analysis engine. The server uses emotion analysis means to analyze the text data and identify the user's emotional state. Specifically, the emotion "confused" is identified from the text data "I feel like I can't get my work done well today. I'm in trouble." The input is the generated text data, and the output is the identified emotional state.

[0850] Step 4:

[0851] Feedback Selection

[0852] Based on the results of the emotion analysis, the server selects appropriate feedback. For example, if the emotion "confused" is identified, the server selects the feedback "Please stay calm. Do you need further instructions?" The input is the identified emotional state, and the output is the selected feedback message.

[0853] Step 5:

[0854] Speech synthesis

[0855] The selected feedback message is converted into audio data using a generative AI model such as the Google Cloud Text-to-Speech API. Specifically, the audio data generated is "Please stay calm and do your job. Do you need further instructions?" The input is the selected feedback message, and the output is the generated audio data.

[0856] Step 6:

[0857] Audio playback

[0858] The generated voice data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive appropriate feedback and instructions in real time. The input is the generated voice data, and the output is the played voice.

[0859] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0860] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0861] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0862] [Third embodiment]

[0863] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0864] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0865] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0866] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0867] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0868] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0869] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0870] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0871] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0872] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0873] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0874] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0875] The present invention is a system for supporting the learning of children with mild physical or intellectual disabilities, and its form is described in detail below. The system described below allows users (children) to smoothly communicate their intentions and receive necessary feedback and educational materials.

[0876] System Overview

[0877] This system captures the user's speech into the device and sends the speech data to a server. The server converts the speech data into text data and analyzes the user's intentions and learning needs. It then selects appropriate feedback and learning materials and sends them back to the device. The device then synthesizes the selected feedback and learning materials into speech and plays them back to the user.

[0878] Program processing details

[0879] Receiving and sending audio input

[0880] The user says to the terminal, "I want to study math today."

[0881] The device uses a microphone to capture the user's voice, and the captured voice data is stored in digital form.

[0882] The device generates and sends an API request to send the captured audio data to the server.

[0883] Speech Recognition and Analysis

[0884] The server receives the voice data sent from the device and inputs it into a voice recognition model (e.g., a model using generative AI technology).

[0885] The speech recognition model converts the speech data into text data, for example, in the form "I want to study math today."

[0886] The server analyzes the acquired text data and understands the user's intent using natural language processing (NLP) technology.

[0887] Based on the analysis results, the server selects appropriate feedback and learning materials according to the user's learning needs. For example, if a user requests to "study math," it will select addition problems appropriate to their level.

[0888] Sending feedback and text-to-speech

[0889] The server generates an API response to send the selected feedback and content of the teaching materials to the terminal as text data.

[0890] The device inputs the text data received from the server into a speech synthesis model (e.g., a model using generative AI technology).

[0891] The speech synthesis model converts text data into speech data and generates speech data such as, "Let's try an addition problem today."

[0892] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0893] Specific examples

[0894] 1. Receiving voice input

[0895] A user says, "Today I want to study math."

[0896] The device captures the audio and stores it as audio data.

[0897] The terminal transmits the voice data to the server.

[0898] 2. Speech Recognition and Analysis

[0899] The server inputs the voice data into a voice recognition model and converts it into text data.

[0900] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[0901] The server selects appropriate teaching materials and generates feedback such as "Let's try some addition problems" as text data.

[0902] 3. Sending feedback and speech synthesis

[0903] The server sends the feedback to the device.

[0904] The device inputs the text data into a speech synthesis model and converts it into speech data.

[0905] The terminal plays back the generated voice data and provides feedback to the user.

[0906] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they progress through their studies. It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[0910] Step 2:

[0911] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[0912] Step 3:

[0913] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[0914] Step 4:

[0915] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[0916] Step 5:

[0917] The server inputs the captured voice data into a speech recognition model, which uses generative AI to perform advanced speech recognition.

[0918] Step 6:

[0919] The speech recognition model converts the speech data into text data, for example, text data in the format "I want to study math today."

[0920] Step 7:

[0921] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intent and learning needs.

[0922] Step 8:

[0923] The server selects appropriate feedback and learning materials based on the analysis results. For example, if a user requests "I want to study math," it will select "addition problems."

[0924] Step 9:

[0925] The server generates text data based on the selected feedback and teaching materials, such as "Let's try some addition problems today."

[0926] Step 10:

[0927] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[0928] Step 11:

[0929] The device receives the API response from the server and obtains the feedback text data.

[0930] Step 12:

[0931] The device inputs the captured text data into a speech synthesis model, which uses generative AI to convert the text data into speech data.

[0932] Step 13:

[0933] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's try an addition problem today."

[0934] Step 14:

[0935] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[0936] Example 1

[0937] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0938] With conventional learning support systems, children with mild physical or intellectual disabilities have difficulty expressing themselves smoothly, making it difficult for them to receive appropriate feedback and learning materials. Furthermore, the burden on parents and welfare workers is heavy, making it difficult to establish an effective support system.

[0939] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0940] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means for converting the voice data into text data in the server, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to the terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This allows the user to smoothly communicate their intentions and progress in their studies while receiving appropriate feedback. It also reduces the burden on parents and welfare workers and enables the creation of an effective support system.

[0941] "User" refers to a child with a mild physical or intellectual disability who uses the system.

[0942] "Voice data" refers to digital data of a user's speech captured by a voice input device such as a microphone.

[0943] "Server" refers to a central processing unit that receives voice data, converts it into text data, and then analyzes the text data to identify the user's intent and select appropriate feedback and educational materials.

[0944] "Speech Recognition Method" refers to the technology or model used to convert voice data received by the server into text data, including, for example, generative AI technology.

[0945] "Text data" refers to character string data converted from voice data by a voice recognition means.

[0946] "Natural language processing technology" refers to technology for analyzing text data to understand its meaning and identify the user's intent.

[0947] "Feedback" refers to instructions, advice, or learnings provided based on user input.

[0948] "Instructional Materials" refers to educational content and materials provided to assist users in their learning.

[0949] "Speech synthesis method" refers to the technology or model used to convert text data into speech data, including, for example, generative AI technologies.

[0950] "Terminal" refers to a device that captures what a user says and communicates with a server. It may include a playback device.

[0951] The present invention is a system for supporting learning for children with mild physical or intellectual disabilities, and is described in detail below. This system allows users (children) to smoothly communicate their intentions and receive necessary feedback and learning materials.

[0952] System Overview

[0953] The system operates on the basis of voice data processing and communication between terminals, servers and users.

[0954] Hardware and Software Configuration

[0955] Device:

[0956] The built-in microphone is used to capture the user's voice.

[0957] It includes an API client to communicate with the server, for example using the HTTPS protocol.

[0958] Play audio using the built-in speaker or headphones.

[0959] Use a speech synthesis model (e.g., Google Text-to-Speech API or Amazon Polly, which use generative AI techniques).

[0960] server:

[0961] The voice data is input into a speech recognition model (e.g., Google Speech-to-Text API or IBM Watson Speech to Text, which utilize generative AI technology).

[0962] We use natural language processing (NLP) techniques (e.g., NLTK, spaCy) to analyze the text data obtained by speech recognition.

[0963] Select appropriate feedback and teaching materials from the database.

[0964] Text data of feedback and teaching materials is generated as an API response and sent to the device.

[0965] Capture and transmit audio data

[0966] When a user speaks into the device, such as "I want to study math today," the device's microphone captures the voice, which is then stored digitally and used to generate an API request and send it to the server.

[0967] Speech Recognition and Analysis

[0968] The server receives the voice data sent from the device and inputs it into a voice recognition model. The model converts the voice data into text data, for example, in the format "I want to study math today." The server analyzes this text data and uses natural language processing technology to understand the user's intention. Through this analysis, the server understands that the user's intention is "I want to study math."

[0969] Selecting and sending appropriate feedback and educational materials

[0970] The server selects appropriate feedback and learning materials from the database based on the user's wishes. For example, if the user says, "I want to study math," the server selects addition problems appropriate to the user's level. It then generates and sends an API response to send the text data of the selected feedback and learning materials to the device.

[0971] Speech synthesis and feedback playback

[0972] The device inputs the text data received from the server into a speech synthesis model and converts it into voice data. For example, voice data such as "Let's try an addition problem today" is generated. Finally, the device plays this voice data to the user through a playback device.

[0973] Specific examples

[0974] When a user says, "I want to study math today," the device captures the voice and sends the voice data to the server. The server inputs this voice data into a voice recognition model, converts it into text data, and then analyzes it using natural language processing technology to determine that the user's intention is "I want to study math." The server then selects an appropriate addition problem and sends the content of that problem to the device as text data. The device then inputs this text data into a voice synthesis model, generates voice data saying, "Let's try an addition problem today," and plays it back to the user.

[0975] Prompt Sentence Examples

[0976] "When a user says, 'I want to study math today,' select learning materials and feedback appropriate to that intent and generate them as text data."

[0977] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they learn. Furthermore, by utilizing a generative AI model, highly accurate speech recognition and speech synthesis are achieved.

[0978] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0979] Step 1:

[0980] The user speaks to the terminal.

[0981] Input: User utterance (e.g., "I want to study math today.")

[0982] Output: The user's speech is input to the device's microphone as an analog audio signal.

[0983] Specific action: The user speaks into the built-in microphone.

[0984] Step 2:

[0985] The device captures the audio data and stores it digitally.

[0986] Input: Analog audio signal

[0987] Output: Digital audio data

[0988] Specific operation: The device's microphone driver converts the analog audio signal into digital audio data and temporarily stores it in internal memory.

[0989] Step 3:

[0990] The device generates and sends an API request to send the captured audio data to the server.

[0991] Input: Digital audio data

[0992] Output: API request to the server (including audio data)

[0993] Specific operation: The device sends audio data to the server using the HTTPS protocol.

[0994] Step 4:

[0995] The server receives the voice data transmitted from the terminal.

[0996] Input: API request (including audio data)

[0997] Output: Audio data stored on the server

[0998] Specific operation: The server's API endpoint receives the request, extracts the audio data, and saves it in the server's storage.

[0999] Step 5:

[1000] The server inputs the voice data into a voice recognition model and converts it into text data.

[1001] Input: Digital audio data

[1002] Output: Text data (e.g., "I want to study math today")

[1003] Specific operation: The server inputs voice data into a voice recognition model (which utilizes generative AI technology), and the model analyzes the voice data and converts it into text data.

[1004] Step 6:

[1005] The server analyzes the text data and identifies the user's intent.

[1006] Input: Text data

[1007] Output: User's intention (e.g., "I want to study math")

[1008] Specific operation: The server uses natural language processing (NLP) techniques to analyze the text data and identify the user's intent.

[1009] Step 7:

[1010] The server selects appropriate feedback and educational materials.

[1011] Input: User intent

[1012] Output: Feedback and text data for teaching materials (e.g., "Let's do the addition problem")

[1013] Specific operation: The server selects feedback and learning materials from the database according to the user's wishes.

[1014] Step 8:

[1015] The server generates and sends an API response to send the selected feedback and text data of the teaching materials to the terminal.

[1016] Input: Text data of feedback and teaching materials

[1017] Output: API response (including text data)

[1018] Specific operation: The server sends text data to the terminal using the HTTPS protocol.

[1019] Step 9:

[1020] The text data received by the terminal from the server is input into a speech synthesis model and converted into voice data.

[1021] Input: Text data

[1022] Output: Audio data (e.g., "Let's try some addition problems today.")

[1023] Specific operation: The device inputs text data into a speech synthesis model (using generative AI technology), which then converts the text data into speech data.

[1024] Step 10:

[1025] The terminal plays the generated audio data to the user using a playback device.

[1026] Input: Audio data

[1027] Output: The audio the user hears

[1028] What it does: The device's audio driver sends audio signals to speakers or headphones so the user can hear the sound.

[1029] (Application example 1)

[1030] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1031] When working in a factory, workers with mild physical or intellectual disabilities may find it difficult to receive appropriate work instructions and feedback. This reduces work efficiency and makes it difficult to ensure worker safety. Furthermore, workers with disabilities have difficulty communicating their intentions smoothly, which can lead to problems such as not receiving appropriate support.

[1032] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1033] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means in the server for converting the voice data into text data, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, means for playing the voice data to the user, and means for capturing the voice of workers in a factory and providing appropriate feedback on work procedures and points to note. This enables workers to smoothly communicate their intentions and receive appropriate feedback on work procedures and points to note.

[1034] "Capture" means taking in data such as audio or images and saving it as digital data.

[1035] "Voice data" refers to data that is recorded using a microphone or other device and expressed in digital format.

[1036] A "server" is a computer system that receives and responds to requests from clients over a network.

[1037] "Text data" refers to data that represents character information in digital form.

[1038] "Speech recognition" is a technology that analyzes input voice data and converts it into corresponding text data.

[1039] "Analysis" is the process of examining data in detail to understand its content and trends.

[1040] "Feedback" refers to information or actions that provide a response or reaction to a particular action or request.

[1041] "Instructional Materials" means materials or content used for educational or training purposes.

[1042] "Speech synthesis" is a technology that converts text data into voice data.

[1043] "Playback" means outputting captured or generated audio or video data through an audio or video device.

[1044] "Factory" means the facility or workspace where manufacturing or production takes place.

[1045] "Worker" means a person who performs work in a factory.

[1046] A "work procedure" is the steps or process for accomplishing a particular task.

[1047] "Caution" refers to matters that require special attention in a particular situation or task.

[1048] System Overview

[1049] This invention relates to a system that supports workers with mild physical or intellectual disabilities who work in factories. This system captures the worker's voice input and sends the voice data to a server. The server converts the voice data into text data and analyzes the worker's intentions and work needs. It then selects appropriate work procedures and points to note and sends them back to the terminal. The terminal also supports the worker by synthesizing the selected feedback into voice and playing it back to the worker.

[1050] Hardware and software used

[1051] 1. Hardware:

[1052] Microphone: Used to capture the voice of the worker.

[1053] Terminal (e.g., tablet, smartphone): A device that allows workers to input voice and receive feedback.

[1054] Speaker: Used to play feedback audio.

[1055] 2. Software:

[1056] Python: The programming language used to implement the entire program.

[1057] speech_recognition library: A library for capturing and recognizing speech.

[1058] OpenAI GPT-3 API: A generative AI model that analyzes voice data and generates appropriate feedback.

[1059] gTTS library: A library for synthesizing text data into voice data.

[1060] playsound library: A library for playing generated sound data.

[1061] System processing details

[1062] 1. Receiving audio input:

[1063] The worker talks to the terminal about the work he is doing and the problems he is having.

[1064] The device uses a microphone to capture audio data and store it in digital form.

[1065] The captured audio data is sent to a server.

[1066] 2. Speech Recognition and Analysis:

[1067] The server receives the transmitted voice data and converts it into text data using the speech_recognition library.

[1068] The converted text data is then analyzed using the OpenAI GPT-3 API.

[1069] Based on the analysis results, the server understands the worker's intentions and work needs, and generates appropriate work procedures and points to note.

[1070] 3. Send feedback and speech synthesis:

[1071] The server transmits the generated feedback text data to the terminal.

[1072] The device uses the gTTS library to convert the feedback text data into voice data.

[1073] The converted voice data is played back to the worker through a speaker.

[1074] Specific examples

[1075] Example 1:

[1076] The worker speaks into the terminal, "What should I do next?"

[1077] A microphone captures the audio and sends it to a server.

[1078] The server converts the voice data into text data and analyzes it using the OpenAI GPT-3 API.

[1079] Feedback is generated such as "Next, you need to assemble part A."

[1080] The terminal synthesizes the generated feedback into voice and plays it back to the worker.

[1081] Prompt Sentence Examples

[1082] User: What should I do next?

[1083] A basic import, a mutually exclusive reaction or a step in the process.

[1084] Using this system, workers can smoothly communicate their intentions and receive feedback on appropriate work procedures and points to note. It is also expected to contribute to improving work efficiency and ensuring safety.

[1085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1086] Step 1:

[1087] The user speaks instructions or questions into the device. The device's microphone captures this voice and stores it as digital audio data. The input is the user's voice, and the output is digital audio data. Specifically, the user speaks, "What should I do next?", and the device records the voice and converts it into digital format.

[1088] Step 2:

[1089] The device sends the captured audio data to the server. The input is digital audio data, and the output is the audio data sent to the server. Specifically, the device generates an HTTP request over the network and sends the audio data to the server.

[1090] Step 3:

[1091] The voice data received by the server is input into a voice recognition model (e.g., speech_recognition library) and converted into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "What should I do next?"

[1092] Step 4:

[1093] The server analyzes the text data and uses a generative AI model (e.g., OpenAI GPT-3) to understand the user's intentions. The input is text data, and the output is feedback text based on the analysis results. Specifically, the server inputs the text data into the generative AI model and creates feedback text such as, "Next, you need to assemble part A."

[1094] Step 5:

[1095] The server sends the generated feedback text to the terminal. The input is the feedback text data, and the output is the feedback text sent to the terminal. Specifically, the server generates an HTTP response and sends the feedback text data to the terminal.

[1096] Step 6:

[1097] The feedback text received by the device is input into a speech synthesis model (e.g., gTTS library) and converted into voice data. The input is the feedback text and the output is voice data. Specifically, the device inputs the text data into the gTTS library and generates voice data saying, "Next, you need to assemble part A."

[1098] Step 7:

[1099] The device plays the generated audio data through a speaker. The input is audio data, and the output is the actual audio heard by the worker. Specifically, the device plays the audio data through the playsound library, and the worker receives feedback via audio.

[1100] Through these steps, workers can smoothly communicate their intentions and receive audio feedback on appropriate work procedures and points to note.

[1101] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1102] The present invention relates to a system for supporting the learning of children with mild physical or intellectual disabilities, and in particular to a system that provides more appropriate feedback and educational materials based on the user's emotional state by incorporating an emotion engine.

[1103] System Overview

[1104] This system captures the user's voice as they speak into their device and sends the voice data to a server. The server converts the voice data into text data and analyzes the user's intentions and learning needs. It also uses an emotion engine to analyze the user's emotions and selects appropriate feedback and learning materials based on the analysis results. The selected feedback and learning materials are sent to the device and played back to the user using voice synthesis.

[1105] Program processing details

[1106] Receiving and sending audio input

[1107] The user says to the terminal, "I want to study math today."

[1108] The device captures the user's voice using a microphone and temporarily stores it in digital form.

[1109] The device generates and sends an API request to send the captured audio data to the server.

[1110] Speech Recognition and Analysis

[1111] The server receives the voice data sent from the device and inputs it into a voice recognition model, which uses generative AI to convert the voice data into text data.

[1112] For example, text data such as "I want to study math today" is generated.

[1113] The server analyzes the text data to understand the user's intent, using natural language processing (NLP) techniques.

[1114] Emotion analysis

[1115] The server inputs the voice or text data into an emotion engine to analyze the user's emotions. Generative AI technology is used to perform emotion analysis and identify the user's emotional state, such as whether they are happy, sad, or distressed.

[1116] The feedback and educational content is adjusted based on the results of sentiment analysis. For example, if the analysis shows that the user is in trouble, the feedback will be gentle and responsive.

[1117] Selecting and submitting feedback

[1118] The server selects appropriate feedback and learning materials based on the results of the analysis and sentiment analysis. For example, if the server determines that the user is having trouble with a request such as "I want to study math," it generates gentle feedback such as "Let's start with the basics."

[1119] The server sends the generated feedback and text data of the teaching materials to the terminal.

[1120] Speech synthesis and playback

[1121] The device inputs the text data received from the server into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[1122] For example, voice data saying "Let's start with the basics first" is generated.

[1123] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1124] Specific examples

[1125] 1. Receiving voice input

[1126] A user says, "Today I want to study math."

[1127] The device captures the audio and stores it as audio data.

[1128] The terminal transmits the voice data to the server.

[1129] 2. Speech Recognition and Analysis

[1130] The server inputs the voice data into a voice recognition model and converts it into text data.

[1131] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[1132] 3. Emotion analysis

[1133] The server uses an emotion engine to analyze the user's emotion from the voice data or text data.

[1134] Based on the analysis results, it is recognized that, for example, "the user is feeling anxious."

[1135] 4. Selecting and Submitting Feedback

[1136] Based on the results of the analysis and emotion analysis, the server generates gentle feedback such as, "Let's try an addition problem."

[1137] The generated feedback is sent to the terminal as text data.

[1138] 5. Speech Synthesis and Playback

[1139] The text data received by the terminal is input into a speech synthesis model and converted into voice data.

[1140] The device plays the generated audio and tells the user, "Let's start with the basics."

[1141] This system allows users to smoothly communicate their intentions, receive appropriate feedback, and progress in learning that is tailored to their emotional state.It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[1142] The processing flow will be explained below.

[1143] Step 1:

[1144] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[1145] Step 2:

[1146] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[1147] Step 3:

[1148] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[1149] Step 4:

[1150] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[1151] Step 5:

[1152] The server inputs the acquired voice data into a voice recognition model, which uses generative AI to perform advanced voice recognition.

[1153] Step 6:

[1154] The speech recognition model converts the speech data into text data, for example, "I want to study math today."

[1155] Step 7:

[1156] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning goals.

[1157] Step 8:

[1158] The server inputs the voice data into an emotion engine to analyze the user's emotions. Generative AI technology is used to identify emotional states (such as joy, sadness, or anxiety) from the voice.

[1159] Step 9:

[1160] The server selects feedback and teaching materials based on the emotional data analyzed by the emotion engine. For example, if the server determines that the user is feeling anxious, it will select gentle instructional content.

[1161] Step 10:

[1162] The server generates text data based on the selected feedback and learning materials, such as "Let's start with the basics."

[1163] Step 11:

[1164] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[1165] Step 12:

[1166] The device receives the API response from the server and obtains the feedback text data.

[1167] Step 13:

[1168] The device inputs the acquired text data into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[1169] Step 14:

[1170] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's start with the basics."

[1171] Step 15:

[1172] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1173] Example 2

[1174] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1175] This invention relates to a system for supporting the learning of children with mild disabilities. Conventional learning support systems often provide one-sided feedback and learning materials without considering the user's emotional state, limiting the effectiveness of learning. Furthermore, there are issues with the difficulty of selecting appropriate feedback and learning materials, making it difficult to maintain the user's motivation to learn. The objective of this invention is to solve these issues and provide more effective learning support.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1177] In this invention,

[1178] The server includes means for capturing voice data input by a user, means for transmitting the voice data to a relay device, voice recognition means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for analyzing the voice data or the text data to identify the user's emotional state, means for selecting appropriate feedback and learning materials based on the user's intention and emotional state, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This makes it possible to provide appropriate feedback and learning materials that take the user's emotional state into consideration, thereby improving learning effectiveness and maintaining learning motivation.

[1179] "Mild disability" refers to a minor limitation in physical or intellectual function.

[1180] "Supporting learning" refers to helping users acquire knowledge and skills efficiently.

[1181] "User" refers to individuals who use the system, particularly children with mild disabilities.

[1182] "Voice data" refers to data that has been saved in digital form as the voice of a user speaking into a terminal.

[1183] "Capturing" refers to using a microphone to collect sound and convert it into a digital format.

[1184] "Relay device" refers to a device or software for transmitting data from a terminal to a server.

[1185] "Speech recognition" refers to the technology of converting voice data into text data.

[1186] "Text data" refers to text information converted from voice data using voice recognition technology.

[1187] "Analysis" refers to evaluating text data to understand the user's intentions and emotions.

[1188] "Emotional state" refers to the emotion the user is feeling at that time.

[1189] "Feedback" refers to a response or information to a user's input or state.

[1190] "Instructional Materials" refers to educational resources and materials intended to aid users in their learning.

[1191] "Speech synthesis" refers to the technology of converting text data into voice data.

[1192] "Playback" refers to listening to audio data through speakers or headphones.

[1193] "Generative AI" refers to models and techniques that use artificial intelligence techniques to generate or transform data.

[1194] This invention is a system for supporting the learning of children with mild disabilities. In particular, it improves learning effectiveness by analyzing the user's emotional state and providing appropriate feedback and learning materials based on that. The system has the function of capturing voice data input by the user and transmitting the voice data to a server.

[1195] The device captures the user's speech using a microphone and stores it in digital format. The stored speech data is then sent to a server via a relay device. The server uses speech recognition technology to convert the speech data into text data. For speech recognition, generative AI technology (e.g., a general generative AI speech recognition model) is used. For example, if a user says, "I want to study math today," the server converts this speech into text and generates the text data, "I want to study math today."

[1196] The server then analyzes this text data to identify the user's intent. This analysis uses natural language processing (NLP) techniques (e.g., widely used natural language processing models). After the user's intent is identified, this text data or the original voice data is input into a sentiment analysis engine. Sentiment analysis uses generative AI techniques (e.g., common sentiment analysis models) to identify the user's emotional state. For example, if the analysis finds that the user is in trouble, the system generates appropriate feedback based on this.

[1197] The server selects appropriate feedback and learning materials based on the user's intentions and emotional state. For example, if a user says, "I want to study math," and is also analyzed as "having trouble," the server generates gentle feedback such as, "Let's start with the basics." The generated feedback and learning materials are then sent to the device as text data.

[1198] The device converts the received text data into voice data using speech synthesis technology (for example, a common speech synthesis model). The converted voice data is played back to the user through speakers or headphones. For example, feedback such as "Let's start with the basics" may be played back as voice.

[1199] This system allows users to smoothly communicate their intentions and receive appropriate feedback and learning materials in response to their requests, while also taking their emotional state into consideration when studying. This improves learning effectiveness and maintains motivation to learn.

[1200] Specific examples

[1201] When a user says, "I want to study math today," the device captures this voice and saves it digitally. The saved voice data is sent to a server. The server converts this voice data into text data using generative AI technology, generating the text data "I want to study math today." Natural language processing technology is then used on this text data to identify the user's intention as "I want to study math."

[1202] The server then inputs this voice or text data into an emotion analysis engine to analyze the user's emotional state. If generative AI technology determines that the user is having trouble, it generates gentle feedback such as, "Let's start with the basics." The text data of this feedback is sent to the device, which then converts it into voice data and plays it back to the user.

[1203] Example prompt sentence:

[1204] I want to study math today.

[1205] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1206] Step 1:

[1207] The device captures the user's speech with a microphone and stores it in digital form. The input is the user's speech, and the device uses the microphone to convert analog speech into digital audio data. The output is the captured digital audio data.

[1208] Step 2:

[1209] The device generates an API request to send the stored digital audio data to the server, and sends the audio data to the server. The input is the digital audio data generated in step 1, and the output is the audio data sent to the server. Specifically, the device sends the audio data over the network.

[1210] Step 3:

[1211] The server receives the voice data sent from the device and inputs it into a voice recognition model. The input is the voice data sent from the device, and the voice data is converted into text data using a voice recognition model (for example, a generative AI voice recognition model). The output is the converted text data. For example, the text data generated is "I want to study math today."

[1212] Step 4:

[1213] The server analyzes the generated text data and identifies the user's intention. The input is the text data generated in step 3, and the server analyzes the data using natural language processing (NLP) techniques (e.g., a widely used NLP model). The output is data indicating the user's intention. For example, the intention "I want to study math" is identified.

[1214] Step 5:

[1215] The server inputs the voice or text data into the emotion engine and analyzes the user's emotional state. The input is the data generated in step 3 or step 4, and generative AI technology is used to identify the user's emotion. The output is data that indicates the user's emotional state. For example, the analysis may determine that the user is in trouble.

[1216] Step 6:

[1217] The server selects appropriate feedback and learning materials based on the user's intentions and the results of emotion analysis. The input is the data generated in steps 4 and 5, and an algorithm is used to select appropriate feedback. The output is the text data of the selected feedback and learning materials. For example, the generated feedback is "Let's start with the basics."

[1218] Step 7:

[1219] The server sends the selected feedback and text data of the teaching materials to the terminal. The input is the text data generated in step 6, and the output is the feedback text data sent to the terminal. Specifically, the server sends the text data over the network.

[1220] Step 8:

[1221] The device inputs the received text data into a speech synthesis model and converts it into speech data. The input is the text data received in step 7, and the generative AI speech synthesis model is used to convert the text data into speech data. The output is the converted speech data. For example, speech data such as "Let's start with the basics" is generated.

[1222] Step 9:

[1223] The device plays the generated audio data to the user through speakers or headphones. The input is the audio data generated in step 8, and the specific operation is to deliver the audio to the user using a playback device. The output is the audio the user hears. For example, the audio played to the user is, "Let's start with the basics."

[1224] (Application example 2)

[1225] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] Conventional factory robots are unable to provide feedback or instructions based on the worker's emotions or situation, making it difficult to reduce worker stress or support efficient work. Furthermore, when a worker feels confused or anxious, delays in appropriate responses can lead to reduced work efficiency and an increased risk of mistakes. Therefore, there is a need for a system that can provide feedback and instructions based on the worker's emotional state, improving work efficiency and reducing worker stress.

[1227] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion analysis means for analyzing voice data or text data to identify the user's emotional state, means for adjusting feedback and teaching materials based on the emotional state, and means for capturing voice data. This allows appropriate feedback and instructions to be provided according to the worker's emotional state, improving work efficiency and reducing worker stress.

[1228] "Audio data" refers to data that represents audio in digital form.

[1229] A "server" is a high-performance computer that processes and manages data over a network.

[1230] "Speech recognition means" refers to technology that analyzes voice data and converts it into text data.

[1231] "Text data" refers to data that represents character information in a digital format.

[1232] "Emotion analysis means" refers to a technique for analyzing voice data or text data to identify the user's emotional state.

[1233] "Means for selecting feedback and educational materials" refers to techniques for selecting appropriate information and educational materials based on the user's intentions and emotional state.

[1234] "Speech synthesis means" refers to technology that converts text data into voice data.

[1235] "Playback means" refers to technology that allows users to listen to audio data through devices such as speakers or headphones.

[1236] "Means for analyzing data" refers to the technology for analyzing input data and extracting necessary information.

[1237] "Emotional state" refers to the user's psychological emotional state, and includes, for example, happiness, sadness, confusion, anxiety, and the like.

[1238] System configuration

[1239] The system of the present invention includes a voice input means, a means for transmitting voice data to a server, a voice recognition means, a sentiment analysis means, a means for selecting feedback and teaching materials, a voice synthesis means, and a playback means. Each of these means will be described in detail below.

[1240] Voice input means

[1241] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. The microphone in the smart glasses or HMD is used to capture the voice data. This voice data is then sent to the server in the next step.

[1242] A means of sending audio data to the server

[1243] The captured audio data is sent to the server via the device's communication module, using HTTP or HTTPS as the communication protocol.

[1244] Voice recognition means

[1245] The server uses a generative AI model, such as the Google Cloud Speech API, to convert the audio data into text. This is the step where speech recognition takes place and converts the user's input speech into text form.

[1246] Emotion analysis means

[1247] The generated text data is passed to a sentiment analysis engine, which analyzes the voice or text data to identify the user's emotions. Generative AI technology is used to identify the user's emotional state, such as whether they are happy, sad, or troubled.

[1248] Feedback and materials selection

[1249] Based on the results of the sentiment analysis, the server selects appropriate feedback and educational materials. For example, if the analysis indicates that the user is confused, the server selects the message "Please stay calm and continue working. Do you need further instructions?" as the corresponding feedback.

[1250] Voice synthesis means

[1251] The selected feedback and learning materials are then converted into audio data using generative AI models such as the Google Cloud Text-to-Speech API, which converts the text data into audio data and generates audio to provide appropriate instructions to the user.

[1252] Regeneration means

[1253] The generated audio data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive feedback and instructions in real time.

[1254] Specific examples

[1255] For example, if a user says, "I feel like I'm not making much progress on my work today. I'm having trouble," the voice input means captures this speech. The voice data is sent to the server and converted into text data by the voice recognition means. The generated text is passed to the "emotion analysis engine," which identifies the emotion "confused." In response, the server selects feedback such as "Please stay calm and continue working. Do you need further instructions?" and converts it into voice data by the voice synthesis means. Finally, the generated voice data is played back through the speakers of the smart glasses or HMD.

[1256] Prompt Sentence Examples

[1257] Analyze the following text to identify the user's sentiment: "I feel like I'm not getting anything done today. I'm frustrated."

[1258] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1259] Step 1:

[1260] Receiving and sending audio input

[1261] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. Specifically, the user utters, "I feel like I'm not making much progress on my work today. I'm in trouble." This captures the voice data. The captured voice data is sent to the server through the device's communication module. The input is the user's voice data, and the output is the voice data sent to the server.

[1262] Step 2:

[1263] Voice Recognition

[1264] The server analyzes the received voice data using a generative AI model such as Google Cloud Speech API. The voice data is converted into text data using speech recognition. Specifically, the generated text data is, "I feel like I'm not making much progress on my work today. I'm in trouble." The input is the voice data sent to the server, and the output is the generated text data.

[1265] Step 3:

[1266] Emotion analysis

[1267] The generated text data is passed to an emotion analysis engine. The server uses emotion analysis means to analyze the text data and identify the user's emotional state. Specifically, the emotion "confused" is identified from the text data "I feel like I can't get my work done well today. I'm in trouble." The input is the generated text data, and the output is the identified emotional state.

[1268] Step 4:

[1269] Feedback Selection

[1270] Based on the results of the emotion analysis, the server selects appropriate feedback. For example, if the emotion "confused" is identified, the server selects the feedback "Please stay calm. Do you need further instructions?" The input is the identified emotional state, and the output is the selected feedback message.

[1271] Step 5:

[1272] Speech synthesis

[1273] The selected feedback message is converted into audio data using a generative AI model such as the Google Cloud Text-to-Speech API. Specifically, the audio data generated is "Please stay calm and do your job. Do you need further instructions?" The input is the selected feedback message, and the output is the generated audio data.

[1274] Step 6:

[1275] Audio playback

[1276] The generated voice data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive appropriate feedback and instructions in real time. The input is the generated voice data, and the output is the played voice.

[1277] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1278] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1279] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1280] [Fourth embodiment]

[1281] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1282] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1283] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1284] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1285] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1286] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1287] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1288] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1289] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1290] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1291] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1292] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1293] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1294] The present invention is a system for supporting the learning of children with mild physical or intellectual disabilities, and its form is described in detail below. The system described below allows users (children) to smoothly communicate their intentions and receive necessary feedback and educational materials.

[1295] System Overview

[1296] This system captures the user's speech into the device and sends the speech data to a server. The server converts the speech data into text data and analyzes the user's intentions and learning needs. It then selects appropriate feedback and learning materials and sends them back to the device. The device then synthesizes the selected feedback and learning materials into speech and plays them back to the user.

[1297] Program processing details

[1298] Receiving and sending audio input

[1299] The user says to the terminal, "I want to study math today."

[1300] The device uses a microphone to capture the user's voice, and the captured voice data is stored in digital form.

[1301] The device generates and sends an API request to send the captured audio data to the server.

[1302] Speech Recognition and Analysis

[1303] The server receives the voice data sent from the device and inputs it into a voice recognition model (e.g., a model using generative AI technology).

[1304] The speech recognition model converts the speech data into text data, for example, in the form "I want to study math today."

[1305] The server analyzes the acquired text data and understands the user's intent using natural language processing (NLP) technology.

[1306] Based on the analysis results, the server selects appropriate feedback and learning materials according to the user's learning needs. For example, if a user requests to "study math," it will select addition problems appropriate to their level.

[1307] Sending feedback and text-to-speech

[1308] The server generates an API response to send the selected feedback and content of the teaching materials to the terminal as text data.

[1309] The device inputs the text data received from the server into a speech synthesis model (e.g., a model using generative AI technology).

[1310] The speech synthesis model converts text data into speech data, generating speech data such as "Let's try an addition problem today."

[1311] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1312] Specific examples

[1313] 1. Receiving voice input

[1314] A user says, "Today I want to study math."

[1315] The device captures the audio and stores it as audio data.

[1316] The terminal transmits the voice data to the server.

[1317] 2. Speech Recognition and Analysis

[1318] The server inputs the voice data into a voice recognition model and converts it into text data.

[1319] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[1320] The server selects appropriate teaching materials and generates feedback such as "Let's try some addition problems" as text data.

[1321] 3. Sending feedback and speech synthesis

[1322] The server sends the feedback to the device.

[1323] The device inputs the text data into a speech synthesis model and converts it into speech data.

[1324] The terminal plays back the generated voice data and provides feedback to the user.

[1325] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they progress through their studies. It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[1326] The processing flow will be explained below.

[1327] Step 1:

[1328] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[1329] Step 2:

[1330] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[1331] Step 3:

[1332] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[1333] Step 4:

[1334] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[1335] Step 5:

[1336] The server inputs the captured voice data into a speech recognition model, which uses generative AI to perform advanced speech recognition.

[1337] Step 6:

[1338] The speech recognition model converts the speech data into text data, for example, text data in the format "I want to study math today."

[1339] Step 7:

[1340] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning needs.

[1341] Step 8:

[1342] The server selects appropriate feedback and learning materials based on the analysis results. For example, if a user requests "I want to study math," it will select "addition problems."

[1343] Step 9:

[1344] The server generates text data based on the selected feedback and teaching materials, such as "Let's try some addition problems today."

[1345] Step 10:

[1346] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[1347] Step 11:

[1348] The device receives the API response from the server and obtains the feedback text data.

[1349] Step 12:

[1350] The device inputs the captured text data into a speech synthesis model, which uses generative AI to convert the text data into speech data.

[1351] Step 13:

[1352] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's try an addition problem today."

[1353] Step 14:

[1354] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1355] Example 1

[1356] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] With conventional learning support systems, children with mild physical or intellectual disabilities have difficulty expressing themselves smoothly, making it difficult for them to receive appropriate feedback and learning materials. Furthermore, the burden on parents and welfare workers is heavy, making it difficult to establish an effective support system.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1359] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means for converting the voice data into text data in the server, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to the terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This allows the user to smoothly communicate their intentions and progress in their studies while receiving appropriate feedback. It also reduces the burden on parents and welfare workers and enables the creation of an effective support system.

[1360] "User" refers to a child with a mild physical or intellectual disability who uses the system.

[1361] "Voice data" refers to digital data of a user's speech captured by a voice input device such as a microphone.

[1362] "Server" refers to a central processing unit that receives voice data, converts it into text data, and then analyzes the text data to identify the user's intent and select appropriate feedback and educational materials.

[1363] "Speech Recognition Method" refers to the technology or model used to convert voice data received by the server into text data, including, for example, generative AI technology.

[1364] "Text data" refers to character string data converted from voice data by a voice recognition means.

[1365] "Natural language processing technology" refers to technology for analyzing text data to understand its meaning and identify the user's intent.

[1366] "Feedback" refers to instructions, advice, or learnings provided based on user input.

[1367] "Instructional Materials" refers to educational content and materials provided to assist users in their learning.

[1368] "Speech synthesis method" refers to the technology or model used to convert text data into speech data, including, for example, generative AI technologies.

[1369] "Terminal" refers to a device that captures what a user says and communicates with a server. It may include a playback device.

[1370] The present invention is a system for supporting learning for children with mild physical or intellectual disabilities, and is described in detail below. This system allows users (children) to smoothly communicate their intentions and receive necessary feedback and learning materials.

[1371] System Overview

[1372] The system operates on the basis of voice data processing and communication between terminals, servers and users.

[1373] Hardware and Software Configuration

[1374] Device:

[1375] The built-in microphone is used to capture the user's voice.

[1376] It includes an API client to communicate with the server, for example using the HTTPS protocol.

[1377] Play audio using the built-in speaker or headphones.

[1378] Use a speech synthesis model (e.g., Google Text-to-Speech API or Amazon Polly, which use generative AI techniques).

[1379] server:

[1380] The voice data is input into a speech recognition model (e.g., Google Speech-to-Text API or IBM Watson Speech to Text, which utilize generative AI technology).

[1381] We use natural language processing (NLP) techniques (e.g., NLTK, spaCy) to analyze the text data obtained by speech recognition.

[1382] Select appropriate feedback and teaching materials from the database.

[1383] Text data of feedback and teaching materials is generated as an API response and sent to the device.

[1384] Capture and transmit audio data

[1385] When a user speaks into the device, such as "I want to study math today," the device's microphone captures the voice, which is then stored digitally and used to generate an API request and send it to the server.

[1386] Speech Recognition and Analysis

[1387] The server receives the voice data sent from the device and inputs it into a voice recognition model. The model converts the voice data into text data, for example, in the format "I want to study math today." The server analyzes this text data and uses natural language processing technology to understand the user's intention. Through this analysis, the server understands that the user's intention is "I want to study math."

[1388] Selecting and sending appropriate feedback and educational materials

[1389] The server selects appropriate feedback and learning materials from the database based on the user's wishes. For example, if the user says, "I want to study math," the server selects addition problems appropriate to the user's level. It then generates and sends an API response to send the text data of the selected feedback and learning materials to the device.

[1390] Speech synthesis and feedback playback

[1391] The device inputs the text data received from the server into a speech synthesis model and converts it into voice data. For example, voice data such as "Let's try an addition problem today" is generated. Finally, the device plays this voice data to the user through a playback device.

[1392] Specific examples

[1393] When a user says, "I want to study math today," the device captures the voice and sends the voice data to the server. The server inputs this voice data into a voice recognition model, converts it into text data, and then analyzes it using natural language processing technology to determine that the user's intention is "I want to study math." The server then selects an appropriate addition problem and sends the content of that problem to the device as text data. The device then inputs this text data into a voice synthesis model, generates voice data saying, "Let's try an addition problem today," and plays it back to the user.

[1394] Prompt Sentence Examples

[1395] "When a user says, 'I want to study math today,' select learning materials and feedback appropriate to that intent and generate them as text data."

[1396] This system allows users to smoothly communicate their intentions and receive appropriate feedback as they learn. Furthermore, by utilizing a generative AI model, highly accurate speech recognition and speech synthesis are achieved.

[1397] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1398] Step 1:

[1399] The user speaks to the terminal.

[1400] Input: User utterance (e.g., "I want to study math today.")

[1401] Output: The user's speech is input to the device's microphone as an analog audio signal.

[1402] Specific action: The user speaks into the built-in microphone.

[1403] Step 2:

[1404] The device captures the audio data and stores it digitally.

[1405] Input: Analog audio signal

[1406] Output: Digital audio data

[1407] Specific operation: The device's microphone driver converts the analog audio signal into digital audio data and temporarily stores it in internal memory.

[1408] Step 3:

[1409] The device generates and sends an API request to send the captured audio data to the server.

[1410] Input: Digital audio data

[1411] Output: API request to the server (including audio data)

[1412] Specific operation: The device sends audio data to the server using the HTTPS protocol.

[1413] Step 4:

[1414] The server receives the voice data transmitted from the terminal.

[1415] Input: API request (including audio data)

[1416] Output: Audio data stored on the server

[1417] Specific operation: The server's API endpoint receives the request, extracts the audio data, and saves it in the server's storage.

[1418] Step 5:

[1419] The server inputs the voice data into a voice recognition model and converts it into text data.

[1420] Input: Digital audio data

[1421] Output: Text data (e.g., "I want to study math today")

[1422] Specific operation: The server inputs voice data into a voice recognition model (which utilizes generative AI technology), and the model analyzes the voice data and converts it into text data.

[1423] Step 6:

[1424] The server analyzes the text data and identifies the user's intent.

[1425] Input: Text data

[1426] Output: User's intention (e.g., "I want to study math")

[1427] Specific operation: The server uses natural language processing (NLP) techniques to analyze the text data and identify the user's intent.

[1428] Step 7:

[1429] The server selects appropriate feedback and educational materials.

[1430] Input: User intent

[1431] Output: Feedback and text data for teaching materials (e.g., "Let's do the addition problem")

[1432] Specific operation: The server selects feedback and learning materials from the database according to the user's wishes.

[1433] Step 8:

[1434] The server generates and sends an API response to send the selected feedback and text data of the teaching materials to the terminal.

[1435] Input: Text data of feedback and teaching materials

[1436] Output: API response (including text data)

[1437] Specific operation: The server sends text data to the terminal using the HTTPS protocol.

[1438] Step 9:

[1439] The text data received by the terminal from the server is input into a speech synthesis model and converted into voice data.

[1440] Input: Text data

[1441] Output: Audio data (e.g., "Let's try some addition problems today.")

[1442] Specific operation: The device inputs text data into a speech synthesis model (using generative AI technology), which then converts the text data into speech data.

[1443] Step 10:

[1444] The terminal plays the generated audio data to the user using a playback device.

[1445] Input: Audio data

[1446] Output: The audio the user hears

[1447] What it does: The device's audio driver sends audio signals to speakers or headphones so the user can hear the sound.

[1448] (Application example 1)

[1449] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1450] When working in a factory, workers with mild physical or intellectual disabilities may find it difficult to receive appropriate work instructions and feedback. This reduces work efficiency and makes it difficult to ensure worker safety. Furthermore, workers with disabilities have difficulty communicating their intentions smoothly, which can lead to problems such as not receiving appropriate support.

[1451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1452] In this invention, the server includes means for capturing voice data input by a user, means for transmitting the voice data to the server, voice recognition means in the server for converting the voice data into text data, means for analyzing the text data and identifying the user's intention, means for selecting appropriate feedback and learning materials, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, means for playing the voice data to the user, and means for capturing the voice of workers in a factory and providing appropriate feedback on work procedures and points to note. This enables workers to smoothly communicate their intentions and receive appropriate feedback on work procedures and points to note.

[1453] "Capture" means taking in data such as audio or images and saving it as digital data.

[1454] "Voice data" refers to data that is recorded using a microphone or other device and expressed in digital format.

[1455] A "server" is a computer system that receives and responds to requests from clients over a network.

[1456] "Text data" refers to data that represents character information in digital form.

[1457] "Speech recognition" is a technology that analyzes input voice data and converts it into corresponding text data.

[1458] "Analysis" is the process of examining data in detail to understand its content and trends.

[1459] "Feedback" refers to information or actions that provide a response or reaction to a particular action or request.

[1460] "Instructional Materials" means materials or content used for educational or training purposes.

[1461] "Speech synthesis" is a technology that converts text data into voice data.

[1462] "Playback" means outputting captured or generated audio or video data through an audio or video device.

[1463] "Factory" means the facility or workspace where manufacturing or production takes place.

[1464] "Worker" means a person who performs work in a factory.

[1465] A "work procedure" is the steps or process for accomplishing a particular task.

[1466] "Caution" refers to matters that require special attention in a particular situation or task.

[1467] System Overview

[1468] This invention relates to a system that supports workers with mild physical or intellectual disabilities who work in factories. This system captures the worker's voice input and sends the voice data to a server. The server converts the voice data into text data and analyzes the worker's intentions and work needs. It then selects appropriate work procedures and points to note and sends them back to the terminal. The terminal also supports the worker by synthesizing the selected feedback into voice and playing it back to the worker.

[1469] Hardware and software used

[1470] 1. Hardware:

[1471] Microphone: Used to capture the voice of the worker.

[1472] Terminal (e.g., tablet, smartphone): A device that allows workers to input voice and receive feedback.

[1473] Speaker: Used to play feedback audio.

[1474] 2. Software:

[1475] Python: The programming language used to implement the entire program.

[1476] speech_recognition library: A library for capturing and recognizing speech.

[1477] OpenAI GPT-3 API: A generative AI model that analyzes voice data and generates appropriate feedback.

[1478] gTTS library: A library for synthesizing text data into voice data.

[1479] playsound library: A library for playing generated sound data.

[1480] System processing details

[1481] 1. Receiving audio input:

[1482] The worker talks to the terminal about the work he is doing and the problems he is having.

[1483] The device uses a microphone to capture audio data and store it in digital form.

[1484] The captured audio data is sent to a server.

[1485] 2. Speech Recognition and Analysis:

[1486] The server receives the transmitted voice data and converts it into text data using the speech_recognition library.

[1487] The converted text data is then analyzed using the OpenAI GPT-3 API.

[1488] Based on the analysis results, the server understands the worker's intentions and work needs, and generates appropriate work procedures and points to note.

[1489] 3. Send feedback and speech synthesis:

[1490] The server transmits the generated feedback text data to the terminal.

[1491] The device uses the gTTS library to convert the feedback text data into voice data.

[1492] The converted voice data is played back to the worker through a speaker.

[1493] Specific examples

[1494] Example 1:

[1495] The worker speaks into the terminal, "What should I do next?"

[1496] A microphone captures the audio and sends it to a server.

[1497] The server converts the voice data into text data and analyzes it using the OpenAI GPT-3 API.

[1498] Feedback is generated such as "Next, you need to assemble part A."

[1499] The terminal synthesizes the generated feedback into voice and plays it back to the worker.

[1500] Prompt Sentence Examples

[1501] User: What should I do next?

[1502] A basic import, a mutually exclusive reaction or a step in the process.

[1503] Using this system, workers can smoothly communicate their intentions and receive feedback on appropriate work procedures and points to note. It is also expected to contribute to improving work efficiency and ensuring safety.

[1504] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1505] Step 1:

[1506] The user speaks instructions or questions into the device. The device's microphone captures this voice and stores it as digital audio data. The input is the user's voice, and the output is digital audio data. Specifically, the user speaks, "What should I do next?", and the device records the voice and converts it into digital format.

[1507] Step 2:

[1508] The device sends the captured audio data to the server. The input is digital audio data, and the output is the audio data sent to the server. Specifically, the device generates an HTTP request over the network and sends the audio data to the server.

[1509] Step 3:

[1510] The voice data received by the server is input into a voice recognition model (e.g., speech_recognition library) and converted into text data. The input is digital voice data, and the output is text data. Specifically, the server analyzes the voice data and generates text data such as "What should I do next?"

[1511] Step 4:

[1512] The server analyzes the text data and uses a generative AI model (e.g., OpenAI GPT-3) to understand the user's intentions. The input is text data, and the output is feedback text based on the analysis results. Specifically, the server inputs the text data into the generative AI model and creates feedback text such as, "Next, you need to assemble part A."

[1513] Step 5:

[1514] The server sends the generated feedback text to the terminal. The input is the feedback text data, and the output is the feedback text sent to the terminal. Specifically, the server generates an HTTP response and sends the feedback text data to the terminal.

[1515] Step 6:

[1516] The feedback text received by the device is input into a speech synthesis model (e.g., gTTS library) and converted into voice data. The input is the feedback text and the output is voice data. Specifically, the device inputs the text data into the gTTS library and generates voice data saying, "Next, you need to assemble part A."

[1517] Step 7:

[1518] The device plays the generated audio data through a speaker. The input is audio data, and the output is the actual audio heard by the worker. Specifically, the device plays the audio data through the playsound library, and the worker receives feedback via audio.

[1519] Through these steps, workers can smoothly communicate their intentions and receive audio feedback on appropriate work procedures and points to note.

[1520] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1521] The present invention relates to a system for supporting the learning of children with mild physical or intellectual disabilities, and in particular to a system that provides more appropriate feedback and educational materials based on the user's emotional state by incorporating an emotion engine.

[1522] System Overview

[1523] This system captures the user's voice as they speak into their device and sends the voice data to a server. The server converts the voice data into text data and analyzes the user's intentions and learning needs. It also uses an emotion engine to analyze the user's emotions and selects appropriate feedback and learning materials based on the analysis results. The selected feedback and learning materials are sent to the device and played back to the user using voice synthesis.

[1524] Program processing details

[1525] Receiving and sending audio input

[1526] The user says to the terminal, "I want to study math today."

[1527] The device captures the user's voice using a microphone and temporarily stores it in digital form.

[1528] The device generates and sends an API request to send the captured audio data to the server.

[1529] Speech Recognition and Analysis

[1530] The server receives the voice data sent from the device and inputs it into a voice recognition model, which uses generative AI to convert the voice data into text data.

[1531] For example, text data such as "I want to study math today" is generated.

[1532] The server analyzes the text data to understand the user's intent, using natural language processing (NLP) techniques.

[1533] Emotion analysis

[1534] The server inputs the voice or text data into an emotion engine to analyze the user's emotions. Generative AI technology is used to perform emotion analysis and identify the user's emotional state, such as whether they are happy, sad, or distressed.

[1535] The feedback and educational content is adjusted based on the results of sentiment analysis. For example, if the analysis shows that the user is in trouble, the feedback will be gentle and responsive.

[1536] Selecting and submitting feedback

[1537] The server selects appropriate feedback and learning materials based on the results of the analysis and sentiment analysis. For example, if the server determines that the user is having trouble with a request such as "I want to study math," it generates gentle feedback such as "Let's start with the basics."

[1538] The server sends the generated feedback and text data of the teaching materials to the terminal.

[1539] Speech synthesis and playback

[1540] The device inputs the text data received from the server into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[1541] For example, voice data saying "Let's start with the basics first" is generated.

[1542] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1543] Specific examples

[1544] 1. Receiving voice input

[1545] A user says, "Today I want to study math."

[1546] The device captures the audio and stores it as audio data.

[1547] The terminal transmits the voice data to the server.

[1548] 2. Speech Recognition and Analysis

[1549] The server inputs the voice data into a voice recognition model and converts it into text data.

[1550] The server analyzes the text data and understands that the user's intention is "I want to study mathematics."

[1551] 3. Emotion analysis

[1552] The server uses an emotion engine to analyze the user's emotion from the voice data or text data.

[1553] Based on the analysis results, it is recognized that, for example, "the user is feeling anxious."

[1554] 4. Selecting and Submitting Feedback

[1555] Based on the results of the analysis and emotion analysis, the server generates gentle feedback such as, "Let's try an addition problem."

[1556] The generated feedback is sent to the terminal as text data.

[1557] 5. Speech Synthesis and Playback

[1558] The text data received by the terminal is input into a speech synthesis model and converted into voice data.

[1559] The device plays the generated audio and tells the user, "Let's start with the basics."

[1560] This system allows users to smoothly communicate their intentions, receive appropriate feedback, and progress in learning that is tailored to their emotional state.It also reduces the burden on parents and welfare workers, making it possible to build an effective support system.

[1561] The processing flow will be explained below.

[1562] Step 1:

[1563] The user speaks to the device about their intentions and learning preferences, for example, "I want to study math today."

[1564] Step 2:

[1565] The device captures the user's voice using a microphone, and the captured voice data is temporarily stored in digital format.

[1566] Step 3:

[1567] The device generates an API request to send the captured audio data to the server, and the request contains the audio data.

[1568] Step 4:

[1569] The device sends the generated API request to the server, which receives the API request and retrieves the audio data.

[1570] Step 5:

[1571] The server inputs the acquired voice data into a voice recognition model, which uses generative AI to perform advanced voice recognition.

[1572] Step 6:

[1573] The speech recognition model converts the speech data into text data, for example, "I want to study math today."

[1574] Step 7:

[1575] The server analyzes the converted text data using natural language processing (NLP) techniques to identify the user's intentions and learning goals.

[1576] Step 8:

[1577] The server inputs the voice data into an emotion engine to analyze the user's emotions. Generative AI technology is used to identify emotional states (such as joy, sadness, or anxiety) from the voice.

[1578] Step 9:

[1579] The server selects feedback and teaching materials based on the emotional data analyzed by the emotion engine. For example, if the server determines that the user is feeling anxious, it will select gentle instructional content.

[1580] Step 10:

[1581] The server generates text data based on the selected feedback and learning materials, such as "Let's start with the basics."

[1582] Step 11:

[1583] The server generates an API response to send the generated text data to the device. The response includes the feedback text data.

[1584] Step 12:

[1585] The device receives the API response from the server and obtains the feedback text data.

[1586] Step 13:

[1587] The device inputs the acquired text data into a speech synthesis model, which uses generative AI technology to convert the text data into speech data.

[1588] Step 14:

[1589] The speech synthesis model converts text data into speech data, for example, generating speech data such as "Let's start with the basics."

[1590] Step 15:

[1591] The terminal plays the generated audio data to the user through a playback device (such as a speaker or headphones).

[1592] Example 2

[1593] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1594] This invention relates to a system for supporting the learning of children with mild disabilities. Conventional learning support systems often provide one-sided feedback and learning materials without considering the user's emotional state, limiting the effectiveness of learning. Furthermore, there are issues with the difficulty of selecting appropriate feedback and learning materials, making it difficult to maintain the user's motivation to learn. The objective of this invention is to solve these issues and provide more effective learning support.

[1595] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1596] In this invention,

[1597] The server includes means for capturing voice data input by a user, means for transmitting the voice data to a relay device, voice recognition means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for analyzing the voice data or the text data to identify the user's emotional state, means for selecting appropriate feedback and learning materials based on the user's intention and emotional state, means for transmitting the text data of the selected feedback and learning materials to a terminal, voice synthesis means for converting the text data into voice data, and means for playing the voice data to the user. This makes it possible to provide appropriate feedback and learning materials that take the user's emotional state into consideration, thereby improving learning effectiveness and maintaining learning motivation.

[1598] "Mild disability" refers to a minor limitation in physical or intellectual function.

[1599] "Supporting learning" refers to helping users acquire knowledge and skills efficiently.

[1600] "User" refers to individuals who use the system, particularly children with mild disabilities.

[1601] "Voice data" refers to data that has been saved in digital form as the voice of a user speaking into a terminal.

[1602] "Capturing" refers to using a microphone to collect sound and convert it into a digital format.

[1603] "Relay device" refers to a device or software for transmitting data from a terminal to a server.

[1604] "Speech recognition" refers to the technology of converting voice data into text data.

[1605] "Text data" refers to text information converted from voice data using voice recognition technology.

[1606] "Analysis" refers to evaluating text data to understand the user's intentions and emotions.

[1607] "Emotional state" refers to the emotion the user is feeling at that time.

[1608] "Feedback" refers to a response or information to a user's input or state.

[1609] "Instructional Materials" refers to educational resources and materials intended to aid users in their learning.

[1610] "Speech synthesis" refers to the technology of converting text data into voice data.

[1611] "Playback" refers to listening to audio data through speakers or headphones.

[1612] "Generative AI" refers to models and techniques that use artificial intelligence techniques to generate or transform data.

[1613] This invention is a system for supporting the learning of children with mild disabilities. In particular, it improves learning effectiveness by analyzing the user's emotional state and providing appropriate feedback and learning materials based on that. The system has the function of capturing voice data input by the user and transmitting the voice data to a server.

[1614] The device captures the user's speech using a microphone and stores it in digital format. The stored speech data is then sent to a server via a relay device. The server uses speech recognition technology to convert the speech data into text data. For speech recognition, generative AI technology (e.g., a general generative AI speech recognition model) is used. For example, if a user says, "I want to study math today," the server converts this speech into text and generates the text data, "I want to study math today."

[1615] The server then analyzes this text data to identify the user's intent. This analysis uses natural language processing (NLP) techniques (e.g., widely used natural language processing models). After the user's intent is identified, this text data or the original voice data is input into a sentiment analysis engine. Sentiment analysis uses generative AI techniques (e.g., common sentiment analysis models) to identify the user's emotional state. For example, if the analysis finds that the user is in trouble, the system generates appropriate feedback based on this.

[1616] The server selects appropriate feedback and learning materials based on the user's intentions and emotional state. For example, if a user says, "I want to study math," and is also analyzed as "having trouble," the server generates gentle feedback such as, "Let's start with the basics." The generated feedback and learning materials are then sent to the device as text data.

[1617] The device converts the received text data into voice data using speech synthesis technology (for example, a common speech synthesis model). The converted voice data is played back to the user through speakers or headphones. For example, feedback such as "Let's start with the basics" may be played back as voice.

[1618] This system allows users to smoothly communicate their intentions and receive appropriate feedback and learning materials in response to their requests, while also taking their emotional state into consideration when studying. This improves learning effectiveness and maintains motivation to learn.

[1619] Specific examples

[1620] When a user says, "I want to study math today," the device captures this voice and saves it digitally. The saved voice data is sent to a server. The server converts this voice data into text data using generative AI technology, generating the text data "I want to study math today." Natural language processing technology is then used on this text data to identify the user's intention as "I want to study math."

[1621] The server then inputs this voice or text data into an emotion analysis engine to analyze the user's emotional state. If generative AI technology determines that the user is having trouble, it generates gentle feedback such as, "Let's start with the basics." The text data of this feedback is sent to the device, which then converts it into voice data and plays it back to the user.

[1622] Example prompt sentence:

[1623] I want to study math today.

[1624] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1625] Step 1:

[1626] The device captures the user's speech with a microphone and stores it in digital form. The input is the user's speech, and the device uses the microphone to convert analog speech into digital audio data. The output is the captured digital audio data.

[1627] Step 2:

[1628] The device generates an API request to send the stored digital audio data to the server, and sends the audio data to the server. The input is the digital audio data generated in step 1, and the output is the audio data sent to the server. Specifically, the device sends the audio data over the network.

[1629] Step 3:

[1630] The server receives the voice data sent from the device and inputs it into a voice recognition model. The input is the voice data sent from the device, and the voice data is converted into text data using a voice recognition model (for example, a generative AI voice recognition model). The output is the converted text data. For example, the text data generated is "I want to study math today."

[1631] Step 4:

[1632] The server analyzes the generated text data and identifies the user's intention. The input is the text data generated in step 3, and the server analyzes the data using natural language processing (NLP) techniques (e.g., a widely used NLP model). The output is data indicating the user's intention. For example, the intention "I want to study math" is identified.

[1633] Step 5:

[1634] The server inputs the voice or text data into the emotion engine and analyzes the user's emotional state. The input is the data generated in step 3 or step 4, and generative AI technology is used to identify the user's emotion. The output is data that indicates the user's emotional state. For example, the analysis may determine that the user is in trouble.

[1635] Step 6:

[1636] The server selects appropriate feedback and learning materials based on the user's intentions and the results of emotion analysis. The input is the data generated in steps 4 and 5, and an algorithm is used to select appropriate feedback. The output is the text data of the selected feedback and learning materials. For example, the generated feedback is "Let's start with the basics."

[1637] Step 7:

[1638] The server sends the selected feedback and text data of the teaching materials to the terminal. The input is the text data generated in step 6, and the output is the feedback text data sent to the terminal. Specifically, the server sends the text data over the network.

[1639] Step 8:

[1640] The device inputs the received text data into a speech synthesis model and converts it into speech data. The input is the text data received in step 7, and the generative AI speech synthesis model is used to convert the text data into speech data. The output is the converted speech data. For example, speech data such as "Let's start with the basics" is generated.

[1641] Step 9:

[1642] The device plays the generated audio data to the user through speakers or headphones. The input is the audio data generated in step 8, and the specific operation is to deliver the audio to the user using a playback device. The output is the audio the user hears. For example, the audio played to the user is, "Let's start with the basics."

[1643] (Application example 2)

[1644] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1645] Conventional factory robots are unable to provide feedback or instructions based on the worker's emotions or situation, making it difficult to reduce worker stress or support efficient work. Furthermore, when a worker feels confused or anxious, delays in appropriate responses can lead to reduced work efficiency and an increased risk of mistakes. Therefore, there is a need for a system that can provide feedback and instructions based on the worker's emotional state, improving work efficiency and reducing worker stress.

[1646] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion analysis means for analyzing voice data or text data to identify the user's emotional state, means for adjusting feedback and teaching materials based on the emotional state, and means for capturing voice data. This allows appropriate feedback and instructions to be provided according to the worker's emotional state, improving work efficiency and reducing worker stress.

[1647] "Audio data" refers to data that represents audio in digital form.

[1648] A "server" is a high-performance computer that processes and manages data over a network.

[1649] "Speech recognition means" refers to technology that analyzes voice data and converts it into text data.

[1650] "Text data" refers to data that represents character information in a digital format.

[1651] "Emotion analysis means" refers to a technique for analyzing voice data or text data to identify the user's emotional state.

[1652] "Means for selecting feedback and educational materials" refers to techniques for selecting appropriate information and educational materials based on the user's intentions and emotional state.

[1653] "Speech synthesis means" refers to technology that converts text data into voice data.

[1654] "Playback means" refers to technology that allows users to listen to audio data through devices such as speakers or headphones.

[1655] "Means for analyzing data" refers to the technology for analyzing input data and extracting necessary information.

[1656] "Emotional state" refers to the user's psychological emotional state, and includes, for example, happiness, sadness, confusion, anxiety, and the like.

[1657] System configuration

[1658] The system of the present invention includes a voice input means, a means for transmitting voice data to a server, a voice recognition means, a sentiment analysis means, a means for selecting feedback and teaching materials, a voice synthesis means, and a playback means. Each of these means will be described in detail below.

[1659] Voice input means

[1660] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. The microphone in the smart glasses or HMD is used to capture the voice data. This voice data is then sent to the server in the next step.

[1661] A means of sending audio data to the server

[1662] The captured audio data is sent to the server via the device's communication module, using HTTP or HTTPS as the communication protocol.

[1663] Voice recognition means

[1664] The server uses a generative AI model, such as the Google Cloud Speech API, to convert the audio data into text. This is the step where speech recognition takes place and converts the user's input speech into text form.

[1665] Emotion analysis means

[1666] The generated text data is passed to a sentiment analysis engine, which analyzes the voice or text data to identify the user's emotions. Generative AI technology is used to identify the user's emotional state, such as whether they are happy, sad, or troubled.

[1667] Feedback and materials selection

[1668] Based on the results of the sentiment analysis, the server selects appropriate feedback and educational materials. For example, if the analysis indicates that the user is confused, the server selects the message "Please stay calm and continue working. Do you need further instructions?" as the corresponding feedback.

[1669] Voice synthesis means

[1670] The selected feedback and learning materials are then converted into audio data using generative AI models such as the Google Cloud Text-to-Speech API, which converts the text data into audio data and generates audio to provide appropriate instructions to the user.

[1671] Regeneration means

[1672] The generated audio data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive feedback and instructions in real time.

[1673] Specific examples

[1674] For example, if a user says, "I feel like I'm not making much progress on my work today. I'm having trouble," the voice input means captures this speech. The voice data is sent to the server and converted into text data by the voice recognition means. The generated text is passed to the "emotion analysis engine," which identifies the emotion "confused." In response, the server selects feedback such as "Please stay calm and continue working. Do you need further instructions?" and converts it into voice data by the voice synthesis means. Finally, the generated voice data is played back through the speakers of the smart glasses or HMD.

[1675] Prompt Sentence Examples

[1676] Analyze the following text to identify the user's sentiment: "I feel like I'm not getting anything done today. I'm frustrated."

[1677] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1678] Step 1:

[1679] Receiving and sending audio input

[1680] The user wears smart glasses or a head-mounted display (HMD) and requests work instructions by voice. Specifically, the user utters, "I feel like I'm not making much progress on my work today. I'm in trouble." This captures the voice data. The captured voice data is sent to the server through the device's communication module. The input is the user's voice data, and the output is the voice data sent to the server.

[1681] Step 2:

[1682] Voice Recognition

[1683] The server analyzes the received voice data using a generative AI model such as Google Cloud Speech API. The voice data is converted into text data using speech recognition. Specifically, the generated text data is, "I feel like I'm not making much progress on my work today. I'm in trouble." The input is the voice data sent to the server, and the output is the generated text data.

[1684] Step 3:

[1685] Emotion analysis

[1686] The generated text data is passed to an emotion analysis engine. The server uses emotion analysis means to analyze the text data and identify the user's emotional state. Specifically, the emotion "confused" is identified from the text data "I feel like I can't get my work done well today. I'm in trouble." The input is the generated text data, and the output is the identified emotional state.

[1687] Step 4:

[1688] Feedback Selection

[1689] Based on the results of the emotion analysis, the server selects appropriate feedback. For example, if the emotion "confused" is identified, the server selects the feedback "Please stay calm. Do you need further instructions?" The input is the identified emotional state, and the output is the selected feedback message.

[1690] Step 5:

[1691] Speech synthesis

[1692] The selected feedback message is converted into audio data using a generative AI model such as the Google Cloud Text-to-Speech API. Specifically, the audio data generated is "Please stay calm and do your job. Do you need further instructions?" The input is the selected feedback message, and the output is the generated audio data.

[1693] Step 6:

[1694] Audio playback

[1695] The generated voice data is played back to the user through the speakers of the smart glasses or HMD, allowing the user to receive appropriate feedback and instructions in real time. The input is the generated voice data, and the output is the played voice.

[1696] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1697] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1698] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1699] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1700] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1701] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1702] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1703] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1704] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1705] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1706] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1707] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1708] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1709] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1710] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1711] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1712] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1713] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1714] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1715] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1716] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1717] The following is further disclosed regarding the above embodiment.

[1718] (Claim 1)

[1719] To support the learning of children with mild physical or intellectual disabilities,

[1720] means for capturing voice data input by a user;

[1721] means for transmitting the voice data to a server;

[1722] a voice recognition means in the server for converting the voice data into text data;

[1723] means for analyzing the text data and identifying the user's intention;

[1724] How to select appropriate feedback and materials;

[1725] means for transmitting text data of the selected feedback and educational materials to a terminal;

[1726] a voice synthesis means for converting the text data into voice data;

[1727] means for playing the audio data to a user;

[1728] A system including:

[1729] (Claim 2)

[1730] The system of claim 1, wherein the speech recognition means utilizes generative AI.

[1731] (Claim 3)

[1732] The system of claim 1, wherein the speech synthesis means utilizes a generative AI.

[1733] "Example 1"

[1734] (Claim 1)

[1735] To support the learning of children with mild physical or intellectual disabilities,

[1736] means for capturing voice data input by a user;

[1737] means for transmitting the voice data to a server;

[1738] a voice recognition means in the server for converting the voice data into text data;

[1739] means for analyzing the text data and identifying the user's intention;

[1740] How to select appropriate feedback and materials;

[1741] means for transmitting text data of the selected feedback and educational materials to a terminal;

[1742] a voice synthesis means for converting the text data into voice data;

[1743] means for playing the audio data to a user;

[1744] A system including:

[1745] (Claim 2)

[1746] 10. The system of claim 1, wherein said speech recognition means utilizes generative artificial intelligence.

[1747] (Claim 3)

[1748] The system of claim 1, wherein said speech synthesis means utilizes generative artificial intelligence.

[1749] "Application Example 1"

[1750] (Claim 1)

[1751] To support the learning of children with mild physical or intellectual disabilities,

[1752] means for capturing voice data input by a user;

[1753] means for transmitting the voice data to a server;

[1754] a voice recognition means in the server for converting the voice data into text data;

[1755] means for analyzing the text data and identifying the user's intention;

[1756] How to select appropriate feedback and materials;

[1757] means for transmitting text data of the selected feedback and educational materials to a terminal;

[1758] a voice synthesis means for converting the text data into voice data;

[1759] means for playing the audio data to a user;

[1760] A system that includes a means of capturing the voices of workers in a factory and providing appropriate feedback on work procedures and points to note.

[1761] (Claim 2)

[1762] The system of claim 1, wherein the speech recognition means utilizes generative AI.

[1763] (Claim 3)

[1764] The system of claim 1, wherein the speech synthesis means utilizes a generative AI.

[1765] "Example 2: Combining Emotion Engines"

[1766] (Claim 1)

[1767] To support the learning of children with mild disabilities,

[1768] means for capturing voice data input by a user;

[1769] means for transmitting the audio data to a relay device;

[1770] a voice recognition means for converting the voice data into text data;

[1771] means for analyzing the text data and identifying the user's intention;

[1772] means for analyzing the audio or text data to identify a user's emotional state;

[1773] means for selecting appropriate feedback and educational materials based on the user's intentions and emotional state;

[1774] means for transmitting text data of the selected feedback and educational materials to a terminal;

[1775] a voice synthesis means for converting the text data into voice data;

[1776] means for playing the audio data to a user;

[1777] A system including:

[1778] (Claim 2)

[1779] The system of claim 1, wherein the speech recognition means utilizes generative AI.

[1780] (Claim 3)

[1781] The system of claim 1, wherein the speech synthesis means utilizes a generative AI.

[1782] "Application example 2 when combining emotion engines"

[1783] (Claim 1)

[1784] To support the learning of children with mild physical or intellectual disabilities,

[1785] means for capturing voice data input by a user;

[1786] means for transmitting the voice data to a server;

[1787] a voice recognition means in the server for converting the voice data into text data;

[1788] means for analyzing the text data and identifying the user's intention;

[1789] How to select appropriate feedback and materials;

[1790] means for transmitting text data of the selected feedback and educational materials to a terminal;

[1791] a voice synthesis means for converting the text data into voice data;

[1792] means for playing the audio data to a user;

[1793] emotion analysis means for analyzing the voice data or text data to identify the user's emotional state;

[1794] means for adjusting feedback and educational materials based on said emotional state;

[1795] A system including:

[1796] (Claim 2)

[1797] The system of claim 1, wherein the speech recognition means utilizes generative AI.

[1798] (Claim 3)

[1799] The system of claim 1, wherein the speech synthesis means utilizes a generative AI. [Explanation of symbols]

[1800] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. To support the learning of children with mild physical or intellectual disabilities, means for capturing voice data input by a user; means for transmitting the voice data to a server; a voice recognition means in the server for converting the voice data into text data; means for analyzing the text data and identifying the user's intention; How to select appropriate feedback and materials; means for transmitting text data of the selected feedback and educational materials to a terminal; a voice synthesis means for converting the text data into voice data; means for playing the audio data to a user; A system including:

2. 2. The system of claim 1, wherein said speech recognition means utilizes generative AI.

3. 2. The system of claim 1, wherein said speech synthesis means utilizes a generative AI.

Images