System

Abstract

A system is provided.SOLUTION: The system includes a voice recognition means, a voice-to-text conversion means, a text-to-voice conversion means, a real-time feedback providing means, a visual learning content generation means, and a learning progress tracking means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While it is widely known that learning using speech recognition and visual elements is effective in language learning, there is a lack of an environment that integrates these elements and provides them in real time. Furthermore, there is no system that allows learners to check their own speech and receive appropriate feedback immediately, making it difficult to grasp their learning progress and level of understanding. Therefore, there is a need for the development of a comprehensive language learning system that includes speech-to-text conversion and visual learning support. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes the following elements: a means for converting a user's speech into text in real time using speech recognition means and providing accurate feedback based on the content of that text; a means for converting text to speech that also provides audio feedback to the user; a means for generating visual learning content related to the text entered by the user, thereby supporting effective learning; and a means for tracking learning progress and evaluating and recording the learner's level of understanding. This allows learners to understand their own learning situation in real time and effectively acquire a language.

[0006] "Speech recognition means" refers to a technology that captures the user's speech in real time and converts it into a digital signal.

[0007] "Speech-to-text conversion means" refers to technology that analyzes captured speech signals and converts them into corresponding text data.

[0008] "Text-to-speech conversion means" refers to technology that converts text data into an audio signal and plays it back as audio.

[0009] "Means for providing feedback in real time" refers to a function that instantly generates evaluations and advice based on user input and speech and provides them to the user.

[0010] "Visual learning content generation means" refers to technology that generates related images and visual materials based on text entered by a user.

[0011] "Learning progress tracking means" refers to a function that analyzes a user's learning activities and level of understanding, and records and evaluates progress information. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. Each process of this system is explained in natural language below.

[0034] User registration and account creation

[0035] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[0036] Speech recognition and text conversion

[0037] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[0038] Providing real-time feedback

[0039] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which displays the feedback to the user. Audio feedback is also provided if necessary.

[0040] Visual learning aids

[0041] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[0042] Learning progress and tracking

[0043] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[0044] Specific examples

[0045] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[0046] 1. The user speaks "apple" into the smartphone microphone.

[0047] 2. The device captures the audio and sends it to the server.

[0048] 3. The server converts the speech to text and displays "apple" to the user.

[0049] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[0050] 5. The user types "apple" into the text input field and requests related images.

[0051] 6. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[0052] 7. The generated image is sent to the device and presented to the user.

[0053] This series of processes enhances the user's pronunciation practice and visual comprehension. In this way, the mode for carrying out the invention is embodied.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The user downloads and installs the app on their smartphone.

[0057] Step 2:

[0058] The user launches the app and goes to the new registration screen.

[0059] Step 3:

[0060] The user enters information such as name, email address, password, and initial English conversation level.

[0061] Step 4:

[0062] The terminal transmits the input information to the server.

[0063] Step 5:

[0064] The server receives the transmitted information and stores it in a database.

[0065] Step 6:

[0066] The server generates a confirmation message for completing the registration and sends it to the terminal.

[0067] Step 7:

[0068] The terminal displays a confirmation message to the user.

[0069] Step 8:

[0070] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[0071] Step 9:

[0072] The user begins speaking into the smartphone's microphone.

[0073] Step 10:

[0074] The device captures audio data input from the microphone.

[0075] Step 11:

[0076] The device transmits the captured audio data to the server in real time.

[0077] Step 12:

[0078] The server receives the voice data and converts it into text using voice recognition technology.

[0079] Step 13:

[0080] The server transmits the converted text data to the terminal.

[0081] Step 14:

[0082] The terminal displays the text data to the user.

[0083] Step 15:

[0084] The server analyzes the converted text data.

[0085] Step 16:

[0086] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[0087] Step 17:

[0088] The server transmits the generated feedback data to the terminal.

[0089] Step 18:

[0090] The device displays feedback to the user, and optionally provides audio feedback.

[0091] Step 19:

[0092] The user enters the word or phrase they want to learn into the text entry field.

[0093] Step 20:

[0094] The terminal transmits the input text data to the server.

[0095] Step 21:

[0096] A server receives the text data and generates related images using a visual learning content generator.

[0097] Step 22:

[0098] The server transmits the generated image data to the terminal.

[0099] Step 23:

[0100] The terminal displays the image to the user.

[0101] Step 24:

[0102] The server stores the user's voice utterance history and feedback data in a database.

[0103] Step 25:

[0104] The server continuously tracks the user's learning progress based on the data stored on the server.

[0105] Step 26:

[0106] The server evaluates the learning progress and generates a new learning plan.

[0107] Step 27:

[0108] The server sends the new lesson plan to the device.

[0109] Step 28:

[0110] The device will display your learning progress and new learning plan on a dashboard.

[0111] This series of steps creates a system that supports effective language learning.

[0112] Example 1

[0113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Conventional speech recognition systems and language learning applications have difficulty accurately converting user speech into text and providing appropriate feedback. Furthermore, they lack sufficient visual learning support and make it difficult to accurately track learning progress. This makes it difficult for users to effectively acquire a language.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0116] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a terminal means for transmitting voice-input data, and a server means for analyzing the data and generating results, thereby enabling the server to accurately convert a user's speech into text, provide accurate feedback, provide visual learning support, and accurately track learning progress.

[0117] "Speech recognition means" refers to means for detecting the user's voice and processing it as digital data.

[0118] The "speech-to-text conversion means" is a means for converting detected speech data into text data.

[0119] "Text to speech conversion means" refers to means for converting text data into speech data.

[0120] A "real-time feedback providing means" is a means for generating and providing instant feedback based on voice or text input by a user.

[0121] A "visual learning content generation means" is a means for generating relevant visual learning content (eg, images and graphics) based on text data.

[0122] The "learning progress tracking means" is a means for continuously recording and tracking a user's learning activities and progress.

[0123] The "terminal means for transmitting voice-input data" refers to a device for a user to input voice and means for transmitting that data.

[0124] The "server means for analyzing the data and generating a result" is a server for analyzing the transmitted data and generating a result.

[0125] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. A specific embodiment of the present invention is described in detail below.

[0126] System Overview

[0127] This invention is a system that allows users to input voice using a device such as a smartphone and receive real-time feedback. The server converts the voice data into text, provides related visual learning content, and tracks the user's learning progress.

[0128] Hardware and software used

[0129] Device: A mobile device such as a smartphone or tablet

[0130] Speech recognition technology: Speech recognition services such as Google® Cloud Speech-to-Text API

[0131] Text-to-speech technology: Synthetic speech technology (e.g., Amazon Polly) for text-to-speech conversion

[0132] Visual learning content generation means: Image generation algorithms such as DALL-E 2

[0133] Database: A server-side database system for recording user data and learning progress.

[0134] User registration and account creation

[0135] The user downloads and installs the app onto a device such as a smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. A confirmation message confirming registration completion is sent from the server to the device, which displays it to the user.

[0136] Speech recognition and text conversion

[0137] The user taps the "Conversation Practice" button on the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is sent from the server to the device, which then displays it to the user.

[0138] Providing real-time feedback

[0139] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback can also be provided simultaneously.

[0140] Visual learning aids

[0141] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using DALL-E 2. The generated image data is sent to the device, which displays it to the user.

[0142] Learning progress and tracking

[0143] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[0144] Specific examples

[0145] For example, the specific process for practicing pronunciation of the word "apple" and obtaining visually related images is shown below.

[0146] 1. The user speaks "apple" into the smartphone microphone.

[0147] 2. The device captures the audio and sends it to the server.

[0148] 3. The server converts the speech to text and displays "apple" to the user.

[0149] 4. The server evaluates the accuracy of the pronunciation and generates and displays feedback such as "Your pronunciation is correct!"

[0150] 5. The user types "apple" into the text input field and requests related images.

[0151] 6. The device sends the text to the server, and the server generates an image of "apple" using DALL-E 2.

[0152] 7. The generated image is sent to the device and presented to the user.

[0153] This process enhances the user's pronunciation practice and visual comprehension.

[0154] Prompt Sentence Examples

[0155] How can a user speak the word "apple" into a smartphone microphone, receive real-time feedback, and see related visual images?

[0156] The above is a detailed description of the preferred embodiment of the present invention, which allows users to learn languages ​​more effectively.

[0157] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0158] Program processing flow

[0159] User registration and account creation

[0160] Step 1:

[0161] The user downloads and installs the app on their smartphone.

[0162] Enter: app download

[0163] Output: Installed apps

[0164] Step 2:

[0165] The user enters information such as name, email address, password, and English conversation level on the new registration screen.

[0166] Input: Name, email address, password, English conversation level

[0167] Output: Input data

[0168] Step 3:

[0169] The terminal transmits the input information to the server.

[0170] Input: Input data

[0171] Output: Server sent data

[0172] Step 4:

[0173] The server stores the received information in a database.

[0174] Input: Server transmission data

[0175] Output: Database update

[0176] Step 5:

[0177] The server sends a confirmation message to the terminal confirming the completion of registration.

[0178] Input: None

[0179] Output:Confirmation message

[0180] Step 6:

[0181] The terminal displays a confirmation message to the user.

[0182] Input:Confirmation message

[0183] Output: Registration complete

[0184] Speech recognition and text conversion

[0185] Step 1:

[0186] The user taps the "Practice Conversation" button within the app.

[0187] Input: Tap a button

[0188] Output: Start speech mode

[0189] Step 2:

[0190] The user begins speaking into the smartphone's microphone.

[0191] Input: Speech content

[0192] Output: Audio data

[0193] Step 3:

[0194] The device captures audio data input from the microphone.

[0195] Input: Audio data

[0196] Output: Captured audio data

[0197] Step 4:

[0198] The device transmits the voice data to the server in real time.

[0199] Input: Captured audio data

[0200] Output: Server sent data

[0201] Step 5:

[0202] The server receives the voice data and converts it into text using speech recognition technology.

[0203] Input: Server transmission data

[0204] Output: Converted text data

[0205] Step 6:

[0206] The server transmits the converted text data to the terminal.

[0207] Input: Converted text data

[0208] Output: Terminal sending data

[0209] Step 7:

[0210] The terminal displays the text data to the user.

[0211] Input: Terminal transmission data

[0212] Output: Text display

[0213] Providing real-time feedback

[0214] Step 1:

[0215] The server immediately analyzes the converted text data.

[0216] Input: Converted text data

[0217] Output: Feedback data

[0218] Step 2:

[0219] The server generates feedback based on what the user says.

[0220] Input: Feedback data

[0221] Output: Generated feedback

[0222] Step 3:

[0223] The server transmits the generated feedback data to the terminal.

[0224] Input: Generated feedback

[0225] Output: Terminal sending data

[0226] Step 4:

[0227] The device displays the feedback to the user.

[0228] Input: Terminal transmission data

[0229] Output: Feedback display

[0230] Step 5:

[0231] The device will also provide audio feedback if necessary.

[0232] Input: Feedback data

[0233] Output: Audio feedback

[0234] Visual learning aids

[0235] Step 1:

[0236] The user enters the word or phrase they want to learn into the text entry field.

[0237] Input: Word or phrase

[0238] Output: Input text

[0239] Step 2:

[0240] The terminal transmits the input text data to the server.

[0241] Input: Input text

[0242] Output: Server sent data

[0243] Step 3:

[0244] A server receives the text data and generates an associated image.

[0245] Input: Server transmission data

[0246] Output: Generated image data

[0247] Step 4:

[0248] The server transmits the generated image data to the terminal.

[0249] Input: Generated image data

[0250] Output: Terminal sending data

[0251] Step 5:

[0252] The terminal displays the image to the user.

[0253] Input: Terminal transmission data

[0254] Output: Image display

[0255] Learning progress and tracking

[0256] Step 1:

[0257] The server stores the user's voice utterance history and feedback data in a database.

[0258] Input: Feedback data and speech history

[0259] Output: Database update

[0260] Step 2:

[0261] The server continuously tracks the user's learning activities and understanding.

[0262] Input: Database data

[0263] Output:Tracking information

[0264] Step 3:

[0265] The server evaluates the user's learning progress based on the data stored.

[0266] Input:Tracking information

[0267] Output: Evaluation results

[0268] Step 4:

[0269] The server generates a new learning plan and sends it to the device.

[0270] Input: Evaluation result

[0271] Output: The generated learning plan

[0272] Step 5:

[0273] The device will present the user with their new learning plan and progress.

[0274] Input: Generated lesson plan

[0275] Output: Plan display

[0276] The above are the specific processing steps of the system program.

[0277] (Application example 1)

[0278] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0279] Conventional virtual store systems have difficulty providing product explanations and purchasing support when users speak different languages. There has also been a demand for providing detailed product information in real time, multilingual support, and comprehensive purchasing support. Providing services to international customers without creating a language barrier has been particularly challenging, making improving the user experience a key challenge.

[0280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0281] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, a multilingual translation unit, and an answer generation unit using a generative AI model. This enables real-time multilingual product explanations and purchasing support in the virtual store, making it possible to provide services to international customers without language barriers.

[0282] A "voice recognition means" is a device or software that has the function of receiving voice input, analyzing the voice, and converting it into text data.

[0283] A "speech-to-text converter" is a system or algorithm that performs a process to convert speech data into text data.

[0284] A "text-to-speech converter" is a system or algorithm that performs a process to convert text data into speech data.

[0285] The "means for providing real-time feedback" refers to a device or software that has the function of generating immediate feedback in response to a user's input and providing it to the user.

[0286] A "visual learning content generator" is a device or software that has the functionality to generate visual learning materials based on text data or other input data.

[0287] A "learning progress tracking means" is a device or software that has the function of continuously recording and evaluating a user's learning activities and progress.

[0288] A "multilingual translation means" is a device or software that has the function of translating text and audio data between a wide variety of languages.

[0289] An "answer generation means using a generative AI model" is a system or algorithm that uses artificial intelligence technology to generate answers to users' questions.

[0290] The "means for providing purchasing support functions" refers to a device or software that has the function of allowing users to obtain information about products and services and assisting them in the purchasing process.

[0291] This invention relates to a system that provides real-time multilingual product explanations and purchasing support in a virtual store. The system is realized by communication between users and a server via an application installed on a mobile device such as a smartphone or smart glasses.

[0292] Voice input and recognition

[0293] 1. Terminal processing:

[0294] The user speaks a question about the product into the device's microphone.

[0295] The device captures the audio data and sends it to the server.

[0296] 2. Server processing:

[0297] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text.

[0298] Get product information

[0299] 3. Server processing:

[0300] The text data generated by speech recognition is used to query the product database using Django or Flask.

[0301] Retrieve relevant product information and translate it into the user's native language.

[0302] Multilingual translation and answer generation

[0303] 4. Server processing:

[0304] Use the Google Cloud Translation API to translate user questions into different languages.

[0305] The translated text is fed into a generative AI model (e.g., GPT-3®) to generate a detailed answer.

[0306] The generated answer is translated back into the user's language.

[0307] Visual Content Generation

[0308] 5. Server Processing:

[0309] Based on the transferred product information, relevant visual learning content (e.g., images) is generated using OpenCV and matplotlib.

[0310] 6. Terminal Processing:

[0311] The generated image sent from the server is displayed to the user.

[0312] Configured example

[0313] For example, if a user says "What are the features of this smartphone case?" in a virtual store, the process will proceed as follows:

[0314] 1. The user speaks into the smartphone microphone, "Please tell me the features of this smartphone cover."

[0315] 2. The device captures the audio and sends it to the server.

[0316] 3. The server converts the speech into text and queries the product database to obtain "information about the smartphone cover."

[0317] 4. Translate the question into English via a translation API and provide the prompt "What are the features of this smartphone case?" to GPT-3.

[0318] 5. The server receives the answer generated by GPT-3 and retranslates it into the user's language.

[0319] 6. Display the generated text and related images on your smartphone.

[0320] Prompt Sentence Examples

[0321] "Please tell me the features of this smartphone case."

[0322] This will enable users to access detailed product information in real time in multiple languages ​​in the virtual store, making it possible to provide services to international customers without causing them language barriers.

[0323] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0324] Step 1:

[0325] The user speaks a question about a product into the microphone of their smartphone in the virtual store. This voice is the input data. The device recognizes the user's speech and captures it as digital voice data.

[0326] Step 2:

[0327] The device sends the captured audio data to the server. At this point, the input is audio data, and the output is the transmission of audio data to the server. The smartphone's communication module is used.

[0328] Step 3:

[0329] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process results in voice data as input and text data as output. Specifically, the server sends the voice data to the API and analyzes the returned text data.

[0330] Step 4:

[0331] The server uses the text data to query the product database. The input is the converted text data, and the output is the product information. Using Django or Flask, the corresponding information is retrieved from the product database.

[0332] Step 5:

[0333] The server uses the Google Cloud Translation API to translate the user's question from the user's native language into a target language such as English. The input is text data, and the output is the translated text. Specifically, the server sends the text to be translated and the target language to the API, and receives the translation result.

[0334] Step 6:

[0335] The server uses a generative AI model (e.g., GPT-3) to generate an answer to the question based on the translated text. The translated question is input to the AI ​​model as a prompt. The input is the translated text, and the output is the generated answer text. The server sends the prompt to the generative AI model and receives the returned answer.

[0336] Step 7:

[0337] The server translates the generated answer into the user's native language using the Google Cloud Translation API again. The input is the generated answer text, and the output is the translated answer text into the user's native language. The translation procedure is the same as in step 5.

[0338] Step 8:

[0339] The server generates relevant visual learning content. The input is product data, and the output is the generated visual content (e.g., images). Specifically, it uses OpenCV and matplotlib to generate visual information related to the product.

[0340] Step 9:

[0341] The server sends the generated text and image data to the terminal. The input is the answer text and visual content, and the output is a status indicating completion of transmission to the terminal. The server sends the data using a communication protocol.

[0342] Step 10:

[0343] The terminal displays the received text and visual content to the user. The input is the data sent from the server, and the output is the content displayed on the terminal display. In concrete terms, the terminal reflects the data on the display and provides the user with visual and text information.

[0344] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0345] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, learning progress tracking, and user emotion recognition. The system recognizes emotions from the user's speech and input text and provides feedback based on these emotions, enabling a more effective learning experience.

[0346] User registration and account creation

[0347] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[0348] Speech recognition and text conversion

[0349] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[0350] Providing real-time feedback

[0351] The server immediately analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The server also adjusts the content and tone of the feedback using an emotion engine that recognizes emotions from the user's speech and input text. The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback is also provided if necessary.

[0352] Visual learning aids

[0353] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[0354] Learning progress and tracking

[0355] The server stores the user's speech history and feedback data in a database, and continuously tracks the user's learning activities and comprehension. Furthermore, based on the user's emotional state recognized by the emotion engine, it influences the tracking results of the learning progress and adjusts the learning plan according to the specific emotional state. This new learning plan is sent to the device, which then presents it to the user along with the learning progress.

[0356] Specific examples

[0357] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[0358] 1. The user speaks "apple" into the smartphone microphone.

[0359] 2. The device captures the audio and sends it to the server.

[0360] 3. The server converts the speech to text and displays "apple" to the user.

[0361] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[0362] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[0363] 6. The user types "apple" into the text input field and requests related images.

[0364] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[0365] 8. The generated image is sent to the terminal and presented to the user.

[0366] This series of processes enhances the user's pronunciation practice and visual understanding, and the emotion engine provides personalized feedback, improving the learner's motivation and creating an appropriate learning plan.

[0367] The processing flow will be explained below.

[0368] Step 1:

[0369] The user downloads and installs the app on their smartphone.

[0370] Step 2:

[0371] The user launches the app and goes to the new registration screen.

[0372] Step 3:

[0373] The user enters information such as name, email address, password, and initial English conversation level.

[0374] Step 4:

[0375] The terminal transmits the input information to the server.

[0376] Step 5:

[0377] The server receives the transmitted information and stores it in a database.

[0378] Step 6:

[0379] The server generates a confirmation message for completing the registration and sends it to the terminal.

[0380] Step 7:

[0381] The terminal displays a confirmation message to the user.

[0382] Step 8:

[0383] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[0384] Step 9:

[0385] The user begins speaking into the smartphone's microphone.

[0386] Step 10:

[0387] The device captures audio data input from the microphone.

[0388] Step 11:

[0389] The device transmits the captured audio data to the server in real time.

[0390] Step 12:

[0391] The server receives the voice data and converts it into text using voice recognition technology.

[0392] Step 13:

[0393] The server transmits the converted text data to the terminal.

[0394] Step 14:

[0395] The terminal displays the text data to the user.

[0396] Step 15:

[0397] The server analyzes the converted text data.

[0398] Step 16:

[0399] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[0400] Step 17:

[0401] The server uses an emotion engine that recognizes emotions from the user's speech and input text to adjust the content and tone of the feedback.

[0402] Step 18:

[0403] The server transmits the generated feedback data to the terminal.

[0404] Step 19:

[0405] The device displays feedback to the user, and optionally provides audio feedback.

[0406] Step 20:

[0407] The user enters the word or phrase they want to learn into the text entry field.

[0408] Step 21:

[0409] The terminal transmits the input text data to the server.

[0410] Step 22:

[0411] A server receives the text data and generates related images using a visual learning content generator.

[0412] Step 23:

[0413] The server transmits the generated image data to the terminal.

[0414] Step 24:

[0415] The terminal displays the image to the user.

[0416] Step 25:

[0417] The server stores the user's voice utterance history and feedback data in a database.

[0418] Step 26:

[0419] The server continuously tracks the user's learning progress based on the data stored on the server.

[0420] Step 27:

[0421] The server influences the tracking results of the learning progress based on the emotional state of the user recognized by the emotion engine.

[0422] Step 28:

[0423] The server adjusts the learning plan according to the particular emotional state and generates a new learning plan.

[0424] Step 29:

[0425] The server sends the new lesson plan to the device.

[0426] Step 30:

[0427] The device will present the user with their learning progress and new learning plan.

[0428] This series of steps enhances the user's pronunciation practice and visual comprehension, while the emotion engine provides personalized feedback to improve learner motivation and create appropriate learning plans.

[0429] Example 2

[0430] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0431] Previous language learning systems provided limited feedback on user utterances, and it was particularly difficult to provide feedback that took into account the user's emotional state. Furthermore, learning progress tracking was inconsistent, making it difficult to provide individually optimized learning plans. This resulted in a lack of learner motivation and an ineffective learning experience.

[0432] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0433] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, and an emotion recognition unit. This makes it possible to provide feedback based not only on the user's spoken content but also on the user's emotional state. Furthermore, by continuously tracking the learning progress and generating and providing an individually optimized learning plan, it is possible to increase the learner's motivation and achieve an effective learning experience.

[0434] "Speech recognition means" is a technology that captures a user's speech, analyzes the speech data, and converts it into linguistic text.

[0435] A "voice-to-text converter" is a function that converts captured voice data into a text format in a language.

[0436] "Text-to-speech conversion means" refers to a technology that generates text-format language data as speech output.

[0437] The "means for providing real-time feedback" is a function that provides immediate evaluation and feedback on corrections based on the user's utterances.

[0438] "Visual learning content generation means" is a technology that generates and visually displays related images and diagrams based on text entered by the user.

[0439] A "learning progress tracking means" is a system that continuously tracks and records a user's learning activities and progress.

[0440] "Emotion recognition means" is a technology that analyzes emotions from a user's speech or input text and recognizes their emotional state.

[0441] The present invention is implemented as a smartphone application for language learners, whose main processes include speech recognition, speech-to-text conversion, text-to-speech conversion, real-time feedback provision, visual learning content generation, learning progress tracking, and emotion recognition.

[0442] Hardware and software used

[0443] Hardware:

[0444] Smartphone (microphone, display)

[0445] software:

[0446] Speech recognition engine (Google Speech-to-Text API)

[0447] Emotional Recognition Engine (IBM Watson® Emotional Recognition)

[0448] Database (MySQL (registered trademark))

[0449] Application server (Node.js)

[0450] Learning content generation tool (TENSORFLOW (registered trademark))

[0451] Specific processing of the program

[0452] User registration and account creation

[0453] Users download and install the app on their smartphone. After installation, they launch the app and enter information such as their name, email address, password, and English conversation level. The device collects this information and sends it to the server. The server stores it in a database and sends a message to the device confirming registration.

[0454] Speech recognition and text conversion

[0455] The user selects "Conversation Practice" and begins speaking. The device captures the audio with the microphone and sends the audio data to the server. The server converts the audio into text using the Google Speech-to-Text API and sends the converted text to the device for display.

[0456] Providing real-time feedback

[0457] The server analyzes the converted text and generates feedback on pronunciation accuracy, grammatical compatibility, and appropriate expression examples. Furthermore, the IBM Watson Emotional Recognition engine recognizes the user's emotions and adjusts the content and tone of the feedback accordingly. Feedback data is sent to the device, where it is displayed and provided to the user via voice.

[0458] Visual learning aids

[0459] When a user inputs a word or phrase they want to learn, the device sends the text data to the server, which then uses TensorFlow to generate a related image, which is then sent to the device and displayed to the user.

[0460] Learning progress and tracking

[0461] The server stores the user's speech history and feedback data in a database, tracks the user's learning progress, and adjusts the learning plan based on the emotion recognition results, sending the plan to the device and presenting it to the user.

[0462] Specific examples

[0463] For example, if a user says the word "apple" and wants to get a visually related image:

[0464] 1. The user speaks "apple" into the smartphone microphone.

[0465] 2. The device captures the audio and sends it to the server.

[0466] 3. The server converts the speech to text and displays "apple."

[0467] 4. The server evaluates the accuracy of the pronunciation and generates and sends feedback saying, "Your pronunciation is correct!"

[0468] 5. The server uses the emotion engine to recognize the user's confidence and generates motivating feedback such as, "That's great! Keep it up!"

[0469] 6. The user types "apple" into the text input field and requests related images.

[0470] 7. The server generates an image of an "apple" using the visual learning content generation means.

[0471] 8. The generated image is sent to the terminal and presented to the user.

[0472] Example prompts for generative AI models

[0473] Generate related images for visual understanding of the word "apple." The system converts the audio data of the user pronouncing "apple" into text, provides feedback on the accuracy of the pronunciation, and then analyzes the user's sentiment (confidence) to provide appropriate feedback. Next, the user types "apple" into a text field to request related visual images.

[0474] This prompt was generated based on the specific operations described above and is formatted to clearly communicate the expected behavior of the AI ​​model.

[0475] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0476] Step 1:

[0477] The user downloads and installs the app on their smartphone.

[0478] Input: App download link

[0479] Output: Installed apps

[0480] Step 2:

[0481] The user launches the app and enters information such as their name, email address, password, and English conversation level on the new registration screen.

[0482] Input: User's personal information (name, email address, password, English conversation level)

[0483] Output: User information entered

[0484] Step 3:

[0485] The terminal transmits the user's input information to the server.

[0486] Input: User information

[0487] Output: A request to send data to the server

[0488] Step 4:

[0489] The server receives the transmitted user information and stores it in a database.

[0490] Input: User information

[0491] Output: A database containing user information

[0492] Step 5:

[0493] The server generates a confirmation message for completing the registration and sends it to the terminal.

[0494] Input: None (default confirmation message)

[0495] Output:Confirmation message

[0496] Step 6:

[0497] The terminal receives the confirmation message and displays it to the user.

[0498] Input:Confirmation message

[0499] Output: What is displayed to the user

[0500] Step 7:

[0501] The user taps the "Practice Conversation" button to enter speech mode.

[0502] Input: Button tap event

[0503] Output: Speech mode

[0504] Step 8:

[0505] The user begins speaking into the smartphone's microphone.

[0506] Input: Audio data

[0507] Output: Audio signal

[0508] Step 9:

[0509] The device captures audio data input from the microphone and transmits it to the server in real time.

[0510] Input: Audio signal

[0511] Output: Request to send audio data to the server

[0512] Step 10:

[0513] The server receives the voice data and converts it into text using a voice recognition engine.

[0514] Input: Audio data

[0515] Output: Text data

[0516] Step 11:

[0517] The server transmits the converted text data to the terminal.

[0518] Input: Text data

[0519] Output: Request to send text data to the terminal

[0520] Step 12:

[0521] The terminal receives the text data and displays it to the user.

[0522] Input: Text data

[0523] Output: What is displayed to the user

[0524] Step 13:

[0525] The server analyzes the text data and generates feedback including pronunciation accuracy, grammatical correctness, and appropriate expression.

[0526] Input: Text data

[0527] Output: Feedback message

[0528] Step 14:

[0529] The server uses an emotion recognition engine to recognize the user's emotions and adjust the content and tone of the feedback.

[0530] Input: Text data (speech content), voice data (speech tone)

[0531] Output: Adjusted feedback message

[0532] Step 15:

[0533] The server transmits the generated feedback data to the terminal.

[0534] Input: Feedback message

[0535] Output: Request to send feedback to device

[0536] Step 16:

[0537] The terminal receives the feedback and provides visual and audio feedback to the user.

[0538] Input: Feedback message

[0539] Output: Visual feedback / Audio feedback

[0540] Step 17:

[0541] Users enter the words or phrases they want to learn as text.

[0542] Input: Word or phrase

[0543] Output: Text data

[0544] Step 18:

[0545] The terminal transmits the text data to the server.

[0546] Input: Text data

[0547] Output: Request to send text data to the server

[0548] Step 19:

[0549] A server receives the text data and generates related images using a visual learning content generation tool.

[0550] Input: Text data

[0551] Output: Image data

[0552] Step 20:

[0553] The server transmits the generated image data to the terminal.

[0554] Input: Image data

[0555] Output: Image data transmission request to the device

[0556] Step 21:

[0557] The terminal receives the image data and presents it to the user.

[0558] Input: Image data

[0559] Output: Display image

[0560] Step 22:

[0561] The server stores the user's speech history and feedback data in a database and tracks their learning progress.

[0562] Input: Speech history, feedback data

[0563] Output: Updated database

[0564] Step 23:

[0565] The server adjusts the learning plan based on the emotion recognition results and generates a new learning plan.

[0566] Input: Emotion recognition results, learning progress data

[0567] Output: New lesson plan

[0568] Step 24:

[0569] The server sends the new lesson plan to the device.

[0570] Input: New lesson plan

[0571] Output: Request to send lesson plan to device

[0572] Step 25:

[0573] The terminal receives the study plan and presents it to the user.

[0574] Input: New lesson plan

[0575] Output: Display of the learning plan

[0576] The above is the specific processing flow of this system.

[0577] (Application example 2)

[0578] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0579] In language learning, conventional systems do not provide feedback that takes users' emotions into account, which can lead to a decrease in learner motivation. They also lack a means to effectively track learning progress and provide individually optimized learning plans. Furthermore, when providing an interactive learning experience, learners face challenges due to the lack of interactive story-based dialogue, which can lead to a lack of immersion.

[0580] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0581] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a user emotion recognition means, an emotion-based feedback adjusting means, and a story-style dialogue content providing means, thereby enabling feedback based on the user's emotions and immersive learning through interactive story-style dialogue content.

[0582] Word definition

[0583] "Speech recognition means" is a function that receives voice data as input and converts it into text data.

[0584] The "voice-to-text conversion means" is a function that analyzes the voice data captured by the voice recognition means and converts it into corresponding text data.

[0585] The "text to speech conversion means" is a function that receives text data as input and outputs the content as speech data.

[0586] The "means for providing feedback in real time" is a function that instantly generates appropriate feedback based on the user's input and provides it to the user.

[0587] The "visual learning content generation means" is a function that generates visual content such as related images and videos based on the text and audio input by the user.

[0588] "Learning progress tracking means" is a function that records a user's learning activities and manages and evaluates their progress.

[0589] The "user emotion recognition means" is a function that analyzes emotions from the user's speech and input text.

[0590] The "feedback adjustment means based on emotion" is a function that adjusts the content and tone of the feedback provided based on the user's emotion recognized by the emotion recognition means.

[0591] The "means for providing narrative dialogue content" is a function for generating and providing interactive dialogue in the form of a story.

[0592] MODE FOR CARRYING OUT THE INVENTION

[0593] This invention is a language learning system that recognizes a user's speech in real time, converts it into text, and provides feedback. It also recognizes the user's emotions and provides feedback based on those emotions, making the learning experience more effective.

[0594] The system mainly consists of the following means:

[0595] 1. Voice recognition means:

[0596] The server captures the user's voice and uses a speech recognition algorithm to convert it into text, using tools such as the Google Speech Recognition API.

[0597] 2. Voice to text conversion methods:

[0598] The server converts the voice data received from the voice recognition means into text data in real time.

[0599] 3. Text to speech conversion methods:

[0600] The server uses Text-to-Speech (TTS) technology, such as the Google Text-to-Speech API, to provide audio feedback of the generated text.

[0601] 4. How to provide real-time feedback:

[0602] The server analyzes the converted text data, evaluates pronunciation accuracy and grammatical correctness, and generates feedback that is provided to the user instantly.

[0603] 5. Visual learning content generation methods:

[0604] The server generates relevant visual content (images and videos) based on the text entered by the user. For example, if a user enters that they want to learn the meaning of "apple," the server generates an image of "apple" and sends it to the device.

[0605] 6. Track your progress:

[0606] The server tracks the user's learning progress and stores the user's speech history and feedback data in a database, allowing the user to check their own learning progress.

[0607] 7. User Emotion Recognition Method:

[0608] The server has an emotion engine that analyzes emotions from user utterances and input text, enabling feedback based on the user's emotions.

[0609] 8. Emotion-based feedback regulation measures:

[0610] The server adjusts the content and tone of the feedback based on the emotional information recognized by the emotion engine. For example, if the user is nervous, it provides encouraging feedback.

[0611] 9. Narrative dialogue content delivery methods:

[0612] The server generates interactive story-style dialogue content, allowing users to participate in learning as characters in the story and learn a language through dialogue.

[0613] As a concrete example, consider a user trying to practice pronouncing the word "apple" and get a related visual image:

[0614] 1. The user speaks "apple" into the smartphone microphone.

[0615] 2. The device captures the audio and sends it to the server.

[0616] 3. The server converts the speech to text and displays "apple" to the user.

[0617] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[0618] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[0619] 6. The user types "apple" into the text input field and requests related images.

[0620] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[0621] 8. The generated image is sent to the terminal and presented to the user.

[0622] This series of processes enhances the user's pronunciation practice and visual understanding, while the emotion engine provides individualized feedback, improving learner motivation and enabling appropriate study plans.

[0623] An example of an input prompt is, "As a great knight, you must face a dragon. Are you ready?"

[0624] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0625] Program processing flow

[0626] Step 1:

[0627] The user speaks into the smartphone's microphone. The device captures the voice data from the microphone. The input is the user's voice, and the output is the captured voice data.

[0628] Step 2:

[0629] The device sends the captured audio data to the server. The input is the audio data, and the output is the data sent to the server.

[0630] Step 3:

[0631] The server converts the received voice data into text data using a voice recognition means, with the input being the voice data and the output being the converted text data.

[0632] Step 4:

[0633] The server analyzes the converted text data and generates real-time feedback. Specifically, it evaluates the accuracy of pronunciation and grammar. The input is the text data, and the output is the evaluation result and a feedback message.

[0634] Step 5:

[0635] The server sends text data and feedback messages to the terminal. The input is the evaluation result and the feedback message, and the output is the data sent to the terminal.

[0636] Step 6:

[0637] The terminal displays the received feedback to the user and provides audio feedback if necessary. The input is the data sent from the server, and the output is visual and audio feedback to the user.

[0638] Step 7:

[0639] The server analyzes text data using an emotion recognition engine to recognize the user's emotions. The input is text data, and the output is the emotion analysis result.

[0640] Step 8:

[0641] The server adjusts the content and tone of the feedback based on the results of the emotion analysis. Specifically, it generates encouraging messages to motivate the user. The input is the emotion analysis results, and the output is the adjusted feedback message.

[0642] Step 9:

[0643] The server sends the final feedback message to the terminal. The input is the adjusted feedback message, and the output is the data to be sent to the terminal.

[0644] Step 10:

[0645] A user enters a word or phrase they want to learn into a text input field and requests related visual content. The input is the text entered by the user, and the output is the requested data.

[0646] Step 11:

[0647] The terminal sends request data to the server. The input is the request data, and the output is the data sent to the server.

[0648] Step 12:

[0649] The server receives the request and generates the relevant image using a visual learning content generator, where the input is the user's text data and the output is the generated image data.

[0650] Step 13:

[0651] The generated image data is sent from the server to the terminal. The input is the image data, and the output is the data sent to the terminal.

[0652] Step 14:

[0653] The terminal presents the received image data to the user. The input is the data sent from the server, and the output is visual feedback to the user.

[0654] This process reinforces the user's speech practice and visual comprehension, and the emotion engine provides personalized feedback. For example, the prompt "You must face the dragon as a great knight. Are you ready?" can be used to guide the user through the dialogue.

[0655] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0656] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0657] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0658] [Second embodiment]

[0659] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0660] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0661] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0662] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0663] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0665] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0666] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0667] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0668] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0669] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0670] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0671] This invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. Each process of this system is explained in natural language below.

[0672] User registration and account creation

[0673] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[0674] Speech recognition and text conversion

[0675] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[0676] Providing real-time feedback

[0677] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which displays the feedback to the user. Audio feedback is also provided if necessary.

[0678] Visual learning aids

[0679] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[0680] Learning progress and tracking

[0681] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[0682] Specific examples

[0683] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[0684] 1. The user speaks "apple" into the smartphone microphone.

[0685] 2. The device captures the audio and sends it to the server.

[0686] 3. The server converts the speech to text and displays "apple" to the user.

[0687] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[0688] 5. The user types "apple" into the text input field and requests related images.

[0689] 6. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[0690] 7. The generated image is sent to the device and presented to the user.

[0691] This series of processes enhances the user's pronunciation practice and visual comprehension. In this way, the mode for carrying out the invention is embodied.

[0692] The processing flow will be explained below.

[0693] Step 1:

[0694] The user downloads and installs the app on their smartphone.

[0695] Step 2:

[0696] The user launches the app and goes to the new registration screen.

[0697] Step 3:

[0698] The user enters information such as name, email address, password, and initial English conversation level.

[0699] Step 4:

[0700] The terminal transmits the input information to the server.

[0701] Step 5:

[0702] The server receives the transmitted information and stores it in a database.

[0703] Step 6:

[0704] The server generates a confirmation message for completing the registration and sends it to the terminal.

[0705] Step 7:

[0706] The terminal displays a confirmation message to the user.

[0707] Step 8:

[0708] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[0709] Step 9:

[0710] The user begins speaking into the smartphone's microphone.

[0711] Step 10:

[0712] The device captures audio data input from the microphone.

[0713] Step 11:

[0714] The device transmits the captured audio data to the server in real time.

[0715] Step 12:

[0716] The server receives the voice data and converts it into text using voice recognition technology.

[0717] Step 13:

[0718] The server transmits the converted text data to the terminal.

[0719] Step 14:

[0720] The terminal displays the text data to the user.

[0721] Step 15:

[0722] The server analyzes the converted text data.

[0723] Step 16:

[0724] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[0725] Step 17:

[0726] The server transmits the generated feedback data to the terminal.

[0727] Step 18:

[0728] The device displays feedback to the user, and optionally provides audio feedback.

[0729] Step 19:

[0730] The user enters the word or phrase they want to learn into the text entry field.

[0731] Step 20:

[0732] The terminal transmits the input text data to the server.

[0733] Step 21:

[0734] A server receives the text data and generates related images using a visual learning content generator.

[0735] Step 22:

[0736] The server transmits the generated image data to the terminal.

[0737] Step 23:

[0738] The terminal displays the image to the user.

[0739] Step 24:

[0740] The server stores the user's voice utterance history and feedback data in a database.

[0741] Step 25:

[0742] The server continuously tracks the user's learning progress based on the data stored on the server.

[0743] Step 26:

[0744] The server evaluates the learning progress and generates a new learning plan.

[0745] Step 27:

[0746] The server sends the new lesson plan to the device.

[0747] Step 28:

[0748] The device will display your learning progress and new learning plan on a dashboard.

[0749] This series of steps creates a system that supports effective language learning.

[0750] Example 1

[0751] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Conventional speech recognition systems and language learning applications have difficulty accurately converting user speech into text and providing appropriate feedback. Furthermore, they lack sufficient visual learning support and make it difficult to accurately track learning progress. This makes it difficult for users to effectively acquire a language.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0754] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a terminal means for transmitting voice-input data, and a server means for analyzing the data and generating results, thereby enabling the server to accurately convert a user's speech into text, provide accurate feedback, provide visual learning support, and accurately track learning progress.

[0755] "Speech recognition means" refers to means for detecting the user's voice and processing it as digital data.

[0756] The "speech-to-text conversion means" is a means for converting detected speech data into text data.

[0757] "Text to speech conversion means" refers to means for converting text data into speech data.

[0758] A "real-time feedback providing means" is a means for generating and providing instant feedback based on voice or text input by a user.

[0759] A "visual learning content generation means" is a means for generating relevant visual learning content (eg, images and graphics) based on text data.

[0760] The "learning progress tracking means" is a means for continuously recording and tracking a user's learning activities and progress.

[0761] The "terminal means for transmitting voice-input data" refers to a device for a user to input voice and means for transmitting that data.

[0762] The "server means for analyzing the data and generating a result" is a server for analyzing the transmitted data and generating a result.

[0763] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. A specific embodiment of the present invention is described in detail below.

[0764] System Overview

[0765] This invention is a system that allows users to input voice using a device such as a smartphone and receive real-time feedback. The server converts the voice data into text, provides related visual learning content, and tracks the user's learning progress.

[0766] Hardware and software used

[0767] Device: A mobile device such as a smartphone or tablet

[0768] Speech recognition technology: Speech recognition services such as Google Cloud Speech-to-Text API

[0769] Text-to-speech technology: Synthetic speech technology (e.g., Amazon Polly) for text-to-speech conversion

[0770] Visual learning content generation means: Image generation algorithms such as DALL-E 2

[0771] Database: A server-side database system for recording user data and learning progress.

[0772] User registration and account creation

[0773] The user downloads and installs the app onto a device such as a smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. A confirmation message confirming registration completion is sent from the server to the device, which displays it to the user.

[0774] Speech recognition and text conversion

[0775] The user taps the "Conversation Practice" button on the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is sent from the server to the device, which then displays it to the user.

[0776] Providing real-time feedback

[0777] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback can also be provided simultaneously.

[0778] Visual learning aids

[0779] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using DALL-E 2. The generated image data is sent to the device, which displays it to the user.

[0780] Learning progress and tracking

[0781] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[0782] Specific examples

[0783] For example, the specific process for practicing pronunciation of the word "apple" and obtaining visually related images is shown below.

[0784] 1. The user speaks "apple" into the smartphone microphone.

[0785] 2. The device captures the audio and sends it to the server.

[0786] 3. The server converts the speech to text and displays "apple" to the user.

[0787] 4. The server evaluates the accuracy of the pronunciation and generates and displays feedback such as "Your pronunciation is correct!"

[0788] 5. The user types "apple" into the text input field and requests related images.

[0789] 6. The device sends the text to the server, and the server generates an image of "apple" using DALL-E 2.

[0790] 7. The generated image is sent to the device and presented to the user.

[0791] This process enhances the user's pronunciation practice and visual comprehension.

[0792] Prompt Sentence Examples

[0793] How can a user speak the word "apple" into a smartphone microphone, receive real-time feedback, and see related visual images?

[0794] The above is a detailed description of the preferred embodiment of the present invention, which allows users to learn languages ​​more effectively.

[0795] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0796] Program processing flow

[0797] User registration and account creation

[0798] Step 1:

[0799] The user downloads and installs the app on their smartphone.

[0800] Enter: app download

[0801] Output: Installed apps

[0802] Step 2:

[0803] The user enters information such as name, email address, password, and English conversation level on the new registration screen.

[0804] Input: Name, email address, password, English conversation level

[0805] Output: Input data

[0806] Step 3:

[0807] The terminal transmits the input information to the server.

[0808] Input: Input data

[0809] Output: Server sent data

[0810] Step 4:

[0811] The server stores the received information in a database.

[0812] Input: Server transmission data

[0813] Output: Database update

[0814] Step 5:

[0815] The server sends a confirmation message to the terminal confirming the completion of registration.

[0816] Input: None

[0817] Output:Confirmation message

[0818] Step 6:

[0819] The terminal displays a confirmation message to the user.

[0820] Input:Confirmation message

[0821] Output: Registration complete

[0822] Speech recognition and text conversion

[0823] Step 1:

[0824] The user taps the "Practice Conversation" button within the app.

[0825] Input: Tap a button

[0826] Output: Start speech mode

[0827] Step 2:

[0828] The user begins speaking into the smartphone's microphone.

[0829] Input: Speech content

[0830] Output: Audio data

[0831] Step 3:

[0832] The device captures audio data input from the microphone.

[0833] Input: Audio data

[0834] Output: Captured audio data

[0835] Step 4:

[0836] The device transmits the voice data to the server in real time.

[0837] Input: Captured audio data

[0838] Output: Server sent data

[0839] Step 5:

[0840] The server receives the voice data and converts it into text using speech recognition technology.

[0841] Input: Server transmission data

[0842] Output: Converted text data

[0843] Step 6:

[0844] The server transmits the converted text data to the terminal.

[0845] Input: Converted text data

[0846] Output: Terminal sending data

[0847] Step 7:

[0848] The terminal displays the text data to the user.

[0849] Input: Terminal transmission data

[0850] Output: Text display

[0851] Providing real-time feedback

[0852] Step 1:

[0853] The server immediately analyzes the converted text data.

[0854] Input: Converted text data

[0855] Output: Feedback data

[0856] Step 2:

[0857] The server generates feedback based on what the user says.

[0858] Input: Feedback data

[0859] Output: Generated feedback

[0860] Step 3:

[0861] The server transmits the generated feedback data to the terminal.

[0862] Input: Generated feedback

[0863] Output: Terminal sending data

[0864] Step 4:

[0865] The device displays the feedback to the user.

[0866] Input: Terminal transmission data

[0867] Output: Feedback display

[0868] Step 5:

[0869] The device will also provide audio feedback if necessary.

[0870] Input: Feedback data

[0871] Output: Audio feedback

[0872] Visual learning aids

[0873] Step 1:

[0874] The user enters the word or phrase they want to learn into the text entry field.

[0875] Input: Word or phrase

[0876] Output: Input text

[0877] Step 2:

[0878] The terminal transmits the input text data to the server.

[0879] Input: Input text

[0880] Output: Server sent data

[0881] Step 3:

[0882] A server receives the text data and generates an associated image.

[0883] Input: Server transmission data

[0884] Output: Generated image data

[0885] Step 4:

[0886] The server transmits the generated image data to the terminal.

[0887] Input: Generated image data

[0888] Output: Terminal sending data

[0889] Step 5:

[0890] The terminal displays the image to the user.

[0891] Input: Terminal transmission data

[0892] Output: Image display

[0893] Learning progress and tracking

[0894] Step 1:

[0895] The server stores the user's voice utterance history and feedback data in a database.

[0896] Input: Feedback data and speech history

[0897] Output: Database update

[0898] Step 2:

[0899] The server continuously tracks the user's learning activities and understanding.

[0900] Input: Database data

[0901] Output:Tracking information

[0902] Step 3:

[0903] The server evaluates the user's learning progress based on the data stored.

[0904] Input:Tracking information

[0905] Output: Evaluation results

[0906] Step 4:

[0907] The server generates a new learning plan and sends it to the device.

[0908] Input: Evaluation result

[0909] Output: The generated learning plan

[0910] Step 5:

[0911] The device will present the user with their new learning plan and progress.

[0912] Input: Generated lesson plan

[0913] Output: Plan display

[0914] The above are the specific processing steps of the system program.

[0915] (Application example 1)

[0916] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0917] Conventional virtual store systems have difficulty providing product explanations and purchasing support when users speak different languages. There has also been a demand for providing detailed product information in real time, multilingual support, and comprehensive purchasing support. Providing services to international customers without creating a language barrier has been particularly challenging, making improving the user experience a key challenge.

[0918] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0919] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, a multilingual translation unit, and an answer generation unit using a generative AI model. This enables real-time multilingual product explanations and purchasing support in the virtual store, making it possible to provide services to international customers without language barriers.

[0920] A "voice recognition means" is a device or software that has the function of receiving voice input, analyzing the voice, and converting it into text data.

[0921] A "speech-to-text converter" is a system or algorithm that performs a process to convert speech data into text data.

[0922] A "text-to-speech converter" is a system or algorithm that performs a process to convert text data into speech data.

[0923] The "means for providing real-time feedback" refers to a device or software that has the function of generating immediate feedback in response to a user's input and providing it to the user.

[0924] A "visual learning content generator" is a device or software that has the functionality to generate visual learning materials based on text data or other input data.

[0925] A "learning progress tracking means" is a device or software that has the function of continuously recording and evaluating a user's learning activities and progress.

[0926] A "multilingual translation means" is a device or software that has the function of translating text and audio data between a wide variety of languages.

[0927] An "answer generation means using a generative AI model" is a system or algorithm that uses artificial intelligence technology to generate answers to users' questions.

[0928] The "means for providing purchasing support functions" refers to a device or software that has the function of allowing users to obtain information about products and services and assisting them in the purchasing process.

[0929] This invention relates to a system that provides real-time multilingual product explanations and purchasing support in a virtual store. The system is realized by communication between users and a server via an application installed on a mobile device such as a smartphone or smart glasses.

[0930] Voice input and recognition

[0931] 1. Terminal processing:

[0932] The user speaks a question about the product into the device's microphone.

[0933] The device captures the audio data and sends it to the server.

[0934] 2. Server processing:

[0935] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text.

[0936] Get product information

[0937] 3. Server processing:

[0938] The text data generated by speech recognition is used to query the product database using Django or Flask.

[0939] Retrieve relevant product information and translate it into the user's native language.

[0940] Multilingual translation and answer generation

[0941] 4. Server processing:

[0942] Use the Google Cloud Translation API to translate user questions into different languages.

[0943] The translated text is fed into a generative AI model (e.g., GPT-3) to generate a detailed answer.

[0944] The generated answer is translated back into the user's language.

[0945] Visual Content Generation

[0946] 5. Server Processing:

[0947] Based on the transferred product information, relevant visual learning content (e.g., images) is generated using OpenCV and matplotlib.

[0948] 6. Terminal Processing:

[0949] The generated image sent from the server is displayed to the user.

[0950] Configured example

[0951] For example, if a user says "What are the features of this smartphone case?" in a virtual store, the process will proceed as follows:

[0952] 1. The user speaks into the smartphone microphone, "Please tell me the features of this smartphone cover."

[0953] 2. The device captures the audio and sends it to the server.

[0954] 3. The server converts the speech into text and queries the product database to obtain "information about the smartphone cover."

[0955] 4. Translate the question into English via a translation API and provide the prompt "What are the features of this smartphone case?" to GPT-3.

[0956] 5. The server receives the answer generated by GPT-3 and retranslates it into the user's language.

[0957] 6. Display the generated text and related images on your smartphone.

[0958] Prompt Sentence Examples

[0959] "Please tell me the features of this smartphone case."

[0960] This will enable users to access detailed product information in real time in multiple languages ​​in the virtual store, making it possible to provide services to international customers without causing them language barriers.

[0961] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0962] Step 1:

[0963] The user speaks a question about a product into the microphone of their smartphone in the virtual store. This voice is the input data. The device recognizes the user's speech and captures it as digital voice data.

[0964] Step 2:

[0965] The device sends the captured audio data to the server. At this point, the input is audio data, and the output is the transmission of audio data to the server. The smartphone's communication module is used.

[0966] Step 3:

[0967] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process results in voice data as input and text data as output. Specifically, the server sends the voice data to the API and analyzes the returned text data.

[0968] Step 4:

[0969] The server uses the text data to query the product database. The input is the converted text data, and the output is the product information. Using Django or Flask, the corresponding information is retrieved from the product database.

[0970] Step 5:

[0971] The server uses the Google Cloud Translation API to translate the user's question from the user's native language into a target language such as English. The input is text data, and the output is the translated text. Specifically, the server sends the text to be translated and the target language to the API, and receives the translation result.

[0972] Step 6:

[0973] The server uses a generative AI model (e.g., GPT-3) to generate an answer to the question based on the translated text. The translated question is input to the AI ​​model as a prompt. The input is the translated text, and the output is the generated answer text. The server sends the prompt to the generative AI model and receives the returned answer.

[0974] Step 7:

[0975] The server translates the generated answer into the user's native language using the Google Cloud Translation API again. The input is the generated answer text, and the output is the translated answer text into the user's native language. The translation procedure is the same as in step 5.

[0976] Step 8:

[0977] The server generates relevant visual learning content. The input is product data, and the output is the generated visual content (e.g., images). Specifically, it uses OpenCV and matplotlib to generate visual information related to the product.

[0978] Step 9:

[0979] The server sends the generated text and image data to the terminal. The input is the answer text and visual content, and the output is a status indicating completion of transmission to the terminal. The server sends the data using a communication protocol.

[0980] Step 10:

[0981] The terminal displays the received text and visual content to the user. The input is the data sent from the server, and the output is the content displayed on the terminal display. In concrete terms, the terminal reflects the data on the display and provides the user with visual and text information.

[0982] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0983] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, learning progress tracking, and user emotion recognition. The system recognizes emotions from the user's speech and input text and provides feedback based on these emotions, enabling a more effective learning experience.

[0984] User registration and account creation

[0985] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[0986] Speech recognition and text conversion

[0987] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[0988] Providing real-time feedback

[0989] The server immediately analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The server also adjusts the content and tone of the feedback using an emotion engine that recognizes emotions from the user's speech and input text. The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback is also provided if necessary.

[0990] Visual learning aids

[0991] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[0992] Learning progress and tracking

[0993] The server stores the user's speech history and feedback data in a database, and continuously tracks the user's learning activities and comprehension. Furthermore, based on the user's emotional state recognized by the emotion engine, it influences the tracking results of the learning progress and adjusts the learning plan according to the specific emotional state. This new learning plan is sent to the device, which then presents it to the user along with the learning progress.

[0994] Specific examples

[0995] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[0996] 1. The user speaks "apple" into the smartphone microphone.

[0997] 2. The device captures the audio and sends it to the server.

[0998] 3. The server converts the speech to text and displays "apple" to the user.

[0999] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1000] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[1001] 6. The user types "apple" into the text input field and requests related images.

[1002] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1003] 8. The generated image is sent to the terminal and presented to the user.

[1004] This series of processes enhances the user's pronunciation practice and visual understanding, and the emotion engine provides personalized feedback, improving the learner's motivation and creating an appropriate learning plan.

[1005] The processing flow will be explained below.

[1006] Step 1:

[1007] The user downloads and installs the app on their smartphone.

[1008] Step 2:

[1009] The user launches the app and goes to the new registration screen.

[1010] Step 3:

[1011] The user enters information such as name, email address, password, and initial English conversation level.

[1012] Step 4:

[1013] The terminal transmits the input information to the server.

[1014] Step 5:

[1015] The server receives the transmitted information and stores it in a database.

[1016] Step 6:

[1017] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1018] Step 7:

[1019] The terminal displays a confirmation message to the user.

[1020] Step 8:

[1021] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[1022] Step 9:

[1023] The user begins speaking into the smartphone's microphone.

[1024] Step 10:

[1025] The device captures audio data input from the microphone.

[1026] Step 11:

[1027] The device transmits the captured audio data to the server in real time.

[1028] Step 12:

[1029] The server receives the voice data and converts it into text using voice recognition technology.

[1030] Step 13:

[1031] The server transmits the converted text data to the terminal.

[1032] Step 14:

[1033] The terminal displays the text data to the user.

[1034] Step 15:

[1035] The server analyzes the converted text data.

[1036] Step 16:

[1037] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[1038] Step 17:

[1039] The server uses an emotion engine that recognizes emotions from the user's speech and input text to adjust the content and tone of the feedback.

[1040] Step 18:

[1041] The server transmits the generated feedback data to the terminal.

[1042] Step 19:

[1043] The device displays feedback to the user, and optionally provides audio feedback.

[1044] Step 20:

[1045] The user enters the word or phrase they want to learn into the text entry field.

[1046] Step 21:

[1047] The terminal transmits the input text data to the server.

[1048] Step 22:

[1049] A server receives the text data and generates related images using a visual learning content generator.

[1050] Step 23:

[1051] The server transmits the generated image data to the terminal.

[1052] Step 24:

[1053] The terminal displays the image to the user.

[1054] Step 25:

[1055] The server stores the user's voice utterance history and feedback data in a database.

[1056] Step 26:

[1057] The server continuously tracks the user's learning progress based on the data stored on the server.

[1058] Step 27:

[1059] The server influences the tracking results of the learning progress based on the emotional state of the user recognized by the emotion engine.

[1060] Step 28:

[1061] The server adjusts the learning plan according to the particular emotional state and generates a new learning plan.

[1062] Step 29:

[1063] The server sends the new lesson plan to the device.

[1064] Step 30:

[1065] The device will present the user with their learning progress and new learning plan.

[1066] This series of steps enhances the user's pronunciation practice and visual comprehension, while the emotion engine provides personalized feedback to improve learner motivation and create appropriate learning plans.

[1067] Example 2

[1068] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1069] Previous language learning systems provided limited feedback on user utterances, and it was particularly difficult to provide feedback that took into account the user's emotional state. Furthermore, learning progress tracking was inconsistent, making it difficult to provide individually optimized learning plans. This resulted in a lack of learner motivation and an ineffective learning experience.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1071] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, and an emotion recognition unit. This makes it possible to provide feedback based not only on the user's spoken content but also on the user's emotional state. Furthermore, by continuously tracking the learning progress and generating and providing an individually optimized learning plan, it is possible to increase the learner's motivation and achieve an effective learning experience.

[1072] "Speech recognition means" is a technology that captures a user's speech, analyzes the speech data, and converts it into linguistic text.

[1073] A "voice-to-text converter" is a function that converts captured voice data into a text format in a language.

[1074] "Text-to-speech conversion means" refers to a technology that generates text-format language data as speech output.

[1075] The "means for providing real-time feedback" is a function that provides immediate evaluation and feedback on corrections based on the user's utterances.

[1076] "Visual learning content generation means" is a technology that generates and visually displays related images and diagrams based on text entered by the user.

[1077] A "learning progress tracking means" is a system that continuously tracks and records a user's learning activities and progress.

[1078] "Emotion recognition means" is a technology that analyzes emotions from a user's speech or input text and recognizes their emotional state.

[1079] The present invention is implemented as a smartphone application for language learners, whose main processes include speech recognition, speech-to-text conversion, text-to-speech conversion, real-time feedback provision, visual learning content generation, learning progress tracking, and emotion recognition.

[1080] Hardware and software used

[1081] Hardware:

[1082] Smartphone (microphone, display)

[1083] software:

[1084] Speech recognition engine (Google Speech-to-Text API)

[1085] Emotion recognition engine (IBM Watson Emotional Recognition)

[1086] Database (MySQL)

[1087] Application server (Node.js)

[1088] Learning content generation tool (TensorFlow)

[1089] Specific processing of the program

[1090] User registration and account creation

[1091] Users download and install the app on their smartphone. After installation, they launch the app and enter information such as their name, email address, password, and English conversation level. The device collects this information and sends it to the server. The server stores it in a database and sends a message to the device confirming registration.

[1092] Speech recognition and text conversion

[1093] The user selects "Conversation Practice" and begins speaking. The device captures the audio with the microphone and sends the audio data to the server. The server converts the audio into text using the Google Speech-to-Text API and sends the converted text to the device for display.

[1094] Providing real-time feedback

[1095] The server analyzes the converted text and generates feedback on pronunciation accuracy, grammatical compatibility, and appropriate expression examples. Furthermore, the IBM Watson Emotional Recognition engine recognizes the user's emotions and adjusts the content and tone of the feedback accordingly. Feedback data is sent to the device, where it is displayed and provided to the user via voice.

[1096] Visual learning aids

[1097] When a user inputs a word or phrase they want to learn, the device sends the text data to the server, which then uses TensorFlow to generate a related image, which is then sent to the device and displayed to the user.

[1098] Learning progress and tracking

[1099] The server stores the user's speech history and feedback data in a database, tracks the user's learning progress, and adjusts the learning plan based on the emotion recognition results, sending the plan to the device and presenting it to the user.

[1100] Specific examples

[1101] For example, if a user says the word "apple" and wants to get a visually related image:

[1102] 1. The user speaks "apple" into the smartphone microphone.

[1103] 2. The device captures the audio and sends it to the server.

[1104] 3. The server converts the speech to text and displays "apple."

[1105] 4. The server evaluates the accuracy of the pronunciation and generates and sends feedback saying, "Your pronunciation is correct!"

[1106] 5. The server uses the emotion engine to recognize the user's confidence and generates motivating feedback such as, "That's great! Keep it up!"

[1107] 6. The user types "apple" into the text input field and requests related images.

[1108] 7. The server generates an image of an "apple" using the visual learning content generation means.

[1109] 8. The generated image is sent to the terminal and presented to the user.

[1110] Example prompts for generative AI models

[1111] Generate related images for visual understanding of the word "apple." The system converts the audio data of the user pronouncing "apple" into text, provides feedback on the accuracy of the pronunciation, and then analyzes the user's sentiment (confidence) to provide appropriate feedback. Next, the user types "apple" into a text field to request related visual images.

[1112] This prompt was generated based on the specific operations described above and is formatted to clearly communicate the expected behavior of the AI ​​model.

[1113] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1114] Step 1:

[1115] The user downloads and installs the app on their smartphone.

[1116] Input: App download link

[1117] Output: Installed apps

[1118] Step 2:

[1119] The user launches the app and enters information such as their name, email address, password, and English conversation level on the new registration screen.

[1120] Input: User's personal information (name, email address, password, English conversation level)

[1121] Output: User information entered

[1122] Step 3:

[1123] The terminal transmits the user's input information to the server.

[1124] Input: User information

[1125] Output: A request to send data to the server

[1126] Step 4:

[1127] The server receives the transmitted user information and stores it in a database.

[1128] Input: User information

[1129] Output: A database containing user information

[1130] Step 5:

[1131] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1132] Input: None (default confirmation message)

[1133] Output:Confirmation message

[1134] Step 6:

[1135] The terminal receives the confirmation message and displays it to the user.

[1136] Input:Confirmation message

[1137] Output: What is displayed to the user

[1138] Step 7:

[1139] The user taps the "Practice Conversation" button to enter speech mode.

[1140] Input: Button tap event

[1141] Output: Speech mode

[1142] Step 8:

[1143] The user begins speaking into the smartphone's microphone.

[1144] Input: Audio data

[1145] Output: Audio signal

[1146] Step 9:

[1147] The device captures audio data input from the microphone and transmits it to the server in real time.

[1148] Input: Audio signal

[1149] Output: Request to send audio data to the server

[1150] Step 10:

[1151] The server receives the voice data and converts it into text using a voice recognition engine.

[1152] Input: Audio data

[1153] Output: Text data

[1154] Step 11:

[1155] The server transmits the converted text data to the terminal.

[1156] Input: Text data

[1157] Output: Request to send text data to the terminal

[1158] Step 12:

[1159] The terminal receives the text data and displays it to the user.

[1160] Input: Text data

[1161] Output: What is displayed to the user

[1162] Step 13:

[1163] The server analyzes the text data and generates feedback including pronunciation accuracy, grammatical correctness, and appropriate expression.

[1164] Input: Text data

[1165] Output: Feedback message

[1166] Step 14:

[1167] The server uses an emotion recognition engine to recognize the user's emotions and adjust the content and tone of the feedback.

[1168] Input: Text data (speech content), voice data (speech tone)

[1169] Output: Adjusted feedback message

[1170] Step 15:

[1171] The server transmits the generated feedback data to the terminal.

[1172] Input: Feedback message

[1173] Output: Request to send feedback to device

[1174] Step 16:

[1175] The terminal receives the feedback and provides visual and audio feedback to the user.

[1176] Input: Feedback message

[1177] Output: Visual feedback / Audio feedback

[1178] Step 17:

[1179] Users enter the words or phrases they want to learn as text.

[1180] Input: Word or phrase

[1181] Output: Text data

[1182] Step 18:

[1183] The terminal transmits the text data to the server.

[1184] Input: Text data

[1185] Output: Request to send text data to the server

[1186] Step 19:

[1187] A server receives the text data and generates related images using a visual learning content generation tool.

[1188] Input: Text data

[1189] Output: Image data

[1190] Step 20:

[1191] The server transmits the generated image data to the terminal.

[1192] Input: Image data

[1193] Output: Image data transmission request to the device

[1194] Step 21:

[1195] The terminal receives the image data and presents it to the user.

[1196] Input: Image data

[1197] Output: Display image

[1198] Step 22:

[1199] The server stores the user's speech history and feedback data in a database and tracks their learning progress.

[1200] Input: Speech history, feedback data

[1201] Output: Updated database

[1202] Step 23:

[1203] The server adjusts the learning plan based on the emotion recognition results and generates a new learning plan.

[1204] Input: Emotion recognition results, learning progress data

[1205] Output: New lesson plan

[1206] Step 24:

[1207] The server sends the new lesson plan to the device.

[1208] Input: New lesson plan

[1209] Output: Request to send lesson plan to device

[1210] Step 25:

[1211] The terminal receives the study plan and presents it to the user.

[1212] Input: New lesson plan

[1213] Output: Display of the learning plan

[1214] The above is the specific processing flow of this system.

[1215] (Application example 2)

[1216] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1217] In language learning, conventional systems do not provide feedback that takes users' emotions into account, which can lead to a decrease in learner motivation. They also lack a means to effectively track learning progress and provide individually optimized learning plans. Furthermore, when providing an interactive learning experience, learners face challenges due to the lack of interactive story-based dialogue, which can lead to a lack of immersion.

[1218] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1219] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a user emotion recognition means, an emotion-based feedback adjusting means, and a story-style dialogue content providing means, thereby enabling feedback based on the user's emotions and immersive learning through interactive story-style dialogue content.

[1220] Word definition

[1221] "Speech recognition means" is a function that receives voice data as input and converts it into text data.

[1222] The "voice-to-text conversion means" is a function that analyzes the voice data captured by the voice recognition means and converts it into corresponding text data.

[1223] The "text to speech conversion means" is a function that receives text data as input and outputs the content as speech data.

[1224] The "means for providing feedback in real time" is a function that instantly generates appropriate feedback based on the user's input and provides it to the user.

[1225] The "visual learning content generation means" is a function that generates visual content such as related images and videos based on the text and audio input by the user.

[1226] "Learning progress tracking means" is a function that records a user's learning activities and manages and evaluates their progress.

[1227] The "user emotion recognition means" is a function that analyzes emotions from the user's speech and input text.

[1228] The "feedback adjustment means based on emotion" is a function that adjusts the content and tone of the feedback provided based on the user's emotion recognized by the emotion recognition means.

[1229] The "means for providing narrative dialogue content" is a function for generating and providing interactive dialogue in the form of a story.

[1230] MODE FOR CARRYING OUT THE INVENTION

[1231] This invention is a language learning system that recognizes a user's speech in real time, converts it into text, and provides feedback. It also recognizes the user's emotions and provides feedback based on those emotions, making the learning experience more effective.

[1232] The system mainly consists of the following means:

[1233] 1. Voice recognition means:

[1234] The server captures the user's voice and uses a speech recognition algorithm to convert it into text, using tools such as the Google Speech Recognition API.

[1235] 2. Voice to text conversion methods:

[1236] The server converts the voice data received from the voice recognition means into text data in real time.

[1237] 3. Text to speech conversion methods:

[1238] The server uses Text-to-Speech (TTS) technology, such as the Google Text-to-Speech API, to provide audio feedback of the generated text.

[1239] 4. How to provide real-time feedback:

[1240] The server analyzes the converted text data, evaluates pronunciation accuracy and grammatical correctness, and generates feedback that is provided to the user instantly.

[1241] 5. Visual learning content generation methods:

[1242] The server generates relevant visual content (images and videos) based on the text entered by the user. For example, if a user enters that they want to learn the meaning of "apple," the server generates an image of "apple" and sends it to the device.

[1243] 6. Track your progress:

[1244] The server tracks the user's learning progress and stores the user's speech history and feedback data in a database, allowing the user to check their own learning progress.

[1245] 7. User Emotion Recognition Method:

[1246] The server has an emotion engine that analyzes emotions from user utterances and input text, enabling feedback based on the user's emotions.

[1247] 8. Emotion-based feedback regulation measures:

[1248] The server adjusts the content and tone of the feedback based on the emotional information recognized by the emotion engine. For example, if the user is nervous, it provides encouraging feedback.

[1249] 9. Narrative dialogue content delivery methods:

[1250] The server generates interactive story-style dialogue content, allowing users to participate in learning as characters in the story and learn a language through dialogue.

[1251] As a concrete example, consider a user trying to practice pronouncing the word "apple" and get a related visual image:

[1252] 1. The user speaks "apple" into the smartphone microphone.

[1253] 2. The device captures the audio and sends it to the server.

[1254] 3. The server converts the speech to text and displays "apple" to the user.

[1255] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1256] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[1257] 6. The user types "apple" into the text input field and requests related images.

[1258] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1259] 8. The generated image is sent to the terminal and presented to the user.

[1260] This series of processes enhances the user's pronunciation practice and visual understanding, while the emotion engine provides individualized feedback, improving learner motivation and enabling appropriate study plans.

[1261] An example of an input prompt is, "As a great knight, you must face a dragon. Are you ready?"

[1262] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1263] Program processing flow

[1264] Step 1:

[1265] The user speaks into the smartphone's microphone. The device captures the voice data from the microphone. The input is the user's voice, and the output is the captured voice data.

[1266] Step 2:

[1267] The device sends the captured audio data to the server. The input is the audio data, and the output is the data sent to the server.

[1268] Step 3:

[1269] The server converts the received voice data into text data using a voice recognition means, with the input being the voice data and the output being the converted text data.

[1270] Step 4:

[1271] The server analyzes the converted text data and generates real-time feedback. Specifically, it evaluates the accuracy of pronunciation and grammar. The input is the text data, and the output is the evaluation result and a feedback message.

[1272] Step 5:

[1273] The server sends text data and feedback messages to the terminal. The input is the evaluation result and the feedback message, and the output is the data sent to the terminal.

[1274] Step 6:

[1275] The terminal displays the received feedback to the user and provides audio feedback if necessary. The input is the data sent from the server, and the output is visual and audio feedback to the user.

[1276] Step 7:

[1277] The server analyzes text data using an emotion recognition engine to recognize the user's emotions. The input is text data, and the output is the emotion analysis result.

[1278] Step 8:

[1279] The server adjusts the content and tone of the feedback based on the results of the emotion analysis. Specifically, it generates encouraging messages to motivate the user. The input is the emotion analysis results, and the output is the adjusted feedback message.

[1280] Step 9:

[1281] The server sends the final feedback message to the terminal. The input is the adjusted feedback message, and the output is the data to be sent to the terminal.

[1282] Step 10:

[1283] A user enters a word or phrase they want to learn into a text input field and requests related visual content. The input is the text entered by the user, and the output is the requested data.

[1284] Step 11:

[1285] The terminal sends request data to the server. The input is the request data, and the output is the data sent to the server.

[1286] Step 12:

[1287] The server receives the request and generates the relevant image using a visual learning content generator, where the input is the user's text data and the output is the generated image data.

[1288] Step 13:

[1289] The generated image data is sent from the server to the terminal. The input is the image data, and the output is the data sent to the terminal.

[1290] Step 14:

[1291] The terminal presents the received image data to the user. The input is the data sent from the server, and the output is visual feedback to the user.

[1292] This process reinforces the user's speech practice and visual comprehension, and the emotion engine provides personalized feedback. For example, the prompt "You must face the dragon as a great knight. Are you ready?" can be used to guide the user through the dialogue.

[1293] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1294] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1295] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1296] [Third embodiment]

[1297] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1298] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1299] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1300] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1301] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1302] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1303] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1304] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1305] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1306] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1307] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1308] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1309] This invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. Each process of this system is explained in natural language below.

[1310] User registration and account creation

[1311] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[1312] Speech recognition and text conversion

[1313] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[1314] Providing real-time feedback

[1315] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which displays the feedback to the user. Audio feedback is also provided if necessary.

[1316] Visual learning aids

[1317] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[1318] Learning progress and tracking

[1319] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[1320] Specific examples

[1321] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[1322] 1. The user speaks "apple" into the smartphone microphone.

[1323] 2. The device captures the audio and sends it to the server.

[1324] 3. The server converts the speech to text and displays "apple" to the user.

[1325] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1326] 5. The user types "apple" into the text input field and requests related images.

[1327] 6. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1328] 7. The generated image is sent to the device and presented to the user.

[1329] This series of processes enhances the user's pronunciation practice and visual comprehension. In this way, the mode for carrying out the invention is embodied.

[1330] The processing flow will be explained below.

[1331] Step 1:

[1332] The user downloads and installs the app on their smartphone.

[1333] Step 2:

[1334] The user launches the app and goes to the new registration screen.

[1335] Step 3:

[1336] The user enters information such as name, email address, password, and initial English conversation level.

[1337] Step 4:

[1338] The terminal transmits the input information to the server.

[1339] Step 5:

[1340] The server receives the transmitted information and stores it in a database.

[1341] Step 6:

[1342] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1343] Step 7:

[1344] The terminal displays a confirmation message to the user.

[1345] Step 8:

[1346] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[1347] Step 9:

[1348] The user begins speaking into the smartphone's microphone.

[1349] Step 10:

[1350] The device captures audio data input from the microphone.

[1351] Step 11:

[1352] The device transmits the captured audio data to the server in real time.

[1353] Step 12:

[1354] The server receives the voice data and converts it into text using voice recognition technology.

[1355] Step 13:

[1356] The server transmits the converted text data to the terminal.

[1357] Step 14:

[1358] The terminal displays the text data to the user.

[1359] Step 15:

[1360] The server analyzes the converted text data.

[1361] Step 16:

[1362] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[1363] Step 17:

[1364] The server transmits the generated feedback data to the terminal.

[1365] Step 18:

[1366] The device displays feedback to the user, and optionally provides audio feedback.

[1367] Step 19:

[1368] The user enters the word or phrase they want to learn into the text entry field.

[1369] Step 20:

[1370] The terminal transmits the input text data to the server.

[1371] Step 21:

[1372] A server receives the text data and generates related images using a visual learning content generator.

[1373] Step 22:

[1374] The server transmits the generated image data to the terminal.

[1375] Step 23:

[1376] The terminal displays the image to the user.

[1377] Step 24:

[1378] The server stores the user's voice utterance history and feedback data in a database.

[1379] Step 25:

[1380] The server continuously tracks the user's learning progress based on the data stored on the server.

[1381] Step 26:

[1382] The server evaluates the learning progress and generates a new learning plan.

[1383] Step 27:

[1384] The server sends the new lesson plan to the device.

[1385] Step 28:

[1386] The device will display your learning progress and new learning plan on a dashboard.

[1387] This series of steps creates a system that supports effective language learning.

[1388] Example 1

[1389] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1390] Conventional speech recognition systems and language learning applications have difficulty accurately converting user speech into text and providing appropriate feedback. Furthermore, they lack sufficient visual learning support and make it difficult to accurately track learning progress. This makes it difficult for users to effectively acquire a language.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1392] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a terminal means for transmitting voice-input data, and a server means for analyzing the data and generating results, thereby enabling the server to accurately convert a user's speech into text, provide accurate feedback, provide visual learning support, and accurately track learning progress.

[1393] "Speech recognition means" refers to means for detecting the user's voice and processing it as digital data.

[1394] The "speech-to-text conversion means" is a means for converting detected speech data into text data.

[1395] "Text to speech conversion means" refers to means for converting text data into speech data.

[1396] A "real-time feedback providing means" is a means for generating and providing instant feedback based on voice or text input by a user.

[1397] A "visual learning content generation means" is a means for generating relevant visual learning content (eg, images and graphics) based on text data.

[1398] The "learning progress tracking means" is a means for continuously recording and tracking a user's learning activities and progress.

[1399] The "terminal means for transmitting voice-input data" refers to a device for a user to input voice and means for transmitting that data.

[1400] The "server means for analyzing the data and generating a result" is a server for analyzing the transmitted data and generating a result.

[1401] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. A specific embodiment of the present invention is described in detail below.

[1402] System Overview

[1403] This invention is a system that allows users to input voice using a device such as a smartphone and receive real-time feedback. The server converts the voice data into text, provides related visual learning content, and tracks the user's learning progress.

[1404] Hardware and software used

[1405] Device: A mobile device such as a smartphone or tablet

[1406] Speech recognition technology: Speech recognition services such as Google Cloud Speech-to-Text API

[1407] Text-to-speech technology: Synthetic speech technology (e.g., Amazon Polly) for text-to-speech conversion

[1408] Visual learning content generation means: Image generation algorithms such as DALL-E 2

[1409] Database: A server-side database system for recording user data and learning progress.

[1410] User registration and account creation

[1411] The user downloads and installs the app onto a device such as a smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. A confirmation message confirming registration completion is sent from the server to the device, which displays it to the user.

[1412] Speech recognition and text conversion

[1413] The user taps the "Conversation Practice" button on the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is sent from the server to the device, which then displays it to the user.

[1414] Providing real-time feedback

[1415] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback can also be provided simultaneously.

[1416] Visual learning aids

[1417] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using DALL-E 2. The generated image data is sent to the device, which displays it to the user.

[1418] Learning progress and tracking

[1419] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[1420] Specific examples

[1421] For example, the specific process for practicing pronunciation of the word "apple" and obtaining visually related images is shown below.

[1422] 1. The user speaks "apple" into the smartphone microphone.

[1423] 2. The device captures the audio and sends it to the server.

[1424] 3. The server converts the speech to text and displays "apple" to the user.

[1425] 4. The server evaluates the accuracy of the pronunciation and generates and displays feedback such as "Your pronunciation is correct!"

[1426] 5. The user types "apple" into the text input field and requests related images.

[1427] 6. The device sends the text to the server, and the server generates an image of "apple" using DALL-E 2.

[1428] 7. The generated image is sent to the device and presented to the user.

[1429] This process enhances the user's pronunciation practice and visual comprehension.

[1430] Prompt Sentence Examples

[1431] How can a user speak the word "apple" into a smartphone microphone, receive real-time feedback, and see related visual images?

[1432] The above is a detailed description of the preferred embodiment of the present invention, which allows users to learn languages ​​more effectively.

[1433] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1434] Program processing flow

[1435] User registration and account creation

[1436] Step 1:

[1437] The user downloads and installs the app on their smartphone.

[1438] Enter: app download

[1439] Output: Installed apps

[1440] Step 2:

[1441] The user enters information such as name, email address, password, and English conversation level on the new registration screen.

[1442] Input: Name, email address, password, English conversation level

[1443] Output: Input data

[1444] Step 3:

[1445] The terminal transmits the input information to the server.

[1446] Input: Input data

[1447] Output: Server sent data

[1448] Step 4:

[1449] The server stores the received information in a database.

[1450] Input: Server transmission data

[1451] Output: Database update

[1452] Step 5:

[1453] The server sends a confirmation message to the terminal confirming the completion of registration.

[1454] Input: None

[1455] Output:Confirmation message

[1456] Step 6:

[1457] The terminal displays a confirmation message to the user.

[1458] Input:Confirmation message

[1459] Output: Registration complete

[1460] Speech recognition and text conversion

[1461] Step 1:

[1462] The user taps the "Practice Conversation" button within the app.

[1463] Input: Tap a button

[1464] Output: Start speech mode

[1465] Step 2:

[1466] The user begins speaking into the smartphone's microphone.

[1467] Input: Speech content

[1468] Output: Audio data

[1469] Step 3:

[1470] The device captures audio data input from the microphone.

[1471] Input: Audio data

[1472] Output: Captured audio data

[1473] Step 4:

[1474] The device transmits the voice data to the server in real time.

[1475] Input: Captured audio data

[1476] Output: Server sent data

[1477] Step 5:

[1478] The server receives the voice data and converts it into text using speech recognition technology.

[1479] Input: Server transmission data

[1480] Output: Converted text data

[1481] Step 6:

[1482] The server transmits the converted text data to the terminal.

[1483] Input: Converted text data

[1484] Output: Terminal sending data

[1485] Step 7:

[1486] The terminal displays the text data to the user.

[1487] Input: Terminal transmission data

[1488] Output: Text display

[1489] Providing real-time feedback

[1490] Step 1:

[1491] The server immediately analyzes the converted text data.

[1492] Input: Converted text data

[1493] Output: Feedback data

[1494] Step 2:

[1495] The server generates feedback based on what the user says.

[1496] Input: Feedback data

[1497] Output: Generated feedback

[1498] Step 3:

[1499] The server transmits the generated feedback data to the terminal.

[1500] Input: Generated feedback

[1501] Output: Terminal sending data

[1502] Step 4:

[1503] The device displays the feedback to the user.

[1504] Input: Terminal transmission data

[1505] Output: Feedback display

[1506] Step 5:

[1507] The device will also provide audio feedback if necessary.

[1508] Input: Feedback data

[1509] Output: Audio feedback

[1510] Visual learning aids

[1511] Step 1:

[1512] The user enters the word or phrase they want to learn into the text entry field.

[1513] Input: Word or phrase

[1514] Output: Input text

[1515] Step 2:

[1516] The terminal transmits the input text data to the server.

[1517] Input: Input text

[1518] Output: Server sent data

[1519] Step 3:

[1520] A server receives the text data and generates an associated image.

[1521] Input: Server transmission data

[1522] Output: Generated image data

[1523] Step 4:

[1524] The server transmits the generated image data to the terminal.

[1525] Input: Generated image data

[1526] Output: Terminal sending data

[1527] Step 5:

[1528] The terminal displays the image to the user.

[1529] Input: Terminal transmission data

[1530] Output: Image display

[1531] Learning progress and tracking

[1532] Step 1:

[1533] The server stores the user's voice utterance history and feedback data in a database.

[1534] Input: Feedback data and speech history

[1535] Output: Database update

[1536] Step 2:

[1537] The server continuously tracks the user's learning activities and understanding.

[1538] Input: Database data

[1539] Output:Tracking information

[1540] Step 3:

[1541] The server evaluates the user's learning progress based on the data stored.

[1542] Input:Tracking information

[1543] Output: Evaluation results

[1544] Step 4:

[1545] The server generates a new learning plan and sends it to the device.

[1546] Input: Evaluation result

[1547] Output: The generated learning plan

[1548] Step 5:

[1549] The device will present the user with their new learning plan and progress.

[1550] Input: Generated lesson plan

[1551] Output: Plan display

[1552] The above are the specific processing steps of the system program.

[1553] (Application example 1)

[1554] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1555] Conventional virtual store systems have difficulty providing product explanations and purchasing support when users speak different languages. There has also been a demand for providing detailed product information in real time, multilingual support, and comprehensive purchasing support. Providing services to international customers without creating a language barrier has been particularly challenging, making improving the user experience a key challenge.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1557] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, a multilingual translation unit, and an answer generation unit using a generative AI model. This enables real-time multilingual product explanations and purchasing support in the virtual store, making it possible to provide services to international customers without language barriers.

[1558] A "voice recognition means" is a device or software that has the function of receiving voice input, analyzing the voice, and converting it into text data.

[1559] A "speech-to-text converter" is a system or algorithm that performs a process to convert speech data into text data.

[1560] A "text-to-speech converter" is a system or algorithm that performs a process to convert text data into speech data.

[1561] The "means for providing real-time feedback" refers to a device or software that has the function of generating immediate feedback in response to a user's input and providing it to the user.

[1562] A "visual learning content generator" is a device or software that has the functionality to generate visual learning materials based on text data or other input data.

[1563] A "learning progress tracking means" is a device or software that has the function of continuously recording and evaluating a user's learning activities and progress.

[1564] A "multilingual translation means" is a device or software that has the function of translating text and audio data between a wide variety of languages.

[1565] An "answer generation means using a generative AI model" is a system or algorithm that uses artificial intelligence technology to generate answers to users' questions.

[1566] The "means for providing purchasing support functions" refers to a device or software that has the function of allowing users to obtain information about products and services and assisting them in the purchasing process.

[1567] This invention relates to a system that provides real-time multilingual product explanations and purchasing support in a virtual store. The system is realized by communication between users and a server via an application installed on a mobile device such as a smartphone or smart glasses.

[1568] Voice input and recognition

[1569] 1. Terminal processing:

[1570] The user speaks a question about the product into the device's microphone.

[1571] The device captures the audio data and sends it to the server.

[1572] 2. Server processing:

[1573] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text.

[1574] Get product information

[1575] 3. Server processing:

[1576] The text data generated by speech recognition is used to query the product database using Django or Flask.

[1577] Retrieve relevant product information and translate it into the user's native language.

[1578] Multilingual translation and answer generation

[1579] 4. Server processing:

[1580] Use the Google Cloud Translation API to translate user questions into different languages.

[1581] The translated text is fed into a generative AI model (e.g., GPT-3) to generate a detailed answer.

[1582] The generated answer is translated back into the user's language.

[1583] Visual Content Generation

[1584] 5. Server Processing:

[1585] Based on the transferred product information, relevant visual learning content (e.g., images) is generated using OpenCV and matplotlib.

[1586] 6. Terminal Processing:

[1587] The generated image sent from the server is displayed to the user.

[1588] Configured example

[1589] For example, if a user says "What are the features of this smartphone case?" in a virtual store, the process will proceed as follows:

[1590] 1. The user speaks into the smartphone microphone, "Please tell me the features of this smartphone cover."

[1591] 2. The device captures the audio and sends it to the server.

[1592] 3. The server converts the speech into text and queries the product database to obtain "information about the smartphone cover."

[1593] 4. Translate the question into English via a translation API and provide the prompt "What are the features of this smartphone case?" to GPT-3.

[1594] 5. The server receives the answer generated by GPT-3 and retranslates it into the user's language.

[1595] 6. Display the generated text and related images on your smartphone.

[1596] Prompt Sentence Examples

[1597] "Please tell me the features of this smartphone case."

[1598] This will enable users to access detailed product information in real time in multiple languages ​​in the virtual store, making it possible to provide services to international customers without causing them language barriers.

[1599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1600] Step 1:

[1601] The user speaks a question about a product into the microphone of their smartphone in the virtual store. This voice is the input data. The device recognizes the user's speech and captures it as digital voice data.

[1602] Step 2:

[1603] The device sends the captured audio data to the server. At this point, the input is audio data, and the output is the transmission of audio data to the server. The smartphone's communication module is used.

[1604] Step 3:

[1605] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process results in voice data as input and text data as output. Specifically, the server sends the voice data to the API and analyzes the returned text data.

[1606] Step 4:

[1607] The server uses the text data to query the product database. The input is the converted text data, and the output is the product information. Using Django or Flask, the corresponding information is retrieved from the product database.

[1608] Step 5:

[1609] The server uses the Google Cloud Translation API to translate the user's question from the user's native language into a target language such as English. The input is text data, and the output is the translated text. Specifically, the server sends the text to be translated and the target language to the API, and receives the translation result.

[1610] Step 6:

[1611] The server uses a generative AI model (e.g., GPT-3) to generate an answer to the question based on the translated text. The translated question is input to the AI ​​model as a prompt. The input is the translated text, and the output is the generated answer text. The server sends the prompt to the generative AI model and receives the returned answer.

[1612] Step 7:

[1613] The server translates the generated answer into the user's native language using the Google Cloud Translation API again. The input is the generated answer text, and the output is the translated answer text into the user's native language. The translation procedure is the same as in step 5.

[1614] Step 8:

[1615] The server generates relevant visual learning content. The input is product data, and the output is the generated visual content (e.g., images). Specifically, it uses OpenCV and matplotlib to generate visual information related to the product.

[1616] Step 9:

[1617] The server sends the generated text and image data to the terminal. The input is the answer text and visual content, and the output is a status indicating completion of transmission to the terminal. The server sends the data using a communication protocol.

[1618] Step 10:

[1619] The terminal displays the received text and visual content to the user. The input is the data sent from the server, and the output is the content displayed on the terminal display. In concrete terms, the terminal reflects the data on the display and provides the user with visual and text information.

[1620] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1621] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, learning progress tracking, and user emotion recognition. The system recognizes emotions from the user's speech and input text and provides feedback based on these emotions, enabling a more effective learning experience.

[1622] User registration and account creation

[1623] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[1624] Speech recognition and text conversion

[1625] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[1626] Providing real-time feedback

[1627] The server immediately analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The server also adjusts the content and tone of the feedback using an emotion engine that recognizes emotions from the user's speech and input text. The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback is also provided if necessary.

[1628] Visual learning aids

[1629] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[1630] Learning progress and tracking

[1631] The server stores the user's speech history and feedback data in a database, and continuously tracks the user's learning activities and comprehension. Furthermore, based on the user's emotional state recognized by the emotion engine, it influences the tracking results of the learning progress and adjusts the learning plan according to the specific emotional state. This new learning plan is sent to the device, which then presents it to the user along with the learning progress.

[1632] Specific examples

[1633] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[1634] 1. The user speaks "apple" into the smartphone microphone.

[1635] 2. The device captures the audio and sends it to the server.

[1636] 3. The server converts the speech to text and displays "apple" to the user.

[1637] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1638] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[1639] 6. The user types "apple" into the text input field and requests related images.

[1640] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1641] 8. The generated image is sent to the terminal and presented to the user.

[1642] This series of processes enhances the user's pronunciation practice and visual understanding, and the emotion engine provides personalized feedback, improving the learner's motivation and creating an appropriate learning plan.

[1643] The processing flow will be explained below.

[1644] Step 1:

[1645] The user downloads and installs the app on their smartphone.

[1646] Step 2:

[1647] The user launches the app and goes to the new registration screen.

[1648] Step 3:

[1649] The user enters information such as name, email address, password, and initial English conversation level.

[1650] Step 4:

[1651] The terminal transmits the input information to the server.

[1652] Step 5:

[1653] The server receives the transmitted information and stores it in a database.

[1654] Step 6:

[1655] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1656] Step 7:

[1657] The terminal displays a confirmation message to the user.

[1658] Step 8:

[1659] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[1660] Step 9:

[1661] The user begins speaking into the smartphone's microphone.

[1662] Step 10:

[1663] The device captures audio data input from the microphone.

[1664] Step 11:

[1665] The device transmits the captured audio data to the server in real time.

[1666] Step 12:

[1667] The server receives the voice data and converts it into text using voice recognition technology.

[1668] Step 13:

[1669] The server transmits the converted text data to the terminal.

[1670] Step 14:

[1671] The terminal displays the text data to the user.

[1672] Step 15:

[1673] The server analyzes the converted text data.

[1674] Step 16:

[1675] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[1676] Step 17:

[1677] The server uses an emotion engine that recognizes emotions from the user's speech and input text to adjust the content and tone of the feedback.

[1678] Step 18:

[1679] The server transmits the generated feedback data to the terminal.

[1680] Step 19:

[1681] The device displays feedback to the user, and optionally provides audio feedback.

[1682] Step 20:

[1683] The user enters the word or phrase they want to learn into the text entry field.

[1684] Step 21:

[1685] The terminal transmits the input text data to the server.

[1686] Step 22:

[1687] A server receives the text data and generates related images using a visual learning content generator.

[1688] Step 23:

[1689] The server transmits the generated image data to the terminal.

[1690] Step 24:

[1691] The terminal displays the image to the user.

[1692] Step 25:

[1693] The server stores the user's voice utterance history and feedback data in a database.

[1694] Step 26:

[1695] The server continuously tracks the user's learning progress based on the data stored on the server.

[1696] Step 27:

[1697] The server influences the tracking results of the learning progress based on the emotional state of the user recognized by the emotion engine.

[1698] Step 28:

[1699] The server adjusts the learning plan according to the particular emotional state and generates a new learning plan.

[1700] Step 29:

[1701] The server sends the new lesson plan to the device.

[1702] Step 30:

[1703] The device will present the user with their learning progress and new learning plan.

[1704] This series of steps enhances the user's pronunciation practice and visual comprehension, while the emotion engine provides personalized feedback to improve learner motivation and create appropriate learning plans.

[1705] Example 2

[1706] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1707] Previous language learning systems provided limited feedback on user utterances, and it was particularly difficult to provide feedback that took into account the user's emotional state. Furthermore, learning progress tracking was inconsistent, making it difficult to provide individually optimized learning plans. This resulted in a lack of learner motivation and an ineffective learning experience.

[1708] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1709] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, and an emotion recognition unit. This makes it possible to provide feedback based not only on the user's spoken content but also on the user's emotional state. Furthermore, by continuously tracking the learning progress and generating and providing an individually optimized learning plan, it is possible to increase the learner's motivation and achieve an effective learning experience.

[1710] "Speech recognition means" is a technology that captures a user's speech, analyzes the speech data, and converts it into linguistic text.

[1711] A "voice-to-text converter" is a function that converts captured voice data into a text format in a language.

[1712] "Text-to-speech conversion means" refers to a technology that generates text-format language data as speech output.

[1713] The "means for providing real-time feedback" is a function that provides immediate evaluation and feedback on corrections based on the user's utterances.

[1714] "Visual learning content generation means" is a technology that generates and visually displays related images and diagrams based on text entered by the user.

[1715] A "learning progress tracking means" is a system that continuously tracks and records a user's learning activities and progress.

[1716] "Emotion recognition means" is a technology that analyzes emotions from a user's speech or input text and recognizes their emotional state.

[1717] The present invention is implemented as a smartphone application for language learners, whose main processes include speech recognition, speech-to-text conversion, text-to-speech conversion, real-time feedback provision, visual learning content generation, learning progress tracking, and emotion recognition.

[1718] Hardware and software used

[1719] Hardware:

[1720] Smartphone (microphone, display)

[1721] software:

[1722] Speech recognition engine (Google Speech-to-Text API)

[1723] Emotion recognition engine (IBM Watson Emotional Recognition)

[1724] Database (MySQL)

[1725] Application server (Node.js)

[1726] Learning content generation tool (TensorFlow)

[1727] Specific processing of the program

[1728] User registration and account creation

[1729] Users download and install the app on their smartphone. After installation, they launch the app and enter information such as their name, email address, password, and English conversation level. The device collects this information and sends it to the server. The server stores it in a database and sends a message to the device confirming registration.

[1730] Speech recognition and text conversion

[1731] The user selects "Conversation Practice" and begins speaking. The device captures the audio with the microphone and sends the audio data to the server. The server converts the audio into text using the Google Speech-to-Text API and sends the converted text to the device for display.

[1732] Providing real-time feedback

[1733] The server analyzes the converted text and generates feedback on pronunciation accuracy, grammatical compatibility, and appropriate expression examples. Furthermore, the IBM Watson Emotional Recognition engine recognizes the user's emotions and adjusts the content and tone of the feedback accordingly. Feedback data is sent to the device, where it is displayed and provided to the user via voice.

[1734] Visual learning aids

[1735] When a user inputs a word or phrase they want to learn, the device sends the text data to the server, which then uses TensorFlow to generate a related image, which is then sent to the device and displayed to the user.

[1736] Learning progress and tracking

[1737] The server stores the user's speech history and feedback data in a database, tracks the user's learning progress, and adjusts the learning plan based on the emotion recognition results, sending the plan to the device and presenting it to the user.

[1738] Specific examples

[1739] For example, if a user says the word "apple" and wants to get a visually related image:

[1740] 1. The user speaks "apple" into the smartphone microphone.

[1741] 2. The device captures the audio and sends it to the server.

[1742] 3. The server converts the speech to text and displays "apple."

[1743] 4. The server evaluates the accuracy of the pronunciation and generates and sends feedback saying, "Your pronunciation is correct!"

[1744] 5. The server uses the emotion engine to recognize the user's confidence and generates motivating feedback such as, "That's great! Keep it up!"

[1745] 6. The user types "apple" into the text input field and requests related images.

[1746] 7. The server generates an image of an "apple" using the visual learning content generation means.

[1747] 8. The generated image is sent to the terminal and presented to the user.

[1748] Example prompts for generative AI models

[1749] Generate related images for visual understanding of the word "apple." The system converts the audio data of the user pronouncing "apple" into text, provides feedback on the accuracy of the pronunciation, and then analyzes the user's sentiment (confidence) to provide appropriate feedback. Next, the user types "apple" into a text field to request related visual images.

[1750] This prompt was generated based on the specific operations described above and is formatted to clearly communicate the expected behavior of the AI ​​model.

[1751] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1752] Step 1:

[1753] The user downloads and installs the app on their smartphone.

[1754] Input: App download link

[1755] Output: Installed apps

[1756] Step 2:

[1757] The user launches the app and enters information such as their name, email address, password, and English conversation level on the new registration screen.

[1758] Input: User's personal information (name, email address, password, English conversation level)

[1759] Output: User information entered

[1760] Step 3:

[1761] The terminal transmits the user's input information to the server.

[1762] Input: User information

[1763] Output: A request to send data to the server

[1764] Step 4:

[1765] The server receives the transmitted user information and stores it in a database.

[1766] Input: User information

[1767] Output: A database containing user information

[1768] Step 5:

[1769] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1770] Input: None (default confirmation message)

[1771] Output:Confirmation message

[1772] Step 6:

[1773] The terminal receives the confirmation message and displays it to the user.

[1774] Input:Confirmation message

[1775] Output: What is displayed to the user

[1776] Step 7:

[1777] The user taps the "Practice Conversation" button to enter speech mode.

[1778] Input: Button tap event

[1779] Output: Speech mode

[1780] Step 8:

[1781] The user begins speaking into the smartphone's microphone.

[1782] Input: Audio data

[1783] Output: Audio signal

[1784] Step 9:

[1785] The device captures audio data input from the microphone and transmits it to the server in real time.

[1786] Input: Audio signal

[1787] Output: Request to send audio data to the server

[1788] Step 10:

[1789] The server receives the voice data and converts it into text using a voice recognition engine.

[1790] Input: Audio data

[1791] Output: Text data

[1792] Step 11:

[1793] The server transmits the converted text data to the terminal.

[1794] Input: Text data

[1795] Output: Request to send text data to the terminal

[1796] Step 12:

[1797] The terminal receives the text data and displays it to the user.

[1798] Input: Text data

[1799] Output: What is displayed to the user

[1800] Step 13:

[1801] The server analyzes the text data and generates feedback including pronunciation accuracy, grammatical correctness, and appropriate expression.

[1802] Input: Text data

[1803] Output: Feedback message

[1804] Step 14:

[1805] The server uses an emotion recognition engine to recognize the user's emotions and adjust the content and tone of the feedback.

[1806] Input: Text data (speech content), voice data (speech tone)

[1807] Output: Adjusted feedback message

[1808] Step 15:

[1809] The server transmits the generated feedback data to the terminal.

[1810] Input: Feedback message

[1811] Output: Request to send feedback to device

[1812] Step 16:

[1813] The terminal receives the feedback and provides visual and audio feedback to the user.

[1814] Input: Feedback message

[1815] Output: Visual feedback / Audio feedback

[1816] Step 17:

[1817] Users enter the words or phrases they want to learn as text.

[1818] Input: Word or phrase

[1819] Output: Text data

[1820] Step 18:

[1821] The terminal transmits the text data to the server.

[1822] Input: Text data

[1823] Output: Request to send text data to the server

[1824] Step 19:

[1825] A server receives the text data and generates related images using a visual learning content generation tool.

[1826] Input: Text data

[1827] Output: Image data

[1828] Step 20:

[1829] The server transmits the generated image data to the terminal.

[1830] Input: Image data

[1831] Output: Image data transmission request to the device

[1832] Step 21:

[1833] The terminal receives the image data and presents it to the user.

[1834] Input: Image data

[1835] Output: Display image

[1836] Step 22:

[1837] The server stores the user's speech history and feedback data in a database and tracks their learning progress.

[1838] Input: Speech history, feedback data

[1839] Output: Updated database

[1840] Step 23:

[1841] The server adjusts the learning plan based on the emotion recognition results and generates a new learning plan.

[1842] Input: Emotion recognition results, learning progress data

[1843] Output: New lesson plan

[1844] Step 24:

[1845] The server sends the new lesson plan to the device.

[1846] Input: New lesson plan

[1847] Output: Request to send lesson plan to device

[1848] Step 25:

[1849] The terminal receives the study plan and presents it to the user.

[1850] Input: New lesson plan

[1851] Output: Display of the learning plan

[1852] The above is the specific processing flow of this system.

[1853] (Application example 2)

[1854] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1855] In language learning, conventional systems do not provide feedback that takes users' emotions into account, which can lead to a decrease in learner motivation. They also lack a means to effectively track learning progress and provide individually optimized learning plans. Furthermore, when providing an interactive learning experience, learners face challenges due to the lack of interactive story-based dialogue, which can lead to a lack of immersion.

[1856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1857] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a user emotion recognition means, an emotion-based feedback adjusting means, and a story-style dialogue content providing means, thereby enabling feedback based on the user's emotions and immersive learning through interactive story-style dialogue content.

[1858] Word definition

[1859] "Speech recognition means" is a function that receives voice data as input and converts it into text data.

[1860] The "voice-to-text conversion means" is a function that analyzes the voice data captured by the voice recognition means and converts it into corresponding text data.

[1861] The "text to speech conversion means" is a function that receives text data as input and outputs the content as speech data.

[1862] The "means for providing feedback in real time" is a function that instantly generates appropriate feedback based on the user's input and provides it to the user.

[1863] The "visual learning content generation means" is a function that generates visual content such as related images and videos based on the text and audio input by the user.

[1864] "Learning progress tracking means" is a function that records a user's learning activities and manages and evaluates their progress.

[1865] The "user emotion recognition means" is a function that analyzes emotions from the user's speech and input text.

[1866] The "feedback adjustment means based on emotion" is a function that adjusts the content and tone of the feedback provided based on the user's emotion recognized by the emotion recognition means.

[1867] The "means for providing narrative dialogue content" is a function for generating and providing interactive dialogue in the form of a story.

[1868] MODE FOR CARRYING OUT THE INVENTION

[1869] This invention is a language learning system that recognizes a user's speech in real time, converts it into text, and provides feedback. It also recognizes the user's emotions and provides feedback based on those emotions, making the learning experience more effective.

[1870] The system mainly consists of the following means:

[1871] 1. Voice recognition means:

[1872] The server captures the user's voice and uses a speech recognition algorithm to convert it into text, using tools such as the Google Speech Recognition API.

[1873] 2. Voice to text conversion methods:

[1874] The server converts the voice data received from the voice recognition means into text data in real time.

[1875] 3. Text to speech conversion methods:

[1876] The server uses Text-to-Speech (TTS) technology, such as the Google Text-to-Speech API, to provide audio feedback of the generated text.

[1877] 4. How to provide real-time feedback:

[1878] The server analyzes the converted text data, evaluates pronunciation accuracy and grammatical correctness, and generates feedback that is provided to the user instantly.

[1879] 5. Visual learning content generation methods:

[1880] The server generates relevant visual content (images and videos) based on the text entered by the user. For example, if a user enters that they want to learn the meaning of "apple," the server generates an image of "apple" and sends it to the device.

[1881] 6. Track your progress:

[1882] The server tracks the user's learning progress and stores the user's speech history and feedback data in a database, allowing the user to check their own learning progress.

[1883] 7. User Emotion Recognition Method:

[1884] The server has an emotion engine that analyzes emotions from user utterances and input text, enabling feedback based on the user's emotions.

[1885] 8. Emotion-based feedback regulation measures:

[1886] The server adjusts the content and tone of the feedback based on the emotional information recognized by the emotion engine. For example, if the user is nervous, it provides encouraging feedback.

[1887] 9. Narrative dialogue content delivery methods:

[1888] The server generates interactive story-style dialogue content, allowing users to participate in learning as characters in the story and learn a language through dialogue.

[1889] As a concrete example, consider a user trying to practice pronouncing the word "apple" and get a related visual image:

[1890] 1. The user speaks "apple" into the smartphone microphone.

[1891] 2. The device captures the audio and sends it to the server.

[1892] 3. The server converts the speech to text and displays "apple" to the user.

[1893] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1894] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[1895] 6. The user types "apple" into the text input field and requests related images.

[1896] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1897] 8. The generated image is sent to the terminal and presented to the user.

[1898] This series of processes enhances the user's pronunciation practice and visual understanding, while the emotion engine provides individualized feedback, improving learner motivation and enabling appropriate study plans.

[1899] An example of an input prompt is, "As a great knight, you must face a dragon. Are you ready?"

[1900] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1901] Program processing flow

[1902] Step 1:

[1903] The user speaks into the smartphone's microphone. The device captures the voice data from the microphone. The input is the user's voice, and the output is the captured voice data.

[1904] Step 2:

[1905] The device sends the captured audio data to the server. The input is the audio data, and the output is the data sent to the server.

[1906] Step 3:

[1907] The server converts the received voice data into text data using a voice recognition means, with the input being the voice data and the output being the converted text data.

[1908] Step 4:

[1909] The server analyzes the converted text data and generates real-time feedback. Specifically, it evaluates the accuracy of pronunciation and grammar. The input is the text data, and the output is the evaluation result and a feedback message.

[1910] Step 5:

[1911] The server sends text data and feedback messages to the terminal. The input is the evaluation result and the feedback message, and the output is the data sent to the terminal.

[1912] Step 6:

[1913] The terminal displays the received feedback to the user and provides audio feedback if necessary. The input is the data sent from the server, and the output is visual and audio feedback to the user.

[1914] Step 7:

[1915] The server analyzes text data using an emotion recognition engine to recognize the user's emotions. The input is text data, and the output is the emotion analysis result.

[1916] Step 8:

[1917] The server adjusts the content and tone of the feedback based on the results of the emotion analysis. Specifically, it generates encouraging messages to motivate the user. The input is the emotion analysis results, and the output is the adjusted feedback message.

[1918] Step 9:

[1919] The server sends the final feedback message to the terminal. The input is the adjusted feedback message, and the output is the data to be sent to the terminal.

[1920] Step 10:

[1921] A user enters a word or phrase they want to learn into a text input field and requests related visual content. The input is the text entered by the user, and the output is the requested data.

[1922] Step 11:

[1923] The terminal sends request data to the server. The input is the request data, and the output is the data sent to the server.

[1924] Step 12:

[1925] The server receives the request and generates the relevant image using a visual learning content generator, where the input is the user's text data and the output is the generated image data.

[1926] Step 13:

[1927] The generated image data is sent from the server to the terminal. The input is the image data, and the output is the data sent to the terminal.

[1928] Step 14:

[1929] The terminal presents the received image data to the user. The input is the data sent from the server, and the output is visual feedback to the user.

[1930] This process reinforces the user's speech practice and visual comprehension, and the emotion engine provides personalized feedback. For example, the prompt "You must face the dragon as a great knight. Are you ready?" can be used to guide the user through the dialogue.

[1931] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1932] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1933] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1934] [Fourth embodiment]

[1935] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1936] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1937] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1938] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1939] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1940] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1941] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1942] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1943] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1944] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1945] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1946] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1947] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1948] This invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. Each process of this system is explained in natural language below.

[1949] User registration and account creation

[1950] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[1951] Speech recognition and text conversion

[1952] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[1953] Providing real-time feedback

[1954] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which displays the feedback to the user. Audio feedback is also provided if necessary.

[1955] Visual learning aids

[1956] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[1957] Learning progress and tracking

[1958] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[1959] Specific examples

[1960] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[1961] 1. The user speaks "apple" into the smartphone microphone.

[1962] 2. The device captures the audio and sends it to the server.

[1963] 3. The server converts the speech to text and displays "apple" to the user.

[1964] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[1965] 5. The user types "apple" into the text input field and requests related images.

[1966] 6. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[1967] 7. The generated image is sent to the device and presented to the user.

[1968] This series of processes enhances the user's pronunciation practice and visual comprehension. In this way, the mode for carrying out the invention is embodied.

[1969] The processing flow will be explained below.

[1970] Step 1:

[1971] The user downloads and installs the app on their smartphone.

[1972] Step 2:

[1973] The user launches the app and goes to the new registration screen.

[1974] Step 3:

[1975] The user enters information such as name, email address, password, and initial English conversation level.

[1976] Step 4:

[1977] The terminal transmits the input information to the server.

[1978] Step 5:

[1979] The server receives the transmitted information and stores it in a database.

[1980] Step 6:

[1981] The server generates a confirmation message for completing the registration and sends it to the terminal.

[1982] Step 7:

[1983] The terminal displays a confirmation message to the user.

[1984] Step 8:

[1985] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[1986] Step 9:

[1987] The user begins speaking into the smartphone's microphone.

[1988] Step 10:

[1989] The device captures audio data input from the microphone.

[1990] Step 11:

[1991] The device transmits the captured audio data to the server in real time.

[1992] Step 12:

[1993] The server receives the voice data and converts it into text using voice recognition technology.

[1994] Step 13:

[1995] The server transmits the converted text data to the terminal.

[1996] Step 14:

[1997] The terminal displays the text data to the user.

[1998] Step 15:

[1999] The server analyzes the converted text data.

[2000] Step 16:

[2001] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[2002] Step 17:

[2003] The server transmits the generated feedback data to the terminal.

[2004] Step 18:

[2005] The device displays feedback to the user, and optionally provides audio feedback.

[2006] Step 19:

[2007] The user enters the word or phrase they want to learn into the text entry field.

[2008] Step 20:

[2009] The terminal transmits the input text data to the server.

[2010] Step 21:

[2011] A server receives the text data and generates related images using a visual learning content generator.

[2012] Step 22:

[2013] The server transmits the generated image data to the terminal.

[2014] Step 23:

[2015] The terminal displays the image to the user.

[2016] Step 24:

[2017] The server stores the user's voice utterance history and feedback data in a database.

[2018] Step 25:

[2019] The server continuously tracks the user's learning progress based on the data stored on the server.

[2020] Step 26:

[2021] The server evaluates the learning progress and generates a new learning plan.

[2022] Step 27:

[2023] The server sends the new lesson plan to the device.

[2024] Step 28:

[2025] The device will display your learning progress and new learning plan on a dashboard.

[2026] This series of steps creates a system that supports effective language learning.

[2027] Example 1

[2028] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2029] Conventional speech recognition systems and language learning applications have difficulty accurately converting user speech into text and providing appropriate feedback. Furthermore, they lack sufficient visual learning support and make it difficult to accurately track learning progress. This makes it difficult for users to effectively acquire a language.

[2030] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2031] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a terminal means for transmitting voice-input data, and a server means for analyzing the data and generating results, thereby enabling the server to accurately convert a user's speech into text, provide accurate feedback, provide visual learning support, and accurately track learning progress.

[2032] "Speech recognition means" refers to means for detecting the user's voice and processing it as digital data.

[2033] The "speech-to-text conversion means" is a means for converting detected speech data into text data.

[2034] "Text to speech conversion means" refers to means for converting text data into speech data.

[2035] A "real-time feedback providing means" is a means for generating and providing instant feedback based on voice or text input by a user.

[2036] A "visual learning content generation means" is a means for generating relevant visual learning content (eg, images and graphics) based on text data.

[2037] The "learning progress tracking means" is a means for continuously recording and tracking a user's learning activities and progress.

[2038] The "terminal means for transmitting voice-input data" refers to a device for a user to input voice and means for transmitting that data.

[2039] The "server means for analyzing the data and generating a result" is a server for analyzing the transmitted data and generating a result.

[2040] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, and learning progress tracking. A specific embodiment of the present invention is described in detail below.

[2041] System Overview

[2042] This invention is a system that allows users to input voice using a device such as a smartphone and receive real-time feedback. The server converts the voice data into text, provides related visual learning content, and tracks the user's learning progress.

[2043] Hardware and software used

[2044] Device: A mobile device such as a smartphone or tablet

[2045] Speech recognition technology: Speech recognition services such as Google Cloud Speech-to-Text API

[2046] Text-to-speech technology: Synthetic speech technology (e.g., Amazon Polly) for text-to-speech conversion

[2047] Visual learning content generation means: Image generation algorithms such as DALL-E 2

[2048] Database: A server-side database system for recording user data and learning progress.

[2049] User registration and account creation

[2050] The user downloads and installs the app onto a device such as a smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. A confirmation message confirming registration completion is sent from the server to the device, which displays it to the user.

[2051] Speech recognition and text conversion

[2052] The user taps the "Conversation Practice" button on the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. The converted text data is sent from the server to the device, which then displays it to the user.

[2053] Providing real-time feedback

[2054] The server instantly analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback can also be provided simultaneously.

[2055] Visual learning aids

[2056] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using DALL-E 2. The generated image data is sent to the device, which displays it to the user.

[2057] Learning progress and tracking

[2058] The server stores the user's speech history and feedback data in a database, continuously tracking the user's learning activities and comprehension. Based on the stored data, the server evaluates the user's learning progress and generates a new learning plan. This new learning plan is sent to the device, which then presents it to the user along with their learning progress.

[2059] Specific examples

[2060] For example, the specific process for practicing pronunciation of the word "apple" and obtaining visually related images is shown below.

[2061] 1. The user speaks "apple" into the smartphone microphone.

[2062] 2. The device captures the audio and sends it to the server.

[2063] 3. The server converts the speech to text and displays "apple" to the user.

[2064] 4. The server evaluates the accuracy of the pronunciation and generates and displays feedback such as "Your pronunciation is correct!"

[2065] 5. The user types "apple" into the text input field and requests related images.

[2066] 6. The device sends the text to the server, and the server generates an image of "apple" using DALL-E 2.

[2067] 7. The generated image is sent to the device and presented to the user.

[2068] This process enhances the user's pronunciation practice and visual comprehension.

[2069] Prompt Sentence Examples

[2070] How can a user speak the word "apple" into a smartphone microphone, receive real-time feedback, and see related visual images?

[2071] The above is a detailed description of the preferred embodiment of the present invention, which allows users to learn languages ​​more effectively.

[2072] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2073] Program processing flow

[2074] User registration and account creation

[2075] Step 1:

[2076] The user downloads and installs the app on their smartphone.

[2077] Enter: app download

[2078] Output: Installed apps

[2079] Step 2:

[2080] The user enters information such as name, email address, password, and English conversation level on the new registration screen.

[2081] Input: Name, email address, password, English conversation level

[2082] Output: Input data

[2083] Step 3:

[2084] The terminal transmits the input information to the server.

[2085] Input: Input data

[2086] Output: Server sent data

[2087] Step 4:

[2088] The server stores the received information in a database.

[2089] Input: Server transmission data

[2090] Output: Database update

[2091] Step 5:

[2092] The server sends a confirmation message to the terminal confirming the completion of registration.

[2093] Input: None

[2094] Output:Confirmation message

[2095] Step 6:

[2096] The terminal displays a confirmation message to the user.

[2097] Input:Confirmation message

[2098] Output: Registration complete

[2099] Speech recognition and text conversion

[2100] Step 1:

[2101] The user taps the "Practice Conversation" button within the app.

[2102] Input: Tap a button

[2103] Output: Start speech mode

[2104] Step 2:

[2105] The user begins speaking into the smartphone's microphone.

[2106] Input: Speech content

[2107] Output: Audio data

[2108] Step 3:

[2109] The device captures audio data input from the microphone.

[2110] Input: Audio data

[2111] Output: Captured audio data

[2112] Step 4:

[2113] The device transmits the voice data to the server in real time.

[2114] Input: Captured audio data

[2115] Output: Server sent data

[2116] Step 5:

[2117] The server receives the voice data and converts it into text using speech recognition technology.

[2118] Input: Server transmission data

[2119] Output: Converted text data

[2120] Step 6:

[2121] The server transmits the converted text data to the terminal.

[2122] Input: Converted text data

[2123] Output: Terminal sending data

[2124] Step 7:

[2125] The terminal displays the text data to the user.

[2126] Input: Terminal transmission data

[2127] Output: Text display

[2128] Providing real-time feedback

[2129] Step 1:

[2130] The server immediately analyzes the converted text data.

[2131] Input: Converted text data

[2132] Output: Feedback data

[2133] Step 2:

[2134] The server generates feedback based on what the user says.

[2135] Input: Feedback data

[2136] Output: Generated feedback

[2137] Step 3:

[2138] The server transmits the generated feedback data to the terminal.

[2139] Input: Generated feedback

[2140] Output: Terminal sending data

[2141] Step 4:

[2142] The device displays the feedback to the user.

[2143] Input: Terminal transmission data

[2144] Output: Feedback display

[2145] Step 5:

[2146] The device will also provide audio feedback if necessary.

[2147] Input: Feedback data

[2148] Output: Audio feedback

[2149] Visual learning aids

[2150] Step 1:

[2151] The user enters the word or phrase they want to learn into the text entry field.

[2152] Input: Word or phrase

[2153] Output: Input text

[2154] Step 2:

[2155] The terminal transmits the input text data to the server.

[2156] Input: Input text

[2157] Output: Server sent data

[2158] Step 3:

[2159] A server receives the text data and generates an associated image.

[2160] Input: Server transmission data

[2161] Output: Generated image data

[2162] Step 4:

[2163] The server transmits the generated image data to the terminal.

[2164] Input: Generated image data

[2165] Output: Terminal sending data

[2166] Step 5:

[2167] The terminal displays the image to the user.

[2168] Input: Terminal transmission data

[2169] Output: Image display

[2170] Learning progress and tracking

[2171] Step 1:

[2172] The server stores the user's voice utterance history and feedback data in a database.

[2173] Input: Feedback data and speech history

[2174] Output: Database update

[2175] Step 2:

[2176] The server continuously tracks the user's learning activities and understanding.

[2177] Input: Database data

[2178] Output:Tracking information

[2179] Step 3:

[2180] The server evaluates the user's learning progress based on the data stored.

[2181] Input:Tracking information

[2182] Output: Evaluation results

[2183] Step 4:

[2184] The server generates a new learning plan and sends it to the device.

[2185] Input: Evaluation result

[2186] Output: The generated learning plan

[2187] Step 5:

[2188] The device will present the user with their new learning plan and progress.

[2189] Input: Generated lesson plan

[2190] Output: Plan display

[2191] The above are the specific processing steps of the system program.

[2192] (Application example 1)

[2193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2194] Conventional virtual store systems have difficulty providing product explanations and purchasing support when users speak different languages. There has also been a demand for providing detailed product information in real time, multilingual support, and comprehensive purchasing support. Providing services to international customers without creating a language barrier has been particularly challenging, making improving the user experience a key challenge.

[2195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2196] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, a multilingual translation unit, and an answer generation unit using a generative AI model. This enables real-time multilingual product explanations and purchasing support in the virtual store, making it possible to provide services to international customers without language barriers.

[2197] A "voice recognition means" is a device or software that has the function of receiving voice input, analyzing the voice, and converting it into text data.

[2198] A "speech-to-text converter" is a system or algorithm that performs a process to convert speech data into text data.

[2199] A "text-to-speech converter" is a system or algorithm that performs a process to convert text data into speech data.

[2200] The "means for providing real-time feedback" refers to a device or software that has the function of generating immediate feedback in response to a user's input and providing it to the user.

[2201] A "visual learning content generator" is a device or software that has the functionality to generate visual learning materials based on text data or other input data.

[2202] A "learning progress tracking means" is a device or software that has the function of continuously recording and evaluating a user's learning activities and progress.

[2203] A "multilingual translation means" is a device or software that has the function of translating text and audio data between a wide variety of languages.

[2204] An "answer generation means using a generative AI model" is a system or algorithm that uses artificial intelligence technology to generate answers to users' questions.

[2205] The "means for providing purchasing support functions" refers to a device or software that has the function of allowing users to obtain information about products and services and assisting them in the purchasing process.

[2206] This invention relates to a system that provides real-time multilingual product explanations and purchasing support in a virtual store. The system is realized by communication between users and a server via an application installed on a mobile device such as a smartphone or smart glasses.

[2207] Voice input and recognition

[2208] 1. Terminal processing:

[2209] The user speaks a question about the product into the device's microphone.

[2210] The device captures the audio data and sends it to the server.

[2211] 2. Server processing:

[2212] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text.

[2213] Get product information

[2214] 3. Server processing:

[2215] The text data generated by speech recognition is used to query the product database using Django or Flask.

[2216] Retrieve relevant product information and translate it into the user's native language.

[2217] Multilingual translation and answer generation

[2218] 4. Server processing:

[2219] Use the Google Cloud Translation API to translate user questions into different languages.

[2220] The translated text is fed into a generative AI model (e.g., GPT-3) to generate a detailed answer.

[2221] The generated answer is translated back into the user's language.

[2222] Visual Content Generation

[2223] 5. Server Processing:

[2224] Based on the transferred product information, relevant visual learning content (e.g., images) is generated using OpenCV and matplotlib.

[2225] 6. Terminal Processing:

[2226] The generated image sent from the server is displayed to the user.

[2227] Configured example

[2228] For example, if a user says "What are the features of this smartphone case?" in a virtual store, the process will proceed as follows:

[2229] 1. The user speaks into the smartphone microphone, "Please tell me the features of this smartphone cover."

[2230] 2. The device captures the audio and sends it to the server.

[2231] 3. The server converts the speech into text and queries the product database to obtain "information about the smartphone cover."

[2232] 4. Translate the question into English via a translation API and provide the prompt "What are the features of this smartphone case?" to GPT-3.

[2233] 5. The server receives the answer generated by GPT-3 and retranslates it into the user's language.

[2234] 6. Display the generated text and related images on your smartphone.

[2235] Prompt Sentence Examples

[2236] "Please tell me the features of this smartphone case."

[2237] This will enable users to access detailed product information in real time in multiple languages ​​in the virtual store, making it possible to provide services to international customers without causing them language barriers.

[2238] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2239] Step 1:

[2240] The user speaks a question about a product into the microphone of their smartphone in the virtual store. This voice is the input data. The device recognizes the user's speech and captures it as digital voice data.

[2241] Step 2:

[2242] The device sends the captured audio data to the server. At this point, the input is audio data, and the output is the transmission of audio data to the server. The smartphone's communication module is used.

[2243] Step 3:

[2244] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API. This process results in voice data as input and text data as output. Specifically, the server sends the voice data to the API and analyzes the returned text data.

[2245] Step 4:

[2246] The server uses the text data to query the product database. The input is the converted text data, and the output is the product information. Using Django or Flask, the corresponding information is retrieved from the product database.

[2247] Step 5:

[2248] The server uses the Google Cloud Translation API to translate the user's question from the user's native language into a target language such as English. The input is text data, and the output is the translated text. Specifically, the server sends the text to be translated and the target language to the API, and receives the translation result.

[2249] Step 6:

[2250] The server uses a generative AI model (e.g., GPT-3) to generate an answer to the question based on the translated text. The translated question is input to the AI ​​model as a prompt. The input is the translated text, and the output is the generated answer text. The server sends the prompt to the generative AI model and receives the returned answer.

[2251] Step 7:

[2252] The server translates the generated answer into the user's native language using the Google Cloud Translation API again. The input is the generated answer text, and the output is the translated answer text into the user's native language. The translation procedure is the same as in step 5.

[2253] Step 8:

[2254] The server generates relevant visual learning content. The input is product data, and the output is the generated visual content (e.g., images). Specifically, it uses OpenCV and matplotlib to generate visual information related to the product.

[2255] Step 9:

[2256] The server sends the generated text and image data to the terminal. The input is the answer text and visual content, and the output is a status indicating completion of transmission to the terminal. The server sends the data using a communication protocol.

[2257] Step 10:

[2258] The terminal displays the received text and visual content to the user. The input is the data sent from the server, and the output is the content displayed on the terminal display. In concrete terms, the terminal reflects the data on the display and provides the user with visual and text information.

[2259] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2260] The present invention is implemented as a smartphone app for language learners, and its main processes are speech recognition, text conversion, voice feedback, visual learning content generation, learning progress tracking, and user emotion recognition. The system recognizes emotions from the user's speech and input text and provides feedback based on these emotions, enabling a more effective learning experience.

[2261] User registration and account creation

[2262] The user downloads and installs the app on their smartphone. After installation, the user moves to the new registration screen and enters information such as their name, email address, password, and initial English conversation level. The device collects this information and sends it to the server. The server receives the information and stores it in a database. After this, the server sends a confirmation message to the device confirming registration, which the device displays to the user.

[2263] Speech recognition and text conversion

[2264] The user taps the "Conversation Practice" button in the app to enter speech mode. The user begins speaking into the smartphone's microphone. The device captures the voice data input from the microphone and sends it to the server in real time. The server receives the voice data and converts it into text using speech recognition technology. The converted text data is sent from the server to the device, which then displays it to the user.

[2265] Providing real-time feedback

[2266] The server immediately analyzes the converted text data and generates feedback based on the user's speech. This feedback includes pronunciation accuracy, grammatical correctness, and appropriate expressions (example sentences). The server also adjusts the content and tone of the feedback using an emotion engine that recognizes emotions from the user's speech and input text. The generated feedback data is sent to the device, which then displays the feedback to the user. Audio feedback is also provided if necessary.

[2267] Visual learning aids

[2268] The user enters the words or phrases they want to learn into the text input field. The device sends the entered text data to the server. The server receives the text data and generates a related image using a visual learning content generation means. The generated image data is sent to the device, which displays it to the user.

[2269] Learning progress and tracking

[2270] The server stores the user's speech history and feedback data in a database, and continuously tracks the user's learning activities and comprehension. Furthermore, based on the user's emotional state recognized by the emotion engine, it influences the tracking results of the learning progress and adjusts the learning plan according to the specific emotional state. This new learning plan is sent to the device, which then presents it to the user along with the learning progress.

[2271] Specific examples

[2272] For example, consider a user who wants to practice pronouncing the word "apple" and obtain related visual images.

[2273] 1. The user speaks "apple" into the smartphone microphone.

[2274] 2. The device captures the audio and sends it to the server.

[2275] 3. The server converts the speech to text and displays "apple" to the user.

[2276] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[2277] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[2278] 6. The user types "apple" into the text input field and requests related images.

[2279] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[2280] 8. The generated image is sent to the terminal and presented to the user.

[2281] This series of processes enhances the user's pronunciation practice and visual understanding, and the emotion engine provides personalized feedback, improving the learner's motivation and creating an appropriate learning plan.

[2282] The processing flow will be explained below.

[2283] Step 1:

[2284] The user downloads and installs the app on their smartphone.

[2285] Step 2:

[2286] The user launches the app and goes to the new registration screen.

[2287] Step 3:

[2288] The user enters information such as name, email address, password, and initial English conversation level.

[2289] Step 4:

[2290] The terminal transmits the input information to the server.

[2291] Step 5:

[2292] The server receives the transmitted information and stores it in a database.

[2293] Step 6:

[2294] The server generates a confirmation message for completing the registration and sends it to the terminal.

[2295] Step 7:

[2296] The terminal displays a confirmation message to the user.

[2297] Step 8:

[2298] The user taps the "Practice Conversation" button in the app to enter speaking mode.

[2299] Step 9:

[2300] The user begins speaking into the smartphone's microphone.

[2301] Step 10:

[2302] The device captures audio data input from the microphone.

[2303] Step 11:

[2304] The device transmits the captured audio data to the server in real time.

[2305] Step 12:

[2306] The server receives the voice data and converts it into text using voice recognition technology.

[2307] Step 13:

[2308] The server transmits the converted text data to the terminal.

[2309] Step 14:

[2310] The terminal displays the text data to the user.

[2311] Step 15:

[2312] The server analyzes the converted text data.

[2313] Step 16:

[2314] The server generates feedback based on the user's speech, including pronunciation accuracy, grammar correctness, and appropriate expressions (example sentences).

[2315] Step 17:

[2316] The server uses an emotion engine that recognizes emotions from the user's speech and input text to adjust the content and tone of the feedback.

[2317] Step 18:

[2318] The server transmits the generated feedback data to the terminal.

[2319] Step 19:

[2320] The device displays feedback to the user, and optionally provides audio feedback.

[2321] Step 20:

[2322] The user enters the word or phrase they want to learn into the text entry field.

[2323] Step 21:

[2324] The terminal transmits the input text data to the server.

[2325] Step 22:

[2326] A server receives the text data and generates related images using a visual learning content generator.

[2327] Step 23:

[2328] The server transmits the generated image data to the terminal.

[2329] Step 24:

[2330] The terminal displays the image to the user.

[2331] Step 25:

[2332] The server stores the user's voice utterance history and feedback data in a database.

[2333] Step 26:

[2334] The server continuously tracks the user's learning progress based on the data stored on the server.

[2335] Step 27:

[2336] The server influences the tracking results of the learning progress based on the emotional state of the user recognized by the emotion engine.

[2337] Step 28:

[2338] The server adjusts the learning plan according to the particular emotional state and generates a new learning plan.

[2339] Step 29:

[2340] The server sends the new lesson plan to the device.

[2341] Step 30:

[2342] The device will present the user with their learning progress and new learning plan.

[2343] This series of steps enhances the user's pronunciation practice and visual comprehension, while the emotion engine provides personalized feedback to improve learner motivation and create appropriate learning plans.

[2344] Example 2

[2345] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2346] Previous language learning systems provided limited feedback on user utterances, and it was particularly difficult to provide feedback that took into account the user's emotional state. Furthermore, learning progress tracking was inconsistent, making it difficult to provide individually optimized learning plans. This resulted in a lack of learner motivation and an ineffective learning experience.

[2347] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2348] In this invention, the server includes a speech recognition unit, a speech-to-text conversion unit, a text-to-speech conversion unit, a real-time feedback providing unit, a visual learning content generation unit, a learning progress tracking unit, and an emotion recognition unit. This makes it possible to provide feedback based not only on the user's spoken content but also on the user's emotional state. Furthermore, by continuously tracking the learning progress and generating and providing an individually optimized learning plan, it is possible to increase the learner's motivation and achieve an effective learning experience.

[2349] "Speech recognition means" is a technology that captures a user's speech, analyzes the speech data, and converts it into linguistic text.

[2350] A "voice-to-text converter" is a function that converts captured voice data into a text format in a language.

[2351] "Text-to-speech conversion means" refers to a technology that generates text-format language data as speech output.

[2352] The "means for providing real-time feedback" is a function that provides immediate evaluation and feedback on corrections based on the user's utterances.

[2353] "Visual learning content generation means" is a technology that generates and visually displays related images and diagrams based on text entered by the user.

[2354] A "learning progress tracking means" is a system that continuously tracks and records a user's learning activities and progress.

[2355] "Emotion recognition means" is a technology that analyzes emotions from a user's speech or input text and recognizes their emotional state.

[2356] The present invention is implemented as a smartphone application for language learners, whose main processes include speech recognition, speech-to-text conversion, text-to-speech conversion, real-time feedback provision, visual learning content generation, learning progress tracking, and emotion recognition.

[2357] Hardware and software used

[2358] Hardware:

[2359] Smartphone (microphone, display)

[2360] software:

[2361] Speech recognition engine (Google Speech-to-Text API)

[2362] Emotion recognition engine (IBM Watson Emotional Recognition)

[2363] Database (MySQL)

[2364] Application server (Node.js)

[2365] Learning content generation tool (TensorFlow)

[2366] Specific processing of the program

[2367] User registration and account creation

[2368] Users download and install the app on their smartphone. After installation, they launch the app and enter information such as their name, email address, password, and English conversation level. The device collects this information and sends it to the server. The server stores it in a database and sends a message to the device confirming registration.

[2369] Speech recognition and text conversion

[2370] The user selects "Conversation Practice" and begins speaking. The device captures the audio with the microphone and sends the audio data to the server. The server converts the audio into text using the Google Speech-to-Text API and sends the converted text to the device for display.

[2371] Providing real-time feedback

[2372] The server analyzes the converted text and generates feedback on pronunciation accuracy, grammatical compatibility, and appropriate expression examples. Furthermore, the IBM Watson Emotional Recognition engine recognizes the user's emotions and adjusts the content and tone of the feedback accordingly. Feedback data is sent to the device, where it is displayed and provided to the user via voice.

[2373] Visual learning aids

[2374] When a user inputs a word or phrase they want to learn, the device sends the text data to the server, which then uses TensorFlow to generate a related image, which is then sent to the device and displayed to the user.

[2375] Learning progress and tracking

[2376] The server stores the user's speech history and feedback data in a database, tracks the user's learning progress, and adjusts the learning plan based on the emotion recognition results, sending the plan to the device and presenting it to the user.

[2377] Specific examples

[2378] For example, if a user says the word "apple" and wants to get a visually related image:

[2379] 1. The user speaks "apple" into the smartphone microphone.

[2380] 2. The device captures the audio and sends it to the server.

[2381] 3. The server converts the speech to text and displays "apple."

[2382] 4. The server evaluates the accuracy of the pronunciation and generates and sends feedback saying, "Your pronunciation is correct!"

[2383] 5. The server uses the emotion engine to recognize the user's confidence and generates motivating feedback such as, "That's great! Keep it up!"

[2384] 6. The user types "apple" into the text input field and requests related images.

[2385] 7. The server generates an image of an "apple" using the visual learning content generation means.

[2386] 8. The generated image is sent to the terminal and presented to the user.

[2387] Example prompts for generative AI models

[2388] Generate related images for visual understanding of the word "apple." The system converts the audio data of the user pronouncing "apple" into text, provides feedback on the accuracy of the pronunciation, and then analyzes the user's sentiment (confidence) to provide appropriate feedback. Next, the user types "apple" into a text field to request related visual images.

[2389] This prompt was generated based on the specific operations described above and is formatted to clearly communicate the expected behavior of the AI ​​model.

[2390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2391] Step 1:

[2392] The user downloads and installs the app on their smartphone.

[2393] Input: App download link

[2394] Output: Installed apps

[2395] Step 2:

[2396] The user launches the app and enters information such as their name, email address, password, and English conversation level on the new registration screen.

[2397] Input: User's personal information (name, email address, password, English conversation level)

[2398] Output: User information entered

[2399] Step 3:

[2400] The terminal transmits the user's input information to the server.

[2401] Input: User information

[2402] Output: A request to send data to the server

[2403] Step 4:

[2404] The server receives the transmitted user information and stores it in a database.

[2405] Input: User information

[2406] Output: A database containing user information

[2407] Step 5:

[2408] The server generates a confirmation message for completing the registration and sends it to the terminal.

[2409] Input: None (default confirmation message)

[2410] Output:Confirmation message

[2411] Step 6:

[2412] The terminal receives the confirmation message and displays it to the user.

[2413] Input:Confirmation message

[2414] Output: What is displayed to the user

[2415] Step 7:

[2416] The user taps the "Practice Conversation" button to enter speech mode.

[2417] Input: Button tap event

[2418] Output: Speech mode

[2419] Step 8:

[2420] The user begins speaking into the smartphone's microphone.

[2421] Input: Audio data

[2422] Output: Audio signal

[2423] Step 9:

[2424] The device captures audio data input from the microphone and transmits it to the server in real time.

[2425] Input: Audio signal

[2426] Output: Request to send audio data to the server

[2427] Step 10:

[2428] The server receives the voice data and converts it into text using a voice recognition engine.

[2429] Input: Audio data

[2430] Output: Text data

[2431] Step 11:

[2432] The server transmits the converted text data to the terminal.

[2433] Input: Text data

[2434] Output: Request to send text data to the terminal

[2435] Step 12:

[2436] The terminal receives the text data and displays it to the user.

[2437] Input: Text data

[2438] Output: What is displayed to the user

[2439] Step 13:

[2440] The server analyzes the text data and generates feedback including pronunciation accuracy, grammatical correctness, and appropriate expression.

[2441] Input: Text data

[2442] Output: Feedback message

[2443] Step 14:

[2444] The server uses an emotion recognition engine to recognize the user's emotions and adjust the content and tone of the feedback.

[2445] Input: Text data (speech content), voice data (speech tone)

[2446] Output: Adjusted feedback message

[2447] Step 15:

[2448] The server transmits the generated feedback data to the terminal.

[2449] Input: Feedback message

[2450] Output: Request to send feedback to device

[2451] Step 16:

[2452] The terminal receives the feedback and provides visual and audio feedback to the user.

[2453] Input: Feedback message

[2454] Output: Visual feedback / Audio feedback

[2455] Step 17:

[2456] Users enter the words or phrases they want to learn as text.

[2457] Input: Word or phrase

[2458] Output: Text data

[2459] Step 18:

[2460] The terminal transmits the text data to the server.

[2461] Input: Text data

[2462] Output: Request to send text data to the server

[2463] Step 19:

[2464] A server receives the text data and generates related images using a visual learning content generation tool.

[2465] Input: Text data

[2466] Output: Image data

[2467] Step 20:

[2468] The server transmits the generated image data to the terminal.

[2469] Input: Image data

[2470] Output: Image data transmission request to the device

[2471] Step 21:

[2472] The terminal receives the image data and presents it to the user.

[2473] Input: Image data

[2474] Output: Display image

[2475] Step 22:

[2476] The server stores the user's speech history and feedback data in a database and tracks their learning progress.

[2477] Input: Speech history, feedback data

[2478] Output: Updated database

[2479] Step 23:

[2480] The server adjusts the learning plan based on the emotion recognition results and generates a new learning plan.

[2481] Input: Emotion recognition results, learning progress data

[2482] Output: New lesson plan

[2483] Step 24:

[2484] The server sends the new lesson plan to the device.

[2485] Input: New lesson plan

[2486] Output: Request to send lesson plan to device

[2487] Step 25:

[2488] The terminal receives the study plan and presents it to the user.

[2489] Input: New lesson plan

[2490] Output: Display of the learning plan

[2491] The above is the specific processing flow of this system.

[2492] (Application example 2)

[2493] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2494] In language learning, conventional systems do not provide feedback that takes users' emotions into account, which can lead to a decrease in learner motivation. They also lack a means to effectively track learning progress and provide individually optimized learning plans. Furthermore, when providing an interactive learning experience, learners face challenges due to the lack of interactive story-based dialogue, which can lead to a lack of immersion.

[2495] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2496] In this invention, the server includes a speech recognition means, a speech-to-text conversion means, a text-to-speech conversion means, a real-time feedback providing means, a visual learning content generating means, a learning progress tracking means, a user emotion recognition means, an emotion-based feedback adjusting means, and a story-style dialogue content providing means, thereby enabling feedback based on the user's emotions and immersive learning through interactive story-style dialogue content.

[2497] Word definition

[2498] "Speech recognition means" is a function that receives voice data as input and converts it into text data.

[2499] The "voice-to-text conversion means" is a function that analyzes the voice data captured by the voice recognition means and converts it into corresponding text data.

[2500] The "text to speech conversion means" is a function that receives text data as input and outputs the content as speech data.

[2501] The "means for providing feedback in real time" is a function that instantly generates appropriate feedback based on the user's input and provides it to the user.

[2502] The "visual learning content generation means" is a function that generates visual content such as related images and videos based on the text and audio input by the user.

[2503] "Learning progress tracking means" is a function that records a user's learning activities and manages and evaluates their progress.

[2504] The "user emotion recognition means" is a function that analyzes emotions from the user's speech and input text.

[2505] The "feedback adjustment means based on emotion" is a function that adjusts the content and tone of the feedback provided based on the user's emotion recognized by the emotion recognition means.

[2506] The "means for providing narrative dialogue content" is a function for generating and providing interactive dialogue in the form of a story.

[2507] MODE FOR CARRYING OUT THE INVENTION

[2508] This invention is a language learning system that recognizes a user's speech in real time, converts it into text, and provides feedback. It also recognizes the user's emotions and provides feedback based on those emotions, making the learning experience more effective.

[2509] The system mainly consists of the following means:

[2510] 1. Voice recognition means:

[2511] The server captures the user's voice and uses a speech recognition algorithm to convert it into text, using tools such as the Google Speech Recognition API.

[2512] 2. Voice to text conversion methods:

[2513] The server converts the voice data received from the voice recognition means into text data in real time.

[2514] 3. Text to speech conversion methods:

[2515] The server uses Text-to-Speech (TTS) technology, such as the Google Text-to-Speech API, to provide audio feedback of the generated text.

[2516] 4. How to provide real-time feedback:

[2517] The server analyzes the converted text data, evaluates pronunciation accuracy and grammatical correctness, and generates feedback that is provided to the user instantly.

[2518] 5. Visual learning content generation methods:

[2519] The server generates relevant visual content (images and videos) based on the text entered by the user. For example, if a user enters that they want to learn the meaning of "apple," the server generates an image of "apple" and sends it to the device.

[2520] 6. Track your progress:

[2521] The server tracks the user's learning progress and stores the user's speech history and feedback data in a database, allowing the user to check their own learning progress.

[2522] 7. User Emotion Recognition Method:

[2523] The server has an emotion engine that analyzes emotions from user utterances and input text, enabling feedback based on the user's emotions.

[2524] 8. Emotion-based feedback regulation measures:

[2525] The server adjusts the content and tone of the feedback based on the emotional information recognized by the emotion engine. For example, if the user is nervous, it provides encouraging feedback.

[2526] 9. Narrative dialogue content delivery methods:

[2527] The server generates interactive story-style dialogue content, allowing users to participate in learning as characters in the story and learn a language through dialogue.

[2528] As a concrete example, consider a user trying to practice pronouncing the word "apple" and get a related visual image:

[2529] 1. The user speaks "apple" into the smartphone microphone.

[2530] 2. The device captures the audio and sends it to the server.

[2531] 3. The server converts the speech to text and displays "apple" to the user.

[2532] 4. The server evaluates the accuracy of the pronunciation and displays feedback such as "Your pronunciation is correct!"

[2533] 5. The server uses its emotion engine to recognize that the user is confident from the speech and generates motivational feedback (e.g., "That's great! Keep it up!").

[2534] 6. The user types "apple" into the text input field and requests related images.

[2535] 7. The terminal sends the text to the server, and the server generates an image of "apple" using a visual learning content generation means.

[2536] 8. The generated image is sent to the terminal and presented to the user.

[2537] This series of processes enhances the user's pronunciation practice and visual understanding, while the emotion engine provides individualized feedback, improving learner motivation and enabling appropriate study plans.

[2538] An example of an input prompt is, "As a great knight, you must face a dragon. Are you ready?"

[2539] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2540] Program processing flow

[2541] Step 1:

[2542] The user speaks into the smartphone's microphone. The device captures the voice data from the microphone. The input is the user's voice, and the output is the captured voice data.

[2543] Step 2:

[2544] The device sends the captured audio data to the server. The input is the audio data, and the output is the data sent to the server.

[2545] Step 3:

[2546] The server converts the received voice data into text data using a voice recognition means, with the input being the voice data and the output being the converted text data.

[2547] Step 4:

[2548] The server analyzes the converted text data and generates real-time feedback. Specifically, it evaluates the accuracy of pronunciation and grammar. The input is the text data, and the output is the evaluation result and a feedback message.

[2549] Step 5:

[2550] The server sends text data and feedback messages to the terminal. The input is the evaluation result and the feedback message, and the output is the data sent to the terminal.

[2551] Step 6:

[2552] The terminal displays the received feedback to the user and provides audio feedback if necessary. The input is the data sent from the server, and the output is visual and audio feedback to the user.

[2553] Step 7:

[2554] The server analyzes text data using an emotion recognition engine to recognize the user's emotions. The input is text data, and the output is the emotion analysis result.

[2555] Step 8:

[2556] The server adjusts the content and tone of the feedback based on the results of the emotion analysis. Specifically, it generates encouraging messages to motivate the user. The input is the emotion analysis results, and the output is the adjusted feedback message.

[2557] Step 9:

[2558] The server sends the final feedback message to the terminal. The input is the adjusted feedback message, and the output is the data to be sent to the terminal.

[2559] Step 10:

[2560] A user enters a word or phrase they want to learn into a text input field and requests related visual content. The input is the text entered by the user, and the output is the requested data.

[2561] Step 11:

[2562] The terminal sends request data to the server. The input is the request data, and the output is the data sent to the server.

[2563] Step 12:

[2564] The server receives the request and generates the relevant image using a visual learning content generator, where the input is the user's text data and the output is the generated image data.

[2565] Step 13:

[2566] The generated image data is sent from the server to the terminal. The input is the image data, and the output is the data sent to the terminal.

[2567] Step 14:

[2568] The terminal presents the received image data to the user. The input is the data sent from the server, and the output is visual feedback to the user.

[2569] This process reinforces the user's speech practice and visual comprehension, and the emotion engine provides personalized feedback. For example, the prompt "You must face the dragon as a great knight. Are you ready?" can be used to guide the user through the dialogue.

[2570] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2571] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2572] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2573] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2574] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2575] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2576] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2577] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2578] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2579] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2580] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2581] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2582] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2583] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2584] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2585] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2586] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2587] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2588] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2589] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2590] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2591] The following is further disclosed regarding the above embodiment.

[2592] (Claim 1)

[2593] a speech recognition means;

[2594] a speech-to-text converter;

[2595] a text-to-speech converter;

[2596] A means of providing real-time feedback;

[2597] A visual learning content generation means;

[2598] A system including a learning progress tracking means.

[2599] (Claim 2)

[2600] A means for converting user utterances into text in real time;

[2601] means for providing accurate feedback based on the transcribed speech;

[2602] 10. The system of claim 1, further comprising means for displaying said feedback to a user.

[2603] (Claim 3)

[2604] 10. The system of claim 1, further comprising means for generating relevant visual learning content using image generation means based on text entered by the user, and presenting the generated content to the user.

[2605] "Example 1"

[2606] (Claim 1)

[2607] a speech recognition means;

[2608] a speech-to-text converter;

[2609] a text-to-speech converter;

[2610] A means of providing real-time feedback;

[2611] A visual learning content generation means;

[2612] learning progress tracking means;

[2613] a terminal means for transmitting voice-input data;

[2614] A system including server means for analyzing said data and generating results.

[2615] (Claim 2)

[2616] A means for converting user utterances into text in real time;

[2617] means for providing accurate feedback based on the transcribed speech;

[2618] 10. The system of claim 1, further comprising means for displaying said feedback to a user.

[2619] (Claim 3)

[2620] 10. The system of claim 1, further comprising means for generating relevant visual learning content using image generation means based on text entered by the user, and presenting the generated content to the user.

[2621] "Application Example 1"

[2622] (Claim 1)

[2623] a speech recognition means;

[2624] a speech-to-text converter;

[2625] a text-to-speech converter;

[2626] A means of providing real-time feedback;

[2627] A visual learning content generation means;

[2628] learning progress tracking means;

[2629] Multilingual translation means;

[2630] A system that includes a means for generating answers using a generative AI model.

[2631] (Claim 2)

[2632] A means for converting user utterances into text in real time;

[2633] means for providing accurate feedback based on the transcribed speech;

[2634] means for displaying said feedback to a user;

[2635] Translation tools that enable multilingual support,

[2636] 10. The system of claim 1, further comprising means for generating answers to user questions using the generative AI model.

[2637] (Claim 3)

[2638] means for generating relevant visual learning content using image generation means based on the text entered by the user, and presenting the generated content to the user;

[2639] 10. The system of claim 1, further comprising means for providing a purchasing support function.

[2640] "Example 2: Combining Emotion Engines"

[2641] (Claim 1)

[2642] a speech recognition means;

[2643] a speech-to-text converter;

[2644] a text-to-speech converter;

[2645] A means of providing real-time feedback;

[2646] A visual learning content generation means;

[2647] learning progress tracking means;

[2648] A system including an emotion recognition means.

[2649] (Claim 2)

[2650] A means for converting user utterances into text in real time;

[2651] means for providing accurate feedback based on the transcribed speech;

[2652] means for displaying said feedback to a user;

[2653] 10. The system of claim 1, further comprising means for adjusting the content and tone of the feedback using the emotion recognition means.

[2654] (Claim 3)

[2655] 10. The system of claim 1, further comprising means for generating relevant visual learning content using image generation means based on text entered by the user, and presenting the generated content to the user.

[2656] "Application example 2 when combining emotion engines"

[2657] (Claim 1)

[2658] a speech recognition means;

[2659] a speech-to-text converter;

[2660] a text-to-speech converter;

[2661] A means of providing real-time feedback;

[2662] A visual learning content generation means;

[2663] learning progress tracking means;

[2664] A means for recognizing user emotions;

[2665] an emotion-based feedback adjustment means;

[2666] A system including a means for providing narrative interactive content.

[2667] (Claim 2)

[2668] A means for converting user utterances into text in real time;

[2669] means for providing accurate feedback based on the transcribed speech;

[2670] means for displaying said feedback to a user;

[2671] 10. The system of claim 1, further comprising means for providing said feedback to the user audibly.

[2672] (Claim 3)

[2673] means for generating relevant visual learning content using image generation means based on the text entered by the user, and presenting the generated content to the user;

[2674] 2. The system according to claim 1, further comprising a language learning support means using story-based interactive content. [Explanation of symbols]

[2675] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice recognition means; a speech-to-text converter; a text-to-speech converter; A means of providing real-time feedback; A visual learning content generation means; A system including a learning progress tracking means.

2. A means for converting user utterances into text in real time; means for providing accurate feedback based on the transcribed speech; 10. The system of claim 1, further comprising means for displaying said feedback to a user.

3. 2. The system of claim 1, further comprising means for generating related visual learning content using image generation means based on text entered by the user, and presenting the generated content to the user.

Images