System

Abstract

A system is provided.SOLUTION: A system comprising: a terminal recording a user's reading aloud; the terminal transmitting the recorded voice data to a server; the server transcribing the received voice data; the server analyzing the transcribed text and performing intonation evaluation, misstatement detection, and reading speed calculation; the server generating feedback and scores based on the analysis; and the terminal displaying the feedback and scores to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the number of dual-income households is increasing, making it difficult for parents to find time to support their children with their homework. The burden on parents is particularly heavy when it comes to elementary school children's reading aloud homework. Therefore, there is a need for a system that can help children effectively complete their reading aloud homework and check their learning progress. Furthermore, to increase children's motivation to learn, there is also a need for a system that allows them to practice reading aloud while having fun. [Means for solving the problem]

[0005] The present invention provides a system in which a device records a user's reading aloud and transmits the recorded audio data to a server. The server transcribes the received audio data, evaluates intonation, detects slip-ups, and calculates reading speed. The server then generates feedback and a score based on the analysis results, and the device displays the feedback and score to the user. The system also provides a battle mode, independently analyzing the reading aloud data of multiple users to generate a score and determine the winner of the battle. The server can also record users' progress and trends over a long period of time and provide feedback on improvements and progress over time. This system effectively supports children's reading aloud practice while reducing the burden on parents.

[0006] A "terminal" is a device that allows a user to record a reading aloud and send the audio data to the server, specifically a smartphone or tablet.

[0007] "User" refers to an individual who reads aloud and provides the audio data to the system via a terminal.

[0008] "Server" refers to the central computing device of the system that receives the voice data sent from the device, transcribes and analyzes it, and generates and returns feedback and scores.

[0009] "Audio data" refers to audio files recorded by a user through a terminal and transmitted to a server.

[0010] "Transcription" refers to the process of analyzing audio data and converting it into text data.

[0011] "Intonation" refers to changes in intonation and pitch when reading aloud.

[0012] A "slip-up" refers to a word or part of a sentence that the user mispronounces while reading aloud.

[0013] "Reading speed" refers to the calculated speed at which a user reads text aloud.

[0014] "Feedback" refers to specific improvements and evaluation comments provided by the server based on the analysis results.

[0015] "Score" refers to the overall evaluation value calculated by the server based on the analysis results of the reading aloud.

[0016] "Battle mode" refers to a function in which multiple users each read aloud and the results are compared to determine the winner.

[0017] "Progress" refers to data that shows a user's ongoing learning outcomes and growth in reading aloud. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention is a system in which a terminal records a user's reading aloud and transmits the recorded voice data to a server. How this system is actually implemented will be described below with specific examples.

[0040] Recording and sending readings

[0041] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with a specific text as a reading task. The user reads the presented text aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device sends the recorded voice data to a server via the Internet.

[0042] Analysis of audio data and feedback generation

[0043] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[0044] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0045] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0046] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0047] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[0048] Providing and Viewing Feedback

[0049] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[0050] Implementing Battle Mode

[0051] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[0052] In this way, the present invention is implemented as a system that effectively supports children's reading aloud learning while reducing the burden on parents in a form that is adapted to modern lifestyles.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The user launches a dedicated application on their device, which then presents the user with a reading assignment text.

[0056] Step 2:

[0057] The user reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device saves the recorded audio data.

[0058] Step 3:

[0059] The device transmits the stored audio data to a server over the Internet using a secure protocol (e.g., HTTPS).

[0060] Step 4:

[0061] The server receives the voice data sent from the device and temporarily stores it for subsequent processing.

[0062] Step 5:

[0063] The server extracts the stored voice data and transcribes it using a speech recognition engine, which then analyzes the voice data and generates text data.

[0064] Step 6:

[0065] The server analyzes the transcribed text. The first step in the analysis is to evaluate intonation, which evaluates the frequency of intonation and specific phrases in the text.

[0066] Step 7:

[0067] The server then compares the transcribed text with pre-trained correct text, detecting spelling errors, mispronunciations, and other mistakes.

[0068] Step 8:

[0069] The server calculates the reading speed based on the total length of the text and the playback time of the audio data, and evaluates whether the reading speed is appropriate.

[0070] Step 9:

[0071] The server generates a comprehensive feedback and score based on the results of intonation evaluation, slip-up detection, and reading speed calculations. An example of specific feedback might be, "Your intonation is lacking. Try to read with more emotion."

[0072] Step 10:

[0073] The server sends the generated feedback and score to the terminal using unique identification information for each user.

[0074] Step 11:

[0075] The device receives feedback and displays the score from the server to the user, allowing the user to identify areas for improvement in their reading aloud and improve their learning.

[0076] Step 12:

[0077] If a user selects Battle Mode, a process of reading aloud competition with parents or friends will begin. Reading aloud data is sent from each device to the server and analyzed individually.

[0078] Step 13:

[0079] In battle mode, the server analyzes multiple reading data independently, generates feedback and scores for each, and compares the scores to determine the winner.

[0080] Step 14:

[0081] The server sends the battle results and feedback to the terminal, which displays the results to the user and promotes communication between users.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Traditional reading aloud education requires parents or teachers to directly supervise and provide feedback to children, which requires a lot of time and effort. Furthermore, evaluation of reading aloud intonation, speech mistakes, and reading speed is subjective, making objective assessment difficult. Furthermore, there is a lack of methods to gamify reading aloud practice and increase motivation by competing with other users.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for the user to start reading aloud, means for the terminal to record the user's reading aloud, means for the terminal to transmit the recorded voice data to the server, means for the server to transcribe the received voice data, means for the server to analyze the transcribed text and evaluate intonation, detect errors, and calculate reading speed, means for the server to generate feedback and a score based on the analysis results, means for the terminal to display the feedback and score to the user, and means for selecting a battle mode.This allows the user to effectively practice reading aloud, understand areas for improvement based on feedback, and increase motivation to learn by competing with other users.

[0087] "Reading aloud" is the act of a user reading aloud the presented text.

[0088] "Terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0089] "Voice data" refers to digital data containing the user's speech recorded by the device's built-in microphone or an external microphone.

[0090] "Server" refers to the central processing unit that receives, analyzes, transcribes, and generates feedback from audio data.

[0091] "Transcription" is the process of analyzing audio data and converting what is spoken into text data.

[0092] "Intonation" is an element that evaluates the changes in pitch and strength in the user's vocalization.

[0093] "Errors" refer to mispronunciations or slip-ups that the user makes while reading aloud.

[0094] "Reading speed" is a factor that measures how quickly a user reads a given text aloud.

[0095] "Feedback" refers to comments on improvements and evaluations that the server provides to the user based on the analysis results.

[0096] "Score" refers to a numerical result that comprehensively evaluates the user's reading aloud.

[0097] "Battle mode" refers to a feature in which multiple users compete against each other for the highest reading score.

[0098] This invention is a system in which a terminal records a user's reading aloud and transmits the audio data to a server. This system applies speech recognition technology and analyzes the recorded data to provide accurate feedback to the user. The specific processing procedures and implementation outline of the system are described below.

[0099] Hardware and Software Used

[0100] 1. Hardware:

[0101] Device: A mobile device such as a smartphone or tablet.

[0102] Microphone: The microphone built into the device.

[0103] 2. Software:

[0104] Dedicated application: An application that displays reading assignments and records audio.

[0105] Server-side analysis program: A program that performs audio analysis, transcription, and feedback generation.

[0106] Speech Recognition Engine: Speech recognition software such as Google Cloud Speech-to-Text.

[0107] Recording and sending readings

[0108] The user launches a dedicated application on the device and reads the displayed text aloud. The device uses a built-in microphone to record the user's voice and transmits the recording to a server via the Internet. For example, the following prompt sentence is used:

[0109] "Please record today's reading assignment and send it to the server."

[0110] Analysis of audio data and feedback generation

[0111] The server temporarily stores the received voice data and transcribes it using a speech recognition engine such as Google Cloud Speech-to-Text.The server then analyzes the transcribed data and performs the following evaluations:

[0112] Intonation assessment: Analyzing the intonation and stress of speech.

[0113] Error detection: Identifying errors by comparing the transcribed text with the exact text.

[0114] Reading speed calculation: Calculates reading speed by comparing the length of the text with the recording time.

[0115] The server generates feedback and a score based on the analysis results. For example, if the intonation is insufficient, the server will provide feedback such as "Your intonation is insufficient. Try reading with more emotion." An example of a prompt is as follows:

[0116] "Analyze recorded speech and generate ratings for intonation, errors, and reading speed."

[0117] Providing and Viewing Feedback

[0118] The feedback and score generated by the server are sent to the device, which displays it to the user, allowing the user to understand where they need to improve their reading. For example, the following message is displayed on the device:

[0119] "Your intonation is lacking, please try again."

[0120] Implementing Battle Mode

[0121] The invention also includes a battle mode in which multiple users compete in reading aloud. When a user selects the battle mode, each reading aloud is sent to the server and analyzed independently. The analysis results are then used to generate feedback and a score for each user, and a final winner is determined. The prompt is as follows:

[0122] "Run Battle Mode and compare each user's score to determine the winner."

[0123] As a result, the present invention is implemented as a system that effectively supports the user's reading aloud learning, improves the reading aloud technique based on feedback, and strengthens motivation to learn.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] The device launches a dedicated application.

[0127] Input: User action.

[0128] Output: The home screen of the application.

[0129] Specific operation: The user taps the application icon on the device to launch the application. The app displays the home screen, which displays the reading assignment and settings menu.

[0130] Step 2:

[0131] The terminal presents the text to the user for reading aloud.

[0132] Enter: the application home screen.

[0133] Output: The text that will be displayed as today's reading assignment.

[0134] Specific operation: The device displays a specific text on the screen as "Today's assignment," and the user prepares to read the text.

[0135] Step 3:

[0136] The user reads the presented text aloud.

[0137] Input: The presented text.

[0138] Output: Audio to be read aloud.

[0139] Specific action: The user reads aloud the presented text (e.g., "red shoes"), paying attention to mispronunciations and the volume of the voice during this stage.

[0140] Step 4:

[0141] The device records the user's voice.

[0142] Input: User's speech reading.

[0143] Output: Recorded audio data.

[0144] Specific operation: The device activates the built-in microphone and records the user's reading. When the recording is complete, the device generates an audio file.

[0145] Step 5:

[0146] The device sends the recorded audio data to the server.

[0147] Input: Recorded audio data.

[0148] Output: The audio data sent to the server.

[0149] Specific operation: The device compresses the recorded audio data and sends it to the server via the Internet. The device then verifies that the data was sent successfully.

[0150] Step 6:

[0151] The server temporarily stores the voice data and transcribes it using a voice recognition engine.

[0152] Input: Audio data received from the device.

[0153] Output: Transcribed text data.

[0154] Specific operation: The server temporarily saves the voice data in a directory where it stores it, then calls a voice recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[0155] Step 7:

[0156] The server analyzes the transcription data.

[0157] Input: Transcribed text data.

[0158] Output: Intonation assessment, error detection, and reading speed calculation.

[0159] Specific operation: The server runs an analysis program to evaluate intonation, detect errors, and calculate reading speed. Specifically, it compares the transcription data with the original text and evaluates pronunciation intonation, error locations, and reading speed.

[0160] Step 8:

[0161] The server generates feedback and a score.

[0162] Input: Intonation assessment, error detection, and reading speed results.

[0163] Output: Feedback and score.

[0164] Specific operation: The server generates a feedback message and an overall score based on the analysis results. For example, it may provide feedback such as, "Your intonation is lacking. Try to read with more emotion."

[0165] Step 9:

[0166] The server sends feedback and scores to the device.

[0167] Input: Generated feedback and score.

[0168] Output: Data sent to the terminal.

[0169] Specific operation: The server assembles the feedback message and score into a data packet and sends it to the device via the Internet.

[0170] Step 10:

[0171] The device displays feedback and scores to the user.

[0172] Input: Feedback and score received from the server.

[0173] Output: Feedback and score displayed to the user.

[0174] Specific operation: The device displays the received feedback message and score on the screen. The user can check the feedback and reflect it in the next reading.

[0175] (Application example 1)

[0176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] While existing reading assessment systems aim to improve users' reading skills, they lack practical application in specific situations and contexts, especially when it comes to assessing product explanations and comprehension. Furthermore, they lack the ability to motivate users by allowing them to compete with other users. This leaves no effective tools for improving product comprehension and explanation skills in virtual stores.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0179] In this invention, the server includes means for recording audio data of product information read aloud by a user on a terminal and transmitting the recorded data to the server, means for the server to transcribe and analyze the audio data, means for generating feedback and a score based on the analysis results and providing them to the user, and means for displaying the results of competitions with other users for the read aloud product information. This allows users to read aloud product descriptions in a virtual store, be evaluated on their level of understanding and explanation ability, and compete with other users, thereby improving their product understanding and explanation ability.

[0180] The "means for the terminal to record the user's reading aloud" is a combination of hardware and software for recording the user's voice reading aloud.

[0181] The "means for transmitting audio data recorded by the terminal to a server" is a function for transmitting recorded audio data to a remote server via a network.

[0182] The "means for transcribing voice data received by the server" is a function including a voice recognition engine that converts voice data received by the server into text format.

[0183] "Means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed" refers to a function that runs algorithms based on the transcribed text data to analyze a speaker's intonation, pronunciation errors, and reading speed.

[0184] "Means for the server to generate feedback and a score based on the analysis results" refers to a function that generates points for improvement and an overall evaluation score for the user based on the analysis results.

[0185] The "means for the terminal to display the feedback and score to the user" is a function for displaying the generated feedback and score on the screen of the user's device.

[0186] "A means of having specific product information read aloud in a virtual store, analyzing the level of understanding and explanation ability, and providing feedback" is a function that allows a user to read aloud product information in a virtual space, evaluate the level of understanding of the information and explanation skills, and provide feedback.

[0187] The "means for displaying the results of a competition with other users for the product information read aloud" is a function that allows a plurality of users to compete for scores based on the product information read aloud, and displays the results.

[0188] This invention is a system for evaluating a user's reading skills and product comprehension. Below, we will explain in natural language how this system is actually implemented by explaining the program processing procedures.

[0189] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with specific product information as a reading task. The user reads the presented product information aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device transmits the recorded voice data to a server via the Internet.

[0190] The server temporarily stores the voice data received from the device and transcribes it using a speech recognition engine (for example, Google Cloud Speech-to-Text API). Once transcription is complete, the server analyzes the voice data using the following procedure.

[0191] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0192] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0193] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0194] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[0195] The system also includes a battle mode function. When a user selects battle mode, they can compete in reading aloud with other users. Each user reads aloud and sends each audio data to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and each user's feedback to the user, stimulating a competitive spirit and encouraging improvement of reading aloud skills.

[0196] Examples of concrete examples and prompts

[0197] The user browses product descriptions in a virtual store and reads them aloud. The app records the audio and sends it to the server. The server analyzes the recording and provides feedback to improve the user's reading skills and product comprehension. Specific examples of prompts include:

[0198] Please read aloud the description of the product "Smart Watch A." ​​We will provide feedback on what you read aloud.

[0199] In this way, the present invention provides a system that allows users to gain a deeper understanding of product information in a virtual store and improve their reading skills and product comprehension by competing with other users.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1:

[0202] The user launches a dedicated application on the device and checks the specific product information presented as a reading task. The input is the product text information, and the output is a state ready for reading aloud. In this step, the device displays the product text to the user and provides an interface for preparing for reading aloud.

[0203] Step 2:

[0204] The user reads aloud the presented product information. The input is the user's voice, and the output is the recorded voice data. The device records the reading using the built-in microphone. In this step, the device receives the user's voice and saves it as an audio file (e.g., WAV format).

[0205] Step 3:

[0206] After the recording is complete, the device sends the recorded audio data to the server via the Internet. The input is the recorded audio data, and the output is the completion status of the audio data transfer to the server. The device uploads the audio file to the server using an HTTP request.

[0207] Step 4:

[0208] The server temporarily stores the received voice data and transcribes it using a speech recognition engine. The input is voice data, and the output is transcribed text data. The server converts the voice data into text using speech recognition technology such as the Google Cloud Speech-to-Text API.

[0209] Step 5:

[0210] The server analyzes the transcribed text data, evaluating intonation patterns, pronunciation errors, and reading speed. The input is the transcription data, and the output is the analysis results. The server uses algorithms to evaluate intonation, detect spelling errors, and calculate the reading speed.

[0211] Step 6:

[0212] The server generates specific feedback and an overall score based on the analysis results. The input is the analysis result data, and the output is feedback and a score. The server generates advice and evaluation points for the user based on the analysis data.

[0213] Step 7:

[0214] The server sends the generated feedback and score to the terminal, which displays it to the user. The input is the feedback and score, and the output is the information displayed to the user. The terminal displays the received data on the user interface.

[0215] Step 8:

[0216] When Battle Mode is selected, each user's reading data is analyzed independently to generate a score so that they can compete with other users. The input is the reading data of multiple users, and the output is feedback to each user, a score, and the winner's decision. The server uses a competitive algorithm to determine the winner and sends the result to the device.

[0217] By following these steps, the system can analyze and evaluate the user's reading skills, helping them improve their understanding of product information and their ability to explain it. Furthermore, the battle mode can stimulate competitive spirit and further motivate users.

[0218] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0219] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. Below, we will explain how to actually implement this system, using concrete examples.

[0220] Recording and sending readings

[0221] First, the user launches a dedicated application on their device (smartphone or tablet). The application then presents the user with a reading assignment text. The user then reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device sends the recorded audio data to a server via the Internet.

[0222] Analysis of audio data and feedback generation

[0223] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[0224] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0225] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0226] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0227] Furthermore, the server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[0228] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected and the user is recognized as nervous, the server will generate an overall score (e.g., 78 points) along with feedback such as, "Your intonation is lacking. Try reading more relaxed."

[0229] Providing and Viewing Feedback

[0230] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[0231] Implementing Battle Mode

[0232] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[0233] Long-term progress tracking and feedback

[0234] In addition, the server records the user's analysis results over a long period of time. This allows the server to understand the user's progress and trends, and provide feedback on improvements and progress over time. For example, by comparing past data with current data, the server can provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0235] In this way, the present invention is implemented as a system that is adapted to modern lifestyles, effectively supports children's reading aloud learning while reducing the burden on parents, and by combining it with an emotion engine, can provide more accurate feedback.

[0236] The processing flow will be explained below.

[0237] Step 1:

[0238] The user starts a dedicated application on their device. The application presents the user with a reading assignment text. The user reads the presented text aloud and prepares to begin reading aloud.

[0239] Step 2:

[0240] The device uses the built-in microphone to record the user's reading in real time, and once the recording is complete, the device temporarily stores the recorded audio data in local storage.

[0241] Step 3:

[0242] The device sends the recorded audio data to the server using a secure protocol (e.g., HTTPS). At the same time, the status of the transmission is displayed to the user.

[0243] Step 4:

[0244] The server receives the voice data sent from the device and stores it in the server, where it is prepared for analysis.

[0245] Step 5:

[0246] The server uses a speech recognition engine to transcribe the voice data. The speech recognition engine analyzes the voice data and generates text data. Once transcription is complete, the text data is stored on the server.

[0247] Step 6:

[0248] The server analyzes the transcribed text. The analysis is performed as follows:

[0249] Intonation assessment: The server detects intonation and the frequency of certain phrases in the text and assesses intonation.

[0250] Misspelling detection: The server compares the transcribed text with pre-trained correct text to detect spelling errors and mispronunciations.

[0251] Reading speed calculation: The server calculates the reading speed based on the total length of the text and the playback time of the audio data.

[0252] Step 7:

[0253] The server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[0254] Step 8:

[0255] The server evaluates intonation, detects slip-ups, calculates reading speed, and recognizes emotions, and generates specific feedback and an overall score. For example, it might generate feedback like, "Your intonation is lacking. Try reading more relaxedly," and calculate an overall score (e.g., 78 points).

[0256] Step 9:

[0257] The server sends the generated feedback and score to the device, using each user's identification information to ensure data integrity.

[0258] Step 10:

[0259] The device receives feedback and displays the score from the server to the user, who can then use the feedback to identify areas for improvement and practice for the next time.

[0260] Step 11:

[0261] When a user selects the battle mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and the audio data is sent from their device to the server.

[0262] Step 12:

[0263] In battle mode, the server analyzes each user's voice data independently and generates feedback and scores for each user. Based on the generated scores, the server determines the winner.

[0264] Step 13:

[0265] The server sends the results of the battle mode and feedback to the terminal, which displays the winner and feedback to the user, promoting communication between users.

[0266] Step 14:

[0267] The server records the user's progress and trends over time, allowing it to provide feedback on improvements and progress over time. It compares past data with current data and provides feedback such as, "Your reading speed is improving compared to before. Keep up the good work."

[0268] Example 2

[0269] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0270] Currently, there are systems that evaluate users' reading ability, but they lack detailed analysis of factors such as emotional state, intonation, and reading speed, and provide insufficient comprehensive feedback. Furthermore, they lack a mechanism for accurate and fair analysis and evaluation in battle modes where multiple users compete in reading aloud. Furthermore, a system that records users' long-term progress and provides appropriate feedback is also needed.

[0271] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0272] In this invention, the server includes a means for transcribing voice data, a means for analyzing the transcribed text to evaluate intonation, detect slip-ups, and calculate reading speed, and a means for providing feedback and a score based on the analysis results, as well as a means for analyzing the user's emotional state from the voice data. This enables comprehensive and precise analysis and feedback of the user's reading aloud. Furthermore, when multiple users compete in battle mode, each reading aloud data can be analyzed independently, enabling fair evaluation and winner determination. Furthermore, the server can support the user's continuous growth by recording the user's analysis results over a long period of time and providing progress and areas for improvement over time.

[0273] A "terminal" is an electronic device used by a user, and includes mobile information terminals such as smartphones and tablets.

[0274] "Reading aloud" refers to the act of a user reading a text aloud.

[0275] "Recording" refers to the act of recording audio as digital data using a microphone attached to a terminal.

[0276] "Audio data" refers to data in which recorded audio is saved in digital format.

[0277] "Server" refers to a computer system that receives, stores, and analyzes audio data.

[0278] "Transcription" is the process of converting audio data into text data using speech recognition technology.

[0279] "Intonation" refers to changes in pitch and volume when reading aloud, and is an element that reflects the emotions and intentions behind the words.

[0280] A "mispronunciation" refers to an error in which the words pronounced by the user during reading aloud do not match the presented task text.

[0281] "Reading speed" refers to the speed at which a user reads aloud, and is usually expressed as the number of characters or words divided by the time.

[0282] The "emotional state" indicates the emotion the user felt when reading aloud, and includes states such as joy, sadness, and anger.

[0283] "Feedback" refers to advice and evaluations provided to users based on the analysis results.

[0284] The "score" is a numerical value of the evaluation results of the reading aloud, and is provided as an overall evaluation.

[0285] "Battle mode" refers to a feature that allows multiple users to compete in reading aloud and compare the results.

[0286] "Analyzing independently" refers to analyzing each user's reading data individually without interfering with other data.

[0287] The "winner" refers to the user who is evaluated as having performed the best reading aloud in the battle mode.

[0288] "Long-term recording" refers to continuously saving the user's analysis results for a certain period of time and accumulating past data.

[0289] "Progress" refers to changes or improvements in a user's reading ability, including skill improvement over time.

[0290] MODE FOR CARRYING OUT THE INVENTION

[0291] The present invention is a reading aloud learning support system that combines an emotion engine with a system that provides feedback by recording and analyzing a user's reading aloud. This system includes a terminal, a server, and a dedicated application. Below, we will explain how this system is specifically implemented.

[0292] Hardware and Software Configuration

[0293] Device: A mobile information terminal used by the user, including a smartphone or tablet. A dedicated application is installed on the device, and it has the function of recording reading aloud using a built-in microphone.

[0294] Server: A computer system that receives voice data and performs transcription, speech analysis, feedback generation, emotion recognition, etc. It uses the following software and services:

[0295] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[0296] Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the user's emotional state.

[0297] Program processing

[0298] 1. Recording a reading: The user launches a dedicated application on their smartphone or tablet. The application displays the reading assignment text to the user, and the user reads and records the text aloud using the built-in microphone.

[0299] 2. Sending audio data: Once recording is complete, the device sends the recorded audio data to the server via the Internet.

[0300] 3. Analysis of voice data: The server temporarily stores the received voice data and analyzes it in the following steps.

[0301] Transcription: Transcribing audio data using the Google Cloud Speech-to-Text API.

[0302] Evaluating intonation: Extracting speech features from the transcribed text and evaluating intonation and intonation.

[0303] Misspelling detection: Compare the transcribed text with the assignment text to detect spelling errors and mispronunciations.

[0304] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0305] 4. Emotion Recognition: The server uses IBM Watson Tone Analyzer to recognize the user's emotional state from the voice data. Specifically, it analyzes the tone, speed, and intonation of the voice to detect the user's emotions (e.g., joy, sadness, anger, etc.).

[0306] 5. Feedback and Score Generation: The server generates specific feedback and an overall score based on the analysis results. For example, if the server detects a lack of intonation and tension, it will generate an overall score (e.g., 78 points) along with the feedback, "Your intonation is lacking. Try reading more relaxed."

[0307] 6. Providing feedback: The server sends the generated feedback and score to the device, which then displays the received feedback and score to the user, who can then improve their reading aloud.

[0308] Battle Mode

[0309] When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud and the audio data is sent to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and their feedback to the user, enhancing the competitive spirit and motivation to learn.

[0310] Long-term progress tracking and feedback

[0311] The server records the user's analysis results over a long period of time. This allows it to compare past data with current data and provide feedback to the user on areas for improvement and progress. For example, it could provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0312] Specific examples

[0313] 1. Example 1: Reading Aloud Assessment

[0314] Prompt text: The user reads the text "The weather is nice today."

[0315] Feedback: "Your intonation is appropriate. However, you read too quickly, so we recommend you read a little more slowly." and an overall score of 85 points are displayed.

[0316] 2. Example 2: Emotion Recognition

[0317] Prompt text: "The user seems emotionally exhausted."

[0318] Feedback: "You may be tired that day, so take a break and relax before trying again."

[0319] In this way, the present invention is implemented as a system that comprehensively analyzes a user's reading ability and provides appropriate feedback.

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] System program processing flow

[0322] Step 1: Record your reading

[0323] User: Launches the dedicated application on a smartphone or tablet.

[0324] Specific action: Tap the application icon.

[0325] Terminal: The application starts and displays the reading assignment text to the user.

[0326] Specific operation: Transition to the text display screen.

[0327] Input: Application launch signal.

[0328] Output: Display of the assignment text.

[0329] Step 2: Start recording

[0330] User: Read the presented text aloud.

[0331] Specific behavior: Reads text aloud into the built-in microphone.

[0332] Device: Use the built-in microphone to record your reading aloud.

[0333] Specific operation: Tap the record button to start recording.

[0334] Input: Record button tap signal.

[0335] Output: Start recording.

[0336] Step 3: Sending audio data

[0337] Device: Once recording is complete, the audio data is sent to a server via the Internet.

[0338] Specific operation: Tap the stop recording button and automatically send the audio data.

[0339] Input: Recording stop button tap signal, recording data.

[0340] Output: Sending audio data to the server.

[0341] Step 4: Receiving and storing audio data

[0342] Server: Temporarily stores received audio data.

[0343] Specific operation: Save the audio file to the database.

[0344] Input: Audio data from the device.

[0345] Output: Save audio data.

[0346] Step 5: Transcription

[0347] Server: Transcribing the audio data using a speech recognition engine.

[0348] Specific operation: Converts speech to text using the Google Cloud Speech-to-Text API.

[0349] Input: Stored audio data.

[0350] Output: The transcribed text.

[0351] Step 6: Evaluate intonation

[0352] Server: Evaluates the intonation and inflection of the transcribed text.

[0353] Specific operation: Extract audio features.

[0354] Input: The transcribed text.

[0355] Output: Intonation evaluation results.

[0356] Step 7: Detect slip-ups

[0357] Server: Compares the transcribed text with the assignment text to detect spelling errors and pronunciation mistakes.

[0358] What it does: Uses a text matching algorithm.

[0359] Input: Transcribed text, assignment text.

[0360] Output: Slip-of-the-speech detection results.

[0361] Step 8: Calculate your reading speed

[0362] Server: Calculates reading speed by comparing text length with playback time.

[0363] Specific operation: Calculate the reading speed (number of characters / second) from the playback time.

[0364] Input: Transcribed text, duration.

[0365] Output: Reading speed calculation results.

[0366] Step 9: Emotion Recognition

[0367] Server: Analyzes the user's emotional state from the voice data using an emotion engine.

[0368] Specific operation: Uses IBM Watson Tone Analyzer.

[0369] Input: Audio data.

[0370] Output: Emotion recognition result.

[0371] Step 10: Generate feedback and scores

[0372] Server: Generates specific feedback and an overall score based on the analysis results.

[0373] Specific actions: Evaluation is based on intonation, mistakes, reading speed, and emotional state.

[0374] Input: Intonation evaluation results, slip-up results, reading speed results, emotion recognition results.

[0375] Output: Feedback and overall score.

[0376] Step 11: Submit your feedback and score

[0377] Server: Sends generated feedback and scores to devices.

[0378] Specific operation: Send data in File or JSON format.

[0379] Input: Feedback and overall score.

[0380] Output: Data sent to the terminal.

[0381] Step 12: View your feedback and scores

[0382] Terminal: Displays received feedback and scores to the user.

[0383] Specific operation: Proceed to the feedback result screen.

[0384] Input: Send data.

[0385] Output: Display of feedback and score.

[0386] (Application example 2)

[0387] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] While conventional reading practice systems can provide technical evaluations of reading accuracy, intonation, speed, etc., it is difficult to understand the user's emotional state and provide appropriate feedback. As a result, users may continue practicing while feeling nervous or anxious, which hinders effective improvement of reading skills.

[0389] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0390] In this invention, the server includes means for recording the user's reading aloud on the terminal, means for transmitting the recorded voice data to the server, means for transcribing the received voice data, means for analyzing the transcribed text and evaluating intonation, detecting slip-ups, and calculating reading speed, means for generating feedback and a score based on the analysis results, means for displaying the feedback and score to the user on the terminal, means for recognizing the user's emotional state from the voice data, and means for generating feedback according to the emotion based on the emotion recognition results. This makes it possible to provide appropriate feedback that reflects the user's emotional state and promote improvement in reading aloud skills.

[0391] A "terminal" is an electronic device such as a smartphone, tablet, or personal computer used by a user who reads aloud.

[0392] The "means for recording" is a mechanism for recording the user's voice reading aloud as digital data using a microphone built into the terminal or an externally connected recording device.

[0393] "Audio data" is digital data representing recorded audio readings.

[0394] "Transmission means" refers to the technical mechanism by which the recorded audio data is sent to a server via the Internet or other communications network.

[0395] A "server" is a central processing system for receiving and analyzing audio data and generating feedback.

[0396] "Transcribing means" refers to the process of analyzing received audio data and converting it into corresponding text data.

[0397] The "analysis method" refers to a series of technical processes that use transcribed text data to evaluate intonation, detect slip-ups, calculate reading speed, and so on.

[0398] "Intonation evaluation" is the process of analyzing the intonation and vocal stress of a text read aloud and judging its quality.

[0399] "Slip-up detection" is the process of comparing the transcribed text with the exact text to identify typos and mispronunciations.

[0400] "Reading speed calculation" is the process of measuring reading speed using the total length of the text and the playback time of the audio data.

[0401] The "means for generating feedback and scores" is a mechanism that, based on the analysis results, creates a feedback message indicating an evaluation of the user's reading aloud and areas for improvement, as well as an overall evaluation score.

[0402] The "means for displaying" is a mechanism for displaying the generated feedback and score on the screen of the terminal in a form that can be visually recognized by the user.

[0403] "Emotional state" refers to the user's psychological and emotional state analyzed from voice data, including, for example, joy, sadness, tension, etc.

[0404] "Means for emotion recognition" refers to the technical process for identifying the user's emotional state from voice data.

[0405] "Feedback according to emotions" is a feedback message that provides appropriate advice or areas for improvement based on the results of emotion recognition of the user.

[0406] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. The specific configuration and method for realizing this system are described in detail below.

[0407] 1. Record and send your reading

[0408] The user launches a dedicated application using a device such as a smartphone or tablet. This application displays the assignment text for the user to read aloud. The user reads the assignment text aloud and records the audio using the device's built-in microphone. This recorded audio data is sent to a server via the Internet.

[0409] 2. Analysis of audio data and feedback generation

[0410] The server temporarily stores the received voice data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The transcribed data is analyzed as follows:

[0411] Intonation assessment: Analyze the transcribed text to assess intonation and intonation.

[0412] Misspelling detection: Compares the transcribed text with the correct text to detect typos and mispronunciations.

[0413] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0414] Furthermore, the server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to determine the user's emotional state (e.g., joy, sadness, nervousness, etc.). Based on these analysis results, the server generates specific feedback and a score. For example, if the user is nervous, the server provides feedback such as, "Your intonation is lacking. Try reading more relaxed."

[0415] 3. Providing and Viewing Feedback

[0416] The feedback and score generated by the server are sent to the terminal, which displays the received feedback and score to the user, allowing the user to improve their reading.

[0417] 4. Implementation of Battle Mode

[0418] The system includes a battle mode, which users can select to compete against other users (e.g., parents or friends) in reading aloud. Each user reads aloud individually and sends the audio data to the server. The server analyzes each audio data independently, generates feedback and scores for each, and determines the winner based on these scores.

[0419] 5. Long-term progress tracking and feedback

[0420] The server records the user's analysis results over a long period of time. This allows the server to evaluate the user's progress over time and provide feedback on improvements and progress over time. For example, the server might provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0421] Examples of concrete examples and prompts

[0422] Example: After a child completes one minute of reading aloud practice, the recorded speech is automatically analyzed and feedback such as "Your speed is 2.5 characters per second. You lack intonation. Relax and try again" is displayed on the head-mounted display.

[0423] Example prompts to input to a generative AI model:

[0424] "Use a sentiment analysis engine to recognize emotions as users read and generate feedback on intonation, speed, and spelling errors."

[0425] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0426] Step 1:

[0427] The user launches a dedicated application on their device. The application displays a reading assignment text to the user. The input is the assignment text selection provided by the user, and the output is the assignment text displayed on the application screen.

[0428] Step 2:

[0429] The user reads the assignment text aloud and records the audio using the device's built-in microphone. The input is the user's voice, and the output is audio data captured by the device's recording function. This recorded data is temporarily saved on the device.

[0430] Step 3:

[0431] The device sends the recorded voice data to the server via the Internet. The input is the voice data stored on the device, and the output is the voice data received by the server. The voice data is stored in the server's storage.

[0432] Step 4:

[0433] The server uses the received audio data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The input is the audio data, and the output is the transcribed text data. The server passes this text data to the next analysis step.

[0434] Step 5:

[0435] The server analyzes the transcribed text data. Specifically, it evaluates intonation, detects slip-ups, and calculates reading speed. The input is the transcribed text, and the output is the results of these evaluations.

[0436] Intonation assessment: Analyze the intonation patterns of the text and evaluate their quality.

[0437] Misspelling detection: Detects typos and mispronunciations by comparing with the correct text.

[0438] Reading speed calculation: Reading speed is calculated using the total length of the text and the playback time of the audio data.

[0439] Step 6:

[0440] The server recognizes the user's emotions from the voice data. Using an emotion recognition engine, it analyzes the tone, rate, and intonation of the voice to identify the emotional state (e.g., joy, sadness, tension, etc.). The input is the voice data, and the output is the identification of the user's emotional state.

[0441] Step 7:

[0442] The server generates feedback and a score based on the analysis and emotion recognition results. The inputs are the evaluation results of intonation, speed, typo detection, and emotional state, and the output is a specific feedback message and an overall score for the user.

[0443] Step 8:

[0444] The server sends feedback and scores to the terminal. The input is the generated feedback and scores, and the output is the transmission of data to the user terminal.

[0445] Step 9:

[0446] The terminal displays the received feedback and score to the user. The input is the feedback and score sent from the server, and the output is the feedback and score displayed on the terminal screen. The user can use this information to improve their reading aloud.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0448] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0450] [Second embodiment]

[0451] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0452] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0453] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0455] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0457] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0458] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0459] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0462] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0463] The present invention is a system in which a terminal records a user's reading aloud and transmits the recorded voice data to a server. How this system is actually implemented will be described below with specific examples.

[0464] Recording and sending readings

[0465] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with a specific text as a reading task. The user reads the presented text aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device sends the recorded voice data to a server via the Internet.

[0466] Analysis of audio data and feedback generation

[0467] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[0468] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0469] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0470] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0471] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[0472] Providing and Viewing Feedback

[0473] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[0474] Implementing Battle Mode

[0475] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[0476] In this way, the present invention is implemented as a system that effectively supports children's reading aloud learning while reducing the burden on parents in a form that is adapted to modern lifestyles.

[0477] The processing flow will be explained below.

[0478] Step 1:

[0479] The user launches a dedicated application on their device, which then presents the user with a reading assignment text.

[0480] Step 2:

[0481] The user reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device saves the recorded audio data.

[0482] Step 3:

[0483] The device transmits the stored audio data to a server over the Internet using a secure protocol (e.g., HTTPS).

[0484] Step 4:

[0485] The server receives the voice data sent from the device and temporarily stores it for subsequent processing.

[0486] Step 5:

[0487] The server extracts the stored voice data and transcribes it using a speech recognition engine, which then analyzes the voice data and generates text data.

[0488] Step 6:

[0489] The server analyzes the transcribed text. The first step in the analysis is to evaluate intonation, which evaluates the frequency of intonation and specific phrases in the text.

[0490] Step 7:

[0491] The server then compares the transcribed text with pre-trained correct text, detecting spelling errors, mispronunciations, and other mistakes.

[0492] Step 8:

[0493] The server calculates the reading speed based on the total length of the text and the playback time of the audio data, and evaluates whether the reading speed is appropriate.

[0494] Step 9:

[0495] The server generates a comprehensive feedback and score based on the results of intonation evaluation, slip-up detection, and reading speed calculations. An example of specific feedback might be, "Your intonation is lacking. Try to read with more emotion."

[0496] Step 10:

[0497] The server sends the generated feedback and score to the terminal using unique identification information for each user.

[0498] Step 11:

[0499] The device receives feedback and displays the score from the server to the user, allowing the user to identify areas for improvement in their reading aloud and improve their learning.

[0500] Step 12:

[0501] If a user selects Battle Mode, a process of reading aloud competition with parents or friends will begin. Reading aloud data is sent from each device to the server and analyzed individually.

[0502] Step 13:

[0503] In battle mode, the server analyzes multiple reading data independently, generates feedback and scores for each, and compares the scores to determine the winner.

[0504] Step 14:

[0505] The server sends the battle results and feedback to the terminal, which displays the results to the user and promotes communication between users.

[0506] Example 1

[0507] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0508] Traditional reading aloud education requires parents or teachers to directly supervise and provide feedback to children, which requires a lot of time and effort. Furthermore, evaluation of reading aloud intonation, speech mistakes, and reading speed is subjective, making objective assessment difficult. Furthermore, there is a lack of methods to gamify reading aloud practice and increase motivation by competing with other users.

[0509] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0510] In this invention, the server includes means for the user to start reading aloud, means for the terminal to record the user's reading aloud, means for the terminal to transmit the recorded voice data to the server, means for the server to transcribe the received voice data, means for the server to analyze the transcribed text and evaluate intonation, detect errors, and calculate reading speed, means for the server to generate feedback and a score based on the analysis results, means for the terminal to display the feedback and score to the user, and means for selecting a battle mode.This allows the user to effectively practice reading aloud, understand areas for improvement based on feedback, and increase motivation to learn by competing with other users.

[0511] "Reading aloud" is the act of a user reading aloud the presented text.

[0512] "Terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0513] "Voice data" refers to digital data containing the user's speech recorded by the device's built-in microphone or an external microphone.

[0514] "Server" refers to the central processing unit that receives, analyzes, transcribes, and generates feedback from audio data.

[0515] "Transcription" is the process of analyzing audio data and converting what is spoken into text data.

[0516] "Intonation" is an element that evaluates the changes in pitch and strength in the user's vocalization.

[0517] "Errors" refer to mispronunciations or slip-ups that the user makes while reading aloud.

[0518] "Reading speed" is a factor that measures how quickly a user reads a given text aloud.

[0519] "Feedback" refers to comments on improvements and evaluations that the server provides to the user based on the analysis results.

[0520] "Score" refers to a numerical result that comprehensively evaluates the user's reading aloud.

[0521] "Battle mode" refers to a feature in which multiple users compete against each other for the highest reading score.

[0522] This invention is a system in which a terminal records a user's reading aloud and transmits the audio data to a server. This system applies speech recognition technology and analyzes the recorded data to provide accurate feedback to the user. The specific processing procedures and implementation outline of the system are described below.

[0523] Hardware and Software Used

[0524] 1. Hardware:

[0525] Device: A mobile device such as a smartphone or tablet.

[0526] Microphone: The microphone built into the device.

[0527] 2. Software:

[0528] Dedicated application: An application that displays reading assignments and records audio.

[0529] Server-side analysis program: A program that performs audio analysis, transcription, and feedback generation.

[0530] Speech Recognition Engine: Speech recognition software such as Google Cloud Speech-to-Text.

[0531] Recording and sending readings

[0532] The user launches a dedicated application on the device and reads the displayed text aloud. The device uses a built-in microphone to record the user's voice and transmits the recording to a server via the Internet. For example, the following prompt sentence is used:

[0533] "Please record today's reading assignment and send it to the server."

[0534] Analysis of audio data and feedback generation

[0535] The server temporarily stores the received voice data and transcribes it using a speech recognition engine such as Google Cloud Speech-to-Text.The server then analyzes the transcribed data and performs the following evaluations:

[0536] Intonation assessment: Analyzing the intonation and stress of speech.

[0537] Error detection: Identifying errors by comparing the transcribed text with the exact text.

[0538] Reading speed calculation: Calculates reading speed by comparing the length of the text with the recording time.

[0539] The server generates feedback and a score based on the analysis results. For example, if the intonation is insufficient, the server will provide feedback such as "Your intonation is insufficient. Try reading with more emotion." An example of a prompt is as follows:

[0540] "Analyze recorded speech and generate ratings for intonation, errors, and reading speed."

[0541] Providing and Viewing Feedback

[0542] The feedback and score generated by the server are sent to the device, which displays it to the user, allowing the user to understand where they need to improve their reading. For example, the following message is displayed on the device:

[0543] "Your intonation is lacking, please try again."

[0544] Implementing Battle Mode

[0545] The invention also includes a battle mode in which multiple users compete in reading aloud. When a user selects the battle mode, each reading aloud is sent to the server and analyzed independently. The analysis results are then used to generate feedback and a score for each user, and a final winner is determined. The prompt is as follows:

[0546] "Run Battle Mode and compare each user's score to determine the winner."

[0547] As a result, the present invention is implemented as a system that effectively supports the user's reading aloud learning, improves the reading aloud technique based on feedback, and strengthens motivation to learn.

[0548] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0549] Step 1:

[0550] The device launches a dedicated application.

[0551] Input: User action.

[0552] Output: The home screen of the application.

[0553] Specific operation: The user taps the application icon on the device to launch the application. The app displays the home screen, which displays the reading assignment and settings menu.

[0554] Step 2:

[0555] The terminal presents the text to the user for reading aloud.

[0556] Enter: the application home screen.

[0557] Output: The text that will be displayed as today's reading assignment.

[0558] Specific operation: The device displays a specific text on the screen as "Today's assignment," and the user prepares to read the text.

[0559] Step 3:

[0560] The user reads the presented text aloud.

[0561] Input: The presented text.

[0562] Output: Audio to be read aloud.

[0563] Specific action: The user reads aloud the presented text (e.g., "red shoes"), paying attention to mispronunciations and the volume of the voice during this stage.

[0564] Step 4:

[0565] The device records the user's voice.

[0566] Input: User's speech reading.

[0567] Output: Recorded audio data.

[0568] Specific operation: The device activates the built-in microphone and records the user's reading. When the recording is complete, the device generates an audio file.

[0569] Step 5:

[0570] The device sends the recorded audio data to the server.

[0571] Input: Recorded audio data.

[0572] Output: The audio data sent to the server.

[0573] Specific operation: The device compresses the recorded audio data and sends it to the server via the Internet. The device then verifies that the data was sent successfully.

[0574] Step 6:

[0575] The server temporarily stores the voice data and transcribes it using a voice recognition engine.

[0576] Input: Audio data received from the device.

[0577] Output: Transcribed text data.

[0578] Specific operation: The server temporarily saves the voice data in a directory where it stores it, then calls a voice recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[0579] Step 7:

[0580] The server analyzes the transcription data.

[0581] Input: Transcribed text data.

[0582] Output: Intonation assessment, error detection, and reading speed calculation.

[0583] Specific operation: The server runs an analysis program to evaluate intonation, detect errors, and calculate reading speed. Specifically, it compares the transcription data with the original text and evaluates pronunciation intonation, error locations, and reading speed.

[0584] Step 8:

[0585] The server generates feedback and a score.

[0586] Input: Intonation assessment, error detection, and reading speed results.

[0587] Output: Feedback and score.

[0588] Specific operation: The server generates a feedback message and an overall score based on the analysis results. For example, it may provide feedback such as, "Your intonation is lacking. Try to read with more emotion."

[0589] Step 9:

[0590] The server sends feedback and scores to the device.

[0591] Input: Generated feedback and score.

[0592] Output: Data sent to the terminal.

[0593] Specific operation: The server assembles the feedback message and score into a data packet and sends it to the device via the Internet.

[0594] Step 10:

[0595] The device displays feedback and scores to the user.

[0596] Input: Feedback and score received from the server.

[0597] Output: Feedback and score displayed to the user.

[0598] Specific operation: The device displays the received feedback message and score on the screen. The user can check the feedback and reflect it in the next reading.

[0599] (Application example 1)

[0600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0601] While existing reading assessment systems aim to improve users' reading skills, they lack practical application in specific situations and contexts, especially when it comes to assessing product explanations and comprehension. Furthermore, they lack the ability to motivate users by allowing them to compete with other users. This leaves no effective tools for improving product comprehension and explanation skills in virtual stores.

[0602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0603] In this invention, the server includes means for recording audio data of product information read aloud by a user on a terminal and transmitting the recorded data to the server, means for the server to transcribe and analyze the audio data, means for generating feedback and a score based on the analysis results and providing them to the user, and means for displaying the results of competitions with other users for the read aloud product information. This allows users to read aloud product descriptions in a virtual store, be evaluated on their level of understanding and explanation ability, and compete with other users, thereby improving their product understanding and explanation ability.

[0604] The "means for the terminal to record the user's reading aloud" is a combination of hardware and software for recording the user's voice reading aloud.

[0605] The "means for transmitting audio data recorded by the terminal to a server" is a function for transmitting recorded audio data to a remote server via a network.

[0606] The "means for transcribing voice data received by the server" is a function including a voice recognition engine that converts voice data received by the server into text format.

[0607] "Means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed" refers to a function that runs algorithms based on the transcribed text data to analyze a speaker's intonation, pronunciation errors, and reading speed.

[0608] "Means for the server to generate feedback and a score based on the analysis results" refers to a function that generates points for improvement and an overall evaluation score for the user based on the analysis results.

[0609] The "means for the terminal to display the feedback and score to the user" is a function for displaying the generated feedback and score on the screen of the user's device.

[0610] "A means of having specific product information read aloud in a virtual store, analyzing the level of understanding and explanation ability, and providing feedback" is a function that allows a user to read aloud product information in a virtual space, evaluate the level of understanding of the information and explanation skills, and provide feedback.

[0611] The "means for displaying the results of a competition with other users for the product information read aloud" is a function that allows a plurality of users to compete for scores based on the product information read aloud, and displays the results.

[0612] This invention is a system for evaluating a user's reading skills and product comprehension. Below, we will explain in natural language how this system is actually implemented by explaining the program processing procedures.

[0613] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with specific product information as a reading task. The user reads the presented product information aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device transmits the recorded voice data to a server via the Internet.

[0614] The server temporarily stores the voice data received from the device and transcribes it using a speech recognition engine (for example, Google Cloud Speech-to-Text API). Once transcription is complete, the server analyzes the voice data using the following procedure.

[0615] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0616] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0617] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0618] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[0619] The system also includes a battle mode function. When a user selects battle mode, they can compete in reading aloud with other users. Each user reads aloud and sends each audio data to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and each user's feedback to the user, stimulating a competitive spirit and encouraging improvement of reading aloud skills.

[0620] Examples of concrete examples and prompts

[0621] The user browses product descriptions in a virtual store and reads them aloud. The app records the audio and sends it to the server. The server analyzes the recording and provides feedback to improve the user's reading skills and product comprehension. Specific examples of prompts include:

[0622] Please read aloud the description of the product "Smart Watch A." ​​We will provide feedback on what you read aloud.

[0623] In this way, the present invention provides a system that allows users to gain a deeper understanding of product information in a virtual store and improve their reading skills and product comprehension by competing with other users.

[0624] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0625] Step 1:

[0626] The user launches a dedicated application on the device and checks the specific product information presented as a reading task. The input is the product text information, and the output is a state ready for reading aloud. In this step, the device displays the product text to the user and provides an interface for preparing for reading aloud.

[0627] Step 2:

[0628] The user reads aloud the presented product information. The input is the user's voice, and the output is the recorded voice data. The device records the reading using the built-in microphone. In this step, the device receives the user's voice and saves it as an audio file (e.g., WAV format).

[0629] Step 3:

[0630] After the recording is complete, the device sends the recorded audio data to the server via the Internet. The input is the recorded audio data, and the output is the completion status of the audio data transfer to the server. The device uploads the audio file to the server using an HTTP request.

[0631] Step 4:

[0632] The server temporarily stores the received voice data and transcribes it using a speech recognition engine. The input is voice data, and the output is transcribed text data. The server converts the voice data into text using speech recognition technology such as the Google Cloud Speech-to-Text API.

[0633] Step 5:

[0634] The server analyzes the transcribed text data, evaluating intonation patterns, pronunciation errors, and reading speed. The input is the transcription data, and the output is the analysis results. The server uses algorithms to evaluate intonation, detect spelling errors, and calculate the reading speed.

[0635] Step 6:

[0636] The server generates specific feedback and an overall score based on the analysis results. The input is the analysis result data, and the output is feedback and a score. The server generates advice and evaluation points for the user based on the analysis data.

[0637] Step 7:

[0638] The server sends the generated feedback and score to the terminal, which displays it to the user. The input is the feedback and score, and the output is the information displayed to the user. The terminal displays the received data on the user interface.

[0639] Step 8:

[0640] When Battle Mode is selected, each user's reading data is analyzed independently to generate a score so that they can compete with other users. The input is the reading data of multiple users, and the output is feedback to each user, a score, and the winner's decision. The server uses a competitive algorithm to determine the winner and sends the result to the device.

[0641] By following these steps, the system can analyze and evaluate the user's reading skills, helping them improve their understanding of product information and their ability to explain it. Furthermore, the battle mode can stimulate competitive spirit and further motivate users.

[0642] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0643] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. Below, we will explain how to actually implement this system, using concrete examples.

[0644] Recording and sending readings

[0645] First, the user launches a dedicated application on their device (smartphone or tablet). The application then presents the user with a reading assignment text. The user then reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device sends the recorded audio data to a server via the Internet.

[0646] Analysis of audio data and feedback generation

[0647] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[0648] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0649] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0650] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0651] Furthermore, the server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[0652] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected and the user is recognized as nervous, the server will generate an overall score (e.g., 78 points) along with feedback such as, "Your intonation is lacking. Try reading more relaxed."

[0653] Providing and Viewing Feedback

[0654] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[0655] Implementing Battle Mode

[0656] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[0657] Long-term progress tracking and feedback

[0658] In addition, the server records the user's analysis results over a long period of time. This allows the server to understand the user's progress and trends, and provide feedback on improvements and progress over time. For example, by comparing past data with current data, the server can provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0659] In this way, the present invention is implemented as a system that is adapted to modern lifestyles, effectively supports children's reading aloud learning while reducing the burden on parents, and by combining it with an emotion engine, can provide more accurate feedback.

[0660] The processing flow will be explained below.

[0661] Step 1:

[0662] The user starts a dedicated application on their device. The application presents the user with a reading assignment text. The user reads the presented text aloud and prepares to begin reading aloud.

[0663] Step 2:

[0664] The device uses the built-in microphone to record the user's reading in real time, and once the recording is complete, the device temporarily stores the recorded audio data in local storage.

[0665] Step 3:

[0666] The device sends the recorded audio data to the server using a secure protocol (e.g., HTTPS). At the same time, the status of the transmission is displayed to the user.

[0667] Step 4:

[0668] The server receives the voice data sent from the device and stores it in the server, where it is prepared for analysis.

[0669] Step 5:

[0670] The server uses a speech recognition engine to transcribe the voice data. The speech recognition engine analyzes the voice data and generates text data. Once transcription is complete, the text data is stored on the server.

[0671] Step 6:

[0672] The server analyzes the transcribed text. The analysis is performed as follows:

[0673] Intonation assessment: The server detects intonation and the frequency of certain phrases in the text and assesses intonation.

[0674] Misspelling detection: The server compares the transcribed text with pre-trained correct text to detect spelling errors and mispronunciations.

[0675] Reading speed calculation: The server calculates the reading speed based on the total length of the text and the playback time of the audio data.

[0676] Step 7:

[0677] The server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[0678] Step 8:

[0679] The server evaluates intonation, detects slip-ups, calculates reading speed, and recognizes emotions, and generates specific feedback and an overall score. For example, it might generate feedback like, "Your intonation is lacking. Try reading more relaxedly," and calculate an overall score (e.g., 78 points).

[0680] Step 9:

[0681] The server sends the generated feedback and score to the device, using each user's identification information to ensure data integrity.

[0682] Step 10:

[0683] The device receives feedback and displays the score from the server to the user, who can then use the feedback to identify areas for improvement and practice for the next time.

[0684] Step 11:

[0685] When a user selects the battle mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and the audio data is sent from their device to the server.

[0686] Step 12:

[0687] In battle mode, the server analyzes each user's voice data independently and generates feedback and scores for each user. Based on the generated scores, the server determines the winner.

[0688] Step 13:

[0689] The server sends the results of the battle mode and feedback to the terminal, which displays the winner and feedback to the user, promoting communication between users.

[0690] Step 14:

[0691] The server records the user's progress and trends over time, allowing it to provide feedback on improvements and progress over time. It compares past data with current data and provides feedback such as, "Your reading speed is improving compared to before. Keep up the good work."

[0692] Example 2

[0693] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] Currently, there are systems that evaluate users' reading ability, but they lack detailed analysis of factors such as emotional state, intonation, and reading speed, and provide insufficient comprehensive feedback. Furthermore, they lack a mechanism for accurate and fair analysis and evaluation in battle modes where multiple users compete in reading aloud. Furthermore, a system that records users' long-term progress and provides appropriate feedback is also needed.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0696] In this invention, the server includes a means for transcribing voice data, a means for analyzing the transcribed text to evaluate intonation, detect slip-ups, and calculate reading speed, and a means for providing feedback and a score based on the analysis results, as well as a means for analyzing the user's emotional state from the voice data. This enables comprehensive and precise analysis and feedback of the user's reading aloud. Furthermore, when multiple users compete in battle mode, each reading aloud data can be analyzed independently, enabling fair evaluation and winner determination. Furthermore, the server can support the user's continuous growth by recording the user's analysis results over a long period of time and providing progress and areas for improvement over time.

[0697] A "terminal" is an electronic device used by a user, and includes mobile information terminals such as smartphones and tablets.

[0698] "Reading aloud" refers to the act of a user reading a text aloud.

[0699] "Recording" refers to the act of recording audio as digital data using a microphone attached to a terminal.

[0700] "Audio data" refers to data in which recorded audio is saved in digital format.

[0701] "Server" refers to a computer system that receives, stores, and analyzes audio data.

[0702] "Transcription" is the process of converting audio data into text data using speech recognition technology.

[0703] "Intonation" refers to changes in pitch and volume when reading aloud, and is an element that reflects the emotions and intentions behind the words.

[0704] A "mispronunciation" refers to an error in which the words pronounced by the user during reading aloud do not match the presented task text.

[0705] "Reading speed" refers to the speed at which a user reads aloud, and is usually expressed as the number of characters or words divided by the time.

[0706] The "emotional state" indicates the emotion the user felt when reading aloud, and includes states such as joy, sadness, and anger.

[0707] "Feedback" refers to advice and evaluations provided to users based on the analysis results.

[0708] The "score" is a numerical value of the evaluation results of the reading aloud, and is provided as an overall evaluation.

[0709] "Battle mode" refers to a feature that allows multiple users to compete in reading aloud and compare the results.

[0710] "Analyzing independently" refers to analyzing each user's reading data individually without interfering with other data.

[0711] The "winner" refers to the user who is evaluated as having performed the best reading aloud in the battle mode.

[0712] "Long-term recording" refers to continuously saving the user's analysis results for a certain period of time and accumulating past data.

[0713] "Progress" refers to changes or improvements in a user's reading ability, including skill improvement over time.

[0714] MODE FOR CARRYING OUT THE INVENTION

[0715] The present invention is a reading aloud learning support system that combines an emotion engine with a system that provides feedback by recording and analyzing a user's reading aloud. This system includes a terminal, a server, and a dedicated application. Below, we will explain how this system is specifically implemented.

[0716] Hardware and Software Configuration

[0717] Device: A mobile information terminal used by the user, including a smartphone or tablet. A dedicated application is installed on the device, and it has the function of recording reading aloud using a built-in microphone.

[0718] Server: A computer system that receives voice data and performs transcription, speech analysis, feedback generation, emotion recognition, etc. It uses the following software and services:

[0719] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[0720] Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the user's emotional state.

[0721] Program processing

[0722] 1. Recording a reading: The user launches a dedicated application on their smartphone or tablet. The application displays the reading assignment text to the user, and the user reads and records the text aloud using the built-in microphone.

[0723] 2. Sending audio data: Once recording is complete, the device sends the recorded audio data to the server via the Internet.

[0724] 3. Analysis of voice data: The server temporarily stores the received voice data and analyzes it in the following steps.

[0725] Transcription: Transcribing audio data using the Google Cloud Speech-to-Text API.

[0726] Evaluating intonation: Extracting speech features from the transcribed text and evaluating intonation and intonation.

[0727] Misspelling detection: Compare the transcribed text with the assignment text to detect spelling errors and mispronunciations.

[0728] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0729] 4. Emotion Recognition: The server uses IBM Watson Tone Analyzer to recognize the user's emotional state from the voice data. Specifically, it analyzes the tone, speed, and intonation of the voice to detect the user's emotions (e.g., joy, sadness, anger, etc.).

[0730] 5. Feedback and Score Generation: The server generates specific feedback and an overall score based on the analysis results. For example, if the server detects a lack of intonation and tension, it will generate an overall score (e.g., 78 points) along with the feedback, "Your intonation is lacking. Try reading more relaxed."

[0731] 6. Providing feedback: The server sends the generated feedback and score to the device, which then displays the received feedback and score to the user, who can then improve their reading aloud.

[0732] Battle Mode

[0733] When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud and the audio data is sent to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and their feedback to the user, enhancing the competitive spirit and motivation to learn.

[0734] Long-term progress tracking and feedback

[0735] The server records the user's analysis results over a long period of time. This allows it to compare past data with current data and provide feedback to the user on areas for improvement and progress. For example, it could provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0736] Specific examples

[0737] 1. Example 1: Reading Aloud Assessment

[0738] Prompt text: The user reads the text "The weather is nice today."

[0739] Feedback: "Your intonation is appropriate. However, you read too quickly, so we recommend you read a little more slowly." and an overall score of 85 points are displayed.

[0740] 2. Example 2: Emotion Recognition

[0741] Prompt text: "The user seems emotionally exhausted."

[0742] Feedback: "You may be tired that day, so take a break and relax before trying again."

[0743] In this way, the present invention is implemented as a system that comprehensively analyzes a user's reading ability and provides appropriate feedback.

[0744] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0745] System program processing flow

[0746] Step 1: Record your reading

[0747] User: Launches the dedicated application on a smartphone or tablet.

[0748] Specific action: Tap the application icon.

[0749] Terminal: The application starts and displays the reading assignment text to the user.

[0750] Specific operation: Transition to the text display screen.

[0751] Input: Application launch signal.

[0752] Output: Display of the assignment text.

[0753] Step 2: Start recording

[0754] User: Read the presented text aloud.

[0755] Specific behavior: Reads text aloud into the built-in microphone.

[0756] Device: Use the built-in microphone to record your reading aloud.

[0757] Specific operation: Tap the record button to start recording.

[0758] Input: Record button tap signal.

[0759] Output: Start recording.

[0760] Step 3: Sending audio data

[0761] Device: Once recording is complete, the audio data is sent to a server via the Internet.

[0762] Specific operation: Tap the stop recording button and automatically send the audio data.

[0763] Input: Recording stop button tap signal, recording data.

[0764] Output: Sending audio data to the server.

[0765] Step 4: Receiving and storing audio data

[0766] Server: Temporarily stores received audio data.

[0767] Specific operation: Save the audio file to the database.

[0768] Input: Audio data from the device.

[0769] Output: Save audio data.

[0770] Step 5: Transcription

[0771] Server: Transcribing the audio data using a speech recognition engine.

[0772] Specific operation: Converts speech to text using the Google Cloud Speech-to-Text API.

[0773] Input: Stored audio data.

[0774] Output: The transcribed text.

[0775] Step 6: Evaluate intonation

[0776] Server: Evaluates the intonation and inflection of the transcribed text.

[0777] Specific operation: Extract audio features.

[0778] Input: The transcribed text.

[0779] Output: Intonation evaluation results.

[0780] Step 7: Detect slip-ups

[0781] Server: Compares the transcribed text with the assignment text to detect spelling errors and pronunciation mistakes.

[0782] What it does: Uses a text matching algorithm.

[0783] Input: Transcribed text, assignment text.

[0784] Output: Slip-of-the-speech detection results.

[0785] Step 8: Calculate your reading speed

[0786] Server: Calculates reading speed by comparing text length with playback time.

[0787] Specific operation: Calculate the reading speed (number of characters / second) from the playback time.

[0788] Input: Transcribed text, duration.

[0789] Output: Reading speed calculation results.

[0790] Step 9: Emotion Recognition

[0791] Server: Analyzes the user's emotional state from the voice data using an emotion engine.

[0792] Specific operation: Uses IBM Watson Tone Analyzer.

[0793] Input: Audio data.

[0794] Output: Emotion recognition result.

[0795] Step 10: Generate feedback and scores

[0796] Server: Generates specific feedback and an overall score based on the analysis results.

[0797] Specific actions: Evaluation is based on intonation, mistakes, reading speed, and emotional state.

[0798] Input: Intonation evaluation results, slip-up results, reading speed results, emotion recognition results.

[0799] Output: Feedback and overall score.

[0800] Step 11: Submit your feedback and score

[0801] Server: Sends generated feedback and scores to devices.

[0802] Specific operation: Send data in File or JSON format.

[0803] Input: Feedback and overall score.

[0804] Output: Data sent to the terminal.

[0805] Step 12: View your feedback and scores

[0806] Terminal: Displays received feedback and scores to the user.

[0807] Specific operation: Proceed to the feedback result screen.

[0808] Input: Send data.

[0809] Output: Display of feedback and score.

[0810] (Application example 2)

[0811] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0812] While conventional reading practice systems can provide technical evaluations of reading accuracy, intonation, speed, etc., it is difficult to understand the user's emotional state and provide appropriate feedback. As a result, users may continue practicing while feeling nervous or anxious, which hinders effective improvement of reading skills.

[0813] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0814] In this invention, the server includes means for recording the user's reading aloud on the terminal, means for transmitting the recorded voice data to the server, means for transcribing the received voice data, means for analyzing the transcribed text and evaluating intonation, detecting slip-ups, and calculating reading speed, means for generating feedback and a score based on the analysis results, means for displaying the feedback and score to the user on the terminal, means for recognizing the user's emotional state from the voice data, and means for generating feedback according to the emotion based on the emotion recognition results. This makes it possible to provide appropriate feedback that reflects the user's emotional state and promote improvement in reading aloud skills.

[0815] A "terminal" is an electronic device such as a smartphone, tablet, or personal computer used by a user who reads aloud.

[0816] The "means for recording" is a mechanism for recording the user's voice reading aloud as digital data using a microphone built into the terminal or an externally connected recording device.

[0817] "Audio data" is digital data representing recorded audio readings.

[0818] "Transmission means" refers to the technical mechanism by which the recorded audio data is sent to a server via the Internet or other communications network.

[0819] A "server" is a central processing system for receiving and analyzing audio data and generating feedback.

[0820] "Transcribing means" refers to the process of analyzing received audio data and converting it into corresponding text data.

[0821] The "analysis method" refers to a series of technical processes that use transcribed text data to evaluate intonation, detect slip-ups, calculate reading speed, and so on.

[0822] "Intonation evaluation" is the process of analyzing the intonation and vocal stress of a text read aloud and judging its quality.

[0823] "Slip-up detection" is the process of comparing the transcribed text with the exact text to identify typos and mispronunciations.

[0824] "Reading speed calculation" is the process of measuring reading speed using the total length of the text and the playback time of the audio data.

[0825] The "means for generating feedback and scores" is a mechanism that, based on the analysis results, creates a feedback message indicating an evaluation of the user's reading aloud and areas for improvement, as well as an overall evaluation score.

[0826] The "means for displaying" is a mechanism for displaying the generated feedback and score on the screen of the terminal in a form that can be visually recognized by the user.

[0827] "Emotional state" refers to the user's psychological and emotional state analyzed from voice data, including, for example, joy, sadness, tension, etc.

[0828] "Means for emotion recognition" refers to the technical process for identifying the user's emotional state from voice data.

[0829] "Feedback according to emotions" is a feedback message that provides appropriate advice or areas for improvement based on the results of emotion recognition of the user.

[0830] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. The specific configuration and method for realizing this system are described in detail below.

[0831] 1. Record and send your reading

[0832] The user launches a dedicated application using a device such as a smartphone or tablet. This application displays the assignment text for the user to read aloud. The user reads the assignment text aloud and records the audio using the device's built-in microphone. This recorded audio data is sent to a server via the Internet.

[0833] 2. Analysis of audio data and feedback generation

[0834] The server temporarily stores the received voice data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The transcribed data is analyzed as follows:

[0835] Intonation assessment: Analyze the transcribed text to assess intonation and intonation.

[0836] Misspelling detection: Compares the transcribed text with the correct text to detect typos and mispronunciations.

[0837] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0838] Furthermore, the server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to determine the user's emotional state (e.g., joy, sadness, nervousness, etc.). Based on these analysis results, the server generates specific feedback and a score. For example, if the user is nervous, the server provides feedback such as, "Your intonation is lacking. Try reading more relaxed."

[0839] 3. Providing and Viewing Feedback

[0840] The feedback and score generated by the server are sent to the terminal, which displays the received feedback and score to the user, allowing the user to improve their reading.

[0841] 4. Implementation of Battle Mode

[0842] The system includes a battle mode, which users can select to compete against other users (e.g., parents or friends) in reading aloud. Each user reads aloud individually and sends the audio data to the server. The server analyzes each audio data independently, generates feedback and scores for each, and determines the winner based on these scores.

[0843] 5. Long-term progress tracking and feedback

[0844] The server records the user's analysis results over a long period of time. This allows the server to evaluate the user's progress over time and provide feedback on improvements and progress over time. For example, the server might provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[0845] Examples of concrete examples and prompts

[0846] Example: After a child completes one minute of reading aloud practice, the recorded speech is automatically analyzed and feedback such as "Your speed is 2.5 characters per second. You lack intonation. Relax and try again" is displayed on the head-mounted display.

[0847] Example prompts to input to a generative AI model:

[0848] "Use a sentiment analysis engine to recognize emotions as users read and generate feedback on intonation, speed, and spelling errors."

[0849] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0850] Step 1:

[0851] The user launches a dedicated application on their device. The application displays a reading assignment text to the user. The input is the assignment text selection provided by the user, and the output is the assignment text displayed on the application screen.

[0852] Step 2:

[0853] The user reads the assignment text aloud and records the audio using the device's built-in microphone. The input is the user's voice, and the output is audio data captured by the device's recording function. This recorded data is temporarily saved on the device.

[0854] Step 3:

[0855] The device sends the recorded voice data to the server via the Internet. The input is the voice data stored on the device, and the output is the voice data received by the server. The voice data is stored in the server's storage.

[0856] Step 4:

[0857] The server uses the received audio data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The input is the audio data, and the output is the transcribed text data. The server passes this text data to the next analysis step.

[0858] Step 5:

[0859] The server analyzes the transcribed text data. Specifically, it evaluates intonation, detects slip-ups, and calculates reading speed. The input is the transcribed text, and the output is the results of these evaluations.

[0860] Intonation assessment: Analyze the intonation patterns of the text and evaluate their quality.

[0861] Misspelling detection: Detects typos and mispronunciations by comparing with the correct text.

[0862] Reading speed calculation: Reading speed is calculated using the total length of the text and the playback time of the audio data.

[0863] Step 6:

[0864] The server recognizes the user's emotions from the voice data. Using an emotion recognition engine, it analyzes the tone, rate, and intonation of the voice to identify the emotional state (e.g., joy, sadness, tension, etc.). The input is the voice data, and the output is the identification of the user's emotional state.

[0865] Step 7:

[0866] The server generates feedback and a score based on the analysis and emotion recognition results. The inputs are the evaluation results of intonation, speed, typo detection, and emotional state, and the output is a specific feedback message and an overall score for the user.

[0867] Step 8:

[0868] The server sends feedback and scores to the terminal. The input is the generated feedback and scores, and the output is the transmission of data to the user terminal.

[0869] Step 9:

[0870] The terminal displays the received feedback and score to the user. The input is the feedback and score sent from the server, and the output is the feedback and score displayed on the terminal screen. The user can use this information to improve their reading aloud.

[0871] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0872] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0873] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0874] [Third embodiment]

[0875] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0876] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0877] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0878] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0879] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0880] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0881] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0882] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0883] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0884] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0885] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0886] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0887] The present invention is a system in which a terminal records a user's reading aloud and transmits the recorded voice data to a server. How this system is actually implemented will be described below with specific examples.

[0888] Recording and sending readings

[0889] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with a specific text as a reading task. The user reads the presented text aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device sends the recorded voice data to a server via the Internet.

[0890] Analysis of audio data and feedback generation

[0891] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[0892] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[0893] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[0894] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[0895] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[0896] Providing and Viewing Feedback

[0897] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[0898] Implementing Battle Mode

[0899] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[0900] In this way, the present invention is implemented as a system that effectively supports children's reading aloud learning while reducing the burden on parents in a form that is adapted to modern lifestyles.

[0901] The processing flow will be explained below.

[0902] Step 1:

[0903] The user launches a dedicated application on their device, which then presents the user with a reading assignment text.

[0904] Step 2:

[0905] The user reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device saves the recorded audio data.

[0906] Step 3:

[0907] The device transmits the stored audio data to a server over the Internet using a secure protocol (e.g., HTTPS).

[0908] Step 4:

[0909] The server receives the voice data sent from the device and temporarily stores it for subsequent processing.

[0910] Step 5:

[0911] The server extracts the stored voice data and transcribes it using a speech recognition engine, which then analyzes the voice data and generates text data.

[0912] Step 6:

[0913] The server analyzes the transcribed text. The first step in the analysis is to evaluate intonation, which evaluates the frequency of intonation and specific phrases in the text.

[0914] Step 7:

[0915] The server then compares the transcribed text with pre-trained correct text, detecting spelling errors, mispronunciations, and other mistakes.

[0916] Step 8:

[0917] The server calculates the reading speed based on the total length of the text and the playback time of the audio data, and evaluates whether the reading speed is appropriate.

[0918] Step 9:

[0919] The server generates a comprehensive feedback and score based on the results of intonation evaluation, slip-up detection, and reading speed calculations. An example of specific feedback might be, "Your intonation is lacking. Try to read with more emotion."

[0920] Step 10:

[0921] The server sends the generated feedback and score to the terminal using unique identification information for each user.

[0922] Step 11:

[0923] The device receives feedback and displays the score from the server to the user, allowing the user to identify areas for improvement in their reading aloud and improve their learning.

[0924] Step 12:

[0925] If a user selects Battle Mode, a process of reading aloud competition with parents or friends will begin. Reading aloud data is sent from each device to the server and analyzed individually.

[0926] Step 13:

[0927] In battle mode, the server analyzes multiple reading data independently, generates feedback and scores for each, and compares the scores to determine the winner.

[0928] Step 14:

[0929] The server sends the battle results and feedback to the terminal, which displays the results to the user and promotes communication between users.

[0930] Example 1

[0931] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] Traditional reading aloud education requires parents or teachers to directly supervise and provide feedback to children, which requires a lot of time and effort. Furthermore, evaluation of reading aloud intonation, speech mistakes, and reading speed is subjective, making objective assessment difficult. Furthermore, there is a lack of methods to gamify reading aloud practice and increase motivation by competing with other users.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0934] In this invention, the server includes means for the user to start reading aloud, means for the terminal to record the user's reading aloud, means for the terminal to transmit the recorded voice data to the server, means for the server to transcribe the received voice data, means for the server to analyze the transcribed text and evaluate intonation, detect errors, and calculate reading speed, means for the server to generate feedback and a score based on the analysis results, means for the terminal to display the feedback and score to the user, and means for selecting a battle mode.This allows the user to effectively practice reading aloud, understand areas for improvement based on feedback, and increase motivation to learn by competing with other users.

[0935] "Reading aloud" is the act of a user reading aloud the presented text.

[0936] "Terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[0937] "Voice data" refers to digital data containing the user's speech recorded by the device's built-in microphone or an external microphone.

[0938] "Server" refers to the central processing unit that receives, analyzes, transcribes, and generates feedback from audio data.

[0939] "Transcription" is the process of analyzing audio data and converting what is spoken into text data.

[0940] "Intonation" is an element that evaluates the changes in pitch and strength in the user's vocalization.

[0941] "Errors" refer to mispronunciations or slip-ups that the user makes while reading aloud.

[0942] "Reading speed" is a factor that measures how quickly a user reads a given text aloud.

[0943] "Feedback" refers to comments on improvements and evaluations that the server provides to the user based on the analysis results.

[0944] "Score" refers to a numerical result that comprehensively evaluates the user's reading aloud.

[0945] "Battle mode" refers to a feature in which multiple users compete against each other for the highest reading score.

[0946] This invention is a system in which a terminal records a user's reading aloud and transmits the audio data to a server. This system applies speech recognition technology and analyzes the recorded data to provide accurate feedback to the user. The specific processing procedures and implementation outline of the system are described below.

[0947] Hardware and Software Used

[0948] 1. Hardware:

[0949] Device: A mobile device such as a smartphone or tablet.

[0950] Microphone: The microphone built into the device.

[0951] 2. Software:

[0952] Dedicated application: An application that displays reading assignments and records audio.

[0953] Server-side analysis program: A program that performs audio analysis, transcription, and feedback generation.

[0954] Speech Recognition Engine: Speech recognition software such as Google Cloud Speech-to-Text.

[0955] Recording and sending readings

[0956] The user launches a dedicated application on the device and reads the displayed text aloud. The device uses a built-in microphone to record the user's voice and transmits the recording to a server via the Internet. For example, the following prompt sentence is used:

[0957] "Please record today's reading assignment and send it to the server."

[0958] Analysis of audio data and feedback generation

[0959] The server temporarily stores the received voice data and transcribes it using a speech recognition engine such as Google Cloud Speech-to-Text.The server then analyzes the transcribed data and performs the following evaluations:

[0960] Intonation assessment: Analyzing the intonation and stress of speech.

[0961] Error detection: Identifying errors by comparing the transcribed text with the exact text.

[0962] Reading speed calculation: Calculates reading speed by comparing the length of the text with the recording time.

[0963] The server generates feedback and a score based on the analysis results. For example, if the intonation is insufficient, the server will provide feedback such as "Your intonation is insufficient. Try reading with more emotion." An example of a prompt is as follows:

[0964] "Analyze recorded speech and generate ratings for intonation, errors, and reading speed."

[0965] Providing and Viewing Feedback

[0966] The feedback and score generated by the server are sent to the device, which displays it to the user, allowing the user to understand where they need to improve their reading. For example, the following message is displayed on the device:

[0967] "Your intonation is lacking, please try again."

[0968] Implementing Battle Mode

[0969] The invention also includes a battle mode in which multiple users compete in reading aloud. When a user selects the battle mode, each reading aloud is sent to the server and analyzed independently. The analysis results are then used to generate feedback and a score for each user, and a final winner is determined. The prompt is as follows:

[0970] "Run Battle Mode and compare each user's score to determine the winner."

[0971] As a result, the present invention is implemented as a system that effectively supports the user's reading aloud learning, improves the reading aloud technique based on feedback, and strengthens motivation to learn.

[0972] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0973] Step 1:

[0974] The device launches a dedicated application.

[0975] Input: User action.

[0976] Output: The home screen of the application.

[0977] Specific operation: The user taps the application icon on the device to launch the application. The app displays the home screen, which displays the reading assignment and settings menu.

[0978] Step 2:

[0979] The terminal presents the text to the user for reading aloud.

[0980] Enter: the application home screen.

[0981] Output: The text that will be displayed as today's reading assignment.

[0982] Specific operation: The device displays a specific text on the screen as "Today's assignment," and the user prepares to read the text.

[0983] Step 3:

[0984] The user reads the presented text aloud.

[0985] Input: The presented text.

[0986] Output: Audio to be read aloud.

[0987] Specific action: The user reads aloud the presented text (e.g., "red shoes"), paying attention to mispronunciations and the volume of the voice during this stage.

[0988] Step 4:

[0989] The device records the user's voice.

[0990] Input: User's speech reading.

[0991] Output: Recorded audio data.

[0992] Specific operation: The device activates the built-in microphone and records the user's reading. When the recording is complete, the device generates an audio file.

[0993] Step 5:

[0994] The device sends the recorded audio data to the server.

[0995] Input: Recorded audio data.

[0996] Output: The audio data sent to the server.

[0997] Specific operation: The device compresses the recorded audio data and sends it to the server via the Internet. The device then verifies that the data was sent successfully.

[0998] Step 6:

[0999] The server temporarily stores the voice data and transcribes it using a voice recognition engine.

[1000] Input: Audio data received from the device.

[1001] Output: Transcribed text data.

[1002] Specific operation: The server temporarily saves the voice data in a directory where it stores it, then calls a voice recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[1003] Step 7:

[1004] The server analyzes the transcription data.

[1005] Input: Transcribed text data.

[1006] Output: Intonation assessment, error detection, and reading speed calculation.

[1007] Specific operation: The server runs an analysis program to evaluate intonation, detect errors, and calculate reading speed. Specifically, it compares the transcription data with the original text and evaluates pronunciation intonation, error locations, and reading speed.

[1008] Step 8:

[1009] The server generates feedback and a score.

[1010] Input: Intonation assessment, error detection, and reading speed results.

[1011] Output: Feedback and score.

[1012] Specific operation: The server generates a feedback message and an overall score based on the analysis results. For example, it may provide feedback such as, "Your intonation is lacking. Try to read with more emotion."

[1013] Step 9:

[1014] The server sends feedback and scores to the device.

[1015] Input: Generated feedback and score.

[1016] Output: Data sent to the terminal.

[1017] Specific operation: The server assembles the feedback message and score into a data packet and sends it to the device via the Internet.

[1018] Step 10:

[1019] The device displays feedback and scores to the user.

[1020] Input: Feedback and score received from the server.

[1021] Output: Feedback and score displayed to the user.

[1022] Specific operation: The device displays the received feedback message and score on the screen. The user can check the feedback and reflect it in the next reading.

[1023] (Application example 1)

[1024] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] While existing reading assessment systems aim to improve users' reading skills, they lack practical application in specific situations and contexts, especially when it comes to assessing product explanations and comprehension. Furthermore, they lack the ability to motivate users by allowing them to compete with other users. This leaves no effective tools for improving product comprehension and explanation skills in virtual stores.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1027] In this invention, the server includes means for recording audio data of product information read aloud by a user on a terminal and transmitting the recorded data to the server, means for the server to transcribe and analyze the audio data, means for generating feedback and a score based on the analysis results and providing them to the user, and means for displaying the results of competitions with other users for the read aloud product information. This allows users to read aloud product descriptions in a virtual store, be evaluated on their level of understanding and explanation ability, and compete with other users, thereby improving their product understanding and explanation ability.

[1028] The "means for the terminal to record the user's reading aloud" is a combination of hardware and software for recording the user's voice reading aloud.

[1029] The "means for transmitting audio data recorded by the terminal to a server" is a function for transmitting recorded audio data to a remote server via a network.

[1030] The "means for transcribing voice data received by the server" is a function including a voice recognition engine that converts voice data received by the server into text format.

[1031] "Means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed" refers to a function that runs algorithms based on the transcribed text data to analyze a speaker's intonation, pronunciation errors, and reading speed.

[1032] "Means for the server to generate feedback and a score based on the analysis results" refers to a function that generates points for improvement and an overall evaluation score for the user based on the analysis results.

[1033] The "means for the terminal to display the feedback and score to the user" is a function for displaying the generated feedback and score on the screen of the user's device.

[1034] "A means of having specific product information read aloud in a virtual store, analyzing the level of understanding and explanation ability, and providing feedback" is a function that allows a user to read aloud product information in a virtual space, evaluate the level of understanding of the information and explanation skills, and provide feedback.

[1035] The "means for displaying the results of a competition with other users for the product information read aloud" is a function that allows a plurality of users to compete for scores based on the product information read aloud, and displays the results.

[1036] This invention is a system for evaluating a user's reading skills and product comprehension. Below, we will explain in natural language how this system is actually implemented by explaining the program processing procedures.

[1037] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with specific product information as a reading task. The user reads the presented product information aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device transmits the recorded voice data to a server via the Internet.

[1038] The server temporarily stores the voice data received from the device and transcribes it using a speech recognition engine (for example, Google Cloud Speech-to-Text API). Once transcription is complete, the server analyzes the voice data using the following procedure.

[1039] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[1040] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[1041] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1042] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[1043] The system also includes a battle mode function. When a user selects battle mode, they can compete in reading aloud with other users. Each user reads aloud and sends each audio data to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and each user's feedback to the user, stimulating a competitive spirit and encouraging improvement of reading aloud skills.

[1044] Examples of concrete examples and prompts

[1045] The user browses product descriptions in a virtual store and reads them aloud. The app records the audio and sends it to the server. The server analyzes the recording and provides feedback to improve the user's reading skills and product comprehension. Specific examples of prompts include:

[1046] Please read aloud the description of the product "Smart Watch A." ​​We will provide feedback on what you read aloud.

[1047] In this way, the present invention provides a system that allows users to gain a deeper understanding of product information in a virtual store and improve their reading skills and product comprehension by competing with other users.

[1048] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1049] Step 1:

[1050] The user launches a dedicated application on the device and checks the specific product information presented as a reading task. The input is the product text information, and the output is a state ready for reading aloud. In this step, the device displays the product text to the user and provides an interface for preparing for reading aloud.

[1051] Step 2:

[1052] The user reads aloud the presented product information. The input is the user's voice, and the output is the recorded voice data. The device records the reading using the built-in microphone. In this step, the device receives the user's voice and saves it as an audio file (e.g., WAV format).

[1053] Step 3:

[1054] After the recording is complete, the device sends the recorded audio data to the server via the Internet. The input is the recorded audio data, and the output is the completion status of the audio data transfer to the server. The device uploads the audio file to the server using an HTTP request.

[1055] Step 4:

[1056] The server temporarily stores the received voice data and transcribes it using a speech recognition engine. The input is voice data, and the output is transcribed text data. The server converts the voice data into text using speech recognition technology such as the Google Cloud Speech-to-Text API.

[1057] Step 5:

[1058] The server analyzes the transcribed text data, evaluating intonation patterns, pronunciation errors, and reading speed. The input is the transcription data, and the output is the analysis results. The server uses algorithms to evaluate intonation, detect spelling errors, and calculate the reading speed.

[1059] Step 6:

[1060] The server generates specific feedback and an overall score based on the analysis results. The input is the analysis result data, and the output is feedback and a score. The server generates advice and evaluation points for the user based on the analysis data.

[1061] Step 7:

[1062] The server sends the generated feedback and score to the terminal, which displays it to the user. The input is the feedback and score, and the output is the information displayed to the user. The terminal displays the received data on the user interface.

[1063] Step 8:

[1064] When Battle Mode is selected, each user's reading data is analyzed independently to generate a score so that they can compete with other users. The input is the reading data of multiple users, and the output is feedback to each user, a score, and the winner's decision. The server uses a competitive algorithm to determine the winner and sends the result to the device.

[1065] By following these steps, the system can analyze and evaluate the user's reading skills, helping them improve their understanding of product information and their ability to explain it. Furthermore, the battle mode can stimulate competitive spirit and further motivate users.

[1066] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1067] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. Below, we will explain how to actually implement this system, using concrete examples.

[1068] Recording and sending readings

[1069] First, the user launches a dedicated application on their device (smartphone or tablet). The application then presents the user with a reading assignment text. The user then reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device sends the recorded audio data to a server via the Internet.

[1070] Analysis of audio data and feedback generation

[1071] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[1072] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[1073] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[1074] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1075] Furthermore, the server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[1076] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected and the user is recognized as nervous, the server will generate an overall score (e.g., 78 points) along with feedback such as, "Your intonation is lacking. Try reading more relaxed."

[1077] Providing and Viewing Feedback

[1078] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[1079] Implementing Battle Mode

[1080] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[1081] Long-term progress tracking and feedback

[1082] In addition, the server records the user's analysis results over a long period of time. This allows the server to understand the user's progress and trends, and provide feedback on improvements and progress over time. For example, by comparing past data with current data, the server can provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1083] In this way, the present invention is implemented as a system that is adapted to modern lifestyles, effectively supports children's reading aloud learning while reducing the burden on parents, and by combining it with an emotion engine, can provide more accurate feedback.

[1084] The processing flow will be explained below.

[1085] Step 1:

[1086] The user starts a dedicated application on their device. The application presents the user with a reading assignment text. The user reads the presented text aloud and prepares to begin reading aloud.

[1087] Step 2:

[1088] The device uses the built-in microphone to record the user's reading in real time, and once the recording is complete, the device temporarily stores the recorded audio data in local storage.

[1089] Step 3:

[1090] The device sends the recorded audio data to the server using a secure protocol (e.g., HTTPS). At the same time, the status of the transmission is displayed to the user.

[1091] Step 4:

[1092] The server receives the voice data sent from the device and stores it in the server, where it is prepared for analysis.

[1093] Step 5:

[1094] The server uses a speech recognition engine to transcribe the voice data. The speech recognition engine analyzes the voice data and generates text data. Once transcription is complete, the text data is stored on the server.

[1095] Step 6:

[1096] The server analyzes the transcribed text. The analysis is performed as follows:

[1097] Intonation assessment: The server detects intonation and the frequency of certain phrases in the text and assesses intonation.

[1098] Misspelling detection: The server compares the transcribed text with pre-trained correct text to detect spelling errors and mispronunciations.

[1099] Reading speed calculation: The server calculates the reading speed based on the total length of the text and the playback time of the audio data.

[1100] Step 7:

[1101] The server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[1102] Step 8:

[1103] The server evaluates intonation, detects slip-ups, calculates reading speed, and recognizes emotions, and generates specific feedback and an overall score. For example, it might generate feedback like, "Your intonation is lacking. Try reading more relaxedly," and calculate an overall score (e.g., 78 points).

[1104] Step 9:

[1105] The server sends the generated feedback and score to the device, using each user's identification information to ensure data integrity.

[1106] Step 10:

[1107] The device receives feedback and displays the score from the server to the user, who can then use the feedback to identify areas for improvement and practice for the next time.

[1108] Step 11:

[1109] When a user selects the battle mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and the audio data is sent from their device to the server.

[1110] Step 12:

[1111] In battle mode, the server analyzes each user's voice data independently and generates feedback and scores for each user. Based on the generated scores, the server determines the winner.

[1112] Step 13:

[1113] The server sends the results of the battle mode and feedback to the terminal, which displays the winner and feedback to the user, promoting communication between users.

[1114] Step 14:

[1115] The server records the user's progress and trends over time, allowing it to provide feedback on improvements and progress over time. It compares past data with current data and provides feedback such as, "Your reading speed is improving compared to before. Keep up the good work."

[1116] Example 2

[1117] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] Currently, there are systems that evaluate users' reading ability, but they lack detailed analysis of factors such as emotional state, intonation, and reading speed, and provide insufficient comprehensive feedback. Furthermore, they lack a mechanism for accurate and fair analysis and evaluation in battle modes where multiple users compete in reading aloud. Furthermore, a system that records users' long-term progress and provides appropriate feedback is also needed.

[1119] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1120] In this invention, the server includes a means for transcribing voice data, a means for analyzing the transcribed text to evaluate intonation, detect slip-ups, and calculate reading speed, and a means for providing feedback and a score based on the analysis results, as well as a means for analyzing the user's emotional state from the voice data. This enables comprehensive and precise analysis and feedback of the user's reading aloud. Furthermore, when multiple users compete in battle mode, each reading aloud data can be analyzed independently, enabling fair evaluation and winner determination. Furthermore, the server can support the user's continuous growth by recording the user's analysis results over a long period of time and providing progress and areas for improvement over time.

[1121] A "terminal" is an electronic device used by a user, and includes mobile information terminals such as smartphones and tablets.

[1122] "Reading aloud" refers to the act of a user reading a text aloud.

[1123] "Recording" refers to the act of recording audio as digital data using a microphone attached to a terminal.

[1124] "Audio data" refers to data in which recorded audio is saved in digital format.

[1125] "Server" refers to a computer system that receives, stores, and analyzes audio data.

[1126] "Transcription" is the process of converting audio data into text data using speech recognition technology.

[1127] "Intonation" refers to changes in pitch and volume when reading aloud, and is an element that reflects the emotions and intentions behind the words.

[1128] A "mispronunciation" refers to an error in which the words pronounced by the user during reading aloud do not match the presented task text.

[1129] "Reading speed" refers to the speed at which a user reads aloud, and is usually expressed as the number of characters or words divided by the time.

[1130] The "emotional state" indicates the emotion the user felt when reading aloud, and includes states such as joy, sadness, and anger.

[1131] "Feedback" refers to advice and evaluations provided to users based on the analysis results.

[1132] The "score" is a numerical value of the evaluation results of the reading aloud, and is provided as an overall evaluation.

[1133] "Battle mode" refers to a feature that allows multiple users to compete in reading aloud and compare the results.

[1134] "Analyzing independently" refers to analyzing each user's reading data individually without interfering with other data.

[1135] The "winner" refers to the user who is evaluated as having performed the best reading aloud in the battle mode.

[1136] "Long-term recording" refers to continuously saving the user's analysis results for a certain period of time and accumulating past data.

[1137] "Progress" refers to changes or improvements in a user's reading ability, including skill improvement over time.

[1138] MODE FOR CARRYING OUT THE INVENTION

[1139] The present invention is a reading aloud learning support system that combines an emotion engine with a system that provides feedback by recording and analyzing a user's reading aloud. This system includes a terminal, a server, and a dedicated application. Below, we will explain how this system is specifically implemented.

[1140] Hardware and Software Configuration

[1141] Device: A mobile information terminal used by the user, including a smartphone or tablet. A dedicated application is installed on the device, and it has the function of recording reading aloud using a built-in microphone.

[1142] Server: A computer system that receives voice data and performs transcription, speech analysis, feedback generation, emotion recognition, etc. It uses the following software and services:

[1143] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[1144] Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the user's emotional state.

[1145] Program processing

[1146] 1. Recording a reading: The user launches a dedicated application on their smartphone or tablet. The application displays the reading assignment text to the user, and the user reads and records the text aloud using the built-in microphone.

[1147] 2. Sending audio data: Once recording is complete, the device sends the recorded audio data to the server via the Internet.

[1148] 3. Analysis of voice data: The server temporarily stores the received voice data and analyzes it in the following steps.

[1149] Transcription: Transcribing audio data using the Google Cloud Speech-to-Text API.

[1150] Evaluating intonation: Extracting speech features from the transcribed text and evaluating intonation and intonation.

[1151] Misspelling detection: Compare the transcribed text with the assignment text to detect spelling errors and mispronunciations.

[1152] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1153] 4. Emotion Recognition: The server uses IBM Watson Tone Analyzer to recognize the user's emotional state from the voice data. Specifically, it analyzes the tone, speed, and intonation of the voice to detect the user's emotions (e.g., joy, sadness, anger, etc.).

[1154] 5. Feedback and Score Generation: The server generates specific feedback and an overall score based on the analysis results. For example, if the server detects a lack of intonation and tension, it will generate an overall score (e.g., 78 points) along with the feedback, "Your intonation is lacking. Try reading more relaxed."

[1155] 6. Providing feedback: The server sends the generated feedback and score to the device, which then displays the received feedback and score to the user, who can then improve their reading aloud.

[1156] Battle Mode

[1157] When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud and the audio data is sent to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and their feedback to the user, enhancing the competitive spirit and motivation to learn.

[1158] Long-term progress tracking and feedback

[1159] The server records the user's analysis results over a long period of time. This allows it to compare past data with current data and provide feedback to the user on areas for improvement and progress. For example, it could provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1160] Specific examples

[1161] 1. Example 1: Reading Aloud Assessment

[1162] Prompt text: The user reads the text "The weather is nice today."

[1163] Feedback: "Your intonation is appropriate. However, you read too quickly, so we recommend you read a little more slowly." and an overall score of 85 points are displayed.

[1164] 2. Example 2: Emotion Recognition

[1165] Prompt text: "The user seems emotionally exhausted."

[1166] Feedback: "You may be tired that day, so take a break and relax before trying again."

[1167] In this way, the present invention is implemented as a system that comprehensively analyzes a user's reading ability and provides appropriate feedback.

[1168] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1169] System program processing flow

[1170] Step 1: Record your reading

[1171] User: Launches the dedicated application on a smartphone or tablet.

[1172] Specific action: Tap the application icon.

[1173] Terminal: The application starts and displays the reading assignment text to the user.

[1174] Specific operation: Transition to the text display screen.

[1175] Input: Application launch signal.

[1176] Output: Display of the assignment text.

[1177] Step 2: Start recording

[1178] User: Read the presented text aloud.

[1179] Specific behavior: Reads text aloud into the built-in microphone.

[1180] Device: Use the built-in microphone to record your reading aloud.

[1181] Specific operation: Tap the record button to start recording.

[1182] Input: Record button tap signal.

[1183] Output: Start recording.

[1184] Step 3: Sending audio data

[1185] Device: Once recording is complete, the audio data is sent to a server via the Internet.

[1186] Specific operation: Tap the stop recording button and automatically send the audio data.

[1187] Input: Recording stop button tap signal, recording data.

[1188] Output: Sending audio data to the server.

[1189] Step 4: Receiving and storing audio data

[1190] Server: Temporarily stores received audio data.

[1191] Specific operation: Save the audio file to the database.

[1192] Input: Audio data from the device.

[1193] Output: Save audio data.

[1194] Step 5: Transcription

[1195] Server: Transcribing the audio data using a speech recognition engine.

[1196] Specific operation: Converts speech to text using the Google Cloud Speech-to-Text API.

[1197] Input: Stored audio data.

[1198] Output: The transcribed text.

[1199] Step 6: Evaluate intonation

[1200] Server: Evaluates the intonation and inflection of the transcribed text.

[1201] Specific operation: Extract audio features.

[1202] Input: The transcribed text.

[1203] Output: Intonation evaluation results.

[1204] Step 7: Detect slip-ups

[1205] Server: Compares the transcribed text with the assignment text to detect spelling errors and pronunciation mistakes.

[1206] What it does: Uses a text matching algorithm.

[1207] Input: Transcribed text, assignment text.

[1208] Output: Slip-of-the-speech detection results.

[1209] Step 8: Calculate your reading speed

[1210] Server: Calculates reading speed by comparing text length with playback time.

[1211] Specific operation: Calculate the reading speed (number of characters / second) from the playback time.

[1212] Input: Transcribed text, duration.

[1213] Output: Reading speed calculation results.

[1214] Step 9: Emotion Recognition

[1215] Server: Analyzes the user's emotional state from the voice data using an emotion engine.

[1216] Specific operation: Uses IBM Watson Tone Analyzer.

[1217] Input: Audio data.

[1218] Output: Emotion recognition result.

[1219] Step 10: Generate feedback and scores

[1220] Server: Generates specific feedback and an overall score based on the analysis results.

[1221] Specific actions: Evaluation is based on intonation, mistakes, reading speed, and emotional state.

[1222] Input: Intonation evaluation results, slip-up results, reading speed results, emotion recognition results.

[1223] Output: Feedback and overall score.

[1224] Step 11: Submit your feedback and score

[1225] Server: Sends generated feedback and scores to devices.

[1226] Specific operation: Send data in File or JSON format.

[1227] Input: Feedback and overall score.

[1228] Output: Data sent to the terminal.

[1229] Step 12: View your feedback and scores

[1230] Terminal: Displays received feedback and scores to the user.

[1231] Specific operation: Proceed to the feedback result screen.

[1232] Input: Send data.

[1233] Output: Display of feedback and score.

[1234] (Application example 2)

[1235] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1236] While conventional reading practice systems can provide technical evaluations of reading accuracy, intonation, speed, etc., it is difficult to understand the user's emotional state and provide appropriate feedback. As a result, users may continue practicing while feeling nervous or anxious, which hinders effective improvement of reading skills.

[1237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1238] In this invention, the server includes means for recording the user's reading aloud on the terminal, means for transmitting the recorded voice data to the server, means for transcribing the received voice data, means for analyzing the transcribed text and evaluating intonation, detecting slip-ups, and calculating reading speed, means for generating feedback and a score based on the analysis results, means for displaying the feedback and score to the user on the terminal, means for recognizing the user's emotional state from the voice data, and means for generating feedback according to the emotion based on the emotion recognition results. This makes it possible to provide appropriate feedback that reflects the user's emotional state and promote improvement in reading aloud skills.

[1239] A "terminal" is an electronic device such as a smartphone, tablet, or personal computer used by a user who reads aloud.

[1240] The "means for recording" is a mechanism for recording the user's voice reading aloud as digital data using a microphone built into the terminal or an externally connected recording device.

[1241] "Audio data" is digital data representing recorded audio readings.

[1242] "Transmission means" refers to the technical mechanism by which the recorded audio data is sent to a server via the Internet or other communications network.

[1243] A "server" is a central processing system for receiving and analyzing audio data and generating feedback.

[1244] "Transcribing means" refers to the process of analyzing received audio data and converting it into corresponding text data.

[1245] The "analysis method" refers to a series of technical processes that use transcribed text data to evaluate intonation, detect slip-ups, calculate reading speed, and so on.

[1246] "Intonation evaluation" is the process of analyzing the intonation and vocal stress of a text read aloud and judging its quality.

[1247] "Slip-up detection" is the process of comparing the transcribed text with the exact text to identify typos and mispronunciations.

[1248] "Reading speed calculation" is the process of measuring reading speed using the total length of the text and the playback time of the audio data.

[1249] The "means for generating feedback and scores" is a mechanism that, based on the analysis results, creates a feedback message indicating an evaluation of the user's reading aloud and areas for improvement, as well as an overall evaluation score.

[1250] The "means for displaying" is a mechanism for displaying the generated feedback and score on the screen of the terminal in a form that can be visually recognized by the user.

[1251] "Emotional state" refers to the user's psychological and emotional state analyzed from voice data, including, for example, joy, sadness, tension, etc.

[1252] "Means for emotion recognition" refers to the technical process for identifying the user's emotional state from voice data.

[1253] "Feedback according to emotions" is a feedback message that provides appropriate advice or areas for improvement based on the results of emotion recognition of the user.

[1254] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. The specific configuration and method for realizing this system are described in detail below.

[1255] 1. Record and send your reading

[1256] The user launches a dedicated application using a device such as a smartphone or tablet. This application displays the assignment text for the user to read aloud. The user reads the assignment text aloud and records the audio using the device's built-in microphone. This recorded audio data is sent to a server via the Internet.

[1257] 2. Analysis of audio data and feedback generation

[1258] The server temporarily stores the received voice data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The transcribed data is analyzed as follows:

[1259] Intonation assessment: Analyze the transcribed text to assess intonation and intonation.

[1260] Misspelling detection: Compares the transcribed text with the correct text to detect typos and mispronunciations.

[1261] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1262] Furthermore, the server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to determine the user's emotional state (e.g., joy, sadness, nervousness, etc.). Based on these analysis results, the server generates specific feedback and a score. For example, if the user is nervous, the server provides feedback such as, "Your intonation is lacking. Try reading more relaxed."

[1263] 3. Providing and Viewing Feedback

[1264] The feedback and score generated by the server are sent to the terminal, which displays the received feedback and score to the user, allowing the user to improve their reading.

[1265] 4. Implementation of Battle Mode

[1266] The system includes a battle mode, which users can select to compete against other users (e.g., parents or friends) in reading aloud. Each user reads aloud individually and sends the audio data to the server. The server analyzes each audio data independently, generates feedback and scores for each, and determines the winner based on these scores.

[1267] 5. Long-term progress tracking and feedback

[1268] The server records the user's analysis results over a long period of time. This allows the server to evaluate the user's progress over time and provide feedback on improvements and progress over time. For example, the server might provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1269] Examples of concrete examples and prompts

[1270] Example: After a child completes one minute of reading aloud practice, the recorded speech is automatically analyzed and feedback such as "Your speed is 2.5 characters per second. You lack intonation. Relax and try again" is displayed on the head-mounted display.

[1271] Example prompts to input to a generative AI model:

[1272] "Use a sentiment analysis engine to recognize emotions as users read and generate feedback on intonation, speed, and spelling errors."

[1273] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1274] Step 1:

[1275] The user launches a dedicated application on their device. The application displays a reading assignment text to the user. The input is the assignment text selection provided by the user, and the output is the assignment text displayed on the application screen.

[1276] Step 2:

[1277] The user reads the assignment text aloud and records the audio using the device's built-in microphone. The input is the user's voice, and the output is audio data captured by the device's recording function. This recorded data is temporarily saved on the device.

[1278] Step 3:

[1279] The device sends the recorded voice data to the server via the Internet. The input is the voice data stored on the device, and the output is the voice data received by the server. The voice data is stored in the server's storage.

[1280] Step 4:

[1281] The server uses the received audio data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The input is the audio data, and the output is the transcribed text data. The server passes this text data to the next analysis step.

[1282] Step 5:

[1283] The server analyzes the transcribed text data. Specifically, it evaluates intonation, detects slip-ups, and calculates reading speed. The input is the transcribed text, and the output is the results of these evaluations.

[1284] Intonation assessment: Analyze the intonation patterns of the text and evaluate their quality.

[1285] Misspelling detection: Detects typos and mispronunciations by comparing with the correct text.

[1286] Reading speed calculation: Reading speed is calculated using the total length of the text and the playback time of the audio data.

[1287] Step 6:

[1288] The server recognizes the user's emotions from the voice data. Using an emotion recognition engine, it analyzes the tone, rate, and intonation of the voice to identify the emotional state (e.g., joy, sadness, tension, etc.). The input is the voice data, and the output is the identification of the user's emotional state.

[1289] Step 7:

[1290] The server generates feedback and a score based on the analysis and emotion recognition results. The inputs are the evaluation results of intonation, speed, typo detection, and emotional state, and the output is a specific feedback message and an overall score for the user.

[1291] Step 8:

[1292] The server sends feedback and scores to the terminal. The input is the generated feedback and scores, and the output is the transmission of data to the user terminal.

[1293] Step 9:

[1294] The terminal displays the received feedback and score to the user. The input is the feedback and score sent from the server, and the output is the feedback and score displayed on the terminal screen. The user can use this information to improve their reading aloud.

[1295] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1296] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1297] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1298] [Fourth embodiment]

[1299] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1300] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1301] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1302] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1303] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1304] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1305] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1306] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1307] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1308] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1309] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1310] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1311] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1312] The present invention is a system in which a terminal records a user's reading aloud and transmits the recorded voice data to a server. How this system is actually implemented will be described below with specific examples.

[1313] Recording and sending readings

[1314] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with a specific text as a reading task. The user reads the presented text aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device sends the recorded voice data to a server via the Internet.

[1315] Analysis of audio data and feedback generation

[1316] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[1317] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[1318] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[1319] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1320] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[1321] Providing and Viewing Feedback

[1322] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[1323] Implementing Battle Mode

[1324] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[1325] In this way, the present invention is implemented as a system that effectively supports children's reading aloud learning while reducing the burden on parents in a form that is adapted to modern lifestyles.

[1326] The processing flow will be explained below.

[1327] Step 1:

[1328] The user launches a dedicated application on their device, which then presents the user with a reading assignment text.

[1329] Step 2:

[1330] The user reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device saves the recorded audio data.

[1331] Step 3:

[1332] The device transmits the stored audio data to a server over the Internet using a secure protocol (e.g., HTTPS).

[1333] Step 4:

[1334] The server receives the voice data sent from the device and temporarily stores it for subsequent processing.

[1335] Step 5:

[1336] The server extracts the stored voice data and transcribes it using a speech recognition engine, which then analyzes the voice data and generates text data.

[1337] Step 6:

[1338] The server analyzes the transcribed text. The first step in the analysis is to evaluate intonation, which evaluates the frequency of intonation and specific phrases in the text.

[1339] Step 7:

[1340] The server then compares the transcribed text with pre-trained correct text, detecting spelling errors, mispronunciations, and other mistakes.

[1341] Step 8:

[1342] The server calculates the reading speed based on the total length of the text and the playback time of the audio data, and evaluates whether the reading speed is appropriate.

[1343] Step 9:

[1344] The server generates a comprehensive feedback and score based on the results of intonation evaluation, slip-up detection, and reading speed calculations. An example of specific feedback might be, "Your intonation is lacking. Try to read with more emotion."

[1345] Step 10:

[1346] The server sends the generated feedback and score to the terminal using unique identification information for each user.

[1347] Step 11:

[1348] The device receives feedback and displays the score from the server to the user, allowing the user to identify areas for improvement in their reading aloud and improve their learning.

[1349] Step 12:

[1350] If a user selects Battle Mode, a process of reading aloud competition with parents or friends will begin. Reading aloud data is sent from each device to the server and analyzed individually.

[1351] Step 13:

[1352] In battle mode, the server analyzes multiple reading data independently, generates feedback and scores for each, and compares the scores to determine the winner.

[1353] Step 14:

[1354] The server sends the battle results and feedback to the terminal, which displays the results to the user and promotes communication between users.

[1355] Example 1

[1356] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1357] Traditional reading aloud education requires parents or teachers to directly supervise and provide feedback to children, which requires a lot of time and effort. Furthermore, evaluation of reading aloud intonation, speech mistakes, and reading speed is subjective, making objective assessment difficult. Furthermore, there is a lack of methods to gamify reading aloud practice and increase motivation by competing with other users.

[1358] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1359] In this invention, the server includes means for the user to start reading aloud, means for the terminal to record the user's reading aloud, means for the terminal to transmit the recorded voice data to the server, means for the server to transcribe the received voice data, means for the server to analyze the transcribed text and evaluate intonation, detect errors, and calculate reading speed, means for the server to generate feedback and a score based on the analysis results, means for the terminal to display the feedback and score to the user, and means for selecting a battle mode.This allows the user to effectively practice reading aloud, understand areas for improvement based on feedback, and increase motivation to learn by competing with other users.

[1360] "Reading aloud" is the act of a user reading aloud the presented text.

[1361] "Terminal" refers to an electronic device used by a user, such as a computer, smartphone, or tablet.

[1362] "Voice data" refers to digital data containing the user's speech recorded by the device's built-in microphone or an external microphone.

[1363] "Server" refers to the central processing unit that receives, analyzes, transcribes, and generates feedback from audio data.

[1364] "Transcription" is the process of analyzing audio data and converting what is spoken into text data.

[1365] "Intonation" is an element that evaluates the changes in pitch and strength in the user's vocalization.

[1366] "Errors" refer to mispronunciations or slip-ups that the user makes while reading aloud.

[1367] "Reading speed" is a factor that measures how quickly a user reads a given text aloud.

[1368] "Feedback" refers to comments on improvements and evaluations that the server provides to the user based on the analysis results.

[1369] "Score" refers to a numerical result that comprehensively evaluates the user's reading aloud.

[1370] "Battle mode" refers to a feature in which multiple users compete against each other for the highest reading score.

[1371] This invention is a system in which a terminal records a user's reading aloud and transmits the audio data to a server. This system applies speech recognition technology and analyzes the recorded data to provide accurate feedback to the user. The specific processing procedures and implementation outline of the system are described below.

[1372] Hardware and Software Used

[1373] 1. Hardware:

[1374] Device: A mobile device such as a smartphone or tablet.

[1375] Microphone: The microphone built into the device.

[1376] 2. Software:

[1377] Dedicated application: An application that displays reading assignments and records audio.

[1378] Server-side analysis program: A program that performs audio analysis, transcription, and feedback generation.

[1379] Speech Recognition Engine: Speech recognition software such as Google Cloud Speech-to-Text.

[1380] Recording and sending readings

[1381] The user launches a dedicated application on the device and reads the displayed text aloud. The device uses a built-in microphone to record the user's voice and transmits the recording to a server via the Internet. For example, the following prompt sentence is used:

[1382] "Please record today's reading assignment and send it to the server."

[1383] Analysis of audio data and feedback generation

[1384] The server temporarily stores the received voice data and transcribes it using a speech recognition engine such as Google Cloud Speech-to-Text.The server then analyzes the transcribed data and performs the following evaluations:

[1385] Intonation assessment: Analyzing the intonation and stress of speech.

[1386] Error detection: Identifying errors by comparing the transcribed text with the exact text.

[1387] Reading speed calculation: Calculates reading speed by comparing the length of the text with the recording time.

[1388] The server generates feedback and a score based on the analysis results. For example, if the intonation is insufficient, the server will provide feedback such as "Your intonation is insufficient. Try reading with more emotion." An example of a prompt is as follows:

[1389] "Analyze recorded speech and generate ratings for intonation, errors, and reading speed."

[1390] Providing and Viewing Feedback

[1391] The feedback and score generated by the server are sent to the device, which displays it to the user, allowing the user to understand where they need to improve their reading. For example, the following message is displayed on the device:

[1392] "Your intonation is lacking, please try again."

[1393] Implementing Battle Mode

[1394] The invention also includes a battle mode in which multiple users compete in reading aloud. When a user selects the battle mode, each reading aloud is sent to the server and analyzed independently. The analysis results are then used to generate feedback and a score for each user, and a final winner is determined. The prompt is as follows:

[1395] "Run Battle Mode and compare each user's score to determine the winner."

[1396] As a result, the present invention is implemented as a system that effectively supports the user's reading aloud learning, improves the reading aloud technique based on feedback, and strengthens motivation to learn.

[1397] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1398] Step 1:

[1399] The device launches a dedicated application.

[1400] Input: User action.

[1401] Output: The home screen of the application.

[1402] Specific operation: The user taps the application icon on the device to launch the application. The app displays the home screen, which displays the reading assignment and settings menu.

[1403] Step 2:

[1404] The terminal presents the text to the user for reading aloud.

[1405] Enter: the application home screen.

[1406] Output: The text that will be displayed as today's reading assignment.

[1407] Specific operation: The device displays a specific text on the screen as "Today's assignment," and the user prepares to read the text.

[1408] Step 3:

[1409] The user reads the presented text aloud.

[1410] Input: The presented text.

[1411] Output: Audio to be read aloud.

[1412] Specific action: The user reads aloud the presented text (e.g., "red shoes"), paying attention to mispronunciations and the volume of the voice during this stage.

[1413] Step 4:

[1414] The device records the user's voice.

[1415] Input: User's speech reading.

[1416] Output: Recorded audio data.

[1417] Specific operation: The device activates the built-in microphone and records the user's reading. When the recording is complete, the device generates an audio file.

[1418] Step 5:

[1419] The device sends the recorded audio data to the server.

[1420] Input: Recorded audio data.

[1421] Output: The audio data sent to the server.

[1422] Specific operation: The device compresses the recorded audio data and sends it to the server via the Internet. The device then verifies that the data was sent successfully.

[1423] Step 6:

[1424] The server temporarily stores the voice data and transcribes it using a voice recognition engine.

[1425] Input: Audio data received from the device.

[1426] Output: Transcribed text data.

[1427] Specific operation: The server temporarily saves the voice data in a directory where it stores it, then calls a voice recognition engine (e.g., Google Cloud Speech-to-Text) to convert the voice data into text.

[1428] Step 7:

[1429] The server analyzes the transcription data.

[1430] Input: Transcribed text data.

[1431] Output: Intonation assessment, error detection, and reading speed calculation.

[1432] Specific operation: The server runs an analysis program to evaluate intonation, detect errors, and calculate reading speed. Specifically, it compares the transcription data with the original text and evaluates pronunciation intonation, error locations, and reading speed.

[1433] Step 8:

[1434] The server generates feedback and a score.

[1435] Input: Intonation assessment, error detection, and reading speed results.

[1436] Output: Feedback and score.

[1437] Specific operation: The server generates a feedback message and an overall score based on the analysis results. For example, it may provide feedback such as, "Your intonation is lacking. Try to read with more emotion."

[1438] Step 9:

[1439] The server sends feedback and scores to the device.

[1440] Input: Generated feedback and score.

[1441] Output: Data sent to the terminal.

[1442] Specific operation: The server assembles the feedback message and score into a data packet and sends it to the device via the Internet.

[1443] Step 10:

[1444] The device displays feedback and scores to the user.

[1445] Input: Feedback and score received from the server.

[1446] Output: Feedback and score displayed to the user.

[1447] Specific operation: The device displays the received feedback message and score on the screen. The user can check the feedback and reflect it in the next reading.

[1448] (Application example 1)

[1449] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1450] While existing reading assessment systems aim to improve users' reading skills, they lack practical application in specific situations and contexts, especially when it comes to assessing product explanations and comprehension. Furthermore, they lack the ability to motivate users by allowing them to compete with other users. This leaves no effective tools for improving product comprehension and explanation skills in virtual stores.

[1451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1452] In this invention, the server includes means for recording audio data of product information read aloud by a user on a terminal and transmitting the recorded data to the server, means for the server to transcribe and analyze the audio data, means for generating feedback and a score based on the analysis results and providing them to the user, and means for displaying the results of competitions with other users for the read aloud product information. This allows users to read aloud product descriptions in a virtual store, be evaluated on their level of understanding and explanation ability, and compete with other users, thereby improving their product understanding and explanation ability.

[1453] The "means for the terminal to record the user's reading aloud" is a combination of hardware and software for recording the user's voice reading aloud.

[1454] The "means for transmitting audio data recorded by the terminal to a server" is a function for transmitting recorded audio data to a remote server via a network.

[1455] The "means for transcribing voice data received by the server" is a function including a voice recognition engine that converts voice data received by the server into text format.

[1456] "Means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed" refers to a function that runs algorithms based on the transcribed text data to analyze a speaker's intonation, pronunciation errors, and reading speed.

[1457] "Means for the server to generate feedback and a score based on the analysis results" refers to a function that generates points for improvement and an overall evaluation score for the user based on the analysis results.

[1458] The "means for the terminal to display the feedback and score to the user" is a function for displaying the generated feedback and score on the screen of the user's device.

[1459] "A means of having specific product information read aloud in a virtual store, analyzing the level of understanding and explanation ability, and providing feedback" is a function that allows a user to read aloud product information in a virtual space, evaluate the level of understanding of the information and explanation skills, and provide feedback.

[1460] The "means for displaying the results of a competition with other users for the product information read aloud" is a function that allows a plurality of users to compete for scores based on the product information read aloud, and displays the results.

[1461] This invention is a system for evaluating a user's reading skills and product comprehension. Below, we will explain in natural language how this system is actually implemented by explaining the program processing procedures.

[1462] First, the user launches a dedicated application on their device (smartphone or tablet). The application presents the user with specific product information as a reading task. The user reads the presented product information aloud, and the device records the user's voice using the built-in microphone. Once the recording is complete, the device transmits the recorded voice data to a server via the Internet.

[1463] The server temporarily stores the voice data received from the device and transcribes it using a speech recognition engine (for example, Google Cloud Speech-to-Text API). Once transcription is complete, the server analyzes the voice data using the following procedure.

[1464] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[1465] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[1466] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1467] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected, the server will generate an overall score (e.g., 85 points) along with feedback such as, "Your intonation is lacking. Try reading with more emotion."

[1468] The system also includes a battle mode function. When a user selects battle mode, they can compete in reading aloud with other users. Each user reads aloud and sends each audio data to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and each user's feedback to the user, stimulating a competitive spirit and encouraging improvement of reading aloud skills.

[1469] Examples of concrete examples and prompts

[1470] The user browses product descriptions in a virtual store and reads them aloud. The app records the audio and sends it to the server. The server analyzes the recording and provides feedback to improve the user's reading skills and product comprehension. Specific examples of prompts include:

[1471] Please read aloud the description of the product "Smart Watch A." ​​We will provide feedback on what you read aloud.

[1472] In this way, the present invention provides a system that allows users to gain a deeper understanding of product information in a virtual store and improve their reading skills and product comprehension by competing with other users.

[1473] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1474] Step 1:

[1475] The user launches a dedicated application on the device and checks the specific product information presented as a reading task. The input is the product text information, and the output is a state ready for reading aloud. In this step, the device displays the product text to the user and provides an interface for preparing for reading aloud.

[1476] Step 2:

[1477] The user reads aloud the presented product information. The input is the user's voice, and the output is the recorded voice data. The device records the reading using the built-in microphone. In this step, the device receives the user's voice and saves it as an audio file (e.g., WAV format).

[1478] Step 3:

[1479] After the recording is complete, the device sends the recorded audio data to the server via the Internet. The input is the recorded audio data, and the output is the completion status of the audio data transfer to the server. The device uploads the audio file to the server using an HTTP request.

[1480] Step 4:

[1481] The server temporarily stores the received voice data and transcribes it using a speech recognition engine. The input is voice data, and the output is transcribed text data. The server converts the voice data into text using speech recognition technology such as the Google Cloud Speech-to-Text API.

[1482] Step 5:

[1483] The server analyzes the transcribed text data, evaluating intonation patterns, pronunciation errors, and reading speed. The input is the transcription data, and the output is the analysis results. The server uses algorithms to evaluate intonation, detect spelling errors, and calculate the reading speed.

[1484] Step 6:

[1485] The server generates specific feedback and an overall score based on the analysis results. The input is the analysis result data, and the output is feedback and a score. The server generates advice and evaluation points for the user based on the analysis data.

[1486] Step 7:

[1487] The server sends the generated feedback and score to the terminal, which displays it to the user. The input is the feedback and score, and the output is the information displayed to the user. The terminal displays the received data on the user interface.

[1488] Step 8:

[1489] When Battle Mode is selected, each user's reading data is analyzed independently to generate a score so that they can compete with other users. The input is the reading data of multiple users, and the output is feedback to each user, a score, and the winner's decision. The server uses a competitive algorithm to determine the winner and sends the result to the device.

[1490] By following these steps, the system can analyze and evaluate the user's reading skills, helping them improve their understanding of product information and their ability to explain it. Furthermore, the battle mode can stimulate competitive spirit and further motivate users.

[1491] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1492] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. Below, we will explain how to actually implement this system, using concrete examples.

[1493] Recording and sending readings

[1494] First, the user launches a dedicated application on their device (smartphone or tablet). The application then presents the user with a reading assignment text. The user then reads the presented text aloud and records the reading using the device's built-in microphone. Once the recording is complete, the device sends the recorded audio data to a server via the Internet.

[1495] Analysis of audio data and feedback generation

[1496] The server temporarily stores the voice data received from the device and transcribes it using a voice recognition engine. Once transcription is complete, the server analyzes the voice data using the following procedure.

[1497] Intonation assessment: Analyze the transcribed text and evaluate patterns of inflection and intonation.

[1498] Misspelling detection: Compare the transcribed text with the correct text to detect spelling errors and mispronunciations.

[1499] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1500] Furthermore, the server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[1501] The server generates specific feedback and an overall score based on the analysis results. For example, if a lack of intonation is detected and the user is recognized as nervous, the server will generate an overall score (e.g., 78 points) along with feedback such as, "Your intonation is lacking. Try reading more relaxed."

[1502] Providing and Viewing Feedback

[1503] The feedback and scores generated by the server are sent to the device, which then displays the received feedback and scores to the user. The user can improve their reading aloud based on the displayed feedback. Parents can use this feedback to understand their child's learning progress without being directly involved in the reading aloud.

[1504] Implementing Battle Mode

[1505] The present invention also includes a function called Battle Mode. When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and each voice data is sent to the server. The server analyzes each voice data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the terminal. The terminal displays the winner and their feedback to the user, promoting communication.

[1506] Long-term progress tracking and feedback

[1507] In addition, the server records the user's analysis results over a long period of time. This allows the server to understand the user's progress and trends, and provide feedback on improvements and progress over time. For example, by comparing past data with current data, the server can provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1508] In this way, the present invention is implemented as a system that is adapted to modern lifestyles, effectively supports children's reading aloud learning while reducing the burden on parents, and by combining it with an emotion engine, can provide more accurate feedback.

[1509] The processing flow will be explained below.

[1510] Step 1:

[1511] The user starts a dedicated application on their device. The application presents the user with a reading assignment text. The user reads the presented text aloud and prepares to begin reading aloud.

[1512] Step 2:

[1513] The device uses the built-in microphone to record the user's reading in real time, and once the recording is complete, the device temporarily stores the recorded audio data in local storage.

[1514] Step 3:

[1515] The device sends the recorded audio data to the server using a secure protocol (e.g., HTTPS). At the same time, the status of the transmission is displayed to the user.

[1516] Step 4:

[1517] The server receives the voice data sent from the device and stores it in the server, where it is prepared for analysis.

[1518] Step 5:

[1519] The server uses a speech recognition engine to transcribe the voice data. The speech recognition engine analyzes the voice data and generates text data. Once transcription is complete, the text data is stored on the server.

[1520] Step 6:

[1521] The server analyzes the transcribed text. The analysis is performed as follows:

[1522] Intonation assessment: The server detects intonation and the frequency of certain phrases in the text and assesses intonation.

[1523] Misspelling detection: The server compares the transcribed text with pre-trained correct text to detect spelling errors and mispronunciations.

[1524] Reading speed calculation: The server calculates the reading speed based on the total length of the text and the playback time of the audio data.

[1525] Step 7:

[1526] The server uses an emotion engine to recognize the user's emotion from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to detect the user's emotional state (e.g., joy, sadness, anger, etc.).

[1527] Step 8:

[1528] The server evaluates intonation, detects slip-ups, calculates reading speed, and recognizes emotions, and generates specific feedback and an overall score. For example, it might generate feedback like, "Your intonation is lacking. Try reading more relaxedly," and calculate an overall score (e.g., 78 points).

[1529] Step 9:

[1530] The server sends the generated feedback and score to the device, using each user's identification information to ensure data integrity.

[1531] Step 10:

[1532] The device receives feedback and displays the score from the server to the user, who can then use the feedback to identify areas for improvement and practice for the next time.

[1533] Step 11:

[1534] When a user selects the battle mode, they can compete in reading aloud against their parents or friends. Each user reads aloud, and the audio data is sent from their device to the server.

[1535] Step 12:

[1536] In battle mode, the server analyzes each user's voice data independently and generates feedback and scores for each user. Based on the generated scores, the server determines the winner.

[1537] Step 13:

[1538] The server sends the results of the battle mode and feedback to the terminal, which displays the winner and feedback to the user, promoting communication between users.

[1539] Step 14:

[1540] The server records the user's progress and trends over time, allowing it to provide feedback on improvements and progress over time. It compares past data with current data and provides feedback such as, "Your reading speed is improving compared to before. Keep up the good work."

[1541] Example 2

[1542] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1543] Currently, there are systems that evaluate users' reading ability, but they lack detailed analysis of factors such as emotional state, intonation, and reading speed, and provide insufficient comprehensive feedback. Furthermore, they lack a mechanism for accurate and fair analysis and evaluation in battle modes where multiple users compete in reading aloud. Furthermore, a system that records users' long-term progress and provides appropriate feedback is also needed.

[1544] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1545] In this invention, the server includes a means for transcribing voice data, a means for analyzing the transcribed text to evaluate intonation, detect slip-ups, and calculate reading speed, and a means for providing feedback and a score based on the analysis results, as well as a means for analyzing the user's emotional state from the voice data. This enables comprehensive and precise analysis and feedback of the user's reading aloud. Furthermore, when multiple users compete in battle mode, each reading aloud data can be analyzed independently, enabling fair evaluation and winner determination. Furthermore, the server can support the user's continuous growth by recording the user's analysis results over a long period of time and providing progress and areas for improvement over time.

[1546] A "terminal" is an electronic device used by a user, and includes mobile information terminals such as smartphones and tablets.

[1547] "Reading aloud" refers to the act of a user reading a text aloud.

[1548] "Recording" refers to the act of recording audio as digital data using a microphone attached to a terminal.

[1549] "Audio data" refers to data in which recorded audio is saved in digital format.

[1550] "Server" refers to a computer system that receives, stores, and analyzes audio data.

[1551] "Transcription" is the process of converting audio data into text data using speech recognition technology.

[1552] "Intonation" refers to changes in pitch and volume when reading aloud, and is an element that reflects the emotions and intentions behind the words.

[1553] A "mispronunciation" refers to an error in which the words pronounced by the user during reading aloud do not match the presented task text.

[1554] "Reading speed" refers to the speed at which a user reads aloud, and is usually expressed as the number of characters or words divided by the time.

[1555] The "emotional state" indicates the emotion the user felt when reading aloud, and includes states such as joy, sadness, and anger.

[1556] "Feedback" refers to advice and evaluations provided to users based on the analysis results.

[1557] The "score" is a numerical value of the evaluation results of the reading aloud, and is provided as an overall evaluation.

[1558] "Battle mode" refers to a feature that allows multiple users to compete in reading aloud and compare the results.

[1559] "Analyzing independently" refers to analyzing each user's reading data individually without interfering with other data.

[1560] The "winner" refers to the user who is evaluated as having performed the best reading aloud in the battle mode.

[1561] "Long-term recording" refers to continuously saving the user's analysis results for a certain period of time and accumulating past data.

[1562] "Progress" refers to changes or improvements in a user's reading ability, including skill improvement over time.

[1563] MODE FOR CARRYING OUT THE INVENTION

[1564] The present invention is a reading aloud learning support system that combines an emotion engine with a system that provides feedback by recording and analyzing a user's reading aloud. This system includes a terminal, a server, and a dedicated application. Below, we will explain how this system is specifically implemented.

[1565] Hardware and Software Configuration

[1566] Device: A mobile information terminal used by the user, including a smartphone or tablet. A dedicated application is installed on the device, and it has the function of recording reading aloud using a built-in microphone.

[1567] Server: A computer system that receives voice data and performs transcription, speech analysis, feedback generation, emotion recognition, etc. It uses the following software and services:

[1568] Speech recognition engine: Uses the Google Cloud Speech-to-Text API to convert voice data into text data.

[1569] Emotion Engine: Uses IBM Watson Tone Analyzer to analyze the user's emotional state.

[1570] Program processing

[1571] 1. Recording a reading: The user launches a dedicated application on their smartphone or tablet. The application displays the reading assignment text to the user, and the user reads and records the text aloud using the built-in microphone.

[1572] 2. Sending audio data: Once recording is complete, the device sends the recorded audio data to the server via the Internet.

[1573] 3. Analysis of voice data: The server temporarily stores the received voice data and analyzes it in the following steps.

[1574] Transcription: Transcribing audio data using the Google Cloud Speech-to-Text API.

[1575] Evaluating intonation: Extracting speech features from the transcribed text and evaluating intonation and intonation.

[1576] Misspelling detection: Compare the transcribed text with the assignment text to detect spelling errors and mispronunciations.

[1577] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1578] 4. Emotion Recognition: The server uses IBM Watson Tone Analyzer to recognize the user's emotional state from the voice data. Specifically, it analyzes the tone, speed, and intonation of the voice to detect the user's emotions (e.g., joy, sadness, anger, etc.).

[1579] 5. Feedback and Score Generation: The server generates specific feedback and an overall score based on the analysis results. For example, if the server detects a lack of intonation and tension, it will generate an overall score (e.g., 78 points) along with the feedback, "Your intonation is lacking. Try reading more relaxed."

[1580] 6. Providing feedback: The server sends the generated feedback and score to the device, which then displays the received feedback and score to the user, who can then improve their reading aloud.

[1581] Battle Mode

[1582] When a user selects Battle Mode, they can compete in reading aloud against their parents or friends. Each user reads aloud and the audio data is sent to the server. The server analyzes each audio data independently and generates feedback and a score for each. Based on these scores, the server determines the winner and sends the result to the device. The device displays the winner and their feedback to the user, enhancing the competitive spirit and motivation to learn.

[1583] Long-term progress tracking and feedback

[1584] The server records the user's analysis results over a long period of time. This allows it to compare past data with current data and provide feedback to the user on areas for improvement and progress. For example, it could provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1585] Specific examples

[1586] 1. Example 1: Reading Aloud Assessment

[1587] Prompt text: The user reads the text "The weather is nice today."

[1588] Feedback: "Your intonation is appropriate. However, you read too quickly, so we recommend you read a little more slowly." and an overall score of 85 points are displayed.

[1589] 2. Example 2: Emotion Recognition

[1590] Prompt text: "The user seems emotionally exhausted."

[1591] Feedback: "You may be tired that day, so take a break and relax before trying again."

[1592] In this way, the present invention is implemented as a system that comprehensively analyzes a user's reading ability and provides appropriate feedback.

[1593] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1594] System program processing flow

[1595] Step 1: Record your reading

[1596] User: Launches the dedicated application on a smartphone or tablet.

[1597] Specific action: Tap the application icon.

[1598] Terminal: The application starts and displays the reading assignment text to the user.

[1599] Specific operation: Transition to the text display screen.

[1600] Input: Application launch signal.

[1601] Output: Display of the assignment text.

[1602] Step 2: Start recording

[1603] User: Read the presented text aloud.

[1604] Specific behavior: Reads text aloud into the built-in microphone.

[1605] Device: Use the built-in microphone to record your reading aloud.

[1606] Specific operation: Tap the record button to start recording.

[1607] Input: Record button tap signal.

[1608] Output: Start recording.

[1609] Step 3: Sending audio data

[1610] Device: Once recording is complete, the audio data is sent to a server via the Internet.

[1611] Specific operation: Tap the stop recording button and automatically send the audio data.

[1612] Input: Recording stop button tap signal, recording data.

[1613] Output: Sending audio data to the server.

[1614] Step 4: Receiving and storing audio data

[1615] Server: Temporarily stores received audio data.

[1616] Specific operation: Save the audio file to the database.

[1617] Input: Audio data from the device.

[1618] Output: Save audio data.

[1619] Step 5: Transcription

[1620] Server: Transcribing the audio data using a speech recognition engine.

[1621] Specific operation: Converts speech to text using the Google Cloud Speech-to-Text API.

[1622] Input: Stored audio data.

[1623] Output: The transcribed text.

[1624] Step 6: Evaluate intonation

[1625] Server: Evaluates the intonation and inflection of the transcribed text.

[1626] Specific operation: Extract audio features.

[1627] Input: The transcribed text.

[1628] Output: Intonation evaluation results.

[1629] Step 7: Detect slip-ups

[1630] Server: Compares the transcribed text with the assignment text to detect spelling errors and pronunciation mistakes.

[1631] What it does: Uses a text matching algorithm.

[1632] Input: Transcribed text, assignment text.

[1633] Output: Slip-of-the-speech detection results.

[1634] Step 8: Calculate your reading speed

[1635] Server: Calculates reading speed by comparing text length with playback time.

[1636] Specific operation: Calculate the reading speed (number of characters / second) from the playback time.

[1637] Input: Transcribed text, duration.

[1638] Output: Reading speed calculation results.

[1639] Step 9: Emotion Recognition

[1640] Server: Analyzes the user's emotional state from the voice data using an emotion engine.

[1641] Specific operation: Uses IBM Watson Tone Analyzer.

[1642] Input: Audio data.

[1643] Output: Emotion recognition result.

[1644] Step 10: Generate feedback and scores

[1645] Server: Generates specific feedback and an overall score based on the analysis results.

[1646] Specific actions: Evaluation is based on intonation, mistakes, reading speed, and emotional state.

[1647] Input: Intonation evaluation results, slip-up results, reading speed results, emotion recognition results.

[1648] Output: Feedback and overall score.

[1649] Step 11: Submit your feedback and score

[1650] Server: Sends generated feedback and scores to devices.

[1651] Specific operation: Send data in File or JSON format.

[1652] Input: Feedback and overall score.

[1653] Output: Data sent to the terminal.

[1654] Step 12: View your feedback and scores

[1655] Terminal: Displays received feedback and scores to the user.

[1656] Specific operation: Proceed to the feedback result screen.

[1657] Input: Send data.

[1658] Output: Display of feedback and score.

[1659] (Application example 2)

[1660] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1661] While conventional reading practice systems can provide technical evaluations of reading accuracy, intonation, speed, etc., it is difficult to understand the user's emotional state and provide appropriate feedback. As a result, users may continue practicing while feeling nervous or anxious, which hinders effective improvement of reading skills.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1663] In this invention, the server includes means for recording the user's reading aloud on the terminal, means for transmitting the recorded voice data to the server, means for transcribing the received voice data, means for analyzing the transcribed text and evaluating intonation, detecting slip-ups, and calculating reading speed, means for generating feedback and a score based on the analysis results, means for displaying the feedback and score to the user on the terminal, means for recognizing the user's emotional state from the voice data, and means for generating feedback according to the emotion based on the emotion recognition results. This makes it possible to provide appropriate feedback that reflects the user's emotional state and promote improvement in reading aloud skills.

[1664] A "terminal" is an electronic device such as a smartphone, tablet, or personal computer used by a user who reads aloud.

[1665] The "means for recording" is a mechanism for recording the user's voice reading aloud as digital data using a microphone built into the terminal or an externally connected recording device.

[1666] "Audio data" is digital data representing recorded audio readings.

[1667] "Transmission means" refers to the technical mechanism by which the recorded audio data is sent to a server via the Internet or other communications network.

[1668] A "server" is a central processing system for receiving and analyzing audio data and generating feedback.

[1669] "Transcribing means" refers to the process of analyzing received audio data and converting it into corresponding text data.

[1670] The "analysis method" refers to a series of technical processes that use transcribed text data to evaluate intonation, detect slip-ups, calculate reading speed, and so on.

[1671] "Intonation evaluation" is the process of analyzing the intonation and vocal stress of a text read aloud and judging its quality.

[1672] "Slip-up detection" is the process of comparing the transcribed text with the exact text to identify typos and mispronunciations.

[1673] "Reading speed calculation" is the process of measuring reading speed using the total length of the text and the playback time of the audio data.

[1674] The "means for generating feedback and scores" is a mechanism that, based on the analysis results, creates a feedback message indicating an evaluation of the user's reading aloud and areas for improvement, as well as an overall evaluation score.

[1675] The "means for displaying" is a mechanism for displaying the generated feedback and score on the screen of the terminal in a form that can be visually recognized by the user.

[1676] "Emotional state" refers to the user's psychological and emotional state analyzed from voice data, including, for example, joy, sadness, tension, etc.

[1677] "Means for emotion recognition" refers to the technical process for identifying the user's emotional state from voice data.

[1678] "Feedback according to emotions" is a feedback message that provides appropriate advice or areas for improvement based on the results of emotion recognition of the user.

[1679] The present invention is a system that combines a system that provides feedback by recording and analyzing a user's reading aloud with an emotion engine that recognizes the user's emotions. The specific configuration and method for realizing this system are described in detail below.

[1680] 1. Record and send your reading

[1681] The user launches a dedicated application using a device such as a smartphone or tablet. This application displays the assignment text for the user to read aloud. The user reads the assignment text aloud and records the audio using the device's built-in microphone. This recorded audio data is sent to a server via the Internet.

[1682] 2. Analysis of audio data and feedback generation

[1683] The server temporarily stores the received voice data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The transcribed data is analyzed as follows:

[1684] Intonation assessment: Analyze the transcribed text to assess intonation and intonation.

[1685] Misspelling detection: Compares the transcribed text with the correct text to detect typos and mispronunciations.

[1686] Reading speed calculation: Calculates reading speed by comparing the length of the entire text with the playback time of the audio data.

[1687] Furthermore, the server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the tone, speed, and intonation of the voice to determine the user's emotional state (e.g., joy, sadness, nervousness, etc.). Based on these analysis results, the server generates specific feedback and a score. For example, if the user is nervous, the server provides feedback such as, "Your intonation is lacking. Try reading more relaxed."

[1688] 3. Providing and Viewing Feedback

[1689] The feedback and score generated by the server are sent to the terminal, which displays the received feedback and score to the user, allowing the user to improve their reading.

[1690] 4. Implementation of Battle Mode

[1691] The system includes a battle mode, which users can select to compete against other users (e.g., parents or friends) in reading aloud. Each user reads aloud individually and sends the audio data to the server. The server analyzes each audio data independently, generates feedback and scores for each, and determines the winner based on these scores.

[1692] 5. Long-term progress tracking and feedback

[1693] The server records the user's analysis results over a long period of time. This allows the server to evaluate the user's progress over time and provide feedback on improvements and progress over time. For example, the server might provide feedback such as, "Your reading speed has improved compared to before. Keep up the good work."

[1694] Examples of concrete examples and prompts

[1695] Example: After a child completes one minute of reading aloud practice, the recorded speech is automatically analyzed and feedback such as "Your speed is 2.5 characters per second. You lack intonation. Relax and try again" is displayed on the head-mounted display.

[1696] Example prompts to input to a generative AI model:

[1697] "Use a sentiment analysis engine to recognize emotions as users read and generate feedback on intonation, speed, and spelling errors."

[1698] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1699] Step 1:

[1700] The user launches a dedicated application on their device. The application displays a reading assignment text to the user. The input is the assignment text selection provided by the user, and the output is the assignment text displayed on the application screen.

[1701] Step 2:

[1702] The user reads the assignment text aloud and records the audio using the device's built-in microphone. The input is the user's voice, and the output is audio data captured by the device's recording function. This recorded data is temporarily saved on the device.

[1703] Step 3:

[1704] The device sends the recorded voice data to the server via the Internet. The input is the voice data stored on the device, and the output is the voice data received by the server. The voice data is stored in the server's storage.

[1705] Step 4:

[1706] The server uses the received audio data and transcribes it using a speech recognition engine (e.g., Google Speech Recognition API). The input is the audio data, and the output is the transcribed text data. The server passes this text data to the next analysis step.

[1707] Step 5:

[1708] The server analyzes the transcribed text data. Specifically, it evaluates intonation, detects slip-ups, and calculates reading speed. The input is the transcribed text, and the output is the results of these evaluations.

[1709] Intonation assessment: Analyze the intonation patterns of the text and evaluate their quality.

[1710] Misspelling detection: Detects typos and mispronunciations by comparing with the correct text.

[1711] Reading speed calculation: Reading speed is calculated using the total length of the text and the playback time of the audio data.

[1712] Step 6:

[1713] The server recognizes the user's emotions from the voice data. Using an emotion recognition engine, it analyzes the tone, rate, and intonation of the voice to identify the emotional state (e.g., joy, sadness, tension, etc.). The input is the voice data, and the output is the identification of the user's emotional state.

[1714] Step 7:

[1715] The server generates feedback and a score based on the analysis and emotion recognition results. The inputs are the evaluation results of intonation, speed, typo detection, and emotional state, and the output is a specific feedback message and an overall score for the user.

[1716] Step 8:

[1717] The server sends feedback and scores to the terminal. The input is the generated feedback and scores, and the output is the transmission of data to the user terminal.

[1718] Step 9:

[1719] The terminal displays the received feedback and score to the user. The input is the feedback and score sent from the server, and the output is the feedback and score displayed on the terminal screen. The user can use this information to improve their reading aloud.

[1720] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1721] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1722] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1723] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1724] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1725] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1726] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1727] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1728] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1729] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1730] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1731] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1732] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1733] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1734] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1735] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1736] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1737] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1738] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1739] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1740] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1741] The following is further disclosed regarding the above embodiment.

[1742] (Claim 1)

[1743] a means for the terminal to record the user's reading aloud;

[1744] A means for transmitting the recorded audio data by the terminal to a server;

[1745] A means for transcribing the audio data received by the server;

[1746] a means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed;

[1747] A means for the server to generate feedback and scores based on the analysis results;

[1748] A system in which the terminal includes means for displaying feedback and scores to the user.

[1749] (Claim 2)

[1750] The system of claim 1 includes a means for, when the server analyzes the reading data and generates scores, to independently analyze the reading data of multiple users in battle mode, generate feedback and scores for each user, and determine the winner of the battle based on these scores.

[1751] (Claim 3)

[1752] The system of claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and provide feedback on improvements and progress over time.

[1753] "Example 1"

[1754] (Claim 1)

[1755] a means for a user to initiate reading aloud;

[1756] a means for the terminal to record the user's reading aloud;

[1757] A means for transmitting the recorded audio data by the terminal to a server;

[1758] A means for transcribing the audio data received by the server;

[1759] means for the server to analyze the transcribed text to assess intonation, detect errors, and calculate reading speed;

[1760] A means for the server to generate feedback and scores based on the analysis results;

[1761] means for the terminal to display feedback and scores to the user;

[1762] A system including a means for selecting a battle mode.

[1763] (Claim 2)

[1764] 2. The system according to claim 1, further comprising means for the server to independently analyze the reading data of a plurality of users, generate feedback and scores for each of them, and determine the winner of the battle based on these scores.

[1765] (Claim 3)

[1766] The system of claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and provide feedback on improvements and progress over time.

[1767] "Application Example 1"

[1768] (Claim 1)

[1769] a means for the terminal to record the user's reading aloud;

[1770] A means for transmitting the recorded audio data by the terminal to a server;

[1771] A means for transcribing the audio data received by the server;

[1772] a means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed;

[1773] A means for the server to generate feedback and scores based on the analysis results;

[1774] means for the terminal to display feedback and scores to the user;

[1775] A method to have specific product information read aloud in a virtual store, analyze the level of comprehension and explanation ability, and provide feedback.

[1776] The system includes a means for displaying the results of a competition with other users for the product information that has been read aloud.

[1777] (Claim 2)

[1778] The system of claim 1 includes a means for, when the server analyzes the reading data and generates scores, to independently analyze the reading data of multiple users in battle mode, generate feedback and scores for each user, and determine the winner of the battle based on these scores.

[1779] (Claim 3)

[1780] The system of claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and provide feedback on improvements and progress over time.

[1781] "Example 2: Combining Emotion Engines"

[1782] (Claim 1)

[1783] a means for the terminal to record the user's reading aloud;

[1784] A means for transmitting the recorded audio data by the terminal to a server;

[1785] A means for transcribing the audio data received by the server;

[1786] a means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed;

[1787] A means for the server to generate feedback and scores based on the analysis results;

[1788] A means for the server to analyze the emotional state of the user from the voice data;

[1789] A system in which the terminal includes means for displaying feedback and scores to the user.

[1790] (Claim 2)

[1791] The system of claim 1 includes a means for, when the server analyzes the reading data and generates scores, to independently analyze the reading data of multiple users in battle mode, generate feedback and scores for each user, and determine the winner of the battle based on these scores.

[1792] (Claim 3)

[1793] The system of claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and provide feedback on improvements and progress over time.

[1794] "Application example 2 when combining emotion engines"

[1795] (Claim 1)

[1796] a means for the terminal to record the user's reading aloud;

[1797] A means for transmitting the recorded audio data by the terminal to a server;

[1798] A means for transcribing the audio data received by the server;

[1799] a means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed;

[1800] A means for the server to generate feedback and scores based on the analysis results;

[1801] means for the terminal to display feedback and scores to the user;

[1802] means for the server to recognize the emotional state of the user from the voice data;

[1803] A system including a means for generating emotion-dependent feedback based on emotion recognition results.

[1804] (Claim 2)

[1805] The system of claim 1 includes a means for, when the server analyzes the reading data and generates scores, to independently analyze the reading data of multiple users in battle mode, generate feedback and scores for each user, and determine the winner of the battle based on these scores.

[1806] (Claim 3)

[1807] The system of claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and provide feedback on improvements and progress over time. [Explanation of symbols]

[1808] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for the terminal to record the user's reading aloud; A means for transmitting the recorded audio data by the terminal to a server; A means for transcribing the audio data received by the server; a means for the server to analyze the transcribed text to assess intonation, detect slip-ups, and calculate reading speed; A means for the server to generate feedback and scores based on the analysis results; A system in which the terminal includes means for displaying feedback and scores to the user.

2. The system of claim 1, wherein when the server analyzes the reading data and generates scores, the system includes a means for independently analyzing the reading data of multiple users in battle mode, generating feedback and scores for each user, and determining the winner of the battle based on these scores.

3. 2. The system according to claim 1, further comprising means for the server to record the user's progress and trends over a long period of time based on the analysis results and to provide feedback on improvements and progress over time.

Images