system

Abstract

Provide a system. A speech recognition unit converts speech data into text data; a text-to-speech means for converting text data into voice data; image generation means for generating images related to the learning content; an English conversation learning means for evaluating a user's pronunciation; A learning record means for recording and managing learning data; an interface means for a mobile device; A learning support system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Previous learning support systems had the problem of not providing enough support for learners with visual impairments or who have difficulty reading and writing. In particular, they lacked the ability to convert audio data to text or text data to audio, to improve visual learning motivation, and to provide efficient feedback for pronunciation practice. As a result, learning efficiency declined and it became difficult to maintain learner motivation. [Means for solving the problem]

[0005] To address these challenges, the present invention provides a learning support system that includes a speech recognition unit that converts speech data into text data, a text-to-speech unit that converts text data into speech data, an image generation unit that generates images related to learning content, an English conversation learning unit that evaluates the user's pronunciation, a learning record unit that records and manages learning data, and an interface unit compatible with mobile devices. The speech recognition unit uses a speech recognition API, and the text-to-speech unit uses a text-to-speech API, achieving highly accurate speech recognition and reading, aiding learners' understanding. The image generation unit also incorporates visual elements to enhance learning motivation. This allows all learners, including those with visual impairments and those with difficulty reading and writing, to learn independently and effectively.

[0006] A "voice recognition means" is a device or system that converts voice data into text data.

[0007] A "text-to-speech means" is a device or system that converts text data into audio data and reads it aloud.

[0008] "Image generation means" refers to a device or system that generates images related to the study content.

[0009] An "English conversation learning tool" is a device or system that evaluates a user's pronunciation and provides feedback.

[0010] The "learning recording means" is a device or system that records and manages learning data.

[0011] An "interface means" is a device or system that allows a user to access the system using a mobile device. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The present invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal. An example implementation of this system is described below.

[0034] Voice recognition function

[0035] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[0036] Text-to-speech function

[0037] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[0038] Image generation function

[0039] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[0040] English conversation learning function

[0041] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0042] Learning record function

[0043] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[0044] Smartphone compatible

[0045] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[0046] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[0047] The processing flow will be explained below.

[0048] Voice recognition function

[0049] Step 1:

[0050] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[0051] Step 2:

[0052] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[0053] Step 3:

[0054] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[0055] Step 4:

[0056] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[0057] Step 5:

[0058] The server records the generated text data in a database and returns the data to the terminal.

[0059] Step 6:

[0060] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[0061] Text-to-speech function

[0062] Step 1:

[0063] The user types the text they want to hear into the device. For example, type "Newton's third law."

[0064] Step 2:

[0065] The terminal sends the entered text data to the server using a POST request.

[0066] Step 3:

[0067] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[0068] Step 4:

[0069] The server transmits the generated voice data to the terminal.

[0070] Step 5:

[0071] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[0072] Image generation function

[0073] Step 1:

[0074] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[0075] Step 2:

[0076] The terminal sends the entered keyword to the server via a POST request.

[0077] Step 3:

[0078] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[0079] Step 4:

[0080] The server transmits the generated image data to the terminal.

[0081] Step 5:

[0082] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[0083] English conversation learning function

[0084] Step 1:

[0085] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[0086] Step 2:

[0087] The terminal sends the entered phrase to the server via a POST request.

[0088] Step 3:

[0089] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[0090] Step 4:

[0091] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[0092] Step 5:

[0093] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[0094] Step 6:

[0095] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[0096] Step 7:

[0097] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[0098] Learning record function

[0099] Step 1:

[0100] When the user starts studying, the terminal starts recording the study time and content.

[0101] Step 2:

[0102] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[0103] Step 3:

[0104] The server stores the received data in a database.

[0105] Step 4:

[0106] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[0107] Step 5:

[0108] The server searches the learning history from the database and sends it to the terminal.

[0109] Step 6:

[0110] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[0111] Smartphone compatible

[0112] Step 1:

[0113] The user launches the application on the smartphone.

[0114] Step 2:

[0115] The device displays the main menu and provides access to each function.

[0116] Step 3:

[0117] The user selects any function, such as English conversation learning or image generation, and begins learning.

[0118] Step 4:

[0119] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[0120] Step 5:

[0121] The terminal displays the received results to the user and provides learning according to the instructions.

[0122] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In conventional learning support systems, each function exists independently, making it difficult to provide comprehensive learning support. Furthermore, functions such as voice recognition, text-to-speech, image generation, English conversation learning, and learning records are provided separately, making it difficult for users to centrally manage their learning progress and history. Furthermore, use on mobile devices is limited, meaning that they can only provide a learning environment that is restricted by location and time.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes a speech recognition unit that converts speech data into text data, a text-to-speech unit that converts text data into speech data, an image generation unit that generates images related to the learning content, an English conversation learning unit that evaluates the user's pronunciation, a learning record unit that records and manages learning data, an interface unit compatible with mobile devices, a transmission unit that captures speech data and transmits it to the server, a conversion unit that converts speech data into text data using a speech recognition API in the server, a display unit that displays the generated text data to the user, an image generation and transmission unit that transmits keywords entered by the user to the server and generates images using an image generation API, and a display unit that displays the generated images to the user. This provides integrated functions such as speech recognition, text-to-speech, image generation, English conversation learning, and learning record, allowing users to continue their learning anytime, anywhere.

[0128] A "voice recognition means" is a device or software for receiving voice data and converting it into text data.

[0129] "Text-to-speech means" refers to a device or software that converts input text data into audio data and plays back the audio.

[0130] The "image generating means" is a device or software for generating related images based on keywords entered by a user.

[0131] An "English conversation learning tool" is a device or software that evaluates a user's pronunciation and supports English conversation practice.

[0132] The "learning recording means" is a device or software for recording and managing the user's learning data.

[0133] "Interface means" refers to a means by which a user accesses the system through a mobile device and uses various functions.

[0134] "Transmission means" refers to a device or software for transmitting voice data or text data to a server.

[0135] The "conversion means" is a device or software for converting voice data into text data in the server.

[0136] The "display means" is a device or software for displaying the generated text data or image data to the user.

[0137] The "image generation and transmission means" is a device or software that transmits the keywords entered by the user to the server and generates an image using the image generation API.

[0138] This invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal.

[0139] Voice recognition function

[0140] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server converts the voice data into text data using a speech recognition API (e.g., Google® Cloud Speech-to-Text API). The converted text data is returned from the server to the device, and the device displays the text to the user. For example, if the user says, "Tell me about Newton's laws," the speech recognition API generates and displays the text, "Tell me about Newton's laws."

[0141] Text-to-speech function

[0142] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API (for example, Amazon Polly or Google Text-to-Speech API) to convert the text data into speech data, and the generated speech data is sent from the server to the device and played. For example, if a user enters "Newton's third law," a voice of "Law of action and reaction" is generated and played using the TTS API.

[0143] Image generation function

[0144] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server generates related images using an image generation API (e.g., DALL-E or MidJourney) and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, a diagram of the related forces is generated and displayed.

[0145] English conversation learning function

[0146] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0147] Learning record function

[0148] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[0149] Smartphone compatible

[0150] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[0151] Below is an example of a prompt sentence to input to the generative AI model.

[0152] "Tell me about Newton's laws."

[0153] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Voice recognition function

[0156] Processing Step 1:

[0157] The user uses a microphone to input speech for learning.

[0158] Input: User's voice

[0159] Specific action: The user speaks into the smartphone microphone, "Tell me about Newton's laws."

[0160] Output: Captured audio data

[0161] Processing step 2:

[0162] The device captures the audio data and sends it to the server.

[0163] Input: Captured audio data

[0164] Specific operation: The device uploads the recorded audio data to a server via the Internet.

[0165] Output: Audio data sent to the server

[0166] Processing step 3:

[0167] The server uses a speech recognition API to convert the voice data into text data.

[0168] Input: Audio data sent to the server

[0169] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0170] Specific operation: The voice recognition API analyzes the voice data and converts it into text data.

[0171] Output: Converted text data

[0172] Processing step 4:

[0173] The server transmits the converted text data to the terminal.

[0174] Input: Converted text data

[0175] Specific operation: The server returns text data to the terminal.

[0176] Output: Text data sent to the terminal

[0177] Processing Step 5:

[0178] The terminal displays the received text data to the user.

[0179] Input: Text data sent to the terminal

[0180] Specific action: Display the text "Tell me about Newton's laws" on the device screen.

[0181] Output: Displayed text data

[0182] Text-to-speech function

[0183] Processing Step 1:

[0184] The user enters the text they want to read into the device.

[0185] Input: Text entered by the user

[0186] Specific action: The user types "Newton's third law" on the smartphone keyboard.

[0187] Output: The input text data

[0188] Processing step 2:

[0189] The terminal transmits the input text data to the server.

[0190] Input: Entered text data

[0191] Specific operation: The device uploads text data to the server.

[0192] Output: Text data sent to the server

[0193] Processing step 3:

[0194] The server uses a text-to-speech API to convert the text data into audio data.

[0195] Input: Text data sent to the server

[0196] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[0197] Specific operation: Send an API request and generate audio data for "Newton's Third Law."

[0198] Output: Generated audio data

[0199] Processing step 4:

[0200] The server transmits the generated voice data to the terminal.

[0201] Input: Generated audio data

[0202] Specific operation: The server sends voice data to the terminal.

[0203] Output: Audio data sent to the device

[0204] Processing Step 5:

[0205] The terminal plays back the received audio data to the user.

[0206] Input: Audio data sent to the device

[0207] Specific action: The device plays an audio file and reads out "Newton's Third Law."

[0208] Output: Played audio data

[0209] Image generation function

[0210] Processing Step 1:

[0211] The user inputs keywords related to the content being studied into the terminal.

[0212] Input: The keyword entered by the user

[0213] Specific action: The user enters "action and reaction" into the smartphone.

[0214] Output: Entered keyword data

[0215] Processing step 2:

[0216] The terminal transmits the keyword data to the server.

[0217] Input: Entered keyword data

[0218] Specific operation: The terminal uploads keyword data to the server.

[0219] Output: Keyword data sent to the server

[0220] Processing step 3:

[0221] The server generates the relevant images using an image generation API.

[0222] Input: Keyword data sent to the server

[0223] Data processing: Use image generation API (e.g., DALL-E or MidJourney)

[0224] Specific operation: Send an API request and generate an image related to "action and reaction."

[0225] Output: Generated image data

[0226] Processing step 4:

[0227] The server transmits the generated image data to the terminal.

[0228] Input: Generated image data

[0229] Specific operation: The server sends an image file to the terminal.

[0230] Output: Image data sent to the device

[0231] Processing Step 5:

[0232] The terminal displays the received image data to the user.

[0233] Input: Image data sent to the device

[0234] Specific operation: Display the generated image on the device screen.

[0235] Output: Displayed image data

[0236] English conversation learning function

[0237] Processing Step 1:

[0238] The user inputs the English phrase they want to practice into the device.

[0239] Input: English phrase entered by the user

[0240] Specific action: The user types "How are you?" on the smartphone keyboard.

[0241] Output: Input phrase data

[0242] Processing step 2:

[0243] The device sends the entered phrase to the server.

[0244] Input: Entered phrase data

[0245] Specific operation: The device uploads phrase data to the server.

[0246] Output: Phrase data sent to the server

[0247] Processing step 3:

[0248] The server uses a text-to-speech API to generate audio data with the correct pronunciation.

[0249] Input: Phrase data sent to the server

[0250] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[0251] Specific behavior: Sends an API request to generate the correct pronunciation of "How are you?"

[0252] Output: Generated audio data

[0253] Processing step 4:

[0254] The server transmits the generated voice data to the terminal.

[0255] Input: Generated audio data

[0256] Specific operation: The server sends voice data to the terminal.

[0257] Output: Audio data sent to the device

[0258] Processing Step 5:

[0259] The terminal plays back the received voice data, and the user pronounces and records the voice accordingly.

[0260] Input: Audio data sent to the device

[0261] Specific operation: The correct pronunciation generated on the device is played back, and the user speaks into the microphone and the audio is recorded.

[0262] Output: Recorded audio data

[0263] Processing Step 6:

[0264] The device sends the recorded pronunciation data to the server.

[0265] Input: Recorded audio data

[0266] Specific operation: The device sends the recorded data to the server.

[0267] Output: Recording data sent to the server

[0268] Processing Step 7:

[0269] The server uses a speech recognition API to evaluate the user's pronunciation and generate a result.

[0270] Input: Recording data sent to the server

[0271] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0272] Specific operations: Send an API request, convert the user's pronunciation into text, and perform an evaluation.

[0273] Output: Evaluation result data

[0274] Processing Step 8:

[0275] The server transmits the evaluation results to the terminal, which displays them to the user.

[0276] Input: Evaluation result data

[0277] Specific operation: The server sends the evaluation results to the terminal, and the terminal displays the results on the screen.

[0278] Output: Displayed evaluation results

[0279] Learning record function

[0280] Processing Step 1:

[0281] When a user starts studying, the terminal records the study time and content.

[0282] Input: Study start information (time, content)

[0283] Specific operation: The device stores the user's learning start time and learning content in a log.

[0284] Output: Recorded training data

[0285] Processing step 2:

[0286] The terminal periodically transmits the recorded data to the server.

[0287] Input: Recorded training data

[0288] Specific operation: The device periodically uploads learning data to the server.

[0289] Output: Training data sent to the server

[0290] Processing step 3:

[0291] The server stores the received learning data in a database.

[0292] Input: Training data sent to the server

[0293] Specific operation: The server inserts the received data into the corresponding table in the database.

[0294] Output: Training data stored in a database

[0295] Processing step 4:

[0296] When a user wants to check his / her past learning history, the terminal sends a request to the server, the server searches for learning history data and sends it to the terminal, and the terminal displays the data to the user.

[0297] Input: User request (e.g., learning history for a specific date)

[0298] Specific operation: The device sends a request to the server to confirm "what was learned on October 1st," and the server searches the database for the relevant data and sends it.

[0299] Output: Displayed learning history data

[0300] Mobile device compatible features

[0301] Processing Step 1:

[0302] The user launches the app on their smartphone.

[0303] Input: App launch operation

[0304] Specific action: The user taps the app icon on their smartphone to launch the app.

[0305] Output: App launch status

[0306] Processing step 2:

[0307] The device displays the main menu.

[0308] Input: App running state

[0309] Specific operation: Links to various functions will be displayed on the app's home screen.

[0310] Output: Main menu displayed

[0311] Processing step 3:

[0312] The user selects the required function, and communication is made from the device to the server, providing the necessary data and feedback in real time.

[0313] Input: User selected function

[0314] Specific operation: Select the English conversation practice mode during your commute, and the device will communicate with the server to provide pronunciation practice.

[0315] Output: Real-time data and feedback

[0316] The above is the specific processing flow for each function.

[0317] (Application example 1)

[0318] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] Conventional security systems can make it difficult for users to quickly identify abnormalities and respond appropriately. In particular, delays in detecting suspicious individuals and issuing security alerts can delay effective responses. Furthermore, because these systems are not interactive, users often cannot directly communicate with the system to check the situation or issue instructions.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0321] In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, a voice evaluation means for evaluating the user's pronunciation, a recording means for recording and managing the learning data, an interface means compatible with the mobile device, a notification means for notifying the user of security events by voice, an analysis means for performing image analysis, and a log management means for recording security events in a log file. This allows the user to interact with the security system through voice commands, check surveillance footage in real time, and respond quickly.

[0322] The "voice recognition means" is a means for converting voice data into text data.

[0323] The "text-to-speech means" is a means for converting text data into voice data.

[0324] "Image generation means" refers to means for generating images related to learning content or other content.

[0325] The "voice evaluation means" is a means for evaluating the user's pronunciation.

[0326] "Recording means" refers to a means for recording and managing learning data and other data.

[0327] "Interface means" refers to means for providing a corresponding interface to a mobile device or other device.

[0328] The "notification means" is a means for notifying the user of a security event by voice.

[0329] "Analysis means" refers to means for analyzing image data and other data.

[0330] A "log management means" is a means for recording and managing security events in a log file.

[0331] The present invention relates to a security support system, which is an integrated system that provides multiple functions such as voice recognition, text-to-speech, image generation, voice evaluation, recording and management, mobile device compatibility, notification, and analysis. Specific embodiments of this system are described below.

[0332] Voice recognition means

[0333] When a user issues a voice command, the device captures the voice data and sends it to the server. The server uses a voice recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice data into text data, and returns the text data to the device, which then displays it. For example, if a user says, "Detect a suspicious person," the voice recognition API generates the text "Detect a suspicious person" and displays it on the device.

[0334] Text-to-speech means

[0335] When the server detects an important security notification, it converts the text data into audio data. The server then uses a text-to-speech API (e.g., Google Cloud Text-to-Speech) to send the generated audio data to the device and notify the user by voice. For example, the generated text might say, "A suspicious person has been detected. Please be careful," and be played back as audio.

[0336] Image Generation Means

[0337] When a user wants to check a specific image, they enter a relevant keyword into their device and send it to the server. The server uses an image generation API to generate an image related to the keyword and sends it to the device. For example, if "suspicious person's face" is entered as a keyword, an image that identifies the suspicious person's face will be generated and displayed on the device.

[0338] Audio evaluation tools

[0339] When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, a user can record and send "How are you?" and the pronunciation will be evaluated to see if it is appropriate.

[0340] Recording means

[0341] Every time a user uses the system, data on related security events is sent to the server and recorded. This allows past security history to be checked. For example, the date, time, and location of any suspicious activity are recorded.

[0342] Interface Means

[0343] The system is compatible with mobile devices, allowing users to access the system anytime, anywhere. When users launch the app on their smartphone, an interface is displayed that allows them to access security functions. For example, they can check security camera footage while commuting.

[0344] Notification means

[0345] When a security event occurs, the system notifies the user with an audio alert. The server converts text into audio and sends it to the device where it is played. For example, a voice alert saying "A suspicious person has been detected" is played.

[0346] Analysis means

[0347] The video captured by the device's camera is sent to the server, and the server uses image analysis software (e.g., OpenCV) to detect suspicious individuals. If a suspicious individual is detected, the information is notified to the user. For example, the face of a suspicious individual can be automatically analyzed based on the camera image.

[0348] Log management methods

[0349] Security event logs are managed on the server and recorded in log files. When a user wants to check past security events, they send a request to the server and the corresponding log data is displayed on the terminal. For example, it can be used to "check details of suspicious individuals detected on October 1st."

[0350] Examples of concrete examples and prompts

[0351] As a specific example of use, when a user says to their smartphone, "Detect suspicious person," the app analyzes the camera image, and if a suspicious person is detected, it notifies them with a voice message saying, "Suspicious person detected," and records the event in a log.

[0352] Prompt Sentence Examples

[0353] User: Suspicious person detected

[0354] App: Analyzing camera images...

[0355] App: Suspicious person detected. Please be careful.

[0356] The above is a specific description of the embodiment of the present invention, which enables users to take security measures quickly and effectively.

[0357] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0358] Step 1:

[0359] The user speaks a voice command, which is captured as voice data by the smartphone's microphone. The input is the voice command, and the output is the captured voice data.

[0360] Step 2:

[0361] The device sends the captured voice data to the server for conversion to text data using a speech recognition API. The server then converts the voice data to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the voice data, and the output is the converted text data.

[0362] Step 3:

[0363] The server analyzes the text data and determines a specific action. For example, if the text contains the phrase "detect a suspicious person," the server will instruct the start of image analysis. The input is the text data, and the output is the analysis result and the decision on the next action.

[0364] Step 4:

[0365] The device's camera captures real-time video and sends the image data to the server. The input is the image data captured by the camera, and the output is the image data sent to the server.

[0366] Step 5:

[0367] The server uses image analysis software (e.g., OpenCV) to analyze the captured image data and detect suspicious individuals. The input is the image data, and the output is the analysis result indicating whether or not there is a suspicious individual.

[0368] Step 6:

[0369] When the server detects a suspicious person, it sends the information to the terminal as text and simultaneously converts it into audio data using a text-to-speech API. The input is text data and the output is audio data.

[0370] Step 7:

[0371] The device receives the voice data and issues a voice notification to the user from the speaker. For example, a voice alert saying "A suspicious person has been detected. Please be careful" is played. The input is voice data, and the output is a voice notification.

[0372] Step 8:

[0373] Details of security events are recorded on the server and saved in a log file through a log management tool. The event date and time, detected details, etc. are recorded in the log so that users can check them later. The input is security event information, and the output is a log file.

[0374] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0375] The present invention is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal. An example implementation is described below.

[0376] Voice recognition function

[0377] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[0378] Text-to-speech function

[0379] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[0380] Image generation function

[0381] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[0382] English conversation learning function

[0383] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0384] Learning record function

[0385] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[0386] Emotion recognition function using emotion engine

[0387] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's research and learning situation. For example, if a user mutters, "This is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as, "Try taking a break to resolve this."

[0388] Smartphone compatible

[0389] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[0390] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[0391] The processing flow will be explained below.

[0392] Voice recognition function

[0393] Step 1:

[0394] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[0395] Step 2:

[0396] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[0397] Step 3:

[0398] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[0399] Step 4:

[0400] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[0401] Step 5:

[0402] The server records the generated text data in a database and returns the data to the terminal.

[0403] Step 6:

[0404] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[0405] Text-to-speech function

[0406] Step 1:

[0407] The user types the text they want to hear into the device. For example, type "Newton's third law."

[0408] Step 2:

[0409] The terminal sends the entered text data to the server using a POST request.

[0410] Step 3:

[0411] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[0412] Step 4:

[0413] The server transmits the generated voice data to the terminal.

[0414] Step 5:

[0415] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[0416] Image generation function

[0417] Step 1:

[0418] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[0419] Step 2:

[0420] The terminal sends the entered keyword to the server via a POST request.

[0421] Step 3:

[0422] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[0423] Step 4:

[0424] The server transmits the generated image data to the terminal.

[0425] Step 5:

[0426] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[0427] English conversation learning function

[0428] Step 1:

[0429] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[0430] Step 2:

[0431] The terminal sends the entered phrase to the server via a POST request.

[0432] Step 3:

[0433] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[0434] Step 4:

[0435] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[0436] Step 5:

[0437] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[0438] Step 6:

[0439] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[0440] Step 7:

[0441] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[0442] Learning record function

[0443] Step 1:

[0444] When the user starts studying, the terminal starts recording the study time and content.

[0445] Step 2:

[0446] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[0447] Step 3:

[0448] The server stores the received data in a database.

[0449] Step 4:

[0450] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[0451] Step 5:

[0452] The server searches the learning history from the database and sends it to the terminal.

[0453] Step 6:

[0454] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[0455] Emotion recognition function using emotion engine

[0456] Step 1:

[0457] The user expresses their emotions through voice and facial expressions. For example, they may say, "It's difficult."

[0458] Step 2:

[0459] The device captures and temporarily saves the user's voice and facial expression data. For example, it saves the facial expression data as .jpg and the voice emotion file as .wav.

[0460] Step 3:

[0461] The device sends this data to the server. For example, send facial expression data (.jpg) or voice emotion file (.wav) to the server using a POST request.

[0462] Step 4:

[0463] The server sends the received data to the emotion engine, which analyzes the user's emotions. For example, it recognizes "frustration."

[0464] Step 5:

[0465] The server sends the emotion analysis results to the device and provides feedback to the user, such as "Let's take a break to solve this."

[0466] Smartphone compatible

[0467] Step 1:

[0468] The user launches the application on the smartphone.

[0469] Step 2:

[0470] The device displays the main menu and provides access to each function.

[0471] Step 3:

[0472] The user selects any function, such as English conversation learning or image generation, and begins learning.

[0473] Step 4:

[0474] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[0475] Step 5:

[0476] The terminal displays the received results to the user and provides learning according to the instructions.

[0477] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[0478] Example 2

[0479] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0480] In today's learning environment, learners are required to efficiently acquire knowledge using a variety of media and tools. However, most learning systems are specialized in individual functions, and few provide comprehensive support across a wide range of areas, such as speech recognition, text-to-speech, image generation, English conversation learning, and emotion recognition. Furthermore, functions such as learning support on mobile devices and storing learning records are not consistently integrated. This makes it difficult for learners to obtain a learning experience that meets their diverse needs.

[0481] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an emotion recognition means for analyzing the user's emotions, and an interface means compatible with mobile devices. This makes it possible to provide comprehensive support for a variety of learning needs.

[0482] "Speech recognition means" refers to a device or software that has the function of analyzing voice data and converting it into text data.

[0483] A "text-to-speech means" is a device or software that has the function of analyzing text data and converting it into audio data.

[0484] "Image generation means" refers to a device or software that has the function of automatically generating related images based on keywords or content entered by a user.

[0485] An "English conversation learning tool" is a device or software that has the function of providing audio with correct pronunciation for English phrases entered by a user, evaluating the user's pronunciation, and providing feedback.

[0486] A "learning recording means" refers to a device or software that has the function of recording and managing a user's learning activities and learning data.

[0487] "Emotion recognition means" refers to devices or software that have the function of analyzing and evaluating emotions from the user's voice and facial expressions.

[0488] "Mobile device compatible interface means" refers to user interfaces and related communication means designed for use on mobile devices such as smartphones and tablets.

[0489] This is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal.

[0490] Voice recognition function

[0491] When a user provides voice input for training, the device captures this voice data and sends it to the server. The server then uses a speech recognition API to convert the voice data into text data. This can be done using services such as the Google Cloud Speech-to-Text API or Amazon Transcribe. The converted text data is then sent back from the server to the device, which then displays the text to the user.

[0492] Specific examples

[0493] When a user says, "Tell me about Newton's Law," the speech recognition API generates the text "Tell me about Newton's Law," which is displayed on the device.

[0494] Prompt Sentence Examples

[0495] "Explain Newton's First Law."

[0496] Text-to-speech function

[0497] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server then uses a text-to-speech API to convert the text data into audio data. Specifically, Google Cloud Text-to-Speech API or Amazon Polly can be used. The generated audio data is sent from the server to the device and played back.

[0498] Specific examples

[0499] When a user types "Newton's third law," a text-to-speech API is used to generate and play an audio version of "The Law of Action and Reaction."

[0500] Prompt Sentence Examples

[0501] Please read the following text aloud: "The law of action and reaction states that for every force there is an equal and opposite force acting in the opposite direction."

[0502] Image generation function

[0503] When a user enters keywords related to the content they are learning, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. Specifically, DALL-E or Stable Diffusion can be used. The device then displays the generated images to the user.

[0504] Specific examples

[0505] When the user enters the keyword "action and reaction," a diagram of the relevant forces is generated and displayed.

[0506] Prompt Sentence Examples

[0507] "Create an image that illustrates action and reaction forces."

[0508] English conversation learning function

[0509] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. Google Cloud Text-to-Speech API or Amazon Polly can be used. This audio data is sent to the device and played back. When the user records their own pronunciation and sends the data to the server, the server uses a speech recognition API to evaluate the pronunciation and sends the results to the device.

[0510] Specific examples

[0511] When a user wishes to practice "How are you?", a correct pronunciation audio is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0512] Prompt Sentence Examples

[0513] "Generate audio to learn the correct pronunciation of 'How are you?'"

[0514] Learning record function

[0515] When a user starts studying, the device records the study time and content. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user.

[0516] Specific examples

[0517] When a user checks "What did I learn on October 1st?", the corresponding content is displayed.

[0518] Prompt Sentence Examples

[0519] "Please show me what you learned on October 1st."

[0520] Emotion recognition function using emotion engine

[0521] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. Specifically, it can use Microsoft® Azure® Emotion API or Google Cloud Vision API. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's learning status.

[0522] Specific examples

[0523] If a user says "this is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as "Let's take a break to solve this."

[0524] Prompt Sentence Examples

[0525] "Please analyze the user's emotions from this voice data."

[0526] Smartphone compatible

[0527] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time.

[0528] Specific examples

[0529] The user can select the English conversation practice mode and practice pronunciation while commuting.

[0530] Prompt Sentence Examples

[0531] "Select English conversation practice mode and start practicing your pronunciation during your commute."

[0532] This allows the present invention to meet the needs of various learners and provide efficient and comprehensive learning support. Each function is realized through the cooperation of the server and terminal, and users can access it from anywhere. This system provides a variety of means to improve the learning experience in an integrated manner.

[0533] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0534] Voice recognition function

[0535] Step 1:

[0536] The user performs voice input for learning.

[0537] Specific Action: The user speaks into the microphone, "Tell me about Newton's Laws."

[0538] Step 2:

[0539] The device captures the audio data.

[0540] What it does: The device's microphone records the user's voice in real time.

[0541] Input: User's voice

[0542] Output: Recorded audio data

[0543] Step 3:

[0544] The device sends the captured audio data to the server.

[0545] Specific operation: The device encrypts the recorded data and sends it to the specified endpoint on the server.

[0546] Input: Audio data

[0547] Output: Audio data sent to the server

[0548] Step 4:

[0549] The server uses a speech recognition API to convert the voice data into text data.

[0550] Specific operation: The server passes the audio data to the Google Cloud Speech-to-Text API and retrieves the text data.

[0551] Input: Audio data sent to the server

[0552] Output: Converted text data

[0553] Step 5:

[0554] The server returns the converted text data to the terminal.

[0555] Specific operation: The server encrypts the converted text data and sends it to the device.

[0556] Input: Converted text data

[0557] Output: Text data sent to the terminal

[0558] Step 6:

[0559] The terminal displays the text data to the user.

[0560] What it does: The device displays the received text on the screen, informing the user, "Tell me about Newton's laws."

[0561] Input: Text data sent to the terminal

[0562] Output: Text displayed on the screen

[0563] Text-to-speech function

[0564] Step 1:

[0565] The user enters the text they want to read into the device or selects existing text.

[0566] Specific action: The user types "Newton's third law" into the terminal.

[0567] Step 2:

[0568] The terminal transmits the text data to the server.

[0569] Specific operation: The device encrypts the text data entered and sends it to the server.

[0570] Input: Text data entered by the user into the terminal

[0571] Output: Text data sent to the server

[0572] Step 3:

[0573] The server uses a text-to-speech API to convert the text data into audio data.

[0574] Specific operation: The server sends text data to the Google Cloud Text-to-Speech API, which generates audio data.

[0575] Input: Text data sent to the server

[0576] Output: Generated audio data

[0577] Step 4:

[0578] The server transmits the generated voice data to the terminal.

[0579] Specific operation: The server encrypts the generated audio data and sends it to the device.

[0580] Input: Generated audio data

[0581] Output: Audio data sent to the device

[0582] Step 5:

[0583] The device plays the audio data.

[0584] Specific operation: The device plays the received audio data and reads out the "Law of Action and Reaction" to the user.

[0585] Input: Audio data sent to the device

[0586] Output: Played audio

[0587] Image generation function

[0588] Step 1:

[0589] The user inputs keywords related to the content being studied into the terminal.

[0590] Specific operation: The user enters the keywords "action and reaction" into the terminal.

[0591] Step 2:

[0592] The device sends the keyword to the server.

[0593] Specific operation: The device encrypts the keyword and sends it to the server.

[0594] Input: Keywords entered by the user into the device

[0595] Output: Keywords sent to the server

[0596] Step 3:

[0597] The server generates the relevant images using an image generation API.

[0598] What it does: The server passes keywords to the DALL-E API and generates related images.

[0599] Input: The keyword sent to the server

[0600] Output: Generated image data

[0601] Step 4:

[0602] The server transmits the generated image to the terminal.

[0603] Specific operation: The server encrypts the generated image data and sends it to the terminal.

[0604] Input: Generated image data

[0605] Output: Image data sent to the device

[0606] Step 5:

[0607] The terminal displays the image to the user.

[0608] Specific operation: The device displays the image it receives on the screen, showing the user an "image illustrating the law of action and reaction."

[0609] Input: Image data sent to the device

[0610] Output: Image displayed on the screen

[0611] English conversation learning function

[0612] Step 1:

[0613] The user inputs the English phrase they want to practice into the device.

[0614] What happens: The user types "How are you?" into the terminal.

[0615] Step 2:

[0616] The device sends the entered phrase to the server.

[0617] Specific operation: The device encrypts the entered phrase and sends it to the server.

[0618] Input: The phrase the user types into the terminal

[0619] Output: Phrase sent to server

[0620] Step 3:

[0621] The server uses the TTS API to generate audio data with the correct pronunciation.

[0622] What happens: The server passes the phrase to the Google Cloud Text-to-Speech API, which generates audio data with the correct pronunciation.

[0623] Input: Phrase sent to server

[0624] output: Generated audio data with correct pronunciation

[0625] Step 4:

[0626] The server transmits the generated voice data to the terminal.

[0627] Specific operation: The server encrypts the generated audio data and sends it to the device.

[0628] Input: Generated audio data

[0629] Output: Audio data sent to the device

[0630] Step 5:

[0631] The device plays the audio data.

[0632] Specific operation: The device plays back the received audio data and provides the user with the correct pronunciation.

[0633] Input: Audio data sent to the device

[0634] Output: Played audio

[0635] Step 6:

[0636] The user records their own pronunciation and sends it from the terminal to the server.

[0637] Specific operation: The user records their own pronunciation and sends it to the server via their device.

[0638] Input: User recording data

[0639] Output: Recording data sent to the server

[0640] Step 7:

[0641] The server evaluates the pronunciation using a speech recognition API.

[0642] What happens: The server passes the recording to the Google Cloud Speech-to-Text API and evaluates the pronunciation.

[0643] Input: Recording data sent to the server

[0644] Output: Evaluation result

[0645] Step 8:

[0646] The server transmits the evaluation results to the terminal.

[0647] Specific operation: The server encrypts the evaluation results and sends them to the device.

[0648] Input: Evaluation result

[0649] Output: Evaluation results sent to the device

[0650] Step 9:

[0651] The terminal displays the evaluation results to the user.

[0652] Specific behavior: The device displays the evaluation results it receives on the screen and informs the user that "your pronunciation was accurate" or "your pronunciation needs improvement."

[0653] Input: Evaluation result sent to the terminal

[0654] Output: Evaluation results displayed on the screen

[0655] Learning record function

[0656] Step 1:

[0657] The user begins learning.

[0658] What happens: A user starts a new learning session in the app.

[0659] Step 2:

[0660] The device records the study time and content.

[0661] Specific operation: The device records the start time of learning and saves the learning content as it progresses.

[0662] Input: Start learning information and learning content

[0663] Output: Training data recorded on the device

[0664] Step 3:

[0665] The device periodically transmits learning data to the server.

[0666] Specific operation: The device encrypts and sends learning data to the server once every 15 minutes.

[0667] Input: Training data recorded on the device

[0668] output: Training data sent to the server

[0669] Step 4:

[0670] The server stores the learning data in a database.

[0671] Specific operation: The server stores the received learning data in a database.

[0672] Input: Training data sent to the server

[0673] output: Training data stored in the database

[0674] Step 5:

[0675] When a user wants to check his / her past learning history, he / she sends a request from the terminal to the server.

[0676] Specific operation: The user sends a request to the server from the device to check the learning history for a specified date in the app.

[0677] Input: Request (e.g., Confirm learning content for October 1st)

[0678] Output: Request sent to the server

[0679] Step 6:

[0680] The server searches the database for learning history data and sends it to the terminal.

[0681] Specific operation: The server searches the database for the learning history for the specified date, encrypts it, and sends it to the device.

[0682] Input: Request

[0683] output: Learning history data sent to the device

[0684] Step 7:

[0685] The terminal displays the learning history data to the user.

[0686] Specific operation: The device displays the learning history data it has received on the screen and informs the user, "The content learned on October 1st was as follows: The basics of Newton's laws."

[0687] Input: Learning history data sent to the device

[0688] Output: Learning history data displayed on the screen

[0689] Emotion recognition function using emotion engine

[0690] Step 1:

[0691] The user expresses their feelings through voice and facial expressions while learning.

[0692] Specific action: The user tweets "This is difficult" while studying.

[0693] Step 2:

[0694] The device captures this data and sends it to the server.

[0695] Specific operation: The device records the user's voice, captures facial expression data with the camera, encrypts it, and sends it to the server.

[0696] Input: Voice data and facial expression data

[0697] Output: Voice and facial expression data sent to the server

[0698] Step 3:

[0699] The server uses an emotion engine to analyze the user's emotions from voice data and facial expression data.

[0700] What it does: The server analyzes voice and facial expression data using the Microsoft Azure Emotion API.

[0701] Input: Voice data and facial expression data sent to the server

[0702] output: Analyzed emotion data

[0703] Step 4:

[0704] The server sends the analysis results to the device.

[0705] Specific operation: The server encrypts the analysis results and sends them to the device.

[0706] Input: Parsed emotion data

[0707] Output: Analysis results sent to the device

[0708] Step 5:

[0709] The device provides feedback and advice to the user based on the analysis results.

[0710] Specific operation: Based on the analysis results received by the device, the device displays to the user, "You are feeling frustrated. Please take a break and refresh yourself."

[0711] Input: Analysis results sent to the device

[0712] Output: Feedback provided to the user

[0713] Smartphone compatible

[0714] Step 1:

[0715] The user launches the app on their smartphone.

[0716] Specific behavior: A user launches a learning app on their smartphone while commuting to work.

[0717] Step 2:

[0718] The device displays the main menu.

[0719] Specific operation: The device will display the main menu and present icons for various functions.

[0720] Input: None (when the app starts)

[0721] Output: Main menu displayed on the terminal screen

[0722] Step 3:

[0723] Users access various functions.

[0724] Specific operation: The user selects "English conversation practice."

[0725] Step 4:

[0726] The terminal sends a request to the server according to the selected function.

[0727] What happens: Your device encrypts and sends a request to the server based on your selection.

[0728] Input: User selection (e.g., English conversation practice)

[0729] Output: Request sent to the server

[0730] Step 5:

[0731] The server generates data and feedback according to the request and sends it back to the device.

[0732] Specific operation: The server generates data and feedback based on the request and sends it to the device.

[0733] Input: The request sent to the server

[0734] Output: Data or feedback sent to the device

[0735] Step 6:

[0736] The device provides the received data and feedback to the user.

[0737] Specific operation: The device provides the user with English conversation practice questions and feedback in real time based on the received data.

[0738] Input: Data and feedback sent to the device

[0739] Output: Exercises and feedback displayed on the screen

[0740] (Application example 2)

[0741] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0742] Conventional learning support systems often provide individual functions independently, resulting in a lack of coordination between them. Furthermore, they do not provide feedback that takes into account the user's emotions, which can lead to problems with reduced learning efficiency and motivation. Furthermore, there is a lack of systems that support face-to-face learning in physical stores, preventing improvements to the user experience.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a speech recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an interface means compatible with mobile devices, an emotion recognition means for analyzing the user's emotions, and a means for generating images based on prompts. This allows each function to work in conjunction with each other, providing feedback according to the user's emotional state, improving learning efficiency and motivation, and further enabling the realization of a system that can accommodate face-to-face learning in physical stores.

[0744] A "voice recognition means" is a device or system for converting voice data into text data.

[0745] A "text-to-speech means" is a device or system that converts text data into audio data and provides information by voice.

[0746] An "image generator" is a device or system for generating visual images based on keywords or prompts related to the learning content.

[0747] An "English conversation learning means" is a device or system for evaluating a user's pronunciation and supporting the improvement of their English conversation skills.

[0748] The "learning recording means" is a device or system for recording and managing learning data and saving the user's learning history.

[0749] An "interface means" is a device or system that is compatible with a mobile device and allows a user to access and use the system.

[0750] An "emotion recognition means" is a device or system that analyzes the user's emotions from their voice and facial expressions and provides appropriate feedback and advice.

[0751] A "means for generating images based on prompts" is a device or system for generating associated images in response to prompts entered by a user.

[0752] This invention is a system that integrates multiple learning support functions, such as voice recognition, text-to-speech, image generation, English conversation learning, learning record, mobile device compatibility, and emotion recognition. This system realizes each function by utilizing communication between a server and a terminal.

[0753] Voice Recognition

[0754] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data and returns the text data to the device. For example, if a user says, "Tell me about Newton's Laws," the voice recognition API generates the text "Tell me about Newton's Laws" and displays it.

[0755] Text-to-speech

[0756] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and sends the generated audio data to the device for playback. For example, if a user enters "Newton's third law," the text-to-speech API generates and plays back the audio for "Law of action and reaction."

[0757] Image generation

[0758] When a user inputs a prompt related to the learning content, the device sends it to the server. The server uses an image generation API to generate a related image and sends it to the device. For example, when the user inputs the prompt "action and reaction," a diagram of the related force is generated and displayed.

[0759] English conversation learning

[0760] When a user wants to practice English conversation, they input the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. This audio data is sent to the device and played back. The user can also record their own pronunciation and send it to the server, which then evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user practices "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0761] Learning Record

[0762] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, the device sends a request to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if the user checks "What did they study on October 1st?", the corresponding content will be displayed.

[0763] emotion recognition

[0764] When a user expresses emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion recognition API to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's emotional state. For example, if a user tweets "This is difficult," the emotion recognition API will recognize "frustration" from the voice and provide feedback such as "Try taking a break."

[0765] Specific examples

[0766] An example of a prompt is "Generate an image that illustrates Newton's laws." This prompt causes the system to call an image generation API to generate an image that visually illustrates Newton's laws. The generated image is displayed to the user in real time.

[0767] This allows the system's functions to work together and provide feedback based on the user's emotional state, improving learning efficiency and motivation, and providing comprehensive learning support that can also be used for face-to-face learning in physical stores.

[0768] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0769] Step 1:

[0770] The user inputs a learning question by voice. The device acquires this voice data and sends it to the server. The server uses a speech recognition API to convert the voice data into text data and sends the text data back to the device. The input is the user's voice data, and the output is text data. Specifically, if you say, "Tell me about Newton's laws," the text "Tell me about Newton's laws" is generated.

[0771] Step 2:

[0772] The user checks the text displayed on the device and selects the text they wish to have read aloud. The device sends the text data to the server, which then converts it into audio data using a text-to-speech API. The converted audio data is sent to the device and played back. The input is text data, and the output is audio data. For example, if you enter "Newton's third law," an audio version of "The Law of Action and Reaction" will be generated and played back.

[0773] Step 3:

[0774] The user inputs a prompt sentence related to the learning content. The device sends this prompt sentence to the server, and the server generates a related image using an image generation API. The generated image is sent to the device and displayed to the user. The input is the prompt sentence, and the output is image data. Specifically, if the user inputs "Generate an image that explains Newton's law," an image that visually shows Newton's law is generated.

[0775] Step 4:

[0776] To practice English conversation, the user inputs the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. The generated audio data is sent to the device and played back. The input is the text data of the English phrase, and the output is audio data. For example, if you input "How are you?", audio with the correct pronunciation will be generated.

[0777] Step 5:

[0778] The user records their own pronunciation and sends the audio data from their device to the server. The server evaluates the user's pronunciation using a speech recognition API and sends the evaluation results to the device. The input is the user's voice data, and the output is the evaluation result data. Specifically, when the user says "How are you?", the evaluation results of the pronunciation are fed back.

[0779] Step 6:

[0780] When a user begins studying, the device records the study time and content, and periodically sends this data to the server. The server saves it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data, sends it to the device, and displays it to the user. The input is the study data, and the output is the history data. Specifically, if a user requests "What did I study on October 1st?", the corresponding content will be displayed.

[0781] Step 7:

[0782] When a user expresses emotions through voice or facial expression while studying, the device captures this data and sends it to the server. The server uses an emotion recognition API to analyze emotions from the voice and facial data and sends the analysis results to the device. Based on the analysis results, the device provides appropriate feedback and advice to the user. The input is emotional data, and the output is feedback data. Specifically, if the user mutters "this is difficult," the device will recognize "frustration" from the voice and provide feedback such as "take a break."

[0783] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0784] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0785] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0786] [Second embodiment]

[0787] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0788] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0789] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0790] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0791] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0792] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0793] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0794] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0795] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0796] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0797] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0798] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0799] The present invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal. An example implementation of this system is described below.

[0800] Voice recognition function

[0801] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[0802] Text-to-speech function

[0803] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[0804] Image generation function

[0805] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[0806] English conversation learning function

[0807] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0808] Learning record function

[0809] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[0810] Smartphone compatible

[0811] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[0812] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[0813] The processing flow will be explained below.

[0814] Voice recognition function

[0815] Step 1:

[0816] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[0817] Step 2:

[0818] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[0819] Step 3:

[0820] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[0821] Step 4:

[0822] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[0823] Step 5:

[0824] The server records the generated text data in a database and returns the data to the terminal.

[0825] Step 6:

[0826] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[0827] Text-to-speech function

[0828] Step 1:

[0829] The user types the text they want to hear into the device. For example, type "Newton's third law."

[0830] Step 2:

[0831] The terminal sends the entered text data to the server using a POST request.

[0832] Step 3:

[0833] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[0834] Step 4:

[0835] The server transmits the generated voice data to the terminal.

[0836] Step 5:

[0837] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[0838] Image generation function

[0839] Step 1:

[0840] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[0841] Step 2:

[0842] The terminal sends the entered keyword to the server via a POST request.

[0843] Step 3:

[0844] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[0845] Step 4:

[0846] The server transmits the generated image data to the terminal.

[0847] Step 5:

[0848] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[0849] English conversation learning function

[0850] Step 1:

[0851] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[0852] Step 2:

[0853] The terminal sends the entered phrase to the server via a POST request.

[0854] Step 3:

[0855] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[0856] Step 4:

[0857] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[0858] Step 5:

[0859] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[0860] Step 6:

[0861] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[0862] Step 7:

[0863] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[0864] Learning record function

[0865] Step 1:

[0866] When the user starts studying, the terminal starts recording the study time and content.

[0867] Step 2:

[0868] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[0869] Step 3:

[0870] The server stores the received data in a database.

[0871] Step 4:

[0872] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[0873] Step 5:

[0874] The server searches the learning history from the database and sends it to the terminal.

[0875] Step 6:

[0876] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[0877] Smartphone compatible

[0878] Step 1:

[0879] The user launches the application on the smartphone.

[0880] Step 2:

[0881] The device displays the main menu and provides access to each function.

[0882] Step 3:

[0883] The user selects any function, such as English conversation learning or image generation, and begins learning.

[0884] Step 4:

[0885] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[0886] Step 5:

[0887] The terminal displays the received results to the user and provides learning according to the instructions.

[0888] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[0889] Example 1

[0890] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0891] In conventional learning support systems, each function exists independently, making it difficult to provide comprehensive learning support. Furthermore, functions such as voice recognition, text-to-speech, image generation, English conversation learning, and learning records are provided separately, making it difficult for users to centrally manage their learning progress and history. Furthermore, use on mobile devices is limited, meaning that they can only provide a learning environment that is restricted by location and time.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes a speech recognition unit that converts speech data into text data, a text-to-speech unit that converts text data into speech data, an image generation unit that generates images related to the learning content, an English conversation learning unit that evaluates the user's pronunciation, a learning record unit that records and manages learning data, an interface unit compatible with mobile devices, a transmission unit that captures speech data and transmits it to the server, a conversion unit that converts speech data into text data using a speech recognition API in the server, a display unit that displays the generated text data to the user, an image generation and transmission unit that transmits keywords entered by the user to the server and generates images using an image generation API, and a display unit that displays the generated images to the user. This provides integrated functions such as speech recognition, text-to-speech, image generation, English conversation learning, and learning record, allowing users to continue their learning anytime, anywhere.

[0894] A "voice recognition means" is a device or software for receiving voice data and converting it into text data.

[0895] "Text-to-speech means" refers to a device or software that converts input text data into audio data and plays back the audio.

[0896] The "image generating means" is a device or software for generating related images based on keywords entered by a user.

[0897] An "English conversation learning tool" is a device or software that evaluates a user's pronunciation and supports English conversation practice.

[0898] The "learning recording means" is a device or software for recording and managing the user's learning data.

[0899] "Interface means" refers to a means by which a user accesses the system through a mobile device and uses various functions.

[0900] "Transmission means" refers to a device or software for transmitting voice data or text data to a server.

[0901] The "conversion means" is a device or software for converting voice data into text data in the server.

[0902] The "display means" is a device or software for displaying the generated text data or image data to the user.

[0903] The "image generation and transmission means" is a device or software that transmits the keywords entered by the user to the server and generates an image using the image generation API.

[0904] This invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal.

[0905] Voice recognition function

[0906] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a speech recognition API (for example, Google Cloud Speech-to-Text API) to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the speech recognition API generates and displays the text, "Tell me about Newton's laws."

[0907] Text-to-speech function

[0908] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API (for example, Amazon Polly or Google Text-to-Speech API) to convert the text data into speech data, and the generated speech data is sent from the server to the device and played. For example, if a user enters "Newton's third law," a voice of "Law of action and reaction" is generated and played using the TTS API.

[0909] Image generation function

[0910] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server generates related images using an image generation API (e.g., DALL-E or MidJourney) and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, a diagram of the related forces is generated and displayed.

[0911] English conversation learning function

[0912] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[0913] Learning record function

[0914] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[0915] Smartphone compatible

[0916] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[0917] Below is an example of a prompt sentence to input to the generative AI model.

[0918] "Tell me about Newton's laws."

[0919] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[0920] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0921] Voice recognition function

[0922] Processing Step 1:

[0923] The user uses a microphone to input speech for learning.

[0924] Input: User's voice

[0925] Specific action: The user speaks into the smartphone microphone, "Tell me about Newton's laws."

[0926] Output: Captured audio data

[0927] Processing step 2:

[0928] The device captures the audio data and sends it to the server.

[0929] Input: Captured audio data

[0930] Specific operation: The device uploads the recorded audio data to a server via the Internet.

[0931] Output: Audio data sent to the server

[0932] Processing step 3:

[0933] The server uses a speech recognition API to convert the voice data into text data.

[0934] Input: Audio data sent to the server

[0935] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[0936] Specific operation: The voice recognition API analyzes the voice data and converts it into text data.

[0937] Output: Converted text data

[0938] Processing step 4:

[0939] The server transmits the converted text data to the terminal.

[0940] Input: Converted text data

[0941] Specific operation: The server returns text data to the terminal.

[0942] Output: Text data sent to the terminal

[0943] Processing Step 5:

[0944] The terminal displays the received text data to the user.

[0945] Input: Text data sent to the terminal

[0946] Specific action: Display the text "Tell me about Newton's laws" on the device screen.

[0947] Output: Displayed text data

[0948] Text-to-speech function

[0949] Processing Step 1:

[0950] The user enters the text they want to read into the device.

[0951] Input: Text entered by the user

[0952] Specific action: The user types "Newton's third law" on the smartphone keyboard.

[0953] Output: The input text data

[0954] Processing step 2:

[0955] The terminal transmits the input text data to the server.

[0956] Input: Entered text data

[0957] Specific operation: The device uploads text data to the server.

[0958] Output: Text data sent to the server

[0959] Processing step 3:

[0960] The server uses a text-to-speech API to convert the text data into audio data.

[0961] Input: Text data sent to the server

[0962] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[0963] Specific operation: Send an API request and generate audio data for "Newton's Third Law."

[0964] Output: Generated audio data

[0965] Processing step 4:

[0966] The server transmits the generated voice data to the terminal.

[0967] Input: Generated audio data

[0968] Specific operation: The server sends voice data to the terminal.

[0969] Output: Audio data sent to the device

[0970] Processing Step 5:

[0971] The terminal plays back the received audio data to the user.

[0972] Input: Audio data sent to the device

[0973] Specific action: The device plays an audio file and reads out "Newton's Third Law."

[0974] Output: Played audio data

[0975] Image generation function

[0976] Processing Step 1:

[0977] The user inputs keywords related to the content being studied into the terminal.

[0978] Input: The keyword entered by the user

[0979] Specific action: The user enters "action and reaction" into the smartphone.

[0980] Output: Entered keyword data

[0981] Processing step 2:

[0982] The terminal transmits the keyword data to the server.

[0983] Input: Entered keyword data

[0984] Specific operation: The terminal uploads keyword data to the server.

[0985] Output: Keyword data sent to the server

[0986] Processing step 3:

[0987] The server generates the relevant images using an image generation API.

[0988] Input: Keyword data sent to the server

[0989] Data processing: Use image generation API (e.g., DALL-E or MidJourney)

[0990] Specific operation: Send an API request and generate an image related to "action and reaction."

[0991] Output: Generated image data

[0992] Processing step 4:

[0993] The server transmits the generated image data to the terminal.

[0994] Input: Generated image data

[0995] Specific operation: The server sends an image file to the terminal.

[0996] Output: Image data sent to the device

[0997] Processing Step 5:

[0998] The terminal displays the received image data to the user.

[0999] Input: Image data sent to the device

[1000] Specific operation: Display the generated image on the device screen.

[1001] Output: Displayed image data

[1002] English conversation learning function

[1003] Processing Step 1:

[1004] The user inputs the English phrase they want to practice into the device.

[1005] Input: English phrase entered by the user

[1006] Specific action: The user types "How are you?" on the smartphone keyboard.

[1007] Output: Input phrase data

[1008] Processing step 2:

[1009] The device sends the entered phrase to the server.

[1010] Input: Entered phrase data

[1011] Specific operation: The device uploads phrase data to the server.

[1012] Output: Phrase data sent to the server

[1013] Processing step 3:

[1014] The server uses a text-to-speech API to generate audio data with the correct pronunciation.

[1015] Input: Phrase data sent to the server

[1016] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[1017] Specific behavior: Sends an API request to generate the correct pronunciation of "How are you?"

[1018] Output: Generated audio data

[1019] Processing step 4:

[1020] The server transmits the generated voice data to the terminal.

[1021] Input: Generated audio data

[1022] Specific operation: The server sends voice data to the terminal.

[1023] Output: Audio data sent to the device

[1024] Processing Step 5:

[1025] The terminal plays back the received voice data, and the user pronounces and records the voice accordingly.

[1026] Input: Audio data sent to the device

[1027] Specific operation: The correct pronunciation generated on the device is played back, and the user speaks into the microphone and the audio is recorded.

[1028] Output: Recorded audio data

[1029] Processing Step 6:

[1030] The device sends the recorded pronunciation data to the server.

[1031] Input: Recorded audio data

[1032] Specific operation: The device sends the recorded data to the server.

[1033] Output: Recording data sent to the server

[1034] Processing Step 7:

[1035] The server uses a speech recognition API to evaluate the user's pronunciation and generate a result.

[1036] Input: Recording data sent to the server

[1037] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[1038] Specific operations: Send an API request, convert the user's pronunciation into text, and perform an evaluation.

[1039] Output: Evaluation result data

[1040] Processing Step 8:

[1041] The server transmits the evaluation results to the terminal, which displays them to the user.

[1042] Input: Evaluation result data

[1043] Specific operation: The server sends the evaluation results to the terminal, and the terminal displays the results on the screen.

[1044] Output: Displayed evaluation results

[1045] Learning record function

[1046] Processing Step 1:

[1047] When a user starts studying, the terminal records the study time and content.

[1048] Input: Study start information (time, content)

[1049] Specific operation: The device stores the user's learning start time and learning content in a log.

[1050] Output: Recorded training data

[1051] Processing step 2:

[1052] The terminal periodically transmits the recorded data to the server.

[1053] Input: Recorded training data

[1054] Specific operation: The device periodically uploads learning data to the server.

[1055] Output: Training data sent to the server

[1056] Processing step 3:

[1057] The server stores the received learning data in a database.

[1058] Input: Training data sent to the server

[1059] Specific operation: The server inserts the received data into the corresponding table in the database.

[1060] Output: Training data stored in a database

[1061] Processing step 4:

[1062] When a user wants to check his / her past learning history, the terminal sends a request to the server, the server searches for learning history data and sends it to the terminal, and the terminal displays the data to the user.

[1063] Input: User request (e.g., learning history for a specific date)

[1064] Specific operation: The device sends a request to the server to confirm "what was learned on October 1st," and the server searches the database for the relevant data and sends it.

[1065] Output: Displayed learning history data

[1066] Mobile device compatible features

[1067] Processing Step 1:

[1068] The user launches the app on their smartphone.

[1069] Input: App launch operation

[1070] Specific action: The user taps the app icon on their smartphone to launch the app.

[1071] Output: App launch status

[1072] Processing step 2:

[1073] The device displays the main menu.

[1074] Input: App running state

[1075] Specific operation: Links to various functions will be displayed on the app's home screen.

[1076] Output: Main menu displayed

[1077] Processing step 3:

[1078] The user selects the required function, and communication is made from the device to the server, providing the necessary data and feedback in real time.

[1079] Input: User selected function

[1080] Specific operation: Select the English conversation practice mode during your commute, and the device will communicate with the server to provide pronunciation practice.

[1081] Output: Real-time data and feedback

[1082] The above is the specific processing flow for each function.

[1083] (Application example 1)

[1084] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1085] Conventional security systems can make it difficult for users to quickly identify abnormalities and respond appropriately. In particular, delays in detecting suspicious individuals and issuing security alerts can delay effective responses. Furthermore, because these systems are not interactive, users often cannot directly communicate with the system to check the situation or issue instructions.

[1086] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1087] In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, a voice evaluation means for evaluating the user's pronunciation, a recording means for recording and managing the learning data, an interface means compatible with the mobile device, a notification means for notifying the user of security events by voice, an analysis means for performing image analysis, and a log management means for recording security events in a log file. This allows the user to interact with the security system through voice commands, check surveillance footage in real time, and respond quickly.

[1088] The "voice recognition means" is a means for converting voice data into text data.

[1089] The "text-to-speech means" is a means for converting text data into voice data.

[1090] "Image generation means" refers to means for generating images related to learning content or other content.

[1091] The "voice evaluation means" is a means for evaluating the user's pronunciation.

[1092] "Recording means" refers to a means for recording and managing learning data and other data.

[1093] "Interface means" refers to means for providing a corresponding interface to a mobile device or other device.

[1094] The "notification means" is a means for notifying the user of a security event by voice.

[1095] "Analysis means" refers to means for analyzing image data and other data.

[1096] A "log management means" is a means for recording and managing security events in a log file.

[1097] The present invention relates to a security support system, which is an integrated system that provides multiple functions such as voice recognition, text-to-speech, image generation, voice evaluation, recording and management, mobile device compatibility, notification, and analysis. Specific embodiments of this system are described below.

[1098] Voice recognition means

[1099] When a user issues a voice command, the device captures the voice data and sends it to the server. The server uses a voice recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice data into text data, and returns the text data to the device, which then displays it. For example, if a user says, "Detect a suspicious person," the voice recognition API generates the text "Detect a suspicious person" and displays it on the device.

[1100] Text-to-speech means

[1101] When the server detects an important security notification, it converts the text data into audio data. The server then uses a text-to-speech API (e.g., Google Cloud Text-to-Speech) to send the generated audio data to the device and notify the user by voice. For example, the generated text might say, "A suspicious person has been detected. Please be careful," and be played back as audio.

[1102] Image Generation Means

[1103] When a user wants to check a specific image, they enter a relevant keyword into their device and send it to the server. The server uses an image generation API to generate an image related to the keyword and sends it to the device. For example, if "suspicious person's face" is entered as a keyword, an image that identifies the suspicious person's face will be generated and displayed on the device.

[1104] Audio evaluation tools

[1105] When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, a user can record and send "How are you?" and the pronunciation will be evaluated to see if it is appropriate.

[1106] Recording means

[1107] Every time a user uses the system, data on related security events is sent to the server and recorded. This allows past security history to be checked. For example, the date, time, and location of any suspicious activity are recorded.

[1108] Interface Means

[1109] The system is compatible with mobile devices, allowing users to access the system anytime, anywhere. When users launch the app on their smartphone, an interface is displayed that allows them to access security functions. For example, they can check security camera footage while commuting.

[1110] Notification means

[1111] When a security event occurs, the system notifies the user with an audio alert. The server converts text into audio and sends it to the device where it is played. For example, a voice alert saying "A suspicious person has been detected" is played.

[1112] Analysis means

[1113] The video captured by the device's camera is sent to the server, and the server uses image analysis software (e.g., OpenCV) to detect suspicious individuals. If a suspicious individual is detected, the information is notified to the user. For example, the face of a suspicious individual can be automatically analyzed based on the camera image.

[1114] Log management methods

[1115] Security event logs are managed on the server and recorded in log files. When a user wants to check past security events, they send a request to the server and the corresponding log data is displayed on the terminal. For example, it can be used to "check details of suspicious individuals detected on October 1st."

[1116] Examples of concrete examples and prompts

[1117] As a specific example of use, when a user says to their smartphone, "Detect suspicious person," the app analyzes the camera image, and if a suspicious person is detected, it notifies them with a voice message saying, "Suspicious person detected," and records the event in a log.

[1118] Prompt Sentence Examples

[1119] User: Suspicious person detected

[1120] App: Analyzing camera images...

[1121] App: Suspicious person detected. Please be careful.

[1122] The above is a specific description of the embodiment of the present invention, which enables users to take security measures quickly and effectively.

[1123] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1124] Step 1:

[1125] The user speaks a voice command, which is captured as voice data by the smartphone's microphone. The input is the voice command, and the output is the captured voice data.

[1126] Step 2:

[1127] The device sends the captured voice data to the server for conversion to text data using a speech recognition API. The server then converts the voice data to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the voice data, and the output is the converted text data.

[1128] Step 3:

[1129] The server analyzes the text data and determines a specific action. For example, if the text contains the phrase "detect a suspicious person," the server will instruct the start of image analysis. The input is the text data, and the output is the analysis result and the decision on the next action.

[1130] Step 4:

[1131] The device's camera captures real-time video and sends the image data to the server. The input is the image data captured by the camera, and the output is the image data sent to the server.

[1132] Step 5:

[1133] The server uses image analysis software (e.g., OpenCV) to analyze the captured image data and detect suspicious individuals. The input is the image data, and the output is the analysis result indicating whether or not there is a suspicious individual.

[1134] Step 6:

[1135] When the server detects a suspicious person, it sends the information to the terminal as text and simultaneously converts it into audio data using a text-to-speech API. The input is text data and the output is audio data.

[1136] Step 7:

[1137] The device receives the voice data and issues a voice notification to the user from the speaker. For example, a voice alert saying "A suspicious person has been detected. Please be careful" is played. The input is voice data, and the output is a voice notification.

[1138] Step 8:

[1139] Details of security events are recorded on the server and saved in a log file through a log management tool. The event date and time, detected details, etc. are recorded in the log so that users can check them later. The input is security event information, and the output is a log file.

[1140] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1141] The present invention is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal. An example implementation is described below.

[1142] Voice recognition function

[1143] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[1144] Text-to-speech function

[1145] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[1146] Image generation function

[1147] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[1148] English conversation learning function

[1149] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1150] Learning record function

[1151] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[1152] Emotion recognition function using emotion engine

[1153] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's research and learning situation. For example, if a user mutters, "This is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as, "Try taking a break to resolve this."

[1154] Smartphone compatible

[1155] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[1156] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[1157] The processing flow will be explained below.

[1158] Voice recognition function

[1159] Step 1:

[1160] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[1161] Step 2:

[1162] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[1163] Step 3:

[1164] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[1165] Step 4:

[1166] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[1167] Step 5:

[1168] The server records the generated text data in a database and returns the data to the terminal.

[1169] Step 6:

[1170] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[1171] Text-to-speech function

[1172] Step 1:

[1173] The user types the text they want to hear into the device. For example, type "Newton's third law."

[1174] Step 2:

[1175] The terminal sends the entered text data to the server using a POST request.

[1176] Step 3:

[1177] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[1178] Step 4:

[1179] The server transmits the generated voice data to the terminal.

[1180] Step 5:

[1181] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[1182] Image generation function

[1183] Step 1:

[1184] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[1185] Step 2:

[1186] The terminal sends the entered keyword to the server via a POST request.

[1187] Step 3:

[1188] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[1189] Step 4:

[1190] The server transmits the generated image data to the terminal.

[1191] Step 5:

[1192] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[1193] English conversation learning function

[1194] Step 1:

[1195] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[1196] Step 2:

[1197] The terminal sends the entered phrase to the server via a POST request.

[1198] Step 3:

[1199] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[1200] Step 4:

[1201] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[1202] Step 5:

[1203] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[1204] Step 6:

[1205] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[1206] Step 7:

[1207] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[1208] Learning record function

[1209] Step 1:

[1210] When the user starts studying, the terminal starts recording the study time and content.

[1211] Step 2:

[1212] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[1213] Step 3:

[1214] The server stores the received data in a database.

[1215] Step 4:

[1216] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[1217] Step 5:

[1218] The server searches the learning history from the database and sends it to the terminal.

[1219] Step 6:

[1220] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[1221] Emotion recognition function using emotion engine

[1222] Step 1:

[1223] The user expresses their emotions through voice and facial expressions. For example, they may say, "It's difficult."

[1224] Step 2:

[1225] The device captures and temporarily saves the user's voice and facial expression data. For example, it saves the facial expression data as .jpg and the voice emotion file as .wav.

[1226] Step 3:

[1227] The device sends this data to the server. For example, send facial expression data (.jpg) or voice emotion file (.wav) to the server using a POST request.

[1228] Step 4:

[1229] The server sends the received data to the emotion engine, which analyzes the user's emotions. For example, it recognizes "frustration."

[1230] Step 5:

[1231] The server sends the emotion analysis results to the device and provides feedback to the user, such as "Let's take a break to solve this."

[1232] Smartphone compatible

[1233] Step 1:

[1234] The user launches the application on the smartphone.

[1235] Step 2:

[1236] The device displays the main menu and provides access to each function.

[1237] Step 3:

[1238] The user selects any function, such as English conversation learning or image generation, and begins learning.

[1239] Step 4:

[1240] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[1241] Step 5:

[1242] The terminal displays the received results to the user and provides learning according to the instructions.

[1243] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[1244] Example 2

[1245] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1246] In today's learning environment, learners are required to efficiently acquire knowledge using a variety of media and tools. However, most learning systems are specialized in individual functions, and few provide comprehensive support across a wide range of areas, such as speech recognition, text-to-speech, image generation, English conversation learning, and emotion recognition. Furthermore, functions such as learning support on mobile devices and storing learning records are not consistently integrated. This makes it difficult for learners to obtain a learning experience that meets their diverse needs.

[1247] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an emotion recognition means for analyzing the user's emotions, and an interface means compatible with mobile devices. This makes it possible to provide comprehensive support for a variety of learning needs.

[1248] "Speech recognition means" refers to a device or software that has the function of analyzing voice data and converting it into text data.

[1249] A "text-to-speech means" is a device or software that has the function of analyzing text data and converting it into audio data.

[1250] "Image generation means" refers to a device or software that has the function of automatically generating related images based on keywords or content entered by a user.

[1251] An "English conversation learning tool" is a device or software that has the function of providing audio with correct pronunciation for English phrases entered by a user, evaluating the user's pronunciation, and providing feedback.

[1252] A "learning recording means" refers to a device or software that has the function of recording and managing a user's learning activities and learning data.

[1253] "Emotion recognition means" refers to devices or software that have the function of analyzing and evaluating emotions from the user's voice and facial expressions.

[1254] "Mobile device compatible interface means" refers to user interfaces and related communication means designed for use on mobile devices such as smartphones and tablets.

[1255] This is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal.

[1256] Voice recognition function

[1257] When a user provides voice input for training, the device captures this voice data and sends it to the server. The server then uses a speech recognition API to convert the voice data into text data. This can be done using services such as the Google Cloud Speech-to-Text API or Amazon Transcribe. The converted text data is then sent back from the server to the device, which then displays the text to the user.

[1258] Specific examples

[1259] When a user says, "Tell me about Newton's Law," the speech recognition API generates the text "Tell me about Newton's Law," which is displayed on the device.

[1260] Prompt Sentence Examples

[1261] "Explain Newton's First Law."

[1262] Text-to-speech function

[1263] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server then uses a text-to-speech API to convert the text data into audio data. Specifically, Google Cloud Text-to-Speech API or Amazon Polly can be used. The generated audio data is sent from the server to the device and played back.

[1264] Specific examples

[1265] When a user types "Newton's third law," a text-to-speech API is used to generate and play an audio version of "The Law of Action and Reaction."

[1266] Prompt Sentence Examples

[1267] Please read the following text aloud: "The law of action and reaction states that for every force there is an equal and opposite force acting in the opposite direction."

[1268] Image generation function

[1269] When a user enters keywords related to the content they are learning, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. Specifically, DALL-E or Stable Diffusion can be used. The device then displays the generated images to the user.

[1270] Specific examples

[1271] When the user enters the keyword "action and reaction," a diagram of the relevant forces is generated and displayed.

[1272] Prompt Sentence Examples

[1273] "Create an image that illustrates action and reaction forces."

[1274] English conversation learning function

[1275] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. Google Cloud Text-to-Speech API or Amazon Polly can be used. This audio data is sent to the device and played back. When the user records their own pronunciation and sends the data to the server, the server uses a speech recognition API to evaluate the pronunciation and sends the results to the device.

[1276] Specific examples

[1277] When a user wishes to practice "How are you?", a correct pronunciation audio is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1278] Prompt Sentence Examples

[1279] "Generate audio to learn the correct pronunciation of 'How are you?'"

[1280] Learning record function

[1281] When a user starts studying, the device records the study time and content. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user.

[1282] Specific examples

[1283] When a user checks "What did I learn on October 1st?", the corresponding content is displayed.

[1284] Prompt Sentence Examples

[1285] "Please show me what you learned on October 1st."

[1286] Emotion recognition function using emotion engine

[1287] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. Specifically, it can use the Microsoft Azure Emotion API or Google Cloud Vision API. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's learning status.

[1288] Specific examples

[1289] If a user says "this is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as "Let's take a break to solve this."

[1290] Prompt Sentence Examples

[1291] "Please analyze the user's emotions from this voice data."

[1292] Smartphone compatible

[1293] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time.

[1294] Specific examples

[1295] The user can select the English conversation practice mode and practice pronunciation while commuting.

[1296] Prompt Sentence Examples

[1297] "Select English conversation practice mode and start practicing your pronunciation during your commute."

[1298] This allows the present invention to meet the needs of various learners and provide efficient and comprehensive learning support. Each function is realized through the cooperation of the server and terminal, and users can access it from anywhere. This system provides a variety of means to improve the learning experience in an integrated manner.

[1299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1300] Voice recognition function

[1301] Step 1:

[1302] The user performs voice input for learning.

[1303] Specific Action: The user speaks into the microphone, "Tell me about Newton's Laws."

[1304] Step 2:

[1305] The device captures the audio data.

[1306] What it does: The device's microphone records the user's voice in real time.

[1307] Input: User's voice

[1308] Output: Recorded audio data

[1309] Step 3:

[1310] The device sends the captured audio data to the server.

[1311] Specific operation: The device encrypts the recorded data and sends it to the specified endpoint on the server.

[1312] Input: Audio data

[1313] Output: Audio data sent to the server

[1314] Step 4:

[1315] The server uses a speech recognition API to convert the voice data into text data.

[1316] Specific operation: The server passes the audio data to the Google Cloud Speech-to-Text API and retrieves the text data.

[1317] Input: Audio data sent to the server

[1318] Output: Converted text data

[1319] Step 5:

[1320] The server returns the converted text data to the terminal.

[1321] Specific operation: The server encrypts the converted text data and sends it to the device.

[1322] Input: Converted text data

[1323] Output: Text data sent to the terminal

[1324] Step 6:

[1325] The terminal displays the text data to the user.

[1326] What it does: The device displays the received text on the screen, informing the user, "Tell me about Newton's laws."

[1327] Input: Text data sent to the terminal

[1328] Output: Text displayed on the screen

[1329] Text-to-speech function

[1330] Step 1:

[1331] The user enters the text they want to read into the device or selects existing text.

[1332] Specific action: The user types "Newton's third law" into the terminal.

[1333] Step 2:

[1334] The terminal transmits the text data to the server.

[1335] Specific operation: The device encrypts the text data entered and sends it to the server.

[1336] Input: Text data entered by the user into the terminal

[1337] Output: Text data sent to the server

[1338] Step 3:

[1339] The server uses a text-to-speech API to convert the text data into audio data.

[1340] Specific operation: The server sends text data to the Google Cloud Text-to-Speech API, which generates audio data.

[1341] Input: Text data sent to the server

[1342] Output: Generated audio data

[1343] Step 4:

[1344] The server transmits the generated voice data to the terminal.

[1345] Specific operation: The server encrypts the generated audio data and sends it to the device.

[1346] Input: Generated audio data

[1347] Output: Audio data sent to the device

[1348] Step 5:

[1349] The device plays the audio data.

[1350] Specific operation: The device plays the received audio data and reads out the "Law of Action and Reaction" to the user.

[1351] Input: Audio data sent to the device

[1352] Output: Played audio

[1353] Image generation function

[1354] Step 1:

[1355] The user inputs keywords related to the content being studied into the terminal.

[1356] Specific operation: The user enters the keywords "action and reaction" into the terminal.

[1357] Step 2:

[1358] The device sends the keyword to the server.

[1359] Specific operation: The device encrypts the keyword and sends it to the server.

[1360] Input: Keywords entered by the user into the device

[1361] Output: Keywords sent to the server

[1362] Step 3:

[1363] The server generates the relevant images using an image generation API.

[1364] What it does: The server passes keywords to the DALL-E API and generates related images.

[1365] Input: The keyword sent to the server

[1366] Output: Generated image data

[1367] Step 4:

[1368] The server transmits the generated image to the terminal.

[1369] Specific operation: The server encrypts the generated image data and sends it to the terminal.

[1370] Input: Generated image data

[1371] Output: Image data sent to the device

[1372] Step 5:

[1373] The terminal displays the image to the user.

[1374] Specific operation: The device displays the image it receives on the screen, showing the user an "image illustrating the law of action and reaction."

[1375] Input: Image data sent to the device

[1376] Output: Image displayed on the screen

[1377] English conversation learning function

[1378] Step 1:

[1379] The user inputs the English phrase they want to practice into the device.

[1380] What happens: The user types "How are you?" into the terminal.

[1381] Step 2:

[1382] The device sends the entered phrase to the server.

[1383] Specific operation: The device encrypts the entered phrase and sends it to the server.

[1384] Input: The phrase the user types into the terminal

[1385] Output: Phrase sent to server

[1386] Step 3:

[1387] The server uses the TTS API to generate audio data with the correct pronunciation.

[1388] What happens: The server passes the phrase to the Google Cloud Text-to-Speech API, which generates audio data with the correct pronunciation.

[1389] Input: Phrase sent to server

[1390] output: Generated audio data with correct pronunciation

[1391] Step 4:

[1392] The server transmits the generated voice data to the terminal.

[1393] Specific operation: The server encrypts the generated audio data and sends it to the device.

[1394] Input: Generated audio data

[1395] Output: Audio data sent to the device

[1396] Step 5:

[1397] The device plays the audio data.

[1398] Specific operation: The device plays back the received audio data and provides the user with the correct pronunciation.

[1399] Input: Audio data sent to the device

[1400] Output: Played audio

[1401] Step 6:

[1402] The user records their own pronunciation and sends it from the terminal to the server.

[1403] Specific operation: The user records their own pronunciation and sends it to the server via their device.

[1404] Input: User recording data

[1405] Output: Recording data sent to the server

[1406] Step 7:

[1407] The server evaluates the pronunciation using a speech recognition API.

[1408] What happens: The server passes the recording to the Google Cloud Speech-to-Text API and evaluates the pronunciation.

[1409] Input: Recording data sent to the server

[1410] Output: Evaluation result

[1411] Step 8:

[1412] The server transmits the evaluation results to the terminal.

[1413] Specific operation: The server encrypts the evaluation results and sends them to the device.

[1414] Input: Evaluation result

[1415] Output: Evaluation results sent to the device

[1416] Step 9:

[1417] The terminal displays the evaluation results to the user.

[1418] Specific behavior: The device displays the evaluation results it receives on the screen and informs the user that "your pronunciation was accurate" or "your pronunciation needs improvement."

[1419] Input: Evaluation result sent to the terminal

[1420] Output: Evaluation results displayed on the screen

[1421] Learning record function

[1422] Step 1:

[1423] The user begins learning.

[1424] What happens: A user starts a new learning session in the app.

[1425] Step 2:

[1426] The device records the study time and content.

[1427] Specific operation: The device records the start time of learning and saves the learning content as it progresses.

[1428] Input: Start learning information and learning content

[1429] Output: Training data recorded on the device

[1430] Step 3:

[1431] The device periodically transmits learning data to the server.

[1432] Specific operation: The device encrypts and sends learning data to the server once every 15 minutes.

[1433] Input: Training data recorded on the device

[1434] output: Training data sent to the server

[1435] Step 4:

[1436] The server stores the learning data in a database.

[1437] Specific operation: The server stores the received learning data in a database.

[1438] Input: Training data sent to the server

[1439] output: Training data stored in the database

[1440] Step 5:

[1441] When a user wants to check his / her past learning history, he / she sends a request from the terminal to the server.

[1442] Specific operation: The user sends a request to the server from the device to check the learning history for a specified date in the app.

[1443] Input: Request (e.g., Confirm learning content for October 1st)

[1444] Output: Request sent to the server

[1445] Step 6:

[1446] The server searches the database for learning history data and sends it to the terminal.

[1447] Specific operation: The server searches the database for the learning history for the specified date, encrypts it, and sends it to the device.

[1448] Input: Request

[1449] output: Learning history data sent to the device

[1450] Step 7:

[1451] The terminal displays the learning history data to the user.

[1452] Specific operation: The device displays the learning history data it has received on the screen and informs the user, "The content learned on October 1st was as follows: The basics of Newton's laws."

[1453] Input: Learning history data sent to the device

[1454] Output: Learning history data displayed on the screen

[1455] Emotion recognition function using emotion engine

[1456] Step 1:

[1457] The user expresses their feelings through voice and facial expressions while learning.

[1458] Specific action: The user tweets "This is difficult" while studying.

[1459] Step 2:

[1460] The device captures this data and sends it to the server.

[1461] Specific operation: The device records the user's voice, captures facial expression data with the camera, encrypts it, and sends it to the server.

[1462] Input: Voice data and facial expression data

[1463] Output: Voice and facial expression data sent to the server

[1464] Step 3:

[1465] The server uses an emotion engine to analyze the user's emotions from voice data and facial expression data.

[1466] What it does: The server analyzes voice and facial expression data using the Microsoft Azure Emotion API.

[1467] Input: Voice data and facial expression data sent to the server

[1468] output: Analyzed emotion data

[1469] Step 4:

[1470] The server sends the analysis results to the device.

[1471] Specific operation: The server encrypts the analysis results and sends them to the device.

[1472] Input: Parsed emotion data

[1473] Output: Analysis results sent to the device

[1474] Step 5:

[1475] The device provides feedback and advice to the user based on the analysis results.

[1476] Specific operation: Based on the analysis results received by the device, the device displays to the user, "You are feeling frustrated. Please take a break and refresh yourself."

[1477] Input: Analysis results sent to the device

[1478] Output: Feedback provided to the user

[1479] Smartphone compatible

[1480] Step 1:

[1481] The user launches the app on their smartphone.

[1482] Specific behavior: A user launches a learning app on their smartphone while commuting to work.

[1483] Step 2:

[1484] The device displays the main menu.

[1485] Specific operation: The device will display the main menu and present icons for various functions.

[1486] Input: None (when the app starts)

[1487] Output: Main menu displayed on the terminal screen

[1488] Step 3:

[1489] Users access various functions.

[1490] Specific operation: The user selects "English conversation practice."

[1491] Step 4:

[1492] The terminal sends a request to the server according to the selected function.

[1493] What happens: Your device encrypts and sends a request to the server based on your selection.

[1494] Input: User selection (e.g., English conversation practice)

[1495] Output: Request sent to the server

[1496] Step 5:

[1497] The server generates data and feedback according to the request and sends it back to the device.

[1498] Specific operation: The server generates data and feedback based on the request and sends it to the device.

[1499] Input: The request sent to the server

[1500] Output: Data or feedback sent to the device

[1501] Step 6:

[1502] The device provides the received data and feedback to the user.

[1503] Specific operation: The device provides the user with English conversation practice questions and feedback in real time based on the received data.

[1504] Input: Data and feedback sent to the device

[1505] Output: Exercises and feedback displayed on the screen

[1506] (Application example 2)

[1507] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1508] Conventional learning support systems often provide individual functions independently, resulting in a lack of coordination between them. Furthermore, they do not provide feedback that takes into account the user's emotions, which can lead to problems with reduced learning efficiency and motivation. Furthermore, there is a lack of systems that support face-to-face learning in physical stores, preventing improvements to the user experience.

[1509] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a speech recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an interface means compatible with mobile devices, an emotion recognition means for analyzing the user's emotions, and a means for generating images based on prompts. This allows each function to work in conjunction with each other, providing feedback according to the user's emotional state, improving learning efficiency and motivation, and further enabling the realization of a system that can accommodate face-to-face learning in physical stores.

[1510] A "voice recognition means" is a device or system for converting voice data into text data.

[1511] A "text-to-speech means" is a device or system that converts text data into audio data and provides information by voice.

[1512] An "image generator" is a device or system for generating visual images based on keywords or prompts related to the learning content.

[1513] An "English conversation learning means" is a device or system for evaluating a user's pronunciation and supporting the improvement of their English conversation skills.

[1514] The "learning recording means" is a device or system for recording and managing learning data and saving the user's learning history.

[1515] An "interface means" is a device or system that is compatible with a mobile device and allows a user to access and use the system.

[1516] An "emotion recognition means" is a device or system that analyzes the user's emotions from their voice and facial expressions and provides appropriate feedback and advice.

[1517] A "means for generating images based on prompts" is a device or system for generating associated images in response to prompts entered by a user.

[1518] This invention is a system that integrates multiple learning support functions, such as voice recognition, text-to-speech, image generation, English conversation learning, learning record, mobile device compatibility, and emotion recognition. This system realizes each function by utilizing communication between a server and a terminal.

[1519] Voice Recognition

[1520] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data and returns the text data to the device. For example, if a user says, "Tell me about Newton's Laws," the voice recognition API generates the text "Tell me about Newton's Laws" and displays it.

[1521] Text-to-speech

[1522] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and sends the generated audio data to the device for playback. For example, if a user enters "Newton's third law," the text-to-speech API generates and plays back the audio for "Law of action and reaction."

[1523] Image generation

[1524] When a user inputs a prompt related to the learning content, the device sends it to the server. The server uses an image generation API to generate a related image and sends it to the device. For example, when the user inputs the prompt "action and reaction," a diagram of the related force is generated and displayed.

[1525] English conversation learning

[1526] When a user wants to practice English conversation, they input the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. This audio data is sent to the device and played back. The user can also record their own pronunciation and send it to the server, which then evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user practices "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1527] Learning Record

[1528] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, the device sends a request to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if the user checks "What did they study on October 1st?", the corresponding content will be displayed.

[1529] emotion recognition

[1530] When a user expresses emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion recognition API to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's emotional state. For example, if a user tweets "This is difficult," the emotion recognition API will recognize "frustration" from the voice and provide feedback such as "Try taking a break."

[1531] Specific examples

[1532] An example of a prompt is "Generate an image that illustrates Newton's laws." This prompt causes the system to call an image generation API to generate an image that visually illustrates Newton's laws. The generated image is displayed to the user in real time.

[1533] This allows the system's functions to work together and provide feedback based on the user's emotional state, improving learning efficiency and motivation, and providing comprehensive learning support that can also be used for face-to-face learning in physical stores.

[1534] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1535] Step 1:

[1536] The user inputs a learning question by voice. The device acquires this voice data and sends it to the server. The server uses a speech recognition API to convert the voice data into text data and sends the text data back to the device. The input is the user's voice data, and the output is text data. Specifically, if you say, "Tell me about Newton's laws," the text "Tell me about Newton's laws" is generated.

[1537] Step 2:

[1538] The user checks the text displayed on the device and selects the text they wish to have read aloud. The device sends the text data to the server, which then converts it into audio data using a text-to-speech API. The converted audio data is sent to the device and played back. The input is text data, and the output is audio data. For example, if you enter "Newton's third law," an audio version of "The Law of Action and Reaction" will be generated and played back.

[1539] Step 3:

[1540] The user inputs a prompt sentence related to the learning content. The device sends this prompt sentence to the server, and the server generates a related image using an image generation API. The generated image is sent to the device and displayed to the user. The input is the prompt sentence, and the output is image data. Specifically, if the user inputs "Generate an image that explains Newton's law," an image that visually shows Newton's law is generated.

[1541] Step 4:

[1542] To practice English conversation, the user inputs the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. The generated audio data is sent to the device and played back. The input is the text data of the English phrase, and the output is audio data. For example, if you input "How are you?", audio with the correct pronunciation will be generated.

[1543] Step 5:

[1544] The user records their own pronunciation and sends the audio data from their device to the server. The server evaluates the user's pronunciation using a speech recognition API and sends the evaluation results to the device. The input is the user's voice data, and the output is the evaluation result data. Specifically, when the user says "How are you?", the evaluation results of the pronunciation are fed back.

[1545] Step 6:

[1546] When a user begins studying, the device records the study time and content, and periodically sends this data to the server. The server saves it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data, sends it to the device, and displays it to the user. The input is the study data, and the output is the history data. Specifically, if a user requests "What did I study on October 1st?", the corresponding content will be displayed.

[1547] Step 7:

[1548] When a user expresses emotions through voice or facial expression while studying, the device captures this data and sends it to the server. The server uses an emotion recognition API to analyze emotions from the voice and facial data and sends the analysis results to the device. Based on the analysis results, the device provides appropriate feedback and advice to the user. The input is emotional data, and the output is feedback data. Specifically, if the user mutters "this is difficult," the device will recognize "frustration" from the voice and provide feedback such as "take a break."

[1549] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1550] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1551] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1552] [Third embodiment]

[1553] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1554] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1555] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1556] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1557] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1558] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1559] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1560] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1561] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1562] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1563] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1564] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1565] The present invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal. An example implementation of this system is described below.

[1566] Voice recognition function

[1567] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[1568] Text-to-speech function

[1569] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[1570] Image generation function

[1571] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[1572] English conversation learning function

[1573] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1574] Learning record function

[1575] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[1576] Smartphone compatible

[1577] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[1578] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[1579] The processing flow will be explained below.

[1580] Voice recognition function

[1581] Step 1:

[1582] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[1583] Step 2:

[1584] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[1585] Step 3:

[1586] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[1587] Step 4:

[1588] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[1589] Step 5:

[1590] The server records the generated text data in a database and returns the data to the terminal.

[1591] Step 6:

[1592] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[1593] Text-to-speech function

[1594] Step 1:

[1595] The user types the text they want to hear into the device. For example, type "Newton's third law."

[1596] Step 2:

[1597] The terminal sends the entered text data to the server using a POST request.

[1598] Step 3:

[1599] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[1600] Step 4:

[1601] The server transmits the generated voice data to the terminal.

[1602] Step 5:

[1603] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[1604] Image generation function

[1605] Step 1:

[1606] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[1607] Step 2:

[1608] The terminal sends the entered keyword to the server via a POST request.

[1609] Step 3:

[1610] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[1611] Step 4:

[1612] The server transmits the generated image data to the terminal.

[1613] Step 5:

[1614] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[1615] English conversation learning function

[1616] Step 1:

[1617] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[1618] Step 2:

[1619] The terminal sends the entered phrase to the server via a POST request.

[1620] Step 3:

[1621] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[1622] Step 4:

[1623] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[1624] Step 5:

[1625] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[1626] Step 6:

[1627] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[1628] Step 7:

[1629] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[1630] Learning record function

[1631] Step 1:

[1632] When the user starts studying, the terminal starts recording the study time and content.

[1633] Step 2:

[1634] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[1635] Step 3:

[1636] The server stores the received data in a database.

[1637] Step 4:

[1638] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[1639] Step 5:

[1640] The server searches the learning history from the database and sends it to the terminal.

[1641] Step 6:

[1642] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[1643] Smartphone compatible

[1644] Step 1:

[1645] The user launches the application on the smartphone.

[1646] Step 2:

[1647] The device displays the main menu and provides access to each function.

[1648] Step 3:

[1649] The user selects any function, such as English conversation learning or image generation, and begins learning.

[1650] Step 4:

[1651] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[1652] Step 5:

[1653] The terminal displays the received results to the user and provides learning according to the instructions.

[1654] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[1655] Example 1

[1656] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1657] In conventional learning support systems, each function exists independently, making it difficult to provide comprehensive learning support. Furthermore, functions such as voice recognition, text-to-speech, image generation, English conversation learning, and learning records are provided separately, making it difficult for users to centrally manage their learning progress and history. Furthermore, use on mobile devices is limited, meaning that they can only provide a learning environment that is restricted by location and time.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1659] In this invention, the server includes a speech recognition unit that converts speech data into text data, a text-to-speech unit that converts text data into speech data, an image generation unit that generates images related to the learning content, an English conversation learning unit that evaluates the user's pronunciation, a learning record unit that records and manages learning data, an interface unit compatible with mobile devices, a transmission unit that captures speech data and transmits it to the server, a conversion unit that converts speech data into text data using a speech recognition API in the server, a display unit that displays the generated text data to the user, an image generation and transmission unit that transmits keywords entered by the user to the server and generates images using an image generation API, and a display unit that displays the generated images to the user. This provides integrated functions such as speech recognition, text-to-speech, image generation, English conversation learning, and learning record, allowing users to continue their learning anytime, anywhere.

[1660] A "voice recognition means" is a device or software for receiving voice data and converting it into text data.

[1661] "Text-to-speech means" refers to a device or software that converts input text data into audio data and plays back the audio.

[1662] The "image generating means" is a device or software for generating related images based on keywords entered by a user.

[1663] An "English conversation learning tool" is a device or software that evaluates a user's pronunciation and supports English conversation practice.

[1664] The "learning recording means" is a device or software for recording and managing the user's learning data.

[1665] "Interface means" refers to a means by which a user accesses the system through a mobile device and uses various functions.

[1666] "Transmission means" refers to a device or software for transmitting voice data or text data to a server.

[1667] The "conversion means" is a device or software for converting voice data into text data in the server.

[1668] The "display means" is a device or software for displaying the generated text data or image data to the user.

[1669] The "image generation and transmission means" is a device or software that transmits the keywords entered by the user to the server and generates an image using the image generation API.

[1670] This invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal.

[1671] Voice recognition function

[1672] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a speech recognition API (for example, Google Cloud Speech-to-Text API) to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the speech recognition API generates and displays the text, "Tell me about Newton's laws."

[1673] Text-to-speech function

[1674] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API (for example, Amazon Polly or Google Text-to-Speech API) to convert the text data into speech data, and the generated speech data is sent from the server to the device and played. For example, if a user enters "Newton's third law," a voice of "Law of action and reaction" is generated and played using the TTS API.

[1675] Image generation function

[1676] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server generates related images using an image generation API (e.g., DALL-E or MidJourney) and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, a diagram of the related forces is generated and displayed.

[1677] English conversation learning function

[1678] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1679] Learning record function

[1680] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[1681] Smartphone compatible

[1682] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[1683] Below is an example of a prompt sentence to input to the generative AI model.

[1684] "Tell me about Newton's laws."

[1685] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[1686] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1687] Voice recognition function

[1688] Processing Step 1:

[1689] The user uses a microphone to input speech for learning.

[1690] Input: User's voice

[1691] Specific action: The user speaks into the smartphone microphone, "Tell me about Newton's laws."

[1692] Output: Captured audio data

[1693] Processing step 2:

[1694] The device captures the audio data and sends it to the server.

[1695] Input: Captured audio data

[1696] Specific operation: The device uploads the recorded audio data to a server via the Internet.

[1697] Output: Audio data sent to the server

[1698] Processing step 3:

[1699] The server uses a speech recognition API to convert the voice data into text data.

[1700] Input: Audio data sent to the server

[1701] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[1702] Specific operation: The voice recognition API analyzes the voice data and converts it into text data.

[1703] Output: Converted text data

[1704] Processing step 4:

[1705] The server transmits the converted text data to the terminal.

[1706] Input: Converted text data

[1707] Specific operation: The server returns text data to the terminal.

[1708] Output: Text data sent to the terminal

[1709] Processing Step 5:

[1710] The terminal displays the received text data to the user.

[1711] Input: Text data sent to the terminal

[1712] Specific action: Display the text "Tell me about Newton's laws" on the device screen.

[1713] Output: Displayed text data

[1714] Text-to-speech function

[1715] Processing Step 1:

[1716] The user enters the text they want to read into the device.

[1717] Input: Text entered by the user

[1718] Specific action: The user types "Newton's third law" on the smartphone keyboard.

[1719] Output: The input text data

[1720] Processing step 2:

[1721] The terminal transmits the input text data to the server.

[1722] Input: Entered text data

[1723] Specific operation: The device uploads text data to the server.

[1724] Output: Text data sent to the server

[1725] Processing step 3:

[1726] The server uses a text-to-speech API to convert the text data into audio data.

[1727] Input: Text data sent to the server

[1728] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[1729] Specific operation: Send an API request and generate audio data for "Newton's Third Law."

[1730] Output: Generated audio data

[1731] Processing step 4:

[1732] The server transmits the generated voice data to the terminal.

[1733] Input: Generated audio data

[1734] Specific operation: The server sends voice data to the terminal.

[1735] Output: Audio data sent to the device

[1736] Processing Step 5:

[1737] The terminal plays back the received audio data to the user.

[1738] Input: Audio data sent to the device

[1739] Specific action: The device plays an audio file and reads out "Newton's Third Law."

[1740] Output: Played audio data

[1741] Image generation function

[1742] Processing Step 1:

[1743] The user inputs keywords related to the content being studied into the terminal.

[1744] Input: The keyword entered by the user

[1745] Specific action: The user enters "action and reaction" into the smartphone.

[1746] Output: Entered keyword data

[1747] Processing step 2:

[1748] The terminal transmits the keyword data to the server.

[1749] Input: Entered keyword data

[1750] Specific operation: The terminal uploads keyword data to the server.

[1751] Output: Keyword data sent to the server

[1752] Processing step 3:

[1753] The server generates the relevant images using an image generation API.

[1754] Input: Keyword data sent to the server

[1755] Data processing: Use image generation API (e.g., DALL-E or MidJourney)

[1756] Specific operation: Send an API request and generate an image related to "action and reaction."

[1757] Output: Generated image data

[1758] Processing step 4:

[1759] The server transmits the generated image data to the terminal.

[1760] Input: Generated image data

[1761] Specific operation: The server sends an image file to the terminal.

[1762] Output: Image data sent to the device

[1763] Processing Step 5:

[1764] The terminal displays the received image data to the user.

[1765] Input: Image data sent to the device

[1766] Specific operation: Display the generated image on the device screen.

[1767] Output: Displayed image data

[1768] English conversation learning function

[1769] Processing Step 1:

[1770] The user inputs the English phrase they want to practice into the device.

[1771] Input: English phrase entered by the user

[1772] Specific action: The user types "How are you?" on the smartphone keyboard.

[1773] Output: Input phrase data

[1774] Processing step 2:

[1775] The device sends the entered phrase to the server.

[1776] Input: Entered phrase data

[1777] Specific operation: The device uploads phrase data to the server.

[1778] Output: Phrase data sent to the server

[1779] Processing step 3:

[1780] The server uses a text-to-speech API to generate audio data with the correct pronunciation.

[1781] Input: Phrase data sent to the server

[1782] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[1783] Specific behavior: Sends an API request to generate the correct pronunciation of "How are you?"

[1784] Output: Generated audio data

[1785] Processing step 4:

[1786] The server transmits the generated voice data to the terminal.

[1787] Input: Generated audio data

[1788] Specific operation: The server sends voice data to the terminal.

[1789] Output: Audio data sent to the device

[1790] Processing Step 5:

[1791] The terminal plays back the received voice data, and the user pronounces and records the voice accordingly.

[1792] Input: Audio data sent to the device

[1793] Specific operation: The correct pronunciation generated on the device is played back, and the user speaks into the microphone and the audio is recorded.

[1794] Output: Recorded audio data

[1795] Processing Step 6:

[1796] The device sends the recorded pronunciation data to the server.

[1797] Input: Recorded audio data

[1798] Specific operation: The device sends the recorded data to the server.

[1799] Output: Recording data sent to the server

[1800] Processing Step 7:

[1801] The server uses a speech recognition API to evaluate the user's pronunciation and generate a result.

[1802] Input: Recording data sent to the server

[1803] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[1804] Specific operations: Send an API request, convert the user's pronunciation into text, and perform an evaluation.

[1805] Output: Evaluation result data

[1806] Processing Step 8:

[1807] The server transmits the evaluation results to the terminal, which displays them to the user.

[1808] Input: Evaluation result data

[1809] Specific operation: The server sends the evaluation results to the terminal, and the terminal displays the results on the screen.

[1810] Output: Displayed evaluation results

[1811] Learning record function

[1812] Processing Step 1:

[1813] When a user starts studying, the terminal records the study time and content.

[1814] Input: Study start information (time, content)

[1815] Specific operation: The device stores the user's learning start time and learning content in a log.

[1816] Output: Recorded training data

[1817] Processing step 2:

[1818] The terminal periodically transmits the recorded data to the server.

[1819] Input: Recorded training data

[1820] Specific operation: The device periodically uploads learning data to the server.

[1821] Output: Training data sent to the server

[1822] Processing step 3:

[1823] The server stores the received learning data in a database.

[1824] Input: Training data sent to the server

[1825] Specific operation: The server inserts the received data into the corresponding table in the database.

[1826] Output: Training data stored in a database

[1827] Processing step 4:

[1828] When a user wants to check his / her past learning history, the terminal sends a request to the server, the server searches for learning history data and sends it to the terminal, and the terminal displays the data to the user.

[1829] Input: User request (e.g., learning history for a specific date)

[1830] Specific operation: The device sends a request to the server to confirm "what was learned on October 1st," and the server searches the database for the relevant data and sends it.

[1831] Output: Displayed learning history data

[1832] Mobile device compatible features

[1833] Processing Step 1:

[1834] The user launches the app on their smartphone.

[1835] Input: App launch operation

[1836] Specific action: The user taps the app icon on their smartphone to launch the app.

[1837] Output: App launch status

[1838] Processing step 2:

[1839] The device displays the main menu.

[1840] Input: App running state

[1841] Specific operation: Links to various functions will be displayed on the app's home screen.

[1842] Output: Main menu displayed

[1843] Processing step 3:

[1844] The user selects the required function, and communication is made from the device to the server, providing the necessary data and feedback in real time.

[1845] Input: User selected function

[1846] Specific operation: Select the English conversation practice mode during your commute, and the device will communicate with the server to provide pronunciation practice.

[1847] Output: Real-time data and feedback

[1848] The above is the specific processing flow for each function.

[1849] (Application example 1)

[1850] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1851] Conventional security systems can make it difficult for users to quickly identify abnormalities and respond appropriately. In particular, delays in detecting suspicious individuals and issuing security alerts can delay effective responses. Furthermore, because these systems are not interactive, users often cannot directly communicate with the system to check the situation or issue instructions.

[1852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1853] In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, a voice evaluation means for evaluating the user's pronunciation, a recording means for recording and managing the learning data, an interface means compatible with the mobile device, a notification means for notifying the user of security events by voice, an analysis means for performing image analysis, and a log management means for recording security events in a log file. This allows the user to interact with the security system through voice commands, check surveillance footage in real time, and respond quickly.

[1854] The "voice recognition means" is a means for converting voice data into text data.

[1855] The "text-to-speech means" is a means for converting text data into voice data.

[1856] "Image generation means" refers to means for generating images related to learning content or other content.

[1857] The "voice evaluation means" is a means for evaluating the user's pronunciation.

[1858] "Recording means" refers to a means for recording and managing learning data and other data.

[1859] "Interface means" refers to means for providing a corresponding interface to a mobile device or other device.

[1860] The "notification means" is a means for notifying the user of a security event by voice.

[1861] "Analysis means" refers to means for analyzing image data and other data.

[1862] A "log management means" is a means for recording and managing security events in a log file.

[1863] The present invention relates to a security support system, which is an integrated system that provides multiple functions such as voice recognition, text-to-speech, image generation, voice evaluation, recording and management, mobile device compatibility, notification, and analysis. Specific embodiments of this system are described below.

[1864] Voice recognition means

[1865] When a user issues a voice command, the device captures the voice data and sends it to the server. The server uses a voice recognition API (e.g., Google Cloud Speech-to-Text) to convert the voice data into text data, and returns the text data to the device, which then displays it. For example, if a user says, "Detect a suspicious person," the voice recognition API generates the text "Detect a suspicious person" and displays it on the device.

[1866] Text-to-speech means

[1867] When the server detects an important security notification, it converts the text data into audio data. The server then uses a text-to-speech API (e.g., Google Cloud Text-to-Speech) to send the generated audio data to the device and notify the user by voice. For example, the generated text might say, "A suspicious person has been detected. Please be careful," and be played back as audio.

[1868] Image Generation Means

[1869] When a user wants to check a specific image, they enter a relevant keyword into their device and send it to the server. The server uses an image generation API to generate an image related to the keyword and sends it to the device. For example, if "suspicious person's face" is entered as a keyword, an image that identifies the suspicious person's face will be generated and displayed on the device.

[1870] Audio evaluation tools

[1871] When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, a user can record and send "How are you?" and the pronunciation will be evaluated to see if it is appropriate.

[1872] Recording means

[1873] Every time a user uses the system, data on related security events is sent to the server and recorded. This allows past security history to be checked. For example, the date, time, and location of any suspicious activity are recorded.

[1874] Interface Means

[1875] The system is compatible with mobile devices, allowing users to access the system anytime, anywhere. When users launch the app on their smartphone, an interface is displayed that allows them to access security functions. For example, they can check security camera footage while commuting.

[1876] Notification means

[1877] When a security event occurs, the system notifies the user with an audio alert. The server converts text into audio and sends it to the device where it is played. For example, a voice alert saying "A suspicious person has been detected" is played.

[1878] Analysis means

[1879] The video captured by the device's camera is sent to the server, and the server uses image analysis software (e.g., OpenCV) to detect suspicious individuals. If a suspicious individual is detected, the information is notified to the user. For example, the face of a suspicious individual can be automatically analyzed based on the camera image.

[1880] Log management methods

[1881] Security event logs are managed on the server and recorded in log files. When a user wants to check past security events, they send a request to the server and the corresponding log data is displayed on the terminal. For example, it can be used to "check details of suspicious individuals detected on October 1st."

[1882] Examples of concrete examples and prompts

[1883] As a specific example of use, when a user says to their smartphone, "Detect suspicious person," the app analyzes the camera image, and if a suspicious person is detected, it notifies them with a voice message saying, "Suspicious person detected," and records the event in a log.

[1884] Prompt Sentence Examples

[1885] User: Suspicious person detected

[1886] App: Analyzing camera images...

[1887] App: Suspicious person detected. Please be careful.

[1888] The above is a specific description of the embodiment of the present invention, which enables users to take security measures quickly and effectively.

[1889] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1890] Step 1:

[1891] The user speaks a voice command, which is captured as voice data by the smartphone's microphone. The input is the voice command, and the output is the captured voice data.

[1892] Step 2:

[1893] The device sends the captured voice data to the server for conversion to text data using a speech recognition API. The server then converts the voice data to text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). The input is the voice data, and the output is the converted text data.

[1894] Step 3:

[1895] The server analyzes the text data and determines a specific action. For example, if the text contains the phrase "detect a suspicious person," the server will instruct the start of image analysis. The input is the text data, and the output is the analysis result and the decision on the next action.

[1896] Step 4:

[1897] The device's camera captures real-time video and sends the image data to the server. The input is the image data captured by the camera, and the output is the image data sent to the server.

[1898] Step 5:

[1899] The server uses image analysis software (e.g., OpenCV) to analyze the captured image data and detect suspicious individuals. The input is the image data, and the output is the analysis result indicating whether or not there is a suspicious individual.

[1900] Step 6:

[1901] When the server detects a suspicious person, it sends the information to the terminal as text and simultaneously converts it into audio data using a text-to-speech API. The input is text data and the output is audio data.

[1902] Step 7:

[1903] The device receives the voice data and issues a voice notification to the user from the speaker. For example, a voice alert saying "A suspicious person has been detected. Please be careful" is played. The input is voice data, and the output is a voice notification.

[1904] Step 8:

[1905] Details of security events are recorded on the server and saved in a log file through a log management tool. The event date and time, detected details, etc. are recorded in the log so that users can check them later. The input is security event information, and the output is a log file.

[1906] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1907] The present invention is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal. An example implementation is described below.

[1908] Voice recognition function

[1909] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[1910] Text-to-speech function

[1911] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[1912] Image generation function

[1913] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[1914] English conversation learning function

[1915] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[1916] Learning record function

[1917] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[1918] Emotion recognition function using emotion engine

[1919] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's research and learning situation. For example, if a user mutters, "This is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as, "Try taking a break to resolve this."

[1920] Smartphone compatible

[1921] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[1922] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[1923] The processing flow will be explained below.

[1924] Voice recognition function

[1925] Step 1:

[1926] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[1927] Step 2:

[1928] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[1929] Step 3:

[1930] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[1931] Step 4:

[1932] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[1933] Step 5:

[1934] The server records the generated text data in a database and returns the data to the terminal.

[1935] Step 6:

[1936] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[1937] Text-to-speech function

[1938] Step 1:

[1939] The user types the text they want to hear into the device. For example, type "Newton's third law."

[1940] Step 2:

[1941] The terminal sends the entered text data to the server using a POST request.

[1942] Step 3:

[1943] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[1944] Step 4:

[1945] The server transmits the generated voice data to the terminal.

[1946] Step 5:

[1947] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[1948] Image generation function

[1949] Step 1:

[1950] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[1951] Step 2:

[1952] The terminal sends the entered keyword to the server via a POST request.

[1953] Step 3:

[1954] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[1955] Step 4:

[1956] The server transmits the generated image data to the terminal.

[1957] Step 5:

[1958] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[1959] English conversation learning function

[1960] Step 1:

[1961] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[1962] Step 2:

[1963] The terminal sends the entered phrase to the server via a POST request.

[1964] Step 3:

[1965] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[1966] Step 4:

[1967] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[1968] Step 5:

[1969] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[1970] Step 6:

[1971] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[1972] Step 7:

[1973] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[1974] Learning record function

[1975] Step 1:

[1976] When the user starts studying, the terminal starts recording the study time and content.

[1977] Step 2:

[1978] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[1979] Step 3:

[1980] The server stores the received data in a database.

[1981] Step 4:

[1982] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[1983] Step 5:

[1984] The server searches the learning history from the database and sends it to the terminal.

[1985] Step 6:

[1986] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[1987] Emotion recognition function using emotion engine

[1988] Step 1:

[1989] The user expresses their emotions through voice and facial expressions. For example, they may say, "It's difficult."

[1990] Step 2:

[1991] The device captures and temporarily saves the user's voice and facial expression data. For example, it saves the facial expression data as .jpg and the voice emotion file as .wav.

[1992] Step 3:

[1993] The device sends this data to the server. For example, send facial expression data (.jpg) or voice emotion file (.wav) to the server using a POST request.

[1994] Step 4:

[1995] The server sends the received data to the emotion engine, which analyzes the user's emotions. For example, it recognizes "frustration."

[1996] Step 5:

[1997] The server sends the emotion analysis results to the device and provides feedback to the user, such as "Let's take a break to solve this."

[1998] Smartphone compatible

[1999] Step 1:

[2000] The user launches the application on the smartphone.

[2001] Step 2:

[2002] The device displays the main menu and provides access to each function.

[2003] Step 3:

[2004] The user selects any function, such as English conversation learning or image generation, and begins learning.

[2005] Step 4:

[2006] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[2007] Step 5:

[2008] The terminal displays the received results to the user and provides learning according to the instructions.

[2009] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[2010] Example 2

[2011] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2012] In today's learning environment, learners are required to efficiently acquire knowledge using a variety of media and tools. However, most learning systems are specialized in individual functions, and few provide comprehensive support across a wide range of areas, such as speech recognition, text-to-speech, image generation, English conversation learning, and emotion recognition. Furthermore, functions such as learning support on mobile devices and storing learning records are not consistently integrated. This makes it difficult for learners to obtain a learning experience that meets their diverse needs.

[2013] The specification process by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a voice recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to the learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an emotion recognition means for analyzing the user's emotions, and an interface means compatible with mobile devices. This makes it possible to provide comprehensive support for a variety of learning needs.

[2014] "Speech recognition means" refers to a device or software that has the function of analyzing voice data and converting it into text data.

[2015] A "text-to-speech means" is a device or software that has the function of analyzing text data and converting it into audio data.

[2016] "Image generation means" refers to a device or software that has the function of automatically generating related images based on keywords or content entered by a user.

[2017] An "English conversation learning tool" is a device or software that has the function of providing audio with correct pronunciation for English phrases entered by a user, evaluating the user's pronunciation, and providing feedback.

[2018] A "learning recording means" refers to a device or software that has the function of recording and managing a user's learning activities and learning data.

[2019] "Emotion recognition means" refers to devices or software that have the function of analyzing and evaluating emotions from the user's voice and facial expressions.

[2020] "Mobile device compatible interface means" refers to user interfaces and related communication means designed for use on mobile devices such as smartphones and tablets.

[2021] This is an integrated system for supporting learners' learning, providing functions such as voice recognition, text-to-speech, image generation, English conversation learning, learning records, mobile device compatibility, and emotion recognition using an emotion engine. This system realizes each function by utilizing communication between a server and a terminal.

[2022] Voice recognition function

[2023] When a user provides voice input for training, the device captures this voice data and sends it to the server. The server then uses a speech recognition API to convert the voice data into text data. This can be done using services such as the Google Cloud Speech-to-Text API or Amazon Transcribe. The converted text data is then sent back from the server to the device, which then displays the text to the user.

[2024] Specific examples

[2025] When a user says, "Tell me about Newton's Law," the speech recognition API generates the text "Tell me about Newton's Law," which is displayed on the device.

[2026] Prompt Sentence Examples

[2027] "Explain Newton's First Law."

[2028] Text-to-speech function

[2029] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server then uses a text-to-speech API to convert the text data into audio data. Specifically, Google Cloud Text-to-Speech API or Amazon Polly can be used. The generated audio data is sent from the server to the device and played back.

[2030] Specific examples

[2031] When a user types "Newton's third law," a text-to-speech API is used to generate and play an audio version of "The Law of Action and Reaction."

[2032] Prompt Sentence Examples

[2033] Please read the following text aloud: "The law of action and reaction states that for every force there is an equal and opposite force acting in the opposite direction."

[2034] Image generation function

[2035] When a user enters keywords related to the content they are learning, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. Specifically, DALL-E or Stable Diffusion can be used. The device then displays the generated images to the user.

[2036] Specific examples

[2037] When the user enters the keyword "action and reaction," a diagram of the relevant forces is generated and displayed.

[2038] Prompt Sentence Examples

[2039] "Create an image that illustrates action and reaction forces."

[2040] English conversation learning function

[2041] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. Google Cloud Text-to-Speech API or Amazon Polly can be used. This audio data is sent to the device and played back. When the user records their own pronunciation and sends the data to the server, the server uses a speech recognition API to evaluate the pronunciation and sends the results to the device.

[2042] Specific examples

[2043] When a user wishes to practice "How are you?", a correct pronunciation audio is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[2044] Prompt Sentence Examples

[2045] "Generate audio to learn the correct pronunciation of 'How are you?'"

[2046] Learning record function

[2047] When a user starts studying, the device records the study time and content. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user.

[2048] Specific examples

[2049] When a user checks "What did I learn on October 1st?", the corresponding content is displayed.

[2050] Prompt Sentence Examples

[2051] "Please show me what you learned on October 1st."

[2052] Emotion recognition function using emotion engine

[2053] When a user expresses their emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion engine to analyze the user's emotions from the voice and facial data. Specifically, it can use the Microsoft Azure Emotion API or Google Cloud Vision API. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's learning status.

[2054] Specific examples

[2055] If a user says "this is difficult," the emotion engine will recognize "frustration" from the voice and provide feedback such as "Let's take a break to solve this."

[2056] Prompt Sentence Examples

[2057] "Please analyze the user's emotions from this voice data."

[2058] Smartphone compatible

[2059] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time.

[2060] Specific examples

[2061] The user can select the English conversation practice mode and practice pronunciation while commuting.

[2062] Prompt Sentence Examples

[2063] "Select English conversation practice mode and start practicing your pronunciation during your commute."

[2064] This allows the present invention to meet the needs of various learners and provide efficient and comprehensive learning support. Each function is realized through the cooperation of the server and terminal, and users can access it from anywhere. This system provides a variety of means to improve the learning experience in an integrated manner.

[2065] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2066] Voice recognition function

[2067] Step 1:

[2068] The user performs voice input for learning.

[2069] Specific Action: The user speaks into the microphone, "Tell me about Newton's Laws."

[2070] Step 2:

[2071] The device captures the audio data.

[2072] What it does: The device's microphone records the user's voice in real time.

[2073] Input: User's voice

[2074] Output: Recorded audio data

[2075] Step 3:

[2076] The device sends the captured audio data to the server.

[2077] Specific operation: The device encrypts the recorded data and sends it to the specified endpoint on the server.

[2078] Input: Audio data

[2079] Output: Audio data sent to the server

[2080] Step 4:

[2081] The server uses a speech recognition API to convert the voice data into text data.

[2082] Specific operation: The server passes the audio data to the Google Cloud Speech-to-Text API and retrieves the text data.

[2083] Input: Audio data sent to the server

[2084] Output: Converted text data

[2085] Step 5:

[2086] The server returns the converted text data to the terminal.

[2087] Specific operation: The server encrypts the converted text data and sends it to the device.

[2088] Input: Converted text data

[2089] Output: Text data sent to the terminal

[2090] Step 6:

[2091] The terminal displays the text data to the user.

[2092] What it does: The device displays the received text on the screen, informing the user, "Tell me about Newton's laws."

[2093] Input: Text data sent to the terminal

[2094] Output: Text displayed on the screen

[2095] Text-to-speech function

[2096] Step 1:

[2097] The user enters the text they want to read into the device or selects existing text.

[2098] Specific action: The user types "Newton's third law" into the terminal.

[2099] Step 2:

[2100] The terminal transmits the text data to the server.

[2101] Specific operation: The device encrypts the text data entered and sends it to the server.

[2102] Input: Text data entered by the user into the terminal

[2103] Output: Text data sent to the server

[2104] Step 3:

[2105] The server uses a text-to-speech API to convert the text data into audio data.

[2106] Specific operation: The server sends text data to the Google Cloud Text-to-Speech API, which generates audio data.

[2107] Input: Text data sent to the server

[2108] Output: Generated audio data

[2109] Step 4:

[2110] The server transmits the generated voice data to the terminal.

[2111] Specific operation: The server encrypts the generated audio data and sends it to the device.

[2112] Input: Generated audio data

[2113] Output: Audio data sent to the device

[2114] Step 5:

[2115] The device plays the audio data.

[2116] Specific operation: The device plays the received audio data and reads out the "Law of Action and Reaction" to the user.

[2117] Input: Audio data sent to the device

[2118] Output: Played audio

[2119] Image generation function

[2120] Step 1:

[2121] The user inputs keywords related to the content being studied into the terminal.

[2122] Specific operation: The user enters the keywords "action and reaction" into the terminal.

[2123] Step 2:

[2124] The device sends the keyword to the server.

[2125] Specific operation: The device encrypts the keyword and sends it to the server.

[2126] Input: Keywords entered by the user into the device

[2127] Output: Keywords sent to the server

[2128] Step 3:

[2129] The server generates the relevant images using an image generation API.

[2130] What it does: The server passes keywords to the DALL-E API and generates related images.

[2131] Input: The keyword sent to the server

[2132] Output: Generated image data

[2133] Step 4:

[2134] The server transmits the generated image to the terminal.

[2135] Specific operation: The server encrypts the generated image data and sends it to the terminal.

[2136] Input: Generated image data

[2137] Output: Image data sent to the device

[2138] Step 5:

[2139] The terminal displays the image to the user.

[2140] Specific operation: The device displays the image it receives on the screen, showing the user an "image illustrating the law of action and reaction."

[2141] Input: Image data sent to the device

[2142] Output: Image displayed on the screen

[2143] English conversation learning function

[2144] Step 1:

[2145] The user inputs the English phrase they want to practice into the device.

[2146] What happens: The user types "How are you?" into the terminal.

[2147] Step 2:

[2148] The device sends the entered phrase to the server.

[2149] Specific operation: The device encrypts the entered phrase and sends it to the server.

[2150] Input: The phrase the user types into the terminal

[2151] Output: Phrase sent to server

[2152] Step 3:

[2153] The server uses the TTS API to generate audio data with the correct pronunciation.

[2154] What happens: The server passes the phrase to the Google Cloud Text-to-Speech API, which generates audio data with the correct pronunciation.

[2155] Input: Phrase sent to server

[2156] output: Generated audio data with correct pronunciation

[2157] Step 4:

[2158] The server transmits the generated voice data to the terminal.

[2159] Specific operation: The server encrypts the generated audio data and sends it to the device.

[2160] Input: Generated audio data

[2161] Output: Audio data sent to the device

[2162] Step 5:

[2163] The device plays the audio data.

[2164] Specific operation: The device plays back the received audio data and provides the user with the correct pronunciation.

[2165] Input: Audio data sent to the device

[2166] Output: Played audio

[2167] Step 6:

[2168] The user records their own pronunciation and sends it from the terminal to the server.

[2169] Specific operation: The user records their own pronunciation and sends it to the server via their device.

[2170] Input: User recording data

[2171] Output: Recording data sent to the server

[2172] Step 7:

[2173] The server evaluates the pronunciation using a speech recognition API.

[2174] What happens: The server passes the recording to the Google Cloud Speech-to-Text API and evaluates the pronunciation.

[2175] Input: Recording data sent to the server

[2176] Output: Evaluation result

[2177] Step 8:

[2178] The server transmits the evaluation results to the terminal.

[2179] Specific operation: The server encrypts the evaluation results and sends them to the device.

[2180] Input: Evaluation result

[2181] Output: Evaluation results sent to the device

[2182] Step 9:

[2183] The terminal displays the evaluation results to the user.

[2184] Specific behavior: The device displays the evaluation results it receives on the screen and informs the user that "your pronunciation was accurate" or "your pronunciation needs improvement."

[2185] Input: Evaluation result sent to the terminal

[2186] Output: Evaluation results displayed on the screen

[2187] Learning record function

[2188] Step 1:

[2189] The user begins learning.

[2190] What happens: A user starts a new learning session in the app.

[2191] Step 2:

[2192] The device records the study time and content.

[2193] Specific operation: The device records the start time of learning and saves the learning content as it progresses.

[2194] Input: Start learning information and learning content

[2195] Output: Training data recorded on the device

[2196] Step 3:

[2197] The device periodically transmits learning data to the server.

[2198] Specific operation: The device encrypts and sends learning data to the server once every 15 minutes.

[2199] Input: Training data recorded on the device

[2200] output: Training data sent to the server

[2201] Step 4:

[2202] The server stores the learning data in a database.

[2203] Specific operation: The server stores the received learning data in a database.

[2204] Input: Training data sent to the server

[2205] output: Training data stored in the database

[2206] Step 5:

[2207] When a user wants to check his / her past learning history, he / she sends a request from the terminal to the server.

[2208] Specific operation: The user sends a request to the server from the device to check the learning history for a specified date in the app.

[2209] Input: Request (e.g., Confirm learning content for October 1st)

[2210] Output: Request sent to the server

[2211] Step 6:

[2212] The server searches the database for learning history data and sends it to the terminal.

[2213] Specific operation: The server searches the database for the learning history for the specified date, encrypts it, and sends it to the device.

[2214] Input: Request

[2215] output: Learning history data sent to the device

[2216] Step 7:

[2217] The terminal displays the learning history data to the user.

[2218] Specific operation: The device displays the learning history data it has received on the screen and informs the user, "The content learned on October 1st was as follows: The basics of Newton's laws."

[2219] Input: Learning history data sent to the device

[2220] Output: Learning history data displayed on the screen

[2221] Emotion recognition function using emotion engine

[2222] Step 1:

[2223] The user expresses their feelings through voice and facial expressions while learning.

[2224] Specific action: The user tweets "This is difficult" while studying.

[2225] Step 2:

[2226] The device captures this data and sends it to the server.

[2227] Specific operation: The device records the user's voice, captures facial expression data with the camera, encrypts it, and sends it to the server.

[2228] Input: Voice data and facial expression data

[2229] Output: Voice and facial expression data sent to the server

[2230] Step 3:

[2231] The server uses an emotion engine to analyze the user's emotions from voice data and facial expression data.

[2232] What it does: The server analyzes voice and facial expression data using the Microsoft Azure Emotion API.

[2233] Input: Voice data and facial expression data sent to the server

[2234] output: Analyzed emotion data

[2235] Step 4:

[2236] The server sends the analysis results to the device.

[2237] Specific operation: The server encrypts the analysis results and sends them to the device.

[2238] Input: Parsed emotion data

[2239] Output: Analysis results sent to the device

[2240] Step 5:

[2241] The device provides feedback and advice to the user based on the analysis results.

[2242] Specific operation: Based on the analysis results received by the device, the device displays to the user, "You are feeling frustrated. Please take a break and refresh yourself."

[2243] Input: Analysis results sent to the device

[2244] Output: Feedback provided to the user

[2245] Smartphone compatible

[2246] Step 1:

[2247] The user launches the app on their smartphone.

[2248] Specific behavior: A user launches a learning app on their smartphone while commuting to work.

[2249] Step 2:

[2250] The device displays the main menu.

[2251] Specific operation: The device will display the main menu and present icons for various functions.

[2252] Input: None (when the app starts)

[2253] Output: Main menu displayed on the terminal screen

[2254] Step 3:

[2255] Users access various functions.

[2256] Specific operation: The user selects "English conversation practice."

[2257] Step 4:

[2258] The terminal sends a request to the server according to the selected function.

[2259] What happens: Your device encrypts and sends a request to the server based on your selection.

[2260] Input: User selection (e.g., English conversation practice)

[2261] Output: Request sent to the server

[2262] Step 5:

[2263] The server generates data and feedback according to the request and sends it back to the device.

[2264] Specific operation: The server generates data and feedback based on the request and sends it to the device.

[2265] Input: The request sent to the server

[2266] Output: Data or feedback sent to the device

[2267] Step 6:

[2268] The device provides the received data and feedback to the user.

[2269] Specific operation: The device provides the user with English conversation practice questions and feedback in real time based on the received data.

[2270] Input: Data and feedback sent to the device

[2271] Output: Exercises and feedback displayed on the screen

[2272] (Application example 2)

[2273] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2274] Conventional learning support systems often provide individual functions independently, resulting in a lack of coordination between them. Furthermore, they do not provide feedback that takes into account the user's emotions, which can lead to problems with reduced learning efficiency and motivation. Furthermore, there is a lack of systems that support face-to-face learning in physical stores, preventing improvements to the user experience.

[2275] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a speech recognition means for converting voice data into text data, a text-to-speech means for converting text data into voice data, an image generation means for generating images related to learning content, an English conversation learning means for evaluating the user's pronunciation, a learning recording means for recording and managing learning data, an interface means compatible with mobile devices, an emotion recognition means for analyzing the user's emotions, and a means for generating images based on prompts. This allows each function to work in conjunction with each other, providing feedback according to the user's emotional state, improving learning efficiency and motivation, and further enabling the realization of a system that can accommodate face-to-face learning in physical stores.

[2276] A "voice recognition means" is a device or system for converting voice data into text data.

[2277] A "text-to-speech means" is a device or system that converts text data into audio data and provides information by voice.

[2278] An "image generator" is a device or system for generating visual images based on keywords or prompts related to the learning content.

[2279] An "English conversation learning means" is a device or system for evaluating a user's pronunciation and supporting the improvement of their English conversation skills.

[2280] The "learning recording means" is a device or system for recording and managing learning data and saving the user's learning history.

[2281] An "interface means" is a device or system that is compatible with a mobile device and allows a user to access and use the system.

[2282] An "emotion recognition means" is a device or system that analyzes the user's emotions from their voice and facial expressions and provides appropriate feedback and advice.

[2283] A "means for generating images based on prompts" is a device or system for generating associated images in response to prompts entered by a user.

[2284] This invention is a system that integrates multiple learning support functions, such as voice recognition, text-to-speech, image generation, English conversation learning, learning record, mobile device compatibility, and emotion recognition. This system realizes each function by utilizing communication between a server and a terminal.

[2285] Voice Recognition

[2286] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data and returns the text data to the device. For example, if a user says, "Tell me about Newton's Laws," the voice recognition API generates the text "Tell me about Newton's Laws" and displays it.

[2287] Text-to-speech

[2288] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and sends the generated audio data to the device for playback. For example, if a user enters "Newton's third law," the text-to-speech API generates and plays back the audio for "Law of action and reaction."

[2289] Image generation

[2290] When a user inputs a prompt related to the learning content, the device sends it to the server. The server uses an image generation API to generate a related image and sends it to the device. For example, when the user inputs the prompt "action and reaction," a diagram of the related force is generated and displayed.

[2291] English conversation learning

[2292] When a user wants to practice English conversation, they input the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. This audio data is sent to the device and played back. The user can also record their own pronunciation and send it to the server, which then evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user practices "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[2293] Learning Record

[2294] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, the device sends a request to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if the user checks "What did they study on October 1st?", the corresponding content will be displayed.

[2295] emotion recognition

[2296] When a user expresses emotions through voice or facial expressions while studying, the device captures this data and sends it to the server. The server then uses an emotion recognition API to analyze the user's emotions from the voice and facial data. The analysis results are sent from the server to the device, which provides appropriate feedback and advice based on the user's emotional state. For example, if a user tweets "This is difficult," the emotion recognition API will recognize "frustration" from the voice and provide feedback such as "Try taking a break."

[2297] Specific examples

[2298] An example of a prompt is "Generate an image that illustrates Newton's laws." This prompt causes the system to call an image generation API to generate an image that visually illustrates Newton's laws. The generated image is displayed to the user in real time.

[2299] This allows the system's functions to work together and provide feedback based on the user's emotional state, improving learning efficiency and motivation, and providing comprehensive learning support that can also be used for face-to-face learning in physical stores.

[2300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2301] Step 1:

[2302] The user inputs a learning question by voice. The device acquires this voice data and sends it to the server. The server uses a speech recognition API to convert the voice data into text data and sends the text data back to the device. The input is the user's voice data, and the output is text data. Specifically, if you say, "Tell me about Newton's laws," the text "Tell me about Newton's laws" is generated.

[2303] Step 2:

[2304] The user checks the text displayed on the device and selects the text they wish to have read aloud. The device sends the text data to the server, which then converts it into audio data using a text-to-speech API. The converted audio data is sent to the device and played back. The input is text data, and the output is audio data. For example, if you enter "Newton's third law," an audio version of "The Law of Action and Reaction" will be generated and played back.

[2305] Step 3:

[2306] The user inputs a prompt sentence related to the learning content. The device sends this prompt sentence to the server, and the server generates a related image using an image generation API. The generated image is sent to the device and displayed to the user. The input is the prompt sentence, and the output is image data. Specifically, if the user inputs "Generate an image that explains Newton's law," an image that visually shows Newton's law is generated.

[2307] Step 4:

[2308] To practice English conversation, the user inputs the English phrase they want to practice into their device. The device sends the text to the server, which uses a text-to-speech API to generate audio data with the correct pronunciation. The generated audio data is sent to the device and played back. The input is the text data of the English phrase, and the output is audio data. For example, if you input "How are you?", audio with the correct pronunciation will be generated.

[2309] Step 5:

[2310] The user records their own pronunciation and sends the audio data from their device to the server. The server evaluates the user's pronunciation using a speech recognition API and sends the evaluation results to the device. The input is the user's voice data, and the output is the evaluation result data. Specifically, when the user says "How are you?", the evaluation results of the pronunciation are fed back.

[2311] Step 6:

[2312] When a user begins studying, the device records the study time and content, and periodically sends this data to the server. The server saves it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data, sends it to the device, and displays it to the user. The input is the study data, and the output is the history data. Specifically, if a user requests "What did I study on October 1st?", the corresponding content will be displayed.

[2313] Step 7:

[2314] When a user expresses emotions through voice or facial expression while studying, the device captures this data and sends it to the server. The server uses an emotion recognition API to analyze emotions from the voice and facial data and sends the analysis results to the device. Based on the analysis results, the device provides appropriate feedback and advice to the user. The input is emotional data, and the output is feedback data. Specifically, if the user mutters "this is difficult," the device will recognize "frustration" from the voice and provide feedback such as "take a break."

[2315] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2316] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2317] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2318] [Fourth embodiment]

[2319] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2320] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2321] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2322] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2323] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2324] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2325] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2326] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2327] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2328] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2329] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2330] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2331] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2332] The present invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal. An example implementation of this system is described below.

[2333] Voice recognition function

[2334] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a voice recognition API to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the voice recognition API generates and displays the text, "Tell me about Newton's laws."

[2335] Text-to-speech function

[2336] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API to convert the text data into audio data, and the generated audio data is sent from the server to the device and played. For example, if a user enters "Newton's third law," the TTS API is used to generate and play audio for "Law of action and reaction."

[2337] Image generation function

[2338] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server uses an image generation API to generate related images and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, an illustration of the related forces is generated and displayed.

[2339] English conversation learning function

[2340] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[2341] Learning record function

[2342] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[2343] Smartphone compatible

[2344] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[2345] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[2346] The processing flow will be explained below.

[2347] Voice recognition function

[2348] Step 1:

[2349] The user provides voice input for learning. For example, say, "Today's lesson is about the laws of physics."

[2350] Step 2:

[2351] The device captures the audio input and stores the audio data temporarily, e.g., as a .wav audio file.

[2352] Step 3:

[2353] The device sends this audio data to the server. Example: Send the audio file .wav to the server with a POST request.

[2354] Step 4:

[2355] The server sends the received voice data to the speech recognition API, which converts the voice into text data. Example: The speech recognition API generates the text "Today's lesson is about the laws of physics."

[2356] Step 5:

[2357] The server records the generated text data in a database and returns the data to the terminal.

[2358] Step 6:

[2359] The device displays the received text data to the user. For example, "Today's lesson is about the laws of physics."

[2360] Text-to-speech function

[2361] Step 1:

[2362] The user types the text they want to hear into the device. For example, type "Newton's third law."

[2363] Step 2:

[2364] The terminal sends the entered text data to the server using a POST request.

[2365] Step 3:

[2366] The server sends the received text data to the text-to-speech API to generate audio data. For example, audio data is generated as .mp3.

[2367] Step 4:

[2368] The server transmits the generated voice data to the terminal.

[2369] Step 5:

[2370] The device plays back the audio data it has received. Example: "Newton's Third Law" is read aloud.

[2371] Image generation function

[2372] Step 1:

[2373] The user enters keywords related to the content of the lesson into the terminal. For example, enter "image of action and reaction forces."

[2374] Step 2:

[2375] The terminal sends the entered keyword to the server via a POST request.

[2376] Step 3:

[2377] The server sends the received keywords to the image generation API and generates a related image. For example, it is generated as image data.png.

[2378] Step 4:

[2379] The server transmits the generated image data to the terminal.

[2380] Step 5:

[2381] The device displays the received image data to the user. For example, an illustration of action and reaction forces is displayed.

[2382] English conversation learning function

[2383] Step 1:

[2384] The user enters the English phrase they want to practice into the device. For example, enter "How are you?"

[2385] Step 2:

[2386] The terminal sends the entered phrase to the server via a POST request.

[2387] Step 3:

[2388] The server sends the received phrase to the TTS API and generates audio data with the correct pronunciation. For example, audio data _how_are_you.mp3 is generated.

[2389] Step 4:

[2390] The server sends the generated voice data to the device, which then plays it back. Example: Correct pronunciation voice is played back.

[2391] Step 5:

[2392] The user records their own pronunciation, and the device sends the recorded data to the server via a POST request.

[2393] Step 6:

[2394] The server sends the received recording data to the speech recognition API, converts it into text data, and evaluates the pronunciation. For example, it generates an evaluation result of "Accuracy: 95%."

[2395] Step 7:

[2396] The server sends the evaluation results to the device, which then displays them to the user. For example, "Your pronunciation is 95% accurate."

[2397] Learning record function

[2398] Step 1:

[2399] When the user starts studying, the terminal starts recording the study time and content.

[2400] Step 2:

[2401] The device sends the recorded data to the server at regular intervals. Example: {time: "10 minutes", content: "Laws of physics"}.

[2402] Step 3:

[2403] The server stores the received data in a database.

[2404] Step 4:

[2405] When a user wants to check the learning history, the user sends a request from the terminal to the server.

[2406] Step 5:

[2407] The server searches the learning history from the database and sends it to the terminal.

[2408] Step 6:

[2409] The device displays the learning history received to the user. For example, "October 1st: Laws of Physics - 10 minutes" is displayed.

[2410] Smartphone compatible

[2411] Step 1:

[2412] The user launches the application on the smartphone.

[2413] Step 2:

[2414] The device displays the main menu and provides access to each function.

[2415] Step 3:

[2416] The user selects any function, such as English conversation learning or image generation, and begins learning.

[2417] Step 4:

[2418] The terminal requests the server to perform the processing required for the selected function, and the server performs the processing and returns the results to the terminal.

[2419] Step 5:

[2420] The terminal displays the received results to the user and provides learning according to the instructions.

[2421] The above are the specific program processing steps for each function in the AI ​​Learner Companion system.

[2422] Example 1

[2423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2424] In conventional learning support systems, each function exists independently, making it difficult to provide comprehensive learning support. Furthermore, functions such as voice recognition, text-to-speech, image generation, English conversation learning, and learning records are provided separately, making it difficult for users to centrally manage their learning progress and history. Furthermore, use on mobile devices is limited, meaning that they can only provide a learning environment that is restricted by location and time.

[2425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2426] In this invention, the server includes a speech recognition unit that converts speech data into text data, a text-to-speech unit that converts text data into speech data, an image generation unit that generates images related to the learning content, an English conversation learning unit that evaluates the user's pronunciation, a learning record unit that records and manages learning data, an interface unit compatible with mobile devices, a transmission unit that captures speech data and transmits it to the server, a conversion unit that converts speech data into text data using a speech recognition API in the server, a display unit that displays the generated text data to the user, an image generation and transmission unit that transmits keywords entered by the user to the server and generates images using an image generation API, and a display unit that displays the generated images to the user. This provides integrated functions such as speech recognition, text-to-speech, image generation, English conversation learning, and learning record, allowing users to continue their learning anytime, anywhere.

[2427] A "voice recognition means" is a device or software for receiving voice data and converting it into text data.

[2428] "Text-to-speech means" refers to a device or software that converts input text data into audio data and plays back the audio.

[2429] The "image generating means" is a device or software for generating related images based on keywords entered by a user.

[2430] An "English conversation learning tool" is a device or software that evaluates a user's pronunciation and supports English conversation practice.

[2431] The "learning recording means" is a device or software for recording and managing the user's learning data.

[2432] "Interface means" refers to a means by which a user accesses the system through a mobile device and uses various functions.

[2433] "Transmission means" refers to a device or software for transmitting voice data or text data to a server.

[2434] The "conversion means" is a device or software for converting voice data into text data in the server.

[2435] The "display means" is a device or software for displaying the generated text data or image data to the user.

[2436] The "image generation and transmission means" is a device or software that transmits the keywords entered by the user to the server and generates an image using the image generation API.

[2437] This invention is an integrated system for supporting users' learning, providing functions such as speech recognition, text-to-speech, image generation, English conversation learning, learning records, and support for mobile devices. This system realizes each function by utilizing communication between a server and a terminal.

[2438] Voice recognition function

[2439] When a user provides voice input for learning, the device captures this voice data and sends it to the server. The server uses a speech recognition API (for example, Google Cloud Speech-to-Text API) to convert the voice data into text data. The converted text data is sent back from the server to the device, and the device displays the text to the user. For example, if a user says, "Tell me about Newton's laws," the speech recognition API generates and displays the text, "Tell me about Newton's laws."

[2440] Text-to-speech function

[2441] When a user enters text to be read aloud into the device or selects existing text, the device sends this text data to the server. The server uses a text-to-speech API (for example, Amazon Polly or Google Text-to-Speech API) to convert the text data into speech data, and the generated speech data is sent from the server to the device and played. For example, if a user enters "Newton's third law," a voice of "Law of action and reaction" is generated and played using the TTS API.

[2442] Image generation function

[2443] When a user enters keywords related to the learning content, the device sends these keywords to the server. The server generates related images using an image generation API (e.g., DALL-E or MidJourney) and sends them to the device. The device then displays the generated images to the user. For example, if "action and reaction" is entered as a keyword, a diagram of the related forces is generated and displayed.

[2444] English conversation learning function

[2445] When a user inputs an English phrase they want to practice into their device, the device sends it to the server, which then uses a TTS API to generate audio data with the correct pronunciation. This audio data is then sent to the device and played back. When a user records their own pronunciation and sends the data to the server, the server evaluates the pronunciation using a speech recognition API and sends the results to the device. For example, if a user wants to practice "How are you?", audio with the correct pronunciation is generated, the user's pronunciation is analyzed, and evaluation feedback is provided.

[2446] Learning record function

[2447] When a user begins studying, the device records the time and content of the study. This data is periodically sent to the server, which stores it in a database. When a user wants to check their past study history, they send a request from the device to the server, and the server searches for the study history data and sends it to the device, which then displays the data to the user. For example, if they check "What did I study on October 1st?", the corresponding content will be displayed.

[2448] Smartphone compatible

[2449] The system is compatible with mobile devices, allowing users to study anytime, anywhere. When a user launches the app on their smartphone, the main menu appears and they can access various functions. The device communicates with the server, providing necessary data and feedback in real time. For example, they can select the English conversation practice mode and practice pronunciation while commuting.

[2450] Below is an example of a prompt sentence to input to the generative AI model.

[2451] "Tell me about Newton's laws."

[2452] The above is an outline of the embodiment of the present invention. Each function cooperates with the others to provide comprehensive learning support.

[2453] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2454] Voice recognition function

[2455] Processing Step 1:

[2456] The user uses a microphone to input speech for learning.

[2457] Input: User's voice

[2458] Specific action: The user speaks into the smartphone microphone, "Tell me about Newton's laws."

[2459] Output: Captured audio data

[2460] Processing step 2:

[2461] The device captures the audio data and sends it to the server.

[2462] Input: Captured audio data

[2463] Specific operation: The device uploads the recorded audio data to a server via the Internet.

[2464] Output: Audio data sent to the server

[2465] Processing step 3:

[2466] The server uses a speech recognition API to convert the voice data into text data.

[2467] Input: Audio data sent to the server

[2468] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[2469] Specific operation: The voice recognition API analyzes the voice data and converts it into text data.

[2470] Output: Converted text data

[2471] Processing step 4:

[2472] The server transmits the converted text data to the terminal.

[2473] Input: Converted text data

[2474] Specific operation: The server returns text data to the terminal.

[2475] Output: Text data sent to the terminal

[2476] Processing Step 5:

[2477] The terminal displays the received text data to the user.

[2478] Input: Text data sent to the terminal

[2479] Specific action: Display the text "Tell me about Newton's laws" on the device screen.

[2480] Output: Displayed text data

[2481] Text-to-speech function

[2482] Processing Step 1:

[2483] The user enters the text they want to read into the device.

[2484] Input: Text entered by the user

[2485] Specific action: The user types "Newton's third law" on the smartphone keyboard.

[2486] Output: The input text data

[2487] Processing step 2:

[2488] The terminal transmits the input text data to the server.

[2489] Input: Entered text data

[2490] Specific operation: The device uploads text data to the server.

[2491] Output: Text data sent to the server

[2492] Processing step 3:

[2493] The server uses a text-to-speech API to convert the text data into audio data.

[2494] Input: Text data sent to the server

[2495] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[2496] Specific operation: Send an API request and generate audio data for "Newton's Third Law."

[2497] Output: Generated audio data

[2498] Processing step 4:

[2499] The server transmits the generated voice data to the terminal.

[2500] Input: Generated audio data

[2501] Specific operation: The server sends voice data to the terminal.

[2502] Output: Audio data sent to the device

[2503] Processing Step 5:

[2504] The terminal plays back the received audio data to the user.

[2505] Input: Audio data sent to the device

[2506] Specific action: The device plays an audio file and reads out "Newton's Third Law."

[2507] Output: Played audio data

[2508] Image generation function

[2509] Processing Step 1:

[2510] The user inputs keywords related to the content being studied into the terminal.

[2511] Input: The keyword entered by the user

[2512] Specific action: The user enters "action and reaction" into the smartphone.

[2513] Output: Entered keyword data

[2514] Processing step 2:

[2515] The terminal transmits the keyword data to the server.

[2516] Input: Entered keyword data

[2517] Specific operation: The terminal uploads keyword data to the server.

[2518] Output: Keyword data sent to the server

[2519] Processing step 3:

[2520] The server generates the relevant images using an image generation API.

[2521] Input: Keyword data sent to the server

[2522] Data processing: Use image generation API (e.g., DALL-E or MidJourney)

[2523] Specific operation: Send an API request and generate an image related to "action and reaction."

[2524] Output: Generated image data

[2525] Processing step 4:

[2526] The server transmits the generated image data to the terminal.

[2527] Input: Generated image data

[2528] Specific operation: The server sends an image file to the terminal.

[2529] Output: Image data sent to the device

[2530] Processing Step 5:

[2531] The terminal displays the received image data to the user.

[2532] Input: Image data sent to the device

[2533] Specific operation: Display the generated image on the device screen.

[2534] Output: Displayed image data

[2535] English conversation learning function

[2536] Processing Step 1:

[2537] The user inputs the English phrase they want to practice into the device.

[2538] Input: English phrase entered by the user

[2539] Specific action: The user types "How are you?" on the smartphone keyboard.

[2540] Output: Input phrase data

[2541] Processing step 2:

[2542] The device sends the entered phrase to the server.

[2543] Input: Entered phrase data

[2544] Specific operation: The device uploads phrase data to the server.

[2545] Output: Phrase data sent to the server

[2546] Processing step 3:

[2547] The server uses a text-to-speech API to generate audio data with the correct pronunciation.

[2548] Input: Phrase data sent to the server

[2549] Data processing: Use a text-to-speech API (e.g., Amazon Polly or Google Text-to-Speech API)

[2550] Specific behavior: Sends an API request to generate the correct pronunciation of "How are you?"

[2551] Output: Generated audio data

[2552] Processing step 4:

[2553] The server transmits the generated voice data to the terminal.

[2554] Input: Generated audio data

[2555] Specific operation: The server sends voice data to the terminal.

[2556] Output: Audio data sent to the device

[2557] Processing Step 5:

[2558] The terminal plays back the received voice data, and the user pronounces and records the voice accordingly.

[2559] Input: Audio data sent to the device

[2560] Specific operation: The correct pronunciation generated on the device is played back, and the user speaks into the microphone and the audio is recorded.

[2561] Output: Recorded audio data

[2562] Processing Step 6:

[2563] The device sends the recorded pronunciation data to the server.

[2564] Input: Recorded audio data

[2565] Specific operation: The device sends the recorded data to the server.

[2566] Output: Recording data sent to the server

[2567] Processing Step 7:

[2568] The server uses a speech recognition API to evaluate the user's pronunciation and generate a result.

[2569] Input: Recording data sent to the server

[2570] Data processing: Use a speech recognition API (e.g., Google Cloud Speech-to-Text API)

[2571] Specific operations: Send an API request, convert the user's pronunciation into text, and perform an evaluation.

[2572] Output: Evaluation result data

[2573] Processing Step 8:

[2574] The server transmits the evaluation results to the terminal, which displays them to the user.

[2575] Input: Evaluation result data

[2576] Specific operation: The server sends the evaluation results to the terminal, and the terminal displays the results on the screen.

[2577] Output: Displayed evaluation results

[2578] Learning record function

[2579] Processing Step 1:

[2580] When a user starts studying, the terminal records the study time and content.

[2581] Input: Study start information (time, content)

[2582] Specific operation: The device stores the user's learning start time and learning content in a log.

[2583] Output: Recorded training data

[2584] Processing step 2:

[2585] The terminal periodically transmits the recorded data to the server.

[2586] Input: Recorded training data

[2587] Specific operation: The device periodically uploads learning data to the server.

[2588] Output: Training data sent to the server

[2589] Processing step 3:

[2590] The server stores the received learning data in a database.

[2591] Input: Training data sent to the server

[2592] Specific operation: The server inserts the received data into the corresponding table in the database.

[2593] Output: Training data stored in a database

[2594] Processing step 4:

[2595] ...

Claims

1. a speech recognition means for converting speech data into text data; a text-to-speech means for converting text data into voice data; image generation means for generating images related to the learning content; an English conversation learning means for evaluating a user's pronunciation; A learning record means for recording and managing learning data; an interface means for a mobile device; A learning support system including:

2. 2. The learning support system according to claim 1, wherein the speech recognition means utilizes a speech recognition API when converting speech data into text data.

3. 2. The learning support system according to claim 1, wherein the text-to-speech means utilizes a text-to-speech API when converting text data into voice data.

Images