System

Abstract

A system is provided.SOLUTION: The system includes a means for collecting voice data, a means for uploading the collected voice data, a means for analyzing the uploaded voice data and converting it into a text, a means for presenting an analysis result to a user, a means for storing a final analysis result in a database, and a means for providing learning contents using the stored dialect data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Dialects are important linguistic forms that reflect local culture and identity, but they are endangered due to a decline in speakers and the advancement of urbanization. Therefore, there is a need for methods to effectively record, preserve, and pass on dialects to future generations. It is also important to provide tools for education and learning that utilize collected dialect data. However, conventional methods pose a challenge, as there is no consistent system for collecting, analyzing, preserving, and supporting learning of dialects. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The system includes a means for collecting voice data, a means for uploading the collected voice data, a means for analyzing the uploaded voice data and converting it into text, a means for presenting the analysis results to the user, a means for saving the final analysis results in a database, and a means for providing learning content using the saved dialect data.The system also includes a means for reproducing pronunciation and intonation from the saved dialect data using AI, and a means for providing the reproduced voice to the user.The system further includes a means for reproducing situations in which the dialect is used using virtual reality or augmented reality technology.With the above configuration, dialects can be efficiently recorded, analyzed, saved, and learned.

[0007] "Audio data" refers to data in which audio is recorded in digital format.

[0008] A "collection means" is a device or function for recording and storing audio data.

[0009] The "uploading means" is a device or function for transmitting collected voice data to a server.

[0010] "Means for analyzing" is a function for processing voice data and converting it into text.

[0011] "Means for converting to text" is a function that analyzes voice data and outputs the results as text information.

[0012] The "presentation means" is a device or function that displays the analysis results to the user visually or audibly.

[0013] "Means for saving" refers to a function for storing analysis results and voice data in a database.

[0014] "Learning content" is a collection of teaching materials and activities for learning dialect vocabulary, grammar, etc.

[0015] "Means of providing" refers to the function of delivering learning content to users in a usable form.

[0016] "Means of reproduction" refers to a function that restores voice and pronunciation based on saved dialect data.

[0017] "Reproduced voice" is voice generated based on the original voice data.

[0018] "Virtual reality technology" is a technology that uses computer technology to visually create a virtual world that does not exist in reality.

[0019] "Augmented reality technology" is a technology that displays virtual information overlaid on the real environment.

[0020] A "situation" is a specific situation or environment in which a particular dialect is used. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The system of the present invention collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. Detailed description of the embodiments of the present invention follows.

[0043] The system includes programs that perform various processes, such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. Each process is realized through the cooperative operation of the user, the terminal, and the server.

[0044] 1. Audio data collection:

[0045] User: Launch the application and select the "Record Dialect" section. Press the record button that appears and speak a specific dialect phrase.

[0046] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[0047] 2. Uploading audio data:

[0048] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[0049] Server: Processes and temporarily stores the received audio data.

[0050] 3. Analysis of audio data:

[0051] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[0052] Terminal: The analysis results are returned and the converted text is presented to the user.

[0053] 4. Check and correct text data:

[0054] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal.

[0055] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[0056] 5. Save to database:

[0057] Server: Stores the final text and audio data, as well as related metadata, in a database.

[0058] Device: Sends a notification to the user that the save is complete.

[0059] 6. Providing interactive learning tools:

[0060] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0061] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0062] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[0063] 7. Dialect Reproduction and Simulation:

[0064] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[0065] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0066] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[0067] 8. Dialect experience using VR and AR:

[0068] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0069] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[0070] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[0071] 9. Multilingual Translation and Dialect Preservation:

[0072] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[0073] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[0074] Server: Stores the translated data in the final database.

[0075] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0076] As described above, this system can consistently carry out each process to effectively preserve and inherit dialects.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0080] Step 2:

[0081] User: Press the record button, speak the dialect phrase as instructed, and press the stop button when finished recording.

[0082] Step 3:

[0083] Device: Temporarily saves the audio data in local storage, then sends a request to upload it to the server.

[0084] Step 4:

[0085] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​speech recognition module for analysis.

[0086] Step 5:

[0087] Server: The AI ​​speech recognition module converts the voice data into text, and stores the converted text data in a temporary database.

[0088] Step 6:

[0089] Server: Returns the analysis results to the device.

[0090] Step 7:

[0091] Terminal: The analysis result text is displayed in the UI and the user is asked to confirm.

[0092] Step 8:

[0093] User: Checks the text analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[0094] Step 9:

[0095] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[0096] Step 10:

[0097] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0098] Step 11:

[0099] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0100] Step 12:

[0101] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0102] Step 13:

[0103] User: Interacts with the provided learning content and receives feedback on correct and incorrect answers.

[0104] Step 14:

[0105] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[0106] Step 15:

[0107] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0108] Step 16:

[0109] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[0110] Step 17:

[0111] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0112] Step 18:

[0113] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[0114] Step 19:

[0115] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[0116] Step 20:

[0117] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[0118] Step 21:

[0119] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[0120] Step 22:

[0121] Server: Stores the translated data in the final database.

[0122] Example 1

[0123] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0124] In modern times, regional dialects are rapidly disappearing, making it difficult to pass them on to the next generation. Furthermore, those who want to learn dialects lack the appropriate learning tools, making it difficult to learn them efficiently. Furthermore, there is a lack of systems for consistently collecting, analyzing, and storing dialect speech data, and there is a need to efficiently carry out these tasks.

[0125] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0126] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the converted text to the user and accepting correction requests as necessary, means for saving the final text data and voice data in a database, means for providing learning content using the saved dialect data, means for providing interaction through the learning content and giving feedback to the user, and means for translating the dialect data into other languages, thereby enabling the effective preservation and inheritance of dialects.

[0127] "Audio data" refers to audio information recorded by a user, and is data that includes specific dialects and phrases.

[0128] A "collection means" is a part of a system that provides a method or facility for a user to record audio data.

[0129] "Means of uploading" refers to the method or function for transferring collected audio data to an online server.

[0130] "Means for analyzing" means a method or function that uses technology such as an AI voice recognition module to convert uploaded voice data into text format.

[0131] "Means of presentation" refers to the methods and functions for displaying and confirming analysis results, learning content, translation results, etc. to the user.

[0132] "Means for accepting correction requests" refers to a method or function that allows a user to request correction of errors in the analysis results.

[0133] "Storage means" refers to the method or function for permanently storing the final text data and audio data in a database.

[0134] "Learning Content" means the learning materials and activities (e.g., quizzes, games, vocabulary exercises) provided to users to help them learn a dialect.

[0135] "Means of interaction" refers to the methods and features that allow users to actively engage with the learning content and receive feedback.

[0136] "Means for translating" means a method or function that uses a multilingual translation AI model to convert dialect data into another language.

[0137] "Means of reproduction" refers to the method or function of generating speech using Text-to-Speech (TTS) technology using stored dialect data.

[0138] "Virtual reality technology" refers to technology that allows users to immerse themselves in a virtual environment using a VR headset or similar device.

[0139] "Augmented reality technology" refers to the technology of overlaying digital information onto the real world using AR-enabled devices.

[0140] MODE FOR CARRYING OUT THE INVENTION

[0141] The system according to the present invention performs processes such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. These processes are realized through the cooperative operation of a user, a terminal, and a server. Specific embodiments are described below.

[0142] Audio data collection

[0143] Users launch the smartphone app, select the "Dialect Recording" section, press the record button, and speak a specific dialect phrase. The device detects that the record button has been pressed and begins recording the user's voice using the built-in microphone. The recorded voice data is temporarily stored in local storage.

[0144] Uploading audio data

[0145] After the recording is complete, the device sends an upload request to the server. The recorded audio data is compressed and transferred to the server. The user is informed of the upload progress using a progress bar. The server temporarily stores the received audio data and prepares for the next process.

[0146] Analysis of audio data

[0147] The server passes the voice data to an AI speech recognition module, which converts the voice into text. The text conversion results are stored in a temporary database on the server, and the results are sent back to the device, which then presents the text analysis results to the user.

[0148] Check and correct text data

[0149] The user checks the displayed text analysis results, and if there are any errors, sends a correction request to the server via the terminal. The server receives the correction request and either performs a new analysis or accepts the user's manual correction.

[0150] Saving to a database

[0151] The final text and audio data, along with associated metadata, are then permanently stored in a database by the server, and the device notifies the user when the storage is complete.

[0152] Providing interactive learning tools

[0153] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[0154] Dialect reproduction and simulation

[0155] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology. The generated voice is provided to the user via the terminal, and a play button and a feedback form are displayed. The user can listen to the pronunciation and intonation of the reproduced dialect.

[0156] Dialect experience using VR and AR

[0157] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[0158] Multilingual Translation and Dialect Preservation

[0159] When a user sends a translation request, the device forwards it to the server. The server uses a multilingual translation AI model to translate the dialect data into another language. The translation results are presented to the user via the device, and correction requests are accepted as needed. The final data is stored in a database by the server.

[0160] Specific examples

[0161] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0162] Prompt Sentence Examples

[0163] Example prompt sentence:

[0164] "Please record yourself saying 'The weather is nice' in the Niigata dialect."

[0165] Upload the recorded audio to the server.

[0166] Once the upload is complete, check the text conversion results.

[0167] "Make any necessary corrections."

[0168] Save the final result.

[0169] "Learn the Niigata dialect using the learning module."

[0170] In this way, the system of the present invention can consistently perform each process for effectively preserving and inheriting dialects.

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1:

[0173] The user launches the application and selects the "Dialect Recording" section. Next, they press the record button and speak the specified phrase. The device detects this and begins recording the audio using the built-in microphone. The recording is temporarily saved in local storage.

[0174] Input: Pressing the record button, user voice

[0175] Output: Audio data stored in local storage

[0176] Specific operation: The recording module detects the button press event and starts the recording process. After the recording is completed, the data is saved in a file format.

[0177] Step 2:

[0178] After the device has finished recording, it sends a request to upload the audio data to the server. The audio data is compressed and sent to the server as an HTTP POST request. The upload progress is displayed to the user. The server temporarily stores the received audio data.

[0179] Input: Audio data stored in local storage

[0180] Output: Audio data stored on the server

[0181] Specific operation: The device compresses the audio data and sends it to the server using the HTTP protocol. The server receives the data and stores it in temporary storage.

[0182] Step 3:

[0183] The server passes the uploaded voice data to an AI speech recognition module, which converts the voice into text, and the conversion results are stored in a temporary database on the server.

[0184] Input: Audio data stored on the server

[0185] Output: Conversion result in text format

[0186] Specific operation: The server calls the speech recognition API to convert the voice data into text format, which is then stored in the database.

[0187] Step 4:

[0188] The server notifies the terminal of the conversion results, and the terminal presents the text analysis results to the user.

[0189] Input: Conversion result in text format

[0190] Output: Text analysis results presented to the user

[0191] Specific operation: The server sends text data to the terminal, and the terminal displays it on the user interface.

[0192] Step 5:

[0193] The user checks the analysis results and, if necessary, sends a correction request to the server via the terminal. The server receives the correction request and performs a new analysis or approves manual correction.

[0194] Input: User correction request

[0195] Output: Corrected text data

[0196] Specific operation: The user makes corrections using a text editor, and the device sends the corrections to the server, which saves them in the database.

[0197] Step 6:

[0198] The server stores the final text data, audio data, and related metadata in a permanent storage database. Once the storage is complete, the device notifies the user.

[0199] Input: Final text and audio data

[0200] Output: Notification of save completion

[0201] Specific operation: The server saves the data permanently in the database and notifies the device of the status. The device notifies the user that the save is complete.

[0202] Step 7:

[0203] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[0204] Input: User access to learning content

[0205] Output: User learning progress and feedback

[0206] Specific operation: The device retrieves learning content from the server and displays it to the user. Feedback is provided based on the user's interactions.

[0207] Step 8:

[0208] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology, which is then provided to the user via their device, along with a playback button and a feedback form.

[0209] Input: Stored dialect data

[0210] Output: Reproduced audio provided to the user

[0211] Specific operation: The server generates speech using TTS technology, and the device displays a play button to the user, allowing the user to listen to the generated speech.

[0212] Step 9:

[0213] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[0214] Input: User access to the dialect experience module

[0215] Output: A virtual reality recreation of a dialect scene

[0216] Specific operation: The device uses VR / AR technology to create a virtual environment and fully render it so that users can experience the dialect within it.

[0217] Step 10:

[0218] The device receives translation requests from users and forwards them to the server. The server uses a multilingual translation AI model to translate the dialect data into other languages ​​and presents the results to the user via the device. The user can then submit correction requests as needed. The final data is stored in a database by the server.

[0219] Input: Translation request from user

[0220] Output: Data translated into other languages

[0221] Specific operation: The server translates the dialect data using the translation AI model, displays the translation results to the user through the terminal, and saves the final data in the database based on the user's feedback.

[0222] (Application example 1)

[0223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] Conventional dialect learning systems simply record audio data and display the resulting text, lacking interactive feedback for learners and practical experience. Furthermore, they lack the ability to reuse saved dialect data or translate it into other languages, making it difficult to accommodate a wide range of users. Furthermore, they lack the ability to provide a learning environment that utilizes virtual reality or augmented reality technology.

[0225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0226] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for providing a variety of voice learning content using the saved dialect data, means for recreating situations in which the dialect is used using virtual reality or augmented reality technology, and means for translating the dialect into other languages, thereby enabling learners to learn the dialect through an interactive and realistic experience and to reuse the saved data for translation into other languages.

[0227] "Voice data" refers to digital data that electronically records the voice of a user speaking in a dialect or standard Japanese.

[0228] A "collection instrument" is a combination of hardware and software used to record and store a user's voice using a device.

[0229] The "means for uploading" is a mechanism for transferring collected voice data to a server via a network.

[0230] The "means for analyzing and converting to text" refers to algorithms and software for converting voice data into text information using voice recognition technology.

[0231] The "means for presenting to the user" is an interface for presenting the analysis results to the user visually or audibly.

[0232] The "means for storing in a database" refers to a storage system for temporarily or long-term storage of analyzed text data and audio data.

[0233] "Means for providing learning content" refers to a mechanism that provides interactive learning materials such as quizzes, games, and vocabulary practice to users to help them learn the dialect.

[0234] The "means for providing a variety of phonetic learning content" is a system for generating and providing various forms of phonetic learning content using stored dialect data.

[0235] "Virtual reality technology" or "augmented reality technology" refers to technology that allows users to experience virtual environments and information in real time, simulating specific scenarios or environments.

[0236] "Means for recreating situations in which dialects are used" refers to a system that uses virtual reality and augmented reality technology to recreate situations in which a specific dialect is actually used.

[0237] "Means for translating into other languages" refers to a multilingual translation system or algorithm for converting dialect data into other languages.

[0238] This invention relates to a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. This system includes a program that collects, uploads, analyzes, presents, and stores the speech data, and provides learning content. Specific embodiments of this system are described below.

[0239] 1. Collection of audio data

[0240] The user launches the application on a device such as a smartphone or tablet, selects the recording section, presses the record button, and speaks a dialect phrase. The device uses WebRTC to record the voice data and temporarily stores it in local storage.

[0241] 2. Uploading audio data

[0242] The device sends an HTTP POST request to upload the recorded audio data to the server. The progress is displayed on the screen to inform the user. The server receives the request using Flask and temporarily stores the audio data.

[0243] 3. Analysis of audio data

[0244] The server inputs the voice data into a speech recognition model using TensorFlow and converts it into text. The conversion results are temporarily stored in a MySQL database. The conversion results are then sent back to the device as an HTTP response, and the text result is presented to the user.

[0245] 4. Check and correct the analysis results

[0246] The user checks the text of the analysis result and sends a correction request if there is an error. The server then re-analyzes or manually corrects the text and stores it in the database again.

[0247] 5. Saving to the database

[0248] The server saves the final analysis results in a MySQL database and sends a notification to the terminal that the results have been saved.

[0249] 6. Providing interactive learning content

[0250] Users access the "Dialect Learning" module and use learning content such as quizzes and vocabulary practice. The server dynamically generates this content and provides it to users.

[0251] 7. Dialect Reproduction and Simulation

[0252] The server uses Text-to-Speech (TTS) technology to reproduce the speech of the dialect data stored and transmits it to the terminal. The user listens to the reproduced speech and learns pronunciation and intonation.

[0253] 8. Dialect experience using VR and AR

[0254] Users access the "Dialect Experience" module using a VR headset or AR-enabled device, and the server uses Unity or Unreal Engine to render the virtual environment and recreate specific dialect-using situations.

[0255] 9. Multilingual Translation

[0256] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database.

[0257] (Example)

[0258] For example, if a user follows the instruction "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect," the recorded audio data is uploaded to the server, analyzed, and converted into text. The results of this analysis are presented to the user, and any errors can be corrected. The data is then stored in a database and used as learning content for other users.

[0259] (Example of a prompt)

[0260] Audio collection prompt: "Please record yourself saying 'The weather is nice' in the Niigata dialect. Press the record button and start speaking."

[0261] Correction prompt: "Your voice data has been analyzed. The following text is the analysis result. Please correct any errors."

[0262] Learning Content Prompt: "Learn the following phrases. Check your pronunciation and take the quiz."

[0263] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0264] Step 1:

[0265] A user launches the application on a device such as a smartphone or tablet and selects the "Dialect Recording" section. The input is the user's actions and the dialect phrase to be recorded. Specifically, the user presses the record button and speaks a specific dialect phrase. The device uses WebRTC to record the voice data in real time and temporarily saves it in local storage. The output is the recorded voice data.

[0266] Step 2:

[0267] When the user finishes recording, the device uploads the audio data to the server. The input is the audio data recorded in step 1. The device sends an HTTP POST request to transfer the audio data to the server. The server receives the request using Flask and temporarily stores the audio data. The output is the audio data stored on the server.

[0268] Step 3:

[0269] To analyze the stored voice data, the server inputs the data into a voice recognition model using TensorFlow. The input is the voice data received in step 2. Specifically, the voice recognition model converts the voice data into text and generates analysis results in text format. The output is the converted text data.

[0270] Step 4:

[0271] The server temporarily stores the analysis results in a MySQL database and returns the text data to the terminal as an HTTP response. The input is the text data obtained in step 3. The terminal displays the analysis results to the user. The output is the text analysis results presented to the user.

[0272] Step 5:

[0273] The user checks the text of the analysis results and, if there are any errors, sends a correction request. The input is the displayed text data and the user's correction request. The terminal transfers the correction request to the server. The output is data including the correction request. The server re-analyzes or performs manual corrections and stores the data in the database again. Finally, corrected text data is generated.

[0274] Step 6:

[0275] The server stores the final analysis results in a MySQL database. The input is the data corrected in step 5. This stored data includes the audio data, associated text data, and metadata. The output is the final analysis results stored in the database.

[0276] Step 7:

[0277] A user accesses the "Dialect Learning" module. The input is dialect data stored in the database. The server dynamically generates learning content such as quizzes and vocabulary exercises using the stored dialect data and sends it to the terminal. The output is interactive learning content provided to the user.

[0278] Step 8:

[0279] The server uses Text-to-Speech (TTS) technology to convert the stored dialect data into speech and send it to the terminal. The input is text data from the database. The user listens to the reproduced speech and learns pronunciation and intonation. The output is the reproduced speech data.

[0280] Step 9:

[0281] A user accesses the "Dialect Experience" module using a VR headset or AR-enabled device. The input is the saved dialect data and the corresponding scenario settings. The server uses Unity or Unreal Engine to render the virtual environment and recreate a specific dialect usage scene. The output is the virtual environment provided to the user.

[0282] Step 10:

[0283] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database. The input is the stored dialect data. The user checks and corrects the translation results as needed and uses the final data. The output is the translated data.

[0284] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0285] The system of the present invention collects, analyzes, and stores dialect speech data, provides interactive tools for learning the dialect, and combines it with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[0286] The system includes programs for collecting, uploading, analyzing, presenting, and saving voice data, providing learning content, and recognizing user emotions. Each process is realized through the cooperative operation of the user, the device, and the server.

[0287] 1. Audio data collection:

[0288] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0289] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[0290] 2. Uploading audio data:

[0291] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[0292] Server: Processes and temporarily stores the received audio data.

[0293] 3. Analysis of audio data:

[0294] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[0295] Terminal: The analysis results are returned and the converted text is presented to the user.

[0296] 4. Check and correct text data:

[0297] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[0298] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[0299] 5. Save to database:

[0300] Server: Stores the final text and audio data, as well as related metadata, in a database.

[0301] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0302] 6. Providing interactive learning tools:

[0303] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0304] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0305] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[0306] 7. Dialect Reproduction and Simulation:

[0307] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[0308] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0309] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[0310] 8. Dialect experience using VR and AR:

[0311] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0312] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[0313] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[0314] 9. Multilingual Translation and Dialect Preservation:

[0315] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[0316] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[0317] Server: Stores the translated data in the final database.

[0318] 10. Incorporating an Emotion Engine:

[0319] Device: Incorporates an emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[0320] Server: Stores the recognized emotion data and reflects it in analysis results and learning content.

[0321] 11. Emotion-based learning content adjustment:

[0322] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine. For example, if the user is excited while learning, it provides more challenging questions, and if the user is stressed, it changes to easier questions.

[0323] 12. Emotional Data Storage:

[0324] Server: Receives user emotion data from the emotion engine and stores it in a database along with other dialect data.

[0325] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0326] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions. On the other hand, if the system recognizes that the user is feeling stressed, it will switch to easier questions. Emotional data is also stored in a database and will be used to improve dialect learning in the future.

[0327] In this way, this system can effectively preserve and pass on dialects while providing an interactive learning experience that reflects the user's emotions.

[0328] The processing flow will be explained below.

[0329] Step 1:

[0330] User: Launches the application and selects the "Record Dialect" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0331] Step 2:

[0332] User: Press the record button, speak the dialect phrase provided, and press the stop button when finished.

[0333] Step 3:

[0334] Device: Recorded audio data is temporarily stored in local storage.

[0335] Step 4:

[0336] Terminal: Sends a request to upload audio data to the server. Displays the upload progress to the user.

[0337] Step 5:

[0338] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​voice recognition module.

[0339] Step 6:

[0340] Server: The AI ​​speech recognition module converts the voice data into text, which is then stored in a temporary database.

[0341] Step 7:

[0342] Server: Returns the text analysis results to the device.

[0343] Step 8:

[0344] Terminal: The analysis result text is presented to the user and confirmation is sought.

[0345] Step 9:

[0346] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no errors, presses the confirmation button.

[0347] Step 10:

[0348] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[0349] Step 11:

[0350] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0351] Step 12:

[0352] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0353] Step 13:

[0354] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0355] Step 14:

[0356] User: Interacts with the provided learning content and receives feedback on whether their answers are correct or incorrect.

[0357] Step 15:

[0358] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[0359] Step 16:

[0360] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0361] Step 17:

[0362] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[0363] Step 18:

[0364] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0365] Step 19:

[0366] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[0367] Step 20:

[0368] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[0369] Step 21:

[0370] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[0371] Step 22:

[0372] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[0373] Step 23:

[0374] Server: Stores the translated data in the final database.

[0375] Step 24:

[0376] On the device: Activate the emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[0377] Step 25:

[0378] Server: Receives and temporarily stores the user's emotion data recognized by the emotion engine.

[0379] Step 26:

[0380] Server: Uses emotional data to adjust learning content, for example, making the content less difficult if the user is feeling stressed.

[0381] Step 27:

[0382] Device: Presents tailored learning content to users and supports them to keep them engaged.

[0383] Step 28:

[0384] Server: Stores the user's emotional data in a final database for future analysis and improvement.

[0385] Example 2

[0386] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0387] In recent years, the preservation and inheritance of dialects has become increasingly important from the perspective of cultural value. However, conventional dialect learning systems have limitations in collecting and analyzing speech data and providing learning content, and in particular lack the ability to optimize learning while taking into account the user's emotions. They are also inadequate in reproducing dialect data and providing an interactive learning environment using virtual reality technology. The present invention aims to solve these problems.

[0388] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for the user to confirm and correct the text, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for incorporating an emotion engine that recognizes the user's emotions in real time, means for dynamically adjusting the learning content based on the user's emotions recognized by the emotion engine, and means for saving the user's emotion data in a database. This makes it possible to provide an interactive and optimized learning experience that takes the user's emotions into consideration.

[0389] "Audio data" refers to data that records sound in digital or analog format.

[0390] A "means of collection" is a device or software that provides the function of recording audio and storing it as data.

[0391] An "uploading means" is a device or software that provides the function of transmitting data stored in local storage to a remote server.

[0392] An "analyzing means" is a device or software that provides the functionality to process collected voice data and convert it into text.

[0393] "Means for converting to text" refers to software or a module that has the function of converting voice data into character data.

[0394] The "presentation means" refers to a device or software that has the function of allowing a server or terminal to display the processing results to the user visually or audibly.

[0395] A "verification and correction means" is a device or software that provides an interface or functionality for a user to review the presented information and correct it if necessary.

[0396] "Means for storage" refers to a database or storage device for permanently recording the final data.

[0397] "Learning Content" refers to the educational materials and interactive activities provided to users to help them learn a dialect.

[0398] An "emotion engine" is software or a module that analyzes a user's facial expressions, tone of voice, etc., and recognizes emotions in real time.

[0399] A "dynamic adjustment means" is a device or software that provides the functionality to change the difficulty or content of learning content in real time based on the user's emotional state.

[0400] The "means for converting to text" is software or hardware for converting voice data into text data.

[0401] A "reproduction means" is a device or software that provides the functionality to generate sounds and virtual scenes based on stored data.

[0402] "User" refers to an individual or end user who uses the System.

[0403] "Interactive" refers to the ability of a user to elicit a response from the system through their own actions.

[0404] "Virtual reality technology" refers to technology that allows users to experience computer-generated three-dimensional environments.

[0405] "Augmented reality technology" refers to technology that displays digital information overlaid on the real world.

[0406] The present invention is a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. In addition, by incorporating an emotion engine that recognizes the user's emotions, effective dialect learning is realized. The detailed processing of the system is described below.

[0407] Hardware and software used

[0408] The system uses the following hardware and software:

[0409] Mobile information terminals such as smartphones and tablets

[0410] Servers (including cloud-based storage and processing resources)

[0411] Voice Recording Module

[0412] AI voice recognition module

[0413] Text-to-Speech (TTS) software

[0414] Emotion Recognition Engine

[0415] Database Management Systems

[0416] Audio data collection

[0417] The user starts the application using a mobile information device such as a smartphone or tablet and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording instruction phrase (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect"). When the user presses the record button, the device's built-in microphone is used to record audio, which is then temporarily saved in local storage.

[0418] Uploading audio data

[0419] The recorded audio data is uploaded from the device to the server. The device sends a request to the server and displays the upload progress to the user. The server receives the audio data and stores it in temporary storage.

[0420] Analysis of audio data

[0421] The server passes the uploaded voice data to an AI speech recognition module, which converts it into text. This module analyzes the voice data and generates corresponding text data. The generated text data is stored in a temporary database. The analysis results are sent back from the server to the device, and the converted text is presented to the user.

[0422] Check and correct text data

[0423] The user checks the presented text data and, if there are any errors, sends a correction request to the server via their terminal. If there are no problems, they press the "Confirm" button to proceed. The server receives the correction request and either performs a new analysis or approves the user's manual correction.

[0424] Saving to a database

[0425] The server stores the final text and audio data, along with associated metadata, in a database. Once the data has been saved, the device displays a notification to the user stating, "Data saved successfully."

[0426] Providing interactive learning tools

[0427] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the provided content and receives feedback on whether their answers are correct or incorrect.

[0428] Dialect reproduction and simulation

[0429] The server uses the stored dialect data to generate a reproduced speech using Text-to-Speech (TTS) technology. This speech is provided to the terminal, and the user can listen to the reproduced speech by pressing the play button. In addition, a feedback form can be used to provide a rating.

[0430] Dialect experience using VR and AR

[0431] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device renders a virtual reality or augmented reality environment, recreating specific dialect-using situations (e.g., market, home, school). Within the virtual environment, users are immersed in dialect-based scenarios, providing a realistic dialect experience.

[0432] Multilingual Translation and Dialect Preservation

[0433] The server uses a multilingual translation AI model to translate the dialect data into other languages. The translation results are then sent back to the device, where the user can review them and send correction requests if necessary. The final translation data is stored in a database.

[0434] Incorporating an emotional engine and adjusting learning content

[0435] The device uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice. The recognized emotion data is sent to a server, which then dynamically adjusts the difficulty and content of the learning content. The emotion data is also stored in a database to help improve dialect learning in the future.

[0436] Examples and prompts

[0437] For example, a user might use a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. The data is finally stored in a database and used as learning content. Additionally, an emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and the learning content is adjusted based on that data.

[0438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0439] Step 1:

[0440] Audio data collection

[0441] User: Launches the smartphone app and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0442] Input: The user presses the record button.

[0443] Process: The device will record audio using the built-in microphone. When you finish recording, press the stop button.

[0444] Output: The recorded audio data is temporarily saved in the device's local storage.

[0445] Step 2:

[0446] Uploading audio data

[0447] Device: Sends a request to upload the recorded audio data to the server.

[0448] Input: Temporarily saved audio data.

[0449] Processing: The device uploads the audio data to the server and displays the upload progress to the user in real time.

[0450] Output: The audio data is saved in temporary storage on the server.

[0451] Step 3:

[0452] Analysis of audio data

[0453] Server: Passes the uploaded voice data to the AI ​​voice recognition module.

[0454] Input: Audio data uploaded to the server.

[0455] Processing: The AI ​​speech recognition module analyzes the voice data and converts it into text format.

[0456] Output: Save the converted text data in a temporary database.

[0457] Step 4:

[0458] User presentation of text data

[0459] Server: Sends the conversion results to the terminal.

[0460] Input: Text data stored in a temporary database.

[0461] Processing: The text data is returned to the terminal.

[0462] Output: Text data is displayed on the terminal.

[0463] Step 5:

[0464] Check and correct text data

[0465] User: Checks the presented text and sends a correction request to the server via the terminal if necessary. If there are no problems, the user presses the "Confirm" button.

[0466] Input: The displayed text data.

[0467] Action: The user confirms the error and submits a correction request, or presses the confirm button to proceed.

[0468] Output: The modification request is sent to the server or the confirm button is pressed.

[0469] Step 6:

[0470] Finalize text data

[0471] Server: Receives the correction request and performs a new analysis or approves manual corrections. After the corrections are confirmed, they are saved in the database.

[0472] Input: Correction request or user confirmation.

[0473] Processing: Performing a new analysis or manual correction is processed to approve and save it in the database.

[0474] Output: The final text and audio data and associated metadata are stored in a database.

[0475] Step 7:

[0476] Save completion notification

[0477] Server: Notifies completion of data saving.

[0478] Input: The final data is saved in the database.

[0479] Processing: Send a save completion notification to the terminal.

[0480] Output: The device displays the message "Data saved successfully" to the user.

[0481] Step 8:

[0482] Providing interactive learning content

[0483] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0484] Input: Access from the user.

[0485] Processing: The device provides learning content (e.g., quizzes, games, vocabulary practice).

[0486] Output: User accesses and interacts with learning content.

[0487] Step 9:

[0488] Dialect reproduction and simulation

[0489] Server: Using the stored dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[0490] Input: Dialect data stored in a database.

[0491] Processing: Generate speech using TTS technology.

[0492] Output: The generated audio data.

[0493] Step 10:

[0494] Providing reproduced audio

[0495] Terminal: Provides the user with the reproduced audio and displays a play button and a feedback form.

[0496] Input: The generated audio data.

[0497] Processing: Play audio and collect feedback.

[0498] Output: You can hear the reproduced audio.

[0499] Step 11:

[0500] Dialect experience using VR and AR

[0501] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0502] Input: Access from the user.

[0503] Processing: The device renders a virtual or augmented reality environment, recreating a specific dialect-using situation (e.g., market, home, school).

[0504] Output: The user is immersed in a dialect-based scenario within a virtual environment.

[0505] Step 12:

[0506] Multilingual Translation and Dialect Preservation

[0507] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[0508] Input: Dialect data stored in a database.

[0509] Processing: Performs the translation using a multilingual translation AI model.

[0510] Output: The translated text data.

[0511] Step 13:

[0512] User presentation and correction of translation results

[0513] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[0514] Input: Translated text data.

[0515] Processing: Display the translation results and receive correction requests.

[0516] Output: A correction request or confirmation from the user.

[0517] Step 14:

[0518] Incorporating an emotion engine

[0519] Device: Uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice.

[0520] Input: Data such as the user's facial expressions and tone of voice.

[0521] Processing: Real-time analysis using emotion recognition engine.

[0522] Output: Recognized emotion data.

[0523] Step 15:

[0524] Emotion-based learning content adjustment

[0525] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine.

[0526] Input: Recognized emotion data.

[0527] Processing: Perform processing to dynamically adjust learning content based on emotions.

[0528] Output: Tailored learning content.

[0529] Step 16:

[0530] Storing Emotional Data

[0531] Server: Stores user emotion data in a database.

[0532] Input: Recognized emotion data.

[0533] Processing: The emotion data is stored in the database.

[0534] Output: Saved emotion data.

[0535] (Application example 2)

[0536] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0537] While the preservation and learning of dialects is important for cultural inheritance, existing systems have problems in that they are unable to efficiently collect and analyze dialect speech data and provide learning content. Furthermore, they do not provide appropriate feedback based on the user's learning situation or emotions, making it difficult to maintain motivation and provide an effective learning environment. Furthermore, they are unable to realistically recreate situations in which dialects are used, resulting in insufficient practice in actual communication situations.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0539] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for recognizing the user's emotions in real time, means for dynamically adjusting the difficulty and content of the learning content based on the recognized emotions, and means for providing a dialect experience using virtual reality or augmented reality, thereby enabling more effective dialect learning and providing an interactive learning experience that responds to the user's emotions.

[0540] "Audio data" is data that represents audio information in digital form.

[0541] "Means of collection" refers to the functions of the device or software used to acquire and record audio data.

[0542] "Means for uploading" refers to the functionality of the device or software used to transfer collected voice data to an external server or cloud.

[0543] "Means for analyzing and converting into text" refers to the function of a device or software for converting voice data into language data.

[0544] "Presentation means" refers to the function of a device or software for visually or audibly presenting analysis results or information to a user.

[0545] "Means for storage" refers to the functions of devices and software for recording and storing analysis results and audio data in a database or storage.

[0546] "Means for providing learning content" refers to the functionality of devices or software that provide learning materials and interactive activities for users to learn dialects.

[0547] "Emotion recognition means" refers to a device or software function for recognizing and analyzing emotions from a user's facial expressions and voice.

[0548] "Means for dynamically adjusting difficulty and content" refers to the functionality of devices or software that adjusts the difficulty and content of learning content in real time according to the user's emotions and learning situation.

[0549] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[0550] "Augmented reality" is a technology that overlays computer-generated information onto the real world.

[0551] A "means for providing a dialect experience" is a device or software function that allows a user to experience a scenario in which a dialect is used in a virtual space or an augmented reality environment.

[0552] The system of the present invention collects, analyzes, and stores speech data, and then provides learning content based on that data. The system recognizes users' emotions in real time and dynamically adjusts the learning experience based on that information. It also uses virtual reality (VR) and augmented reality (AR) to provide an interactive environment for users to learn dialects.

[0553] The system mainly includes the following means:

[0554] 1. How to collect audio data

[0555] Users launch the application on their smartphone, tablet, or other device, select the "Dialect Recording" section, press the record button, speak a phrase in the specified dialect, and the device will record the audio data.

[0556] 2. How to upload audio data

[0557] The device uploads the recorded audio data to the server, and the upload progress is displayed to the user.

[0558] 3. Methods for analyzing audio data

[0559] The server receives the uploaded voice data, passes it to an AI voice recognition module (such as Google Cloud's "Speech-to-Text API"), and converts the voice into text.

[0560] 4. Presentation of analysis results

[0561] The terminal presents the analysis results returned from the server to the user, who then checks the analysis results and sends a correction request if necessary.

[0562] 5. Database storage method

[0563] The server stores the final text and audio data, as well as associated metadata, in a database (e.g., Firebase).

[0564] 6. Means of providing learning content

[0565] The terminal uses the stored dialect data to provide the user with learning content (such as quizzes and games).

[0566] 7. Emotion recognition means

[0567] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time.

[0568] 8. Dynamic adjustment of learning content

[0569] The server dynamically adjusts the difficulty and content of learning content based on the user's recognized emotions, for example, providing more challenging problems if the user is excited, and easier problems if the user is stressed.

[0570] 9. Means of providing dialect experiences

[0571] The device uses Unity or the Oculus SDK to render virtual reality or augmented reality environments, allowing users to experience specific dialect settings (e.g., a market or home). Users can immerse themselves in these scenarios using a VR headset or AR-enabled device.

[0572] As a specific use case, consider a case where a user uses a smartphone app and follows the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0573] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions, but if they are feeling stressed, it will switch to easier questions.

[0574] Example prompt sentence:

[0575] "Press the record button and say, 'The weather is nice, isn't it?' in Niigata dialect."

[0576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0577] Step 1:

[0578] A user uses a device such as a smartphone or tablet to launch the application and select the "Dialect Recording" section. Then, the user presses the record button and speaks the specified phrase in the dialect. In this process, the device acquires and records audio data through the microphone input. The input is the user's voice, and the output is the recorded audio data.

[0579] Step 2:

[0580] When the user stops recording, the device uploads the recorded audio data to the server. Specifically, the device sends an HTTP request to the server and attaches the audio file. The input is the recorded audio data, and the output is the audio file uploaded to the server.

[0581] Step 3:

[0582] The server passes the received voice data to an AI voice recognition module, which converts the voice into text data. In this process, Google Cloud's "Speech-to-Text API" is used to analyze the voice data and convert it into text format. The input is the voice data uploaded to the server, and the output is text data.

[0583] Step 4:

[0584] The server returns the analyzed text data to the terminal, and the terminal displays the analysis results to the user. At this time, the user can check the analysis results and send a correction request if there are any errors. The input is the text data returned from the server, and the output is the text data displayed on the terminal.

[0585] Step 5:

[0586] The user checks the text data and presses the confirm button if there are no problems. The terminal sends a confirmation notification to the server, and the server stores the final analysis results in a database. The input is the user's confirmation operation and the text data, and the output is the final analysis results stored in the database.

[0587] Step 6:

[0588] The server generates learning content using the stored dialect data and provides it to the device. Specifically, it generates learning content in the form of quizzes and games, allowing users to learn dialects interactively. The input is the dialect data stored in the database, and the output is the learning content provided to the device.

[0589] Step 7:

[0590] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time. The input is the user's facial expression and voice data, and the output is the recognized emotion data.

[0591] Step 8:

[0592] The server dynamically adjusts the difficulty and content of learning content based on emotion recognition. If the user is excited, it provides challenging questions, and if the user is stressed, it provides easy questions. The input is the recognized emotion data, and the output is dynamically adjusted learning content.

[0593] Step 9:

[0594] The device uses Unity or the Oculus SDK to render a virtual reality or augmented reality environment, providing a dialect experience to the user. The user can immerse themselves in this virtual environment using a VR headset or an AR-enabled device. The input is dialect data and scenario data stored in the database, and the output is the VR or AR environment provided to the user.

[0595] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0597] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0598] [Second embodiment]

[0599] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0600] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0601] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0602] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0603] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0604] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0605] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0606] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0607] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0608] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0609] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0610] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0611] The system of the present invention collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. Detailed description of the embodiments of the present invention follows.

[0612] The system includes programs that perform various processes, such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. Each process is realized through the cooperative operation of the user, the terminal, and the server.

[0613] 1. Audio data collection:

[0614] User: Launch the application and select the "Record Dialect" section. Press the record button that appears and speak a specific dialect phrase.

[0615] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[0616] 2. Uploading audio data:

[0617] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[0618] Server: Processes and temporarily stores the received audio data.

[0619] 3. Analysis of audio data:

[0620] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[0621] Terminal: The analysis results are returned and the converted text is presented to the user.

[0622] 4. Check and correct text data:

[0623] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal.

[0624] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[0625] 5. Save to database:

[0626] Server: Stores the final text and audio data, as well as related metadata, in a database.

[0627] Device: Sends a notification to the user that the save is complete.

[0628] 6. Providing interactive learning tools:

[0629] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0630] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0631] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[0632] 7. Dialect Reproduction and Simulation:

[0633] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[0634] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0635] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[0636] 8. Dialect experience using VR and AR:

[0637] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0638] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[0639] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[0640] 9. Multilingual Translation and Dialect Preservation:

[0641] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[0642] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[0643] Server: Stores the translated data in the final database.

[0644] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0645] As described above, this system can consistently carry out each process to effectively preserve and inherit dialects.

[0646] The processing flow will be explained below.

[0647] Step 1:

[0648] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0649] Step 2:

[0650] User: Press the record button, speak the dialect phrase as instructed, and press the stop button when finished recording.

[0651] Step 3:

[0652] Device: Temporarily saves the audio data in local storage, then sends a request to upload it to the server.

[0653] Step 4:

[0654] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​speech recognition module for analysis.

[0655] Step 5:

[0656] Server: The AI ​​speech recognition module converts the voice data into text, and stores the converted text data in a temporary database.

[0657] Step 6:

[0658] Server: Returns the analysis results to the device.

[0659] Step 7:

[0660] Terminal: The analysis result text is displayed in the UI and the user is asked to confirm.

[0661] Step 8:

[0662] User: Checks the text analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[0663] Step 9:

[0664] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[0665] Step 10:

[0666] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0667] Step 11:

[0668] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0669] Step 12:

[0670] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0671] Step 13:

[0672] User: Interacts with the provided learning content and receives feedback on correct and incorrect answers.

[0673] Step 14:

[0674] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[0675] Step 15:

[0676] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0677] Step 16:

[0678] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[0679] Step 17:

[0680] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0681] Step 18:

[0682] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[0683] Step 19:

[0684] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[0685] Step 20:

[0686] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[0687] Step 21:

[0688] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[0689] Step 22:

[0690] Server: Stores the translated data in the final database.

[0691] Example 1

[0692] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] In modern times, regional dialects are rapidly disappearing, making it difficult to pass them on to the next generation. Furthermore, those who want to learn dialects lack the appropriate learning tools, making it difficult to learn them efficiently. Furthermore, there is a lack of systems for consistently collecting, analyzing, and storing dialect speech data, and there is a need to efficiently carry out these tasks.

[0694] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0695] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the converted text to the user and accepting correction requests as necessary, means for saving the final text data and voice data in a database, means for providing learning content using the saved dialect data, means for providing interaction through the learning content and giving feedback to the user, and means for translating the dialect data into other languages, thereby enabling the effective preservation and inheritance of dialects.

[0696] "Audio data" refers to audio information recorded by a user, and is data that includes specific dialects and phrases.

[0697] A "collection means" is a part of a system that provides a method or facility for a user to record audio data.

[0698] "Means of uploading" refers to the method or function for transferring collected audio data to an online server.

[0699] "Means for analyzing" means a method or function that uses technology such as an AI voice recognition module to convert uploaded voice data into text format.

[0700] "Means of presentation" refers to the methods and functions for displaying and confirming analysis results, learning content, translation results, etc. to the user.

[0701] "Means for accepting correction requests" refers to a method or function that allows a user to request correction of errors in the analysis results.

[0702] "Storage means" refers to the method or function for permanently storing the final text data and audio data in a database.

[0703] "Learning Content" means the learning materials and activities (e.g., quizzes, games, vocabulary exercises) provided to users to help them learn a dialect.

[0704] "Means of interaction" refers to the methods and features that allow users to actively engage with the learning content and receive feedback.

[0705] "Means for translating" means a method or function that uses a multilingual translation AI model to convert dialect data into another language.

[0706] "Means of reproduction" refers to the method or function of generating speech using Text-to-Speech (TTS) technology using stored dialect data.

[0707] "Virtual reality technology" refers to technology that allows users to immerse themselves in a virtual environment using a VR headset or similar device.

[0708] "Augmented reality technology" refers to the technology of overlaying digital information onto the real world using AR-enabled devices.

[0709] MODE FOR CARRYING OUT THE INVENTION

[0710] The system according to the present invention performs processes such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. These processes are realized through the cooperative operation of a user, a terminal, and a server. Specific embodiments are described below.

[0711] Audio data collection

[0712] Users launch the smartphone app, select the "Dialect Recording" section, press the record button, and speak a specific dialect phrase. The device detects that the record button has been pressed and begins recording the user's voice using the built-in microphone. The recorded voice data is temporarily stored in local storage.

[0713] Uploading audio data

[0714] After the recording is complete, the device sends an upload request to the server. The recorded audio data is compressed and transferred to the server. The user is informed of the upload progress using a progress bar. The server temporarily stores the received audio data and prepares for the next process.

[0715] Analysis of audio data

[0716] The server passes the voice data to an AI speech recognition module, which converts the voice into text. The text conversion results are stored in a temporary database on the server, and the results are sent back to the device, which then presents the text analysis results to the user.

[0717] Check and correct text data

[0718] The user checks the displayed text analysis results, and if there are any errors, sends a correction request to the server via the terminal. The server receives the correction request and either performs a new analysis or accepts the user's manual correction.

[0719] Saving to a database

[0720] The final text and audio data, along with associated metadata, are then permanently stored in a database by the server, and the device notifies the user when the storage is complete.

[0721] Providing interactive learning tools

[0722] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[0723] Dialect reproduction and simulation

[0724] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology. The generated voice is provided to the user via the terminal, and a play button and a feedback form are displayed. The user can listen to the pronunciation and intonation of the reproduced dialect.

[0725] Dialect experience using VR and AR

[0726] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[0727] Multilingual Translation and Dialect Preservation

[0728] When a user sends a translation request, the device forwards it to the server. The server uses a multilingual translation AI model to translate the dialect data into another language. The translation results are presented to the user via the device, and correction requests are accepted as needed. The final data is stored in a database by the server.

[0729] Specific examples

[0730] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0731] Prompt Sentence Examples

[0732] Example prompt sentence:

[0733] "Please record yourself saying 'The weather is nice' in the Niigata dialect."

[0734] Upload the recorded audio to the server.

[0735] Once the upload is complete, check the text conversion results.

[0736] "Make any necessary corrections."

[0737] Save the final result.

[0738] "Learn the Niigata dialect using the learning module."

[0739] In this way, the system of the present invention can consistently perform each process for effectively preserving and inheriting dialects.

[0740] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0741] Step 1:

[0742] The user launches the application and selects the "Dialect Recording" section. Next, they press the record button and speak the specified phrase. The device detects this and begins recording the audio using the built-in microphone. The recording is temporarily saved in local storage.

[0743] Input: Pressing the record button, user voice

[0744] Output: Audio data stored in local storage

[0745] Specific operation: The recording module detects the button press event and starts the recording process. After the recording is completed, the data is saved in a file format.

[0746] Step 2:

[0747] After the device has finished recording, it sends a request to upload the audio data to the server. The audio data is compressed and sent to the server as an HTTP POST request. The upload progress is displayed to the user. The server temporarily stores the received audio data.

[0748] Input: Audio data stored in local storage

[0749] Output: Audio data stored on the server

[0750] Specific operation: The device compresses the audio data and sends it to the server using the HTTP protocol. The server receives the data and stores it in temporary storage.

[0751] Step 3:

[0752] The server passes the uploaded voice data to an AI speech recognition module, which converts the voice into text, and the conversion results are stored in a temporary database on the server.

[0753] Input: Audio data stored on the server

[0754] Output: Conversion result in text format

[0755] Specific operation: The server calls the speech recognition API to convert the voice data into text format, which is then stored in the database.

[0756] Step 4:

[0757] The server notifies the terminal of the conversion results, and the terminal presents the text analysis results to the user.

[0758] Input: Conversion result in text format

[0759] Output: Text analysis results presented to the user

[0760] Specific operation: The server sends text data to the terminal, and the terminal displays it on the user interface.

[0761] Step 5:

[0762] The user checks the analysis results and, if necessary, sends a correction request to the server via the terminal. The server receives the correction request and performs a new analysis or approves manual correction.

[0763] Input: User correction request

[0764] Output: Corrected text data

[0765] Specific operation: The user makes corrections using a text editor, and the device sends the corrections to the server, which saves them in the database.

[0766] Step 6:

[0767] The server stores the final text data, audio data, and related metadata in a permanent storage database. Once the storage is complete, the device notifies the user.

[0768] Input: Final text and audio data

[0769] Output: Notification of save completion

[0770] Specific operation: The server saves the data permanently in the database and notifies the device of the status. The device notifies the user that the save is complete.

[0771] Step 7:

[0772] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[0773] Input: User access to learning content

[0774] Output: User learning progress and feedback

[0775] Specific operation: The device retrieves learning content from the server and displays it to the user. Feedback is provided based on the user's interactions.

[0776] Step 8:

[0777] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology, which is then provided to the user via their device, along with a playback button and a feedback form.

[0778] Input: Stored dialect data

[0779] Output: Reproduced audio provided to the user

[0780] Specific operation: The server generates speech using TTS technology, and the device displays a play button to the user, allowing the user to listen to the generated speech.

[0781] Step 9:

[0782] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[0783] Input: User access to the dialect experience module

[0784] Output: A virtual reality recreation of a dialect scene

[0785] Specific operation: The device uses VR / AR technology to create a virtual environment and fully render it so that users can experience the dialect within it.

[0786] Step 10:

[0787] The device receives translation requests from users and forwards them to the server. The server uses a multilingual translation AI model to translate the dialect data into other languages ​​and presents the results to the user via the device. The user can then submit correction requests as needed. The final data is stored in a database by the server.

[0788] Input: Translation request from user

[0789] Output: Data translated into other languages

[0790] Specific operation: The server translates the dialect data using the translation AI model, displays the translation results to the user through the terminal, and saves the final data in the database based on the user's feedback.

[0791] (Application example 1)

[0792] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0793] Conventional dialect learning systems simply record audio data and display the resulting text, lacking interactive feedback for learners and practical experience. Furthermore, they lack the ability to reuse saved dialect data or translate it into other languages, making it difficult to accommodate a wide range of users. Furthermore, they lack the ability to provide a learning environment that utilizes virtual reality or augmented reality technology.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0795] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for providing a variety of voice learning content using the saved dialect data, means for recreating situations in which the dialect is used using virtual reality or augmented reality technology, and means for translating the dialect into other languages, thereby enabling learners to learn the dialect through an interactive and realistic experience and to reuse the saved data for translation into other languages.

[0796] "Voice data" refers to digital data that electronically records the voice of a user speaking in a dialect or standard Japanese.

[0797] A "collection instrument" is a combination of hardware and software used to record and store a user's voice using a device.

[0798] The "means for uploading" is a mechanism for transferring collected voice data to a server via a network.

[0799] The "means for analyzing and converting to text" refers to algorithms and software for converting voice data into text information using voice recognition technology.

[0800] The "means for presenting to the user" is an interface for presenting the analysis results to the user visually or audibly.

[0801] The "means for storing in a database" refers to a storage system for temporarily or long-term storage of analyzed text data and audio data.

[0802] "Means for providing learning content" refers to a mechanism that provides interactive learning materials such as quizzes, games, and vocabulary practice to users to help them learn the dialect.

[0803] The "means for providing a variety of phonetic learning content" is a system for generating and providing various forms of phonetic learning content using stored dialect data.

[0804] "Virtual reality technology" or "augmented reality technology" refers to technology that allows users to experience virtual environments and information in real time, simulating specific scenarios or environments.

[0805] "Means for recreating situations in which dialects are used" refers to a system that uses virtual reality and augmented reality technology to recreate situations in which a specific dialect is actually used.

[0806] "Means for translating into other languages" refers to a multilingual translation system or algorithm for converting dialect data into other languages.

[0807] This invention relates to a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. This system includes a program that collects, uploads, analyzes, presents, and stores the speech data, and provides learning content. Specific embodiments of this system are described below.

[0808] 1. Collection of audio data

[0809] The user launches the application on a device such as a smartphone or tablet, selects the recording section, presses the record button, and speaks a dialect phrase. The device uses WebRTC to record the voice data and temporarily stores it in local storage.

[0810] 2. Uploading audio data

[0811] The device sends an HTTP POST request to upload the recorded audio data to the server. The progress is displayed on the screen to inform the user. The server receives the request using Flask and temporarily stores the audio data.

[0812] 3. Analysis of audio data

[0813] The server inputs the voice data into a speech recognition model using TensorFlow and converts it into text. The conversion results are temporarily stored in a MySQL database. The conversion results are then sent back to the device as an HTTP response, and the text result is presented to the user.

[0814] 4. Check and correct the analysis results

[0815] The user checks the text of the analysis result and sends a correction request if there is an error. The server then re-analyzes or manually corrects the text and stores it in the database again.

[0816] 5. Saving to the database

[0817] The server saves the final analysis results in a MySQL database and sends a notification to the terminal that the results have been saved.

[0818] 6. Providing interactive learning content

[0819] Users access the "Dialect Learning" module and use learning content such as quizzes and vocabulary practice. The server dynamically generates this content and provides it to users.

[0820] 7. Dialect Reproduction and Simulation

[0821] The server uses Text-to-Speech (TTS) technology to reproduce the speech of the dialect data stored and transmits it to the terminal. The user listens to the reproduced speech and learns pronunciation and intonation.

[0822] 8. Dialect experience using VR and AR

[0823] Users access the "Dialect Experience" module using a VR headset or AR-enabled device, and the server uses Unity or Unreal Engine to render the virtual environment and recreate specific dialect-using situations.

[0824] 9. Multilingual Translation

[0825] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database.

[0826] (Example)

[0827] For example, if a user follows the instruction "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect," the recorded audio data is uploaded to the server, analyzed, and converted into text. The results of this analysis are presented to the user, and any errors can be corrected. The data is then stored in a database and used as learning content for other users.

[0828] (Example of a prompt)

[0829] Audio collection prompt: "Please record yourself saying 'The weather is nice' in the Niigata dialect. Press the record button and start speaking."

[0830] Correction prompt: "Your voice data has been analyzed. The following text is the analysis result. Please correct any errors."

[0831] Learning Content Prompt: "Learn the following phrases. Check your pronunciation and take the quiz."

[0832] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0833] Step 1:

[0834] A user launches the application on a device such as a smartphone or tablet and selects the "Dialect Recording" section. The input is the user's actions and the dialect phrase to be recorded. Specifically, the user presses the record button and speaks a specific dialect phrase. The device uses WebRTC to record the voice data in real time and temporarily saves it in local storage. The output is the recorded voice data.

[0835] Step 2:

[0836] When the user finishes recording, the device uploads the audio data to the server. The input is the audio data recorded in step 1. The device sends an HTTP POST request to transfer the audio data to the server. The server receives the request using Flask and temporarily stores the audio data. The output is the audio data stored on the server.

[0837] Step 3:

[0838] To analyze the stored voice data, the server inputs the data into a voice recognition model using TensorFlow. The input is the voice data received in step 2. Specifically, the voice recognition model converts the voice data into text and generates analysis results in text format. The output is the converted text data.

[0839] Step 4:

[0840] The server temporarily stores the analysis results in a MySQL database and returns the text data to the terminal as an HTTP response. The input is the text data obtained in step 3. The terminal displays the analysis results to the user. The output is the text analysis results presented to the user.

[0841] Step 5:

[0842] The user checks the text of the analysis results and, if there are any errors, sends a correction request. The input is the displayed text data and the user's correction request. The terminal transfers the correction request to the server. The output is data including the correction request. The server re-analyzes or performs manual corrections and stores the data in the database again. Finally, corrected text data is generated.

[0843] Step 6:

[0844] The server stores the final analysis results in a MySQL database. The input is the data corrected in step 5. This stored data includes the audio data, associated text data, and metadata. The output is the final analysis results stored in the database.

[0845] Step 7:

[0846] A user accesses the "Dialect Learning" module. The input is dialect data stored in the database. The server dynamically generates learning content such as quizzes and vocabulary exercises using the stored dialect data and sends it to the terminal. The output is interactive learning content provided to the user.

[0847] Step 8:

[0848] The server uses Text-to-Speech (TTS) technology to convert the stored dialect data into speech and send it to the terminal. The input is text data from the database. The user listens to the reproduced speech and learns pronunciation and intonation. The output is the reproduced speech data.

[0849] Step 9:

[0850] A user accesses the "Dialect Experience" module using a VR headset or AR-enabled device. The input is the saved dialect data and the corresponding scenario settings. The server uses Unity or Unreal Engine to render the virtual environment and recreate a specific dialect usage scene. The output is the virtual environment provided to the user.

[0851] Step 10:

[0852] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database. The input is the stored dialect data. The user checks and corrects the translation results as needed and uses the final data. The output is the translated data.

[0853] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0854] The system of the present invention collects, analyzes, and stores dialect speech data, provides interactive tools for learning the dialect, and combines it with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[0855] The system includes programs for collecting, uploading, analyzing, presenting, and saving voice data, providing learning content, and recognizing user emotions. Each process is realized through the cooperative operation of the user, the device, and the server.

[0856] 1. Audio data collection:

[0857] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0858] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[0859] 2. Uploading audio data:

[0860] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[0861] Server: Processes and temporarily stores the received audio data.

[0862] 3. Analysis of audio data:

[0863] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[0864] Terminal: The analysis results are returned and the converted text is presented to the user.

[0865] 4. Check and correct text data:

[0866] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[0867] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[0868] 5. Save to database:

[0869] Server: Stores the final text and audio data, as well as related metadata, in a database.

[0870] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0871] 6. Providing interactive learning tools:

[0872] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0873] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0874] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[0875] 7. Dialect Reproduction and Simulation:

[0876] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[0877] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0878] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[0879] 8. Dialect experience using VR and AR:

[0880] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0881] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[0882] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[0883] 9. Multilingual Translation and Dialect Preservation:

[0884] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[0885] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[0886] Server: Stores the translated data in the final database.

[0887] 10. Incorporating an Emotion Engine:

[0888] Device: Incorporates an emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[0889] Server: Stores the recognized emotion data and reflects it in analysis results and learning content.

[0890] 11. Emotion-based learning content adjustment:

[0891] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine. For example, if the user is excited while learning, it provides more challenging questions, and if the user is stressed, it changes to easier questions.

[0892] 12. Emotional Data Storage:

[0893] Server: Receives user emotion data from the emotion engine and stores it in a database along with other dialect data.

[0894] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[0895] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions. On the other hand, if the system recognizes that the user is feeling stressed, it will switch to easier questions. Emotional data is also stored in a database and will be used to improve dialect learning in the future.

[0896] In this way, this system can effectively preserve and pass on dialects while providing an interactive learning experience that reflects the user's emotions.

[0897] The processing flow will be explained below.

[0898] Step 1:

[0899] User: Launches the application and selects the "Record Dialect" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[0900] Step 2:

[0901] User: Press the record button, speak the dialect phrase provided, and press the stop button when finished.

[0902] Step 3:

[0903] Device: Recorded audio data is temporarily stored in local storage.

[0904] Step 4:

[0905] Terminal: Sends a request to upload audio data to the server. Displays the upload progress to the user.

[0906] Step 5:

[0907] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​voice recognition module.

[0908] Step 6:

[0909] Server: The AI ​​speech recognition module converts the voice data into text, which is then stored in a temporary database.

[0910] Step 7:

[0911] Server: Returns the text analysis results to the device.

[0912] Step 8:

[0913] Terminal: The analysis result text is presented to the user and confirmation is sought.

[0914] Step 9:

[0915] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no errors, presses the confirmation button.

[0916] Step 10:

[0917] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[0918] Step 11:

[0919] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[0920] Step 12:

[0921] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[0922] Step 13:

[0923] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[0924] Step 14:

[0925] User: Interacts with the provided learning content and receives feedback on whether their answers are correct or incorrect.

[0926] Step 15:

[0927] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[0928] Step 16:

[0929] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[0930] Step 17:

[0931] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[0932] Step 18:

[0933] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[0934] Step 19:

[0935] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[0936] Step 20:

[0937] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[0938] Step 21:

[0939] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[0940] Step 22:

[0941] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[0942] Step 23:

[0943] Server: Stores the translated data in the final database.

[0944] Step 24:

[0945] On the device: Activate the emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[0946] Step 25:

[0947] Server: Receives and temporarily stores the user's emotion data recognized by the emotion engine.

[0948] Step 26:

[0949] Server: Uses emotional data to adjust learning content, for example, making the content less difficult if the user is feeling stressed.

[0950] Step 27:

[0951] Device: Presents tailored learning content to users and supports them to keep them engaged.

[0952] Step 28:

[0953] Server: Stores the user's emotional data in a final database for future analysis and improvement.

[0954] Example 2

[0955] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0956] In recent years, the preservation and inheritance of dialects has become increasingly important from the perspective of cultural value. However, conventional dialect learning systems have limitations in collecting and analyzing speech data and providing learning content, and in particular lack the ability to optimize learning while taking into account the user's emotions. They are also inadequate in reproducing dialect data and providing an interactive learning environment using virtual reality technology. The present invention aims to solve these problems.

[0957] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for the user to confirm and correct the text, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for incorporating an emotion engine that recognizes the user's emotions in real time, means for dynamically adjusting the learning content based on the user's emotions recognized by the emotion engine, and means for saving the user's emotion data in a database. This makes it possible to provide an interactive and optimized learning experience that takes the user's emotions into consideration.

[0958] "Audio data" refers to data that records sound in digital or analog format.

[0959] A "means of collection" is a device or software that provides the function of recording audio and storing it as data.

[0960] An "uploading means" is a device or software that provides the function of transmitting data stored in local storage to a remote server.

[0961] An "analyzing means" is a device or software that provides the functionality to process collected voice data and convert it into text.

[0962] "Means for converting to text" refers to software or a module that has the function of converting voice data into character data.

[0963] The "presentation means" refers to a device or software that has the function of allowing a server or terminal to display the processing results to the user visually or audibly.

[0964] A "verification and correction means" is a device or software that provides an interface or functionality for a user to review the presented information and correct it if necessary.

[0965] "Means for storage" refers to a database or storage device for permanently recording the final data.

[0966] "Learning Content" refers to the educational materials and interactive activities provided to users to help them learn a dialect.

[0967] An "emotion engine" is software or a module that analyzes a user's facial expressions, tone of voice, etc., and recognizes emotions in real time.

[0968] A "dynamic adjustment means" is a device or software that provides the functionality to change the difficulty or content of learning content in real time based on the user's emotional state.

[0969] The "means for converting to text" is software or hardware for converting voice data into text data.

[0970] A "reproduction means" is a device or software that provides the functionality to generate sounds and virtual scenes based on stored data.

[0971] "User" refers to an individual or end user who uses the System.

[0972] "Interactive" refers to the ability of a user to elicit a response from the system through their own actions.

[0973] "Virtual reality technology" refers to technology that allows users to experience computer-generated three-dimensional environments.

[0974] "Augmented reality technology" refers to technology that displays digital information overlaid on the real world.

[0975] The present invention is a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. In addition, by incorporating an emotion engine that recognizes the user's emotions, effective dialect learning is realized. The detailed processing of the system is described below.

[0976] Hardware and software used

[0977] The system uses the following hardware and software:

[0978] Mobile information terminals such as smartphones and tablets

[0979] Servers (including cloud-based storage and processing resources)

[0980] Voice Recording Module

[0981] AI voice recognition module

[0982] Text-to-Speech (TTS) software

[0983] Emotion Recognition Engine

[0984] Database Management Systems

[0985] Audio data collection

[0986] The user starts the application using a mobile information device such as a smartphone or tablet and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording instruction phrase (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect"). When the user presses the record button, the device's built-in microphone is used to record audio, which is then temporarily saved in local storage.

[0987] Uploading audio data

[0988] The recorded audio data is uploaded from the device to the server. The device sends a request to the server and displays the upload progress to the user. The server receives the audio data and stores it in temporary storage.

[0989] Analysis of audio data

[0990] The server passes the uploaded voice data to an AI speech recognition module, which converts it into text. This module analyzes the voice data and generates corresponding text data. The generated text data is stored in a temporary database. The analysis results are sent back from the server to the device, and the converted text is presented to the user.

[0991] Check and correct text data

[0992] The user checks the presented text data and, if there are any errors, sends a correction request to the server via their terminal. If there are no problems, they press the "Confirm" button to proceed. The server receives the correction request and either performs a new analysis or approves the user's manual correction.

[0993] Saving to a database

[0994] The server stores the final text and audio data, along with associated metadata, in a database. Once the data has been saved, the device displays a notification to the user stating, "Data saved successfully."

[0995] Providing interactive learning tools

[0996] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the provided content and receives feedback on whether their answers are correct or incorrect.

[0997] Dialect reproduction and simulation

[0998] The server uses the stored dialect data to generate a reproduced speech using Text-to-Speech (TTS) technology. This speech is provided to the terminal, and the user can listen to the reproduced speech by pressing the play button. In addition, a feedback form can be used to provide a rating.

[0999] Dialect experience using VR and AR

[1000] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device renders a virtual reality or augmented reality environment, recreating specific dialect-using situations (e.g., market, home, school). Within the virtual environment, users are immersed in dialect-based scenarios, providing a realistic dialect experience.

[1001] Multilingual Translation and Dialect Preservation

[1002] The server uses a multilingual translation AI model to translate the dialect data into other languages. The translation results are then sent back to the device, where the user can review them and send correction requests if necessary. The final translation data is stored in a database.

[1003] Incorporating an emotional engine and adjusting learning content

[1004] The device uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice. The recognized emotion data is sent to a server, which then dynamically adjusts the difficulty and content of the learning content. The emotion data is also stored in a database to help improve dialect learning in the future.

[1005] Examples and prompts

[1006] For example, a user might use a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. The data is finally stored in a database and used as learning content. Additionally, an emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and the learning content is adjusted based on that data.

[1007] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1008] Step 1:

[1009] Audio data collection

[1010] User: Launches the smartphone app and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1011] Input: The user presses the record button.

[1012] Process: The device will record audio using the built-in microphone. When you finish recording, press the stop button.

[1013] Output: The recorded audio data is temporarily saved in the device's local storage.

[1014] Step 2:

[1015] Uploading audio data

[1016] Device: Sends a request to upload the recorded audio data to the server.

[1017] Input: Temporarily saved audio data.

[1018] Processing: The device uploads the audio data to the server and displays the upload progress to the user in real time.

[1019] Output: The audio data is saved in temporary storage on the server.

[1020] Step 3:

[1021] Analysis of audio data

[1022] Server: Passes the uploaded voice data to the AI ​​voice recognition module.

[1023] Input: Audio data uploaded to the server.

[1024] Processing: The AI ​​speech recognition module analyzes the voice data and converts it into text format.

[1025] Output: Save the converted text data in a temporary database.

[1026] Step 4:

[1027] User presentation of text data

[1028] Server: Sends the conversion results to the terminal.

[1029] Input: Text data stored in a temporary database.

[1030] Processing: The text data is returned to the terminal.

[1031] Output: Text data is displayed on the terminal.

[1032] Step 5:

[1033] Check and correct text data

[1034] User: Checks the presented text and sends a correction request to the server via the terminal if necessary. If there are no problems, the user presses the "Confirm" button.

[1035] Input: The displayed text data.

[1036] Action: The user confirms the error and submits a correction request, or presses the confirm button to proceed.

[1037] Output: The modification request is sent to the server or the confirm button is pressed.

[1038] Step 6:

[1039] Finalize text data

[1040] Server: Receives the correction request and performs a new analysis or approves manual corrections. After the corrections are confirmed, they are saved in the database.

[1041] Input: Correction request or user confirmation.

[1042] Processing: Performing a new analysis or manual correction is processed to approve and save it in the database.

[1043] Output: The final text and audio data and associated metadata are stored in a database.

[1044] Step 7:

[1045] Save completion notification

[1046] Server: Notifies completion of data saving.

[1047] Input: The final data is saved in the database.

[1048] Processing: Send a save completion notification to the terminal.

[1049] Output: The device displays the message "Data saved successfully" to the user.

[1050] Step 8:

[1051] Providing interactive learning content

[1052] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1053] Input: Access from the user.

[1054] Processing: The device provides learning content (e.g., quizzes, games, vocabulary practice).

[1055] Output: User accesses and interacts with learning content.

[1056] Step 9:

[1057] Dialect reproduction and simulation

[1058] Server: Using the stored dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[1059] Input: Dialect data stored in a database.

[1060] Processing: Generate speech using TTS technology.

[1061] Output: The generated audio data.

[1062] Step 10:

[1063] Providing reproduced audio

[1064] Terminal: Provides the user with the reproduced audio and displays a play button and a feedback form.

[1065] Input: The generated audio data.

[1066] Processing: Play audio and collect feedback.

[1067] Output: You can hear the reproduced audio.

[1068] Step 11:

[1069] Dialect experience using VR and AR

[1070] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1071] Input: Access from the user.

[1072] Processing: The device renders a virtual or augmented reality environment, recreating a specific dialect-using situation (e.g., market, home, school).

[1073] Output: The user is immersed in a dialect-based scenario within a virtual environment.

[1074] Step 12:

[1075] Multilingual Translation and Dialect Preservation

[1076] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[1077] Input: Dialect data stored in a database.

[1078] Processing: Performs the translation using a multilingual translation AI model.

[1079] Output: The translated text data.

[1080] Step 13:

[1081] User presentation and correction of translation results

[1082] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[1083] Input: Translated text data.

[1084] Processing: Display the translation results and receive correction requests.

[1085] Output: A correction request or confirmation from the user.

[1086] Step 14:

[1087] Incorporating an emotion engine

[1088] Device: Uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice.

[1089] Input: Data such as the user's facial expressions and tone of voice.

[1090] Processing: Real-time analysis using emotion recognition engine.

[1091] Output: Recognized emotion data.

[1092] Step 15:

[1093] Emotion-based learning content adjustment

[1094] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine.

[1095] Input: Recognized emotion data.

[1096] Processing: Perform processing to dynamically adjust learning content based on emotions.

[1097] Output: Tailored learning content.

[1098] Step 16:

[1099] Storing Emotional Data

[1100] Server: Stores user emotion data in a database.

[1101] Input: Recognized emotion data.

[1102] Processing: The emotion data is stored in the database.

[1103] Output: Saved emotion data.

[1104] (Application example 2)

[1105] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1106] While the preservation and learning of dialects is important for cultural inheritance, existing systems have problems in that they are unable to efficiently collect and analyze dialect speech data and provide learning content. Furthermore, they do not provide appropriate feedback based on the user's learning situation or emotions, making it difficult to maintain motivation and provide an effective learning environment. Furthermore, they are unable to realistically recreate situations in which dialects are used, resulting in insufficient practice in actual communication situations.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1108] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for recognizing the user's emotions in real time, means for dynamically adjusting the difficulty and content of the learning content based on the recognized emotions, and means for providing a dialect experience using virtual reality or augmented reality, thereby enabling more effective dialect learning and providing an interactive learning experience that responds to the user's emotions.

[1109] "Audio data" is data that represents audio information in digital form.

[1110] "Means of collection" refers to the functions of the device or software used to acquire and record audio data.

[1111] "Means for uploading" refers to the functionality of the device or software used to transfer collected voice data to an external server or cloud.

[1112] "Means for analyzing and converting into text" refers to the function of a device or software for converting voice data into language data.

[1113] "Presentation means" refers to the function of a device or software for visually or audibly presenting analysis results or information to a user.

[1114] "Means for storage" refers to the functions of devices and software for recording and storing analysis results and audio data in a database or storage.

[1115] "Means for providing learning content" refers to the functionality of devices or software that provide learning materials and interactive activities for users to learn dialects.

[1116] "Emotion recognition means" refers to a device or software function for recognizing and analyzing emotions from a user's facial expressions and voice.

[1117] "Means for dynamically adjusting difficulty and content" refers to the functionality of devices or software that adjusts the difficulty and content of learning content in real time according to the user's emotions and learning situation.

[1118] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[1119] "Augmented reality" is a technology that overlays computer-generated information onto the real world.

[1120] A "means for providing a dialect experience" is a device or software function that allows a user to experience a scenario in which a dialect is used in a virtual space or an augmented reality environment.

[1121] The system of the present invention collects, analyzes, and stores speech data, and then provides learning content based on that data. The system recognizes users' emotions in real time and dynamically adjusts the learning experience based on that information. It also uses virtual reality (VR) and augmented reality (AR) to provide an interactive environment for users to learn dialects.

[1122] The system mainly includes the following means:

[1123] 1. How to collect audio data

[1124] Users launch the application on their smartphone, tablet, or other device, select the "Dialect Recording" section, press the record button, speak a phrase in the specified dialect, and the device will record the audio data.

[1125] 2. How to upload audio data

[1126] The device uploads the recorded audio data to the server, and the upload progress is displayed to the user.

[1127] 3. Methods for analyzing audio data

[1128] The server receives the uploaded voice data, passes it to an AI voice recognition module (such as Google Cloud's "Speech-to-Text API"), and converts the voice into text.

[1129] 4. Presentation of analysis results

[1130] The terminal presents the analysis results returned from the server to the user, who then checks the analysis results and sends a correction request if necessary.

[1131] 5. Database storage method

[1132] The server stores the final text and audio data, as well as associated metadata, in a database (e.g., Firebase).

[1133] 6. Means of providing learning content

[1134] The terminal uses the stored dialect data to provide the user with learning content (such as quizzes and games).

[1135] 7. Emotion recognition means

[1136] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time.

[1137] 8. Dynamic adjustment of learning content

[1138] The server dynamically adjusts the difficulty and content of learning content based on the user's recognized emotions, for example, providing more challenging problems if the user is excited, and easier problems if the user is stressed.

[1139] 9. Means of providing dialect experiences

[1140] The device uses Unity or the Oculus SDK to render virtual reality or augmented reality environments, allowing users to experience specific dialect settings (e.g., a market or home). Users can immerse themselves in these scenarios using a VR headset or AR-enabled device.

[1141] As a specific use case, consider a case where a user uses a smartphone app and follows the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1142] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions, but if they are feeling stressed, it will switch to easier questions.

[1143] Example prompt sentence:

[1144] "Press the record button and say, 'The weather is nice, isn't it?' in Niigata dialect."

[1145] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1146] Step 1:

[1147] A user uses a device such as a smartphone or tablet to launch the application and select the "Dialect Recording" section. Then, the user presses the record button and speaks the specified phrase in the dialect. In this process, the device acquires and records audio data through the microphone input. The input is the user's voice, and the output is the recorded audio data.

[1148] Step 2:

[1149] When the user stops recording, the device uploads the recorded audio data to the server. Specifically, the device sends an HTTP request to the server and attaches the audio file. The input is the recorded audio data, and the output is the audio file uploaded to the server.

[1150] Step 3:

[1151] The server passes the received voice data to an AI voice recognition module, which converts the voice into text data. In this process, Google Cloud's "Speech-to-Text API" is used to analyze the voice data and convert it into text format. The input is the voice data uploaded to the server, and the output is text data.

[1152] Step 4:

[1153] The server returns the analyzed text data to the terminal, and the terminal displays the analysis results to the user. At this time, the user can check the analysis results and send a correction request if there are any errors. The input is the text data returned from the server, and the output is the text data displayed on the terminal.

[1154] Step 5:

[1155] The user checks the text data and presses the confirm button if there are no problems. The terminal sends a confirmation notification to the server, and the server stores the final analysis results in a database. The input is the user's confirmation operation and the text data, and the output is the final analysis results stored in the database.

[1156] Step 6:

[1157] The server generates learning content using the stored dialect data and provides it to the device. Specifically, it generates learning content in the form of quizzes and games, allowing users to learn dialects interactively. The input is the dialect data stored in the database, and the output is the learning content provided to the device.

[1158] Step 7:

[1159] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time. The input is the user's facial expression and voice data, and the output is the recognized emotion data.

[1160] Step 8:

[1161] The server dynamically adjusts the difficulty and content of learning content based on emotion recognition. If the user is excited, it provides challenging questions, and if the user is stressed, it provides easy questions. The input is the recognized emotion data, and the output is dynamically adjusted learning content.

[1162] Step 9:

[1163] The device uses Unity or the Oculus SDK to render a virtual reality or augmented reality environment, providing a dialect experience to the user. The user can immerse themselves in this virtual environment using a VR headset or an AR-enabled device. The input is dialect data and scenario data stored in the database, and the output is the VR or AR environment provided to the user.

[1164] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1165] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1166] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1167] [Third embodiment]

[1168] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1169] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1170] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1171] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1172] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1173] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1174] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1175] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1176] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1177] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1178] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1179] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1180] The system of the present invention collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. Detailed description of the embodiments of the present invention follows.

[1181] The system includes programs that perform various processes, such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. Each process is realized through the cooperative operation of the user, the terminal, and the server.

[1182] 1. Audio data collection:

[1183] User: Launch the application and select the "Record Dialect" section. Press the record button that appears and speak a specific dialect phrase.

[1184] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[1185] 2. Uploading audio data:

[1186] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[1187] Server: Processes and temporarily stores the received audio data.

[1188] 3. Analysis of audio data:

[1189] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[1190] Terminal: The analysis results are returned and the converted text is presented to the user.

[1191] 4. Check and correct text data:

[1192] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal.

[1193] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[1194] 5. Save to database:

[1195] Server: Stores the final text and audio data, as well as related metadata, in a database.

[1196] Device: Sends a notification to the user that the save is complete.

[1197] 6. Providing interactive learning tools:

[1198] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1199] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1200] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[1201] 7. Dialect Reproduction and Simulation:

[1202] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[1203] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1204] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[1205] 8. Dialect experience using VR and AR:

[1206] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1207] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[1208] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[1209] 9. Multilingual Translation and Dialect Preservation:

[1210] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[1211] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[1212] Server: Stores the translated data in the final database.

[1213] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1214] As described above, this system can consistently carry out each process to effectively preserve and inherit dialects.

[1215] The processing flow will be explained below.

[1216] Step 1:

[1217] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1218] Step 2:

[1219] User: Press the record button, speak the dialect phrase as instructed, and press the stop button when finished recording.

[1220] Step 3:

[1221] Device: Temporarily saves the audio data in local storage, then sends a request to upload it to the server.

[1222] Step 4:

[1223] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​speech recognition module for analysis.

[1224] Step 5:

[1225] Server: The AI ​​speech recognition module converts the voice data into text, and stores the converted text data in a temporary database.

[1226] Step 6:

[1227] Server: Returns the analysis results to the device.

[1228] Step 7:

[1229] Terminal: The analysis result text is displayed in the UI and the user is asked to confirm.

[1230] Step 8:

[1231] User: Checks the text analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[1232] Step 9:

[1233] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[1234] Step 10:

[1235] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[1236] Step 11:

[1237] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1238] Step 12:

[1239] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1240] Step 13:

[1241] User: Interacts with the provided learning content and receives feedback on correct and incorrect answers.

[1242] Step 14:

[1243] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[1244] Step 15:

[1245] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1246] Step 16:

[1247] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[1248] Step 17:

[1249] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1250] Step 18:

[1251] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[1252] Step 19:

[1253] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[1254] Step 20:

[1255] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[1256] Step 21:

[1257] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[1258] Step 22:

[1259] Server: Stores the translated data in the final database.

[1260] Example 1

[1261] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1262] In modern times, regional dialects are rapidly disappearing, making it difficult to pass them on to the next generation. Furthermore, those who want to learn dialects lack the appropriate learning tools, making it difficult to learn them efficiently. Furthermore, there is a lack of systems for consistently collecting, analyzing, and storing dialect speech data, and there is a need to efficiently carry out these tasks.

[1263] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1264] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the converted text to the user and accepting correction requests as necessary, means for saving the final text data and voice data in a database, means for providing learning content using the saved dialect data, means for providing interaction through the learning content and giving feedback to the user, and means for translating the dialect data into other languages, thereby enabling the effective preservation and inheritance of dialects.

[1265] "Audio data" refers to audio information recorded by a user, and is data that includes specific dialects and phrases.

[1266] A "collection means" is a part of a system that provides a method or facility for a user to record audio data.

[1267] "Means of uploading" refers to the method or function for transferring collected audio data to an online server.

[1268] "Means for analyzing" means a method or function that uses technology such as an AI voice recognition module to convert uploaded voice data into text format.

[1269] "Means of presentation" refers to the methods and functions for displaying and confirming analysis results, learning content, translation results, etc. to the user.

[1270] "Means for accepting correction requests" refers to a method or function that allows a user to request correction of errors in the analysis results.

[1271] "Storage means" refers to the method or function for permanently storing the final text data and audio data in a database.

[1272] "Learning Content" means the learning materials and activities (e.g., quizzes, games, vocabulary exercises) provided to users to help them learn a dialect.

[1273] "Means of interaction" refers to the methods and features that allow users to actively engage with the learning content and receive feedback.

[1274] "Means for translating" means a method or function that uses a multilingual translation AI model to convert dialect data into another language.

[1275] "Means of reproduction" refers to the method or function of generating speech using Text-to-Speech (TTS) technology using stored dialect data.

[1276] "Virtual reality technology" refers to technology that allows users to immerse themselves in a virtual environment using a VR headset or similar device.

[1277] "Augmented reality technology" refers to the technology of overlaying digital information onto the real world using AR-enabled devices.

[1278] MODE FOR CARRYING OUT THE INVENTION

[1279] The system according to the present invention performs processes such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. These processes are realized through the cooperative operation of a user, a terminal, and a server. Specific embodiments are described below.

[1280] Audio data collection

[1281] Users launch the smartphone app, select the "Dialect Recording" section, press the record button, and speak a specific dialect phrase. The device detects that the record button has been pressed and begins recording the user's voice using the built-in microphone. The recorded voice data is temporarily stored in local storage.

[1282] Uploading audio data

[1283] After the recording is complete, the device sends an upload request to the server. The recorded audio data is compressed and transferred to the server. The user is informed of the upload progress using a progress bar. The server temporarily stores the received audio data and prepares for the next process.

[1284] Analysis of audio data

[1285] The server passes the voice data to an AI speech recognition module, which converts the voice into text. The text conversion results are stored in a temporary database on the server, and the results are sent back to the device, which then presents the text analysis results to the user.

[1286] Check and correct text data

[1287] The user checks the displayed text analysis results, and if there are any errors, sends a correction request to the server via the terminal. The server receives the correction request and either performs a new analysis or accepts the user's manual correction.

[1288] Saving to a database

[1289] The final text and audio data, along with associated metadata, are then permanently stored in a database by the server, and the device notifies the user when the storage is complete.

[1290] Providing interactive learning tools

[1291] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[1292] Dialect reproduction and simulation

[1293] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology. The generated voice is provided to the user via the terminal, and a play button and a feedback form are displayed. The user can listen to the pronunciation and intonation of the reproduced dialect.

[1294] Dialect experience using VR and AR

[1295] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[1296] Multilingual Translation and Dialect Preservation

[1297] When a user sends a translation request, the device forwards it to the server. The server uses a multilingual translation AI model to translate the dialect data into another language. The translation results are presented to the user via the device, and correction requests are accepted as needed. The final data is stored in a database by the server.

[1298] Specific examples

[1299] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1300] Prompt Sentence Examples

[1301] Example prompt sentence:

[1302] "Please record yourself saying 'The weather is nice' in the Niigata dialect."

[1303] Upload the recorded audio to the server.

[1304] Once the upload is complete, check the text conversion results.

[1305] "Make any necessary corrections."

[1306] Save the final result.

[1307] "Learn the Niigata dialect using the learning module."

[1308] In this way, the system of the present invention can consistently perform each process for effectively preserving and inheriting dialects.

[1309] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1310] Step 1:

[1311] The user launches the application and selects the "Dialect Recording" section. Next, they press the record button and speak the specified phrase. The device detects this and begins recording the audio using the built-in microphone. The recording is temporarily saved in local storage.

[1312] Input: Pressing the record button, user voice

[1313] Output: Audio data stored in local storage

[1314] Specific operation: The recording module detects the button press event and starts the recording process. After the recording is completed, the data is saved in a file format.

[1315] Step 2:

[1316] After the device has finished recording, it sends a request to upload the audio data to the server. The audio data is compressed and sent to the server as an HTTP POST request. The upload progress is displayed to the user. The server temporarily stores the received audio data.

[1317] Input: Audio data stored in local storage

[1318] Output: Audio data stored on the server

[1319] Specific operation: The device compresses the audio data and sends it to the server using the HTTP protocol. The server receives the data and stores it in temporary storage.

[1320] Step 3:

[1321] The server passes the uploaded voice data to an AI speech recognition module, which converts the voice into text, and the conversion results are stored in a temporary database on the server.

[1322] Input: Audio data stored on the server

[1323] Output: Conversion result in text format

[1324] Specific operation: The server calls the speech recognition API to convert the voice data into text format, which is then stored in the database.

[1325] Step 4:

[1326] The server notifies the terminal of the conversion results, and the terminal presents the text analysis results to the user.

[1327] Input: Conversion result in text format

[1328] Output: Text analysis results presented to the user

[1329] Specific operation: The server sends text data to the terminal, and the terminal displays it on the user interface.

[1330] Step 5:

[1331] The user checks the analysis results and, if necessary, sends a correction request to the server via the terminal. The server receives the correction request and performs a new analysis or approves manual correction.

[1332] Input: User correction request

[1333] Output: Corrected text data

[1334] Specific operation: The user makes corrections using a text editor, and the device sends the corrections to the server, which saves them in the database.

[1335] Step 6:

[1336] The server stores the final text data, audio data, and related metadata in a permanent storage database. Once the storage is complete, the device notifies the user.

[1337] Input: Final text and audio data

[1338] Output: Notification of save completion

[1339] Specific operation: The server saves the data permanently in the database and notifies the device of the status. The device notifies the user that the save is complete.

[1340] Step 7:

[1341] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[1342] Input: User access to learning content

[1343] Output: User learning progress and feedback

[1344] Specific operation: The device retrieves learning content from the server and displays it to the user. Feedback is provided based on the user's interactions.

[1345] Step 8:

[1346] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology, which is then provided to the user via their device, along with a playback button and a feedback form.

[1347] Input: Stored dialect data

[1348] Output: Reproduced audio provided to the user

[1349] Specific operation: The server generates speech using TTS technology, and the device displays a play button to the user, allowing the user to listen to the generated speech.

[1350] Step 9:

[1351] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[1352] Input: User access to the dialect experience module

[1353] Output: A virtual reality recreation of a dialect scene

[1354] Specific operation: The device uses VR / AR technology to create a virtual environment and fully render it so that users can experience the dialect within it.

[1355] Step 10:

[1356] The device receives translation requests from users and forwards them to the server. The server uses a multilingual translation AI model to translate the dialect data into other languages ​​and presents the results to the user via the device. The user can then submit correction requests as needed. The final data is stored in a database by the server.

[1357] Input: Translation request from user

[1358] Output: Data translated into other languages

[1359] Specific operation: The server translates the dialect data using the translation AI model, displays the translation results to the user through the terminal, and saves the final data in the database based on the user's feedback.

[1360] (Application example 1)

[1361] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1362] Conventional dialect learning systems simply record audio data and display the resulting text, lacking interactive feedback for learners and practical experience. Furthermore, they lack the ability to reuse saved dialect data or translate it into other languages, making it difficult to accommodate a wide range of users. Furthermore, they lack the ability to provide a learning environment that utilizes virtual reality or augmented reality technology.

[1363] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1364] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for providing a variety of voice learning content using the saved dialect data, means for recreating situations in which the dialect is used using virtual reality or augmented reality technology, and means for translating the dialect into other languages, thereby enabling learners to learn the dialect through an interactive and realistic experience and to reuse the saved data for translation into other languages.

[1365] "Voice data" refers to digital data that electronically records the voice of a user speaking in a dialect or standard Japanese.

[1366] A "collection instrument" is a combination of hardware and software used to record and store a user's voice using a device.

[1367] The "means for uploading" is a mechanism for transferring collected voice data to a server via a network.

[1368] The "means for analyzing and converting to text" refers to algorithms and software for converting voice data into text information using voice recognition technology.

[1369] The "means for presenting to the user" is an interface for presenting the analysis results to the user visually or audibly.

[1370] The "means for storing in a database" refers to a storage system for temporarily or long-term storage of analyzed text data and audio data.

[1371] "Means for providing learning content" refers to a mechanism that provides interactive learning materials such as quizzes, games, and vocabulary practice to users to help them learn the dialect.

[1372] The "means for providing a variety of phonetic learning content" is a system for generating and providing various forms of phonetic learning content using stored dialect data.

[1373] "Virtual reality technology" or "augmented reality technology" refers to technology that allows users to experience virtual environments and information in real time, simulating specific scenarios or environments.

[1374] "Means for recreating situations in which dialects are used" refers to a system that uses virtual reality and augmented reality technology to recreate situations in which a specific dialect is actually used.

[1375] "Means for translating into other languages" refers to a multilingual translation system or algorithm for converting dialect data into other languages.

[1376] This invention relates to a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. This system includes a program that collects, uploads, analyzes, presents, and stores the speech data, and provides learning content. Specific embodiments of this system are described below.

[1377] 1. Collection of audio data

[1378] The user launches the application on a device such as a smartphone or tablet, selects the recording section, presses the record button, and speaks a dialect phrase. The device uses WebRTC to record the voice data and temporarily stores it in local storage.

[1379] 2. Uploading audio data

[1380] The device sends an HTTP POST request to upload the recorded audio data to the server. The progress is displayed on the screen to inform the user. The server receives the request using Flask and temporarily stores the audio data.

[1381] 3. Analysis of audio data

[1382] The server inputs the voice data into a speech recognition model using TensorFlow and converts it into text. The conversion results are temporarily stored in a MySQL database. The conversion results are then sent back to the device as an HTTP response, and the text result is presented to the user.

[1383] 4. Check and correct the analysis results

[1384] The user checks the text of the analysis result and sends a correction request if there is an error. The server then re-analyzes or manually corrects the text and stores it in the database again.

[1385] 5. Saving to the database

[1386] The server saves the final analysis results in a MySQL database and sends a notification to the terminal that the results have been saved.

[1387] 6. Providing interactive learning content

[1388] Users access the "Dialect Learning" module and use learning content such as quizzes and vocabulary practice. The server dynamically generates this content and provides it to users.

[1389] 7. Dialect Reproduction and Simulation

[1390] The server uses Text-to-Speech (TTS) technology to reproduce the speech of the dialect data stored and transmits it to the terminal. The user listens to the reproduced speech and learns pronunciation and intonation.

[1391] 8. Dialect experience using VR and AR

[1392] Users access the "Dialect Experience" module using a VR headset or AR-enabled device, and the server uses Unity or Unreal Engine to render the virtual environment and recreate specific dialect-using situations.

[1393] 9. Multilingual Translation

[1394] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database.

[1395] (Example)

[1396] For example, if a user follows the instruction "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect," the recorded audio data is uploaded to the server, analyzed, and converted into text. The results of this analysis are presented to the user, and any errors can be corrected. The data is then stored in a database and used as learning content for other users.

[1397] (Example of a prompt)

[1398] Audio collection prompt: "Please record yourself saying 'The weather is nice' in the Niigata dialect. Press the record button and start speaking."

[1399] Correction prompt: "Your voice data has been analyzed. The following text is the analysis result. Please correct any errors."

[1400] Learning Content Prompt: "Learn the following phrases. Check your pronunciation and take the quiz."

[1401] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1402] Step 1:

[1403] A user launches the application on a device such as a smartphone or tablet and selects the "Dialect Recording" section. The input is the user's actions and the dialect phrase to be recorded. Specifically, the user presses the record button and speaks a specific dialect phrase. The device uses WebRTC to record the voice data in real time and temporarily saves it in local storage. The output is the recorded voice data.

[1404] Step 2:

[1405] When the user finishes recording, the device uploads the audio data to the server. The input is the audio data recorded in step 1. The device sends an HTTP POST request to transfer the audio data to the server. The server receives the request using Flask and temporarily stores the audio data. The output is the audio data stored on the server.

[1406] Step 3:

[1407] To analyze the stored voice data, the server inputs the data into a voice recognition model using TensorFlow. The input is the voice data received in step 2. Specifically, the voice recognition model converts the voice data into text and generates analysis results in text format. The output is the converted text data.

[1408] Step 4:

[1409] The server temporarily stores the analysis results in a MySQL database and returns the text data to the terminal as an HTTP response. The input is the text data obtained in step 3. The terminal displays the analysis results to the user. The output is the text analysis results presented to the user.

[1410] Step 5:

[1411] The user checks the text of the analysis results and, if there are any errors, sends a correction request. The input is the displayed text data and the user's correction request. The terminal transfers the correction request to the server. The output is data including the correction request. The server re-analyzes or performs manual corrections and stores the data in the database again. Finally, corrected text data is generated.

[1412] Step 6:

[1413] The server stores the final analysis results in a MySQL database. The input is the data corrected in step 5. This stored data includes the audio data, associated text data, and metadata. The output is the final analysis results stored in the database.

[1414] Step 7:

[1415] A user accesses the "Dialect Learning" module. The input is dialect data stored in the database. The server dynamically generates learning content such as quizzes and vocabulary exercises using the stored dialect data and sends it to the terminal. The output is interactive learning content provided to the user.

[1416] Step 8:

[1417] The server uses Text-to-Speech (TTS) technology to convert the stored dialect data into speech and send it to the terminal. The input is text data from the database. The user listens to the reproduced speech and learns pronunciation and intonation. The output is the reproduced speech data.

[1418] Step 9:

[1419] A user accesses the "Dialect Experience" module using a VR headset or AR-enabled device. The input is the saved dialect data and the corresponding scenario settings. The server uses Unity or Unreal Engine to render the virtual environment and recreate a specific dialect usage scene. The output is the virtual environment provided to the user.

[1420] Step 10:

[1421] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database. The input is the stored dialect data. The user checks and corrects the translation results as needed and uses the final data. The output is the translated data.

[1422] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1423] The system of the present invention collects, analyzes, and stores dialect speech data, provides interactive tools for learning the dialect, and combines it with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[1424] The system includes programs for collecting, uploading, analyzing, presenting, and saving voice data, providing learning content, and recognizing user emotions. Each process is realized through the cooperative operation of the user, the device, and the server.

[1425] 1. Audio data collection:

[1426] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1427] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[1428] 2. Uploading audio data:

[1429] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[1430] Server: Processes and temporarily stores the received audio data.

[1431] 3. Analysis of audio data:

[1432] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[1433] Terminal: The analysis results are returned and the converted text is presented to the user.

[1434] 4. Check and correct text data:

[1435] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[1436] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[1437] 5. Save to database:

[1438] Server: Stores the final text and audio data, as well as related metadata, in a database.

[1439] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[1440] 6. Providing interactive learning tools:

[1441] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1442] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1443] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[1444] 7. Dialect Reproduction and Simulation:

[1445] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[1446] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1447] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[1448] 8. Dialect experience using VR and AR:

[1449] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1450] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[1451] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[1452] 9. Multilingual Translation and Dialect Preservation:

[1453] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[1454] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[1455] Server: Stores the translated data in the final database.

[1456] 10. Incorporating an Emotion Engine:

[1457] Device: Incorporates an emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[1458] Server: Stores the recognized emotion data and reflects it in analysis results and learning content.

[1459] 11. Emotion-based learning content adjustment:

[1460] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine. For example, if the user is excited while learning, it provides more challenging questions, and if the user is stressed, it changes to easier questions.

[1461] 12. Emotional Data Storage:

[1462] Server: Receives user emotion data from the emotion engine and stores it in a database along with other dialect data.

[1463] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1464] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions. On the other hand, if the system recognizes that the user is feeling stressed, it will switch to easier questions. Emotional data is also stored in a database and will be used to improve dialect learning in the future.

[1465] In this way, this system can effectively preserve and pass on dialects while providing an interactive learning experience that reflects the user's emotions.

[1466] The processing flow will be explained below.

[1467] Step 1:

[1468] User: Launches the application and selects the "Record Dialect" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1469] Step 2:

[1470] User: Press the record button, speak the dialect phrase provided, and press the stop button when finished.

[1471] Step 3:

[1472] Device: Recorded audio data is temporarily stored in local storage.

[1473] Step 4:

[1474] Terminal: Sends a request to upload audio data to the server. Displays the upload progress to the user.

[1475] Step 5:

[1476] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​voice recognition module.

[1477] Step 6:

[1478] Server: The AI ​​speech recognition module converts the voice data into text, which is then stored in a temporary database.

[1479] Step 7:

[1480] Server: Returns the text analysis results to the device.

[1481] Step 8:

[1482] Terminal: The analysis result text is presented to the user and confirmation is sought.

[1483] Step 9:

[1484] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no errors, presses the confirmation button.

[1485] Step 10:

[1486] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[1487] Step 11:

[1488] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[1489] Step 12:

[1490] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1491] Step 13:

[1492] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1493] Step 14:

[1494] User: Interacts with the provided learning content and receives feedback on whether their answers are correct or incorrect.

[1495] Step 15:

[1496] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[1497] Step 16:

[1498] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1499] Step 17:

[1500] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[1501] Step 18:

[1502] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1503] Step 19:

[1504] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[1505] Step 20:

[1506] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[1507] Step 21:

[1508] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[1509] Step 22:

[1510] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[1511] Step 23:

[1512] Server: Stores the translated data in the final database.

[1513] Step 24:

[1514] On the device: Activate the emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[1515] Step 25:

[1516] Server: Receives and temporarily stores the user's emotion data recognized by the emotion engine.

[1517] Step 26:

[1518] Server: Uses emotional data to adjust learning content, for example, making the content less difficult if the user is feeling stressed.

[1519] Step 27:

[1520] Device: Presents tailored learning content to users and supports them to keep them engaged.

[1521] Step 28:

[1522] Server: Stores the user's emotional data in a final database for future analysis and improvement.

[1523] Example 2

[1524] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1525] In recent years, the preservation and inheritance of dialects has become increasingly important from the perspective of cultural value. However, conventional dialect learning systems have limitations in collecting and analyzing speech data and providing learning content, and in particular lack the ability to optimize learning while taking into account the user's emotions. They are also inadequate in reproducing dialect data and providing an interactive learning environment using virtual reality technology. The present invention aims to solve these problems.

[1526] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for the user to confirm and correct the text, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for incorporating an emotion engine that recognizes the user's emotions in real time, means for dynamically adjusting the learning content based on the user's emotions recognized by the emotion engine, and means for saving the user's emotion data in a database. This makes it possible to provide an interactive and optimized learning experience that takes the user's emotions into consideration.

[1527] "Audio data" refers to data that records sound in digital or analog format.

[1528] A "means of collection" is a device or software that provides the function of recording audio and storing it as data.

[1529] An "uploading means" is a device or software that provides the function of transmitting data stored in local storage to a remote server.

[1530] An "analyzing means" is a device or software that provides the functionality to process collected voice data and convert it into text.

[1531] "Means for converting to text" refers to software or a module that has the function of converting voice data into character data.

[1532] The "presentation means" refers to a device or software that has the function of allowing a server or terminal to display the processing results to the user visually or audibly.

[1533] A "verification and correction means" is a device or software that provides an interface or functionality for a user to review the presented information and correct it if necessary.

[1534] "Means for storage" refers to a database or storage device for permanently recording the final data.

[1535] "Learning Content" refers to the educational materials and interactive activities provided to users to help them learn a dialect.

[1536] An "emotion engine" is software or a module that analyzes a user's facial expressions, tone of voice, etc., and recognizes emotions in real time.

[1537] A "dynamic adjustment means" is a device or software that provides the functionality to change the difficulty or content of learning content in real time based on the user's emotional state.

[1538] The "means for converting to text" is software or hardware for converting voice data into text data.

[1539] A "reproduction means" is a device or software that provides the functionality to generate sounds and virtual scenes based on stored data.

[1540] "User" refers to an individual or end user who uses the System.

[1541] "Interactive" refers to the ability of a user to elicit a response from the system through their own actions.

[1542] "Virtual reality technology" refers to technology that allows users to experience computer-generated three-dimensional environments.

[1543] "Augmented reality technology" refers to technology that displays digital information overlaid on the real world.

[1544] The present invention is a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. In addition, by incorporating an emotion engine that recognizes the user's emotions, effective dialect learning is realized. The detailed processing of the system is described below.

[1545] Hardware and software used

[1546] The system uses the following hardware and software:

[1547] Mobile information terminals such as smartphones and tablets

[1548] Servers (including cloud-based storage and processing resources)

[1549] Voice Recording Module

[1550] AI voice recognition module

[1551] Text-to-Speech (TTS) software

[1552] Emotion Recognition Engine

[1553] Database Management Systems

[1554] Audio data collection

[1555] The user starts the application using a mobile information device such as a smartphone or tablet and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording instruction phrase (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect"). When the user presses the record button, the device's built-in microphone is used to record audio, which is then temporarily saved in local storage.

[1556] Uploading audio data

[1557] The recorded audio data is uploaded from the device to the server. The device sends a request to the server and displays the upload progress to the user. The server receives the audio data and stores it in temporary storage.

[1558] Analysis of audio data

[1559] The server passes the uploaded voice data to an AI speech recognition module, which converts it into text. This module analyzes the voice data and generates corresponding text data. The generated text data is stored in a temporary database. The analysis results are sent back from the server to the device, and the converted text is presented to the user.

[1560] Check and correct text data

[1561] The user checks the presented text data and, if there are any errors, sends a correction request to the server via their terminal. If there are no problems, they press the "Confirm" button to proceed. The server receives the correction request and either performs a new analysis or approves the user's manual correction.

[1562] Saving to a database

[1563] The server stores the final text and audio data, along with associated metadata, in a database. Once the data has been saved, the device displays a notification to the user stating, "Data saved successfully."

[1564] Providing interactive learning tools

[1565] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the provided content and receives feedback on whether their answers are correct or incorrect.

[1566] Dialect reproduction and simulation

[1567] The server uses the stored dialect data to generate a reproduced speech using Text-to-Speech (TTS) technology. This speech is provided to the terminal, and the user can listen to the reproduced speech by pressing the play button. In addition, a feedback form can be used to provide a rating.

[1568] Dialect experience using VR and AR

[1569] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device renders a virtual reality or augmented reality environment, recreating specific dialect-using situations (e.g., market, home, school). Within the virtual environment, users are immersed in dialect-based scenarios, providing a realistic dialect experience.

[1570] Multilingual Translation and Dialect Preservation

[1571] The server uses a multilingual translation AI model to translate the dialect data into other languages. The translation results are then sent back to the device, where the user can review them and send correction requests if necessary. The final translation data is stored in a database.

[1572] Incorporating an emotional engine and adjusting learning content

[1573] The device uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice. The recognized emotion data is sent to a server, which then dynamically adjusts the difficulty and content of the learning content. The emotion data is also stored in a database to help improve dialect learning in the future.

[1574] Examples and prompts

[1575] For example, a user might use a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. The data is finally stored in a database and used as learning content. Additionally, an emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and the learning content is adjusted based on that data.

[1576] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1577] Step 1:

[1578] Audio data collection

[1579] User: Launches the smartphone app and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1580] Input: The user presses the record button.

[1581] Process: The device will record audio using the built-in microphone. When you finish recording, press the stop button.

[1582] Output: The recorded audio data is temporarily saved in the device's local storage.

[1583] Step 2:

[1584] Uploading audio data

[1585] Device: Sends a request to upload the recorded audio data to the server.

[1586] Input: Temporarily saved audio data.

[1587] Processing: The device uploads the audio data to the server and displays the upload progress to the user in real time.

[1588] Output: The audio data is saved in temporary storage on the server.

[1589] Step 3:

[1590] Analysis of audio data

[1591] Server: Passes the uploaded voice data to the AI ​​voice recognition module.

[1592] Input: Audio data uploaded to the server.

[1593] Processing: The AI ​​speech recognition module analyzes the voice data and converts it into text format.

[1594] Output: Save the converted text data in a temporary database.

[1595] Step 4:

[1596] User presentation of text data

[1597] Server: Sends the conversion results to the terminal.

[1598] Input: Text data stored in a temporary database.

[1599] Processing: The text data is returned to the terminal.

[1600] Output: Text data is displayed on the terminal.

[1601] Step 5:

[1602] Check and correct text data

[1603] User: Checks the presented text and sends a correction request to the server via the terminal if necessary. If there are no problems, the user presses the "Confirm" button.

[1604] Input: The displayed text data.

[1605] Action: The user confirms the error and submits a correction request, or presses the confirm button to proceed.

[1606] Output: The modification request is sent to the server or the confirm button is pressed.

[1607] Step 6:

[1608] Finalize text data

[1609] Server: Receives the correction request and performs a new analysis or approves manual corrections. After the corrections are confirmed, they are saved in the database.

[1610] Input: Correction request or user confirmation.

[1611] Processing: Performing a new analysis or manual correction is processed to approve and save it in the database.

[1612] Output: The final text and audio data and associated metadata are stored in a database.

[1613] Step 7:

[1614] Save completion notification

[1615] Server: Notifies completion of data saving.

[1616] Input: The final data is saved in the database.

[1617] Processing: Send a save completion notification to the terminal.

[1618] Output: The device displays the message "Data saved successfully" to the user.

[1619] Step 8:

[1620] Providing interactive learning content

[1621] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1622] Input: Access from the user.

[1623] Processing: The device provides learning content (e.g., quizzes, games, vocabulary practice).

[1624] Output: User accesses and interacts with learning content.

[1625] Step 9:

[1626] Dialect reproduction and simulation

[1627] Server: Using the stored dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[1628] Input: Dialect data stored in a database.

[1629] Processing: Generate speech using TTS technology.

[1630] Output: The generated audio data.

[1631] Step 10:

[1632] Providing reproduced audio

[1633] Terminal: Provides the user with the reproduced audio and displays a play button and a feedback form.

[1634] Input: The generated audio data.

[1635] Processing: Play audio and collect feedback.

[1636] Output: You can hear the reproduced audio.

[1637] Step 11:

[1638] Dialect experience using VR and AR

[1639] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1640] Input: Access from the user.

[1641] Processing: The device renders a virtual or augmented reality environment, recreating a specific dialect-using situation (e.g., market, home, school).

[1642] Output: The user is immersed in a dialect-based scenario within a virtual environment.

[1643] Step 12:

[1644] Multilingual Translation and Dialect Preservation

[1645] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[1646] Input: Dialect data stored in a database.

[1647] Processing: Performs the translation using a multilingual translation AI model.

[1648] Output: The translated text data.

[1649] Step 13:

[1650] User presentation and correction of translation results

[1651] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[1652] Input: Translated text data.

[1653] Processing: Display the translation results and receive correction requests.

[1654] Output: A correction request or confirmation from the user.

[1655] Step 14:

[1656] Incorporating an emotion engine

[1657] Device: Uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice.

[1658] Input: Data such as the user's facial expressions and tone of voice.

[1659] Processing: Real-time analysis using emotion recognition engine.

[1660] Output: Recognized emotion data.

[1661] Step 15:

[1662] Emotion-based learning content adjustment

[1663] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine.

[1664] Input: Recognized emotion data.

[1665] Processing: Perform processing to dynamically adjust learning content based on emotions.

[1666] Output: Tailored learning content.

[1667] Step 16:

[1668] Storing Emotional Data

[1669] Server: Stores user emotion data in a database.

[1670] Input: Recognized emotion data.

[1671] Processing: The emotion data is stored in the database.

[1672] Output: Saved emotion data.

[1673] (Application example 2)

[1674] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1675] While the preservation and learning of dialects is important for cultural inheritance, existing systems have problems in that they are unable to efficiently collect and analyze dialect speech data and provide learning content. Furthermore, they do not provide appropriate feedback based on the user's learning situation or emotions, making it difficult to maintain motivation and provide an effective learning environment. Furthermore, they are unable to realistically recreate situations in which dialects are used, resulting in insufficient practice in actual communication situations.

[1676] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1677] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for recognizing the user's emotions in real time, means for dynamically adjusting the difficulty and content of the learning content based on the recognized emotions, and means for providing a dialect experience using virtual reality or augmented reality, thereby enabling more effective dialect learning and providing an interactive learning experience that responds to the user's emotions.

[1678] "Audio data" is data that represents audio information in digital form.

[1679] "Means of collection" refers to the functions of the device or software used to acquire and record audio data.

[1680] "Means for uploading" refers to the functionality of the device or software used to transfer collected voice data to an external server or cloud.

[1681] "Means for analyzing and converting into text" refers to the function of a device or software for converting voice data into language data.

[1682] "Presentation means" refers to the function of a device or software for visually or audibly presenting analysis results or information to a user.

[1683] "Means for storage" refers to the functions of devices and software for recording and storing analysis results and audio data in a database or storage.

[1684] "Means for providing learning content" refers to the functionality of devices or software that provide learning materials and interactive activities for users to learn dialects.

[1685] "Emotion recognition means" refers to a device or software function for recognizing and analyzing emotions from a user's facial expressions and voice.

[1686] "Means for dynamically adjusting difficulty and content" refers to the functionality of devices or software that adjusts the difficulty and content of learning content in real time according to the user's emotions and learning situation.

[1687] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[1688] "Augmented reality" is a technology that overlays computer-generated information onto the real world.

[1689] A "means for providing a dialect experience" is a device or software function that allows a user to experience a scenario in which a dialect is used in a virtual space or an augmented reality environment.

[1690] The system of the present invention collects, analyzes, and stores speech data, and then provides learning content based on that data. The system recognizes users' emotions in real time and dynamically adjusts the learning experience based on that information. It also uses virtual reality (VR) and augmented reality (AR) to provide an interactive environment for users to learn dialects.

[1691] The system mainly includes the following means:

[1692] 1. How to collect audio data

[1693] Users launch the application on their smartphone, tablet, or other device, select the "Dialect Recording" section, press the record button, speak a phrase in the specified dialect, and the device will record the audio data.

[1694] 2. How to upload audio data

[1695] The device uploads the recorded audio data to the server, and the upload progress is displayed to the user.

[1696] 3. Methods for analyzing audio data

[1697] The server receives the uploaded voice data, passes it to an AI voice recognition module (such as Google Cloud's "Speech-to-Text API"), and converts the voice into text.

[1698] 4. Presentation of analysis results

[1699] The terminal presents the analysis results returned from the server to the user, who then checks the analysis results and sends a correction request if necessary.

[1700] 5. Database storage method

[1701] The server stores the final text and audio data, as well as associated metadata, in a database (e.g., Firebase).

[1702] 6. Means of providing learning content

[1703] The terminal uses the stored dialect data to provide the user with learning content (such as quizzes and games).

[1704] 7. Emotion recognition means

[1705] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time.

[1706] 8. Dynamic adjustment of learning content

[1707] The server dynamically adjusts the difficulty and content of learning content based on the user's recognized emotions, for example, providing more challenging problems if the user is excited, and easier problems if the user is stressed.

[1708] 9. Means of providing dialect experiences

[1709] The device uses Unity or the Oculus SDK to render virtual reality or augmented reality environments, allowing users to experience specific dialect settings (e.g., a market or home). Users can immerse themselves in these scenarios using a VR headset or AR-enabled device.

[1710] As a specific use case, consider a case where a user uses a smartphone app and follows the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1711] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions, but if they are feeling stressed, it will switch to easier questions.

[1712] Example prompt sentence:

[1713] "Press the record button and say, 'The weather is nice, isn't it?' in Niigata dialect."

[1714] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1715] Step 1:

[1716] A user uses a device such as a smartphone or tablet to launch the application and select the "Dialect Recording" section. Then, the user presses the record button and speaks the specified phrase in the dialect. In this process, the device acquires and records audio data through the microphone input. The input is the user's voice, and the output is the recorded audio data.

[1717] Step 2:

[1718] When the user stops recording, the device uploads the recorded audio data to the server. Specifically, the device sends an HTTP request to the server and attaches the audio file. The input is the recorded audio data, and the output is the audio file uploaded to the server.

[1719] Step 3:

[1720] The server passes the received voice data to an AI voice recognition module, which converts the voice into text data. In this process, Google Cloud's "Speech-to-Text API" is used to analyze the voice data and convert it into text format. The input is the voice data uploaded to the server, and the output is text data.

[1721] Step 4:

[1722] The server returns the analyzed text data to the terminal, and the terminal displays the analysis results to the user. At this time, the user can check the analysis results and send a correction request if there are any errors. The input is the text data returned from the server, and the output is the text data displayed on the terminal.

[1723] Step 5:

[1724] The user checks the text data and presses the confirm button if there are no problems. The terminal sends a confirmation notification to the server, and the server stores the final analysis results in a database. The input is the user's confirmation operation and the text data, and the output is the final analysis results stored in the database.

[1725] Step 6:

[1726] The server generates learning content using the stored dialect data and provides it to the device. Specifically, it generates learning content in the form of quizzes and games, allowing users to learn dialects interactively. The input is the dialect data stored in the database, and the output is the learning content provided to the device.

[1727] Step 7:

[1728] The device uses Microsoft Azure's "Emotion API" and Google Cloud's "Cloud Vision API" to recognize emotions from the user's facial expressions and voice in real time. The input is the user's facial expression and voice data, and the output is the recognized emotion data.

[1729] Step 8:

[1730] The server dynamically adjusts the difficulty and content of learning content based on emotion recognition. If the user is excited, it provides challenging questions, and if the user is stressed, it provides easy questions. The input is the recognized emotion data, and the output is dynamically adjusted learning content.

[1731] Step 9:

[1732] The device uses Unity or the Oculus SDK to render a virtual reality or augmented reality environment, providing a dialect experience to the user. The user can immerse themselves in this virtual environment using a VR headset or an AR-enabled device. The input is dialect data and scenario data stored in the database, and the output is the VR or AR environment provided to the user.

[1733] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1734] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1735] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1736] [Fourth embodiment]

[1737] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1738] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1739] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1740] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1741] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1742] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1743] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1744] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1745] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1746] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1747] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1748] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1749] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] The system of the present invention collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. Detailed description of the embodiments of the present invention follows.

[1751] The system includes programs that perform various processes, such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. Each process is realized through the cooperative operation of the user, the terminal, and the server.

[1752] 1. Audio data collection:

[1753] User: Launch the application and select the "Record Dialect" section. Press the record button that appears and speak a specific dialect phrase.

[1754] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[1755] 2. Uploading audio data:

[1756] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[1757] Server: Processes and temporarily stores the received audio data.

[1758] 3. Analysis of audio data:

[1759] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[1760] Terminal: The analysis results are returned and the converted text is presented to the user.

[1761] 4. Check and correct text data:

[1762] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal.

[1763] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[1764] 5. Save to database:

[1765] Server: Stores the final text and audio data, as well as related metadata, in a database.

[1766] Device: Sends a notification to the user that the save is complete.

[1767] 6. Providing interactive learning tools:

[1768] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1769] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1770] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[1771] 7. Dialect Reproduction and Simulation:

[1772] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[1773] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1774] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[1775] 8. Dialect experience using VR and AR:

[1776] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1777] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[1778] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[1779] 9. Multilingual Translation and Dialect Preservation:

[1780] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[1781] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[1782] Server: Stores the translated data in the final database.

[1783] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1784] As described above, this system can consistently carry out each process to effectively preserve and inherit dialects.

[1785] The processing flow will be explained below.

[1786] Step 1:

[1787] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1788] Step 2:

[1789] User: Press the record button, speak the dialect phrase as instructed, and press the stop button when finished recording.

[1790] Step 3:

[1791] Device: Temporarily saves the audio data in local storage, then sends a request to upload it to the server.

[1792] Step 4:

[1793] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​speech recognition module for analysis.

[1794] Step 5:

[1795] Server: The AI ​​speech recognition module converts the voice data into text, and stores the converted text data in a temporary database.

[1796] Step 6:

[1797] Server: Returns the analysis results to the device.

[1798] Step 7:

[1799] Terminal: The analysis result text is displayed in the UI and the user is asked to confirm.

[1800] Step 8:

[1801] User: Checks the text analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[1802] Step 9:

[1803] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[1804] Step 10:

[1805] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[1806] Step 11:

[1807] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[1808] Step 12:

[1809] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[1810] Step 13:

[1811] User: Interacts with the provided learning content and receives feedback on correct and incorrect answers.

[1812] Step 14:

[1813] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[1814] Step 15:

[1815] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[1816] Step 16:

[1817] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[1818] Step 17:

[1819] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[1820] Step 18:

[1821] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[1822] Step 19:

[1823] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[1824] Step 20:

[1825] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[1826] Step 21:

[1827] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[1828] Step 22:

[1829] Server: Stores the translated data in the final database.

[1830] Example 1

[1831] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1832] In modern times, regional dialects are rapidly disappearing, making it difficult to pass them on to the next generation. Furthermore, those who want to learn dialects lack the appropriate learning tools, making it difficult to learn them efficiently. Furthermore, there is a lack of systems for consistently collecting, analyzing, and storing dialect speech data, and there is a need to efficiently carry out these tasks.

[1833] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1834] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the converted text to the user and accepting correction requests as necessary, means for saving the final text data and voice data in a database, means for providing learning content using the saved dialect data, means for providing interaction through the learning content and giving feedback to the user, and means for translating the dialect data into other languages, thereby enabling the effective preservation and inheritance of dialects.

[1835] "Audio data" refers to audio information recorded by a user, and is data that includes specific dialects and phrases.

[1836] A "collection means" is a part of a system that provides a method or facility for a user to record audio data.

[1837] "Means of uploading" refers to the method or function for transferring collected audio data to an online server.

[1838] "Means for analyzing" means a method or function that uses technology such as an AI voice recognition module to convert uploaded voice data into text format.

[1839] "Means of presentation" refers to the methods and functions for displaying and confirming analysis results, learning content, translation results, etc. to the user.

[1840] "Means for accepting correction requests" refers to a method or function that allows a user to request correction of errors in the analysis results.

[1841] "Storage means" refers to the method or function for permanently storing the final text data and audio data in a database.

[1842] "Learning Content" means the learning materials and activities (e.g., quizzes, games, vocabulary exercises) provided to users to help them learn a dialect.

[1843] "Means of interaction" refers to the methods and features that allow users to actively engage with the learning content and receive feedback.

[1844] "Means for translating" means a method or function that uses a multilingual translation AI model to convert dialect data into another language.

[1845] "Means of reproduction" refers to the method or function of generating speech using Text-to-Speech (TTS) technology using stored dialect data.

[1846] "Virtual reality technology" refers to technology that allows users to immerse themselves in a virtual environment using a VR headset or similar device.

[1847] "Augmented reality technology" refers to the technology of overlaying digital information onto the real world using AR-enabled devices.

[1848] MODE FOR CARRYING OUT THE INVENTION

[1849] The system according to the present invention performs processes such as collecting, uploading, analyzing, presenting, and saving voice data, and providing learning content. These processes are realized through the cooperative operation of a user, a terminal, and a server. Specific embodiments are described below.

[1850] Audio data collection

[1851] Users launch the smartphone app, select the "Dialect Recording" section, press the record button, and speak a specific dialect phrase. The device detects that the record button has been pressed and begins recording the user's voice using the built-in microphone. The recorded voice data is temporarily stored in local storage.

[1852] Uploading audio data

[1853] After the recording is complete, the device sends an upload request to the server. The recorded audio data is compressed and transferred to the server. The user is informed of the upload progress using a progress bar. The server temporarily stores the received audio data and prepares for the next process.

[1854] Analysis of audio data

[1855] The server passes the voice data to an AI speech recognition module, which converts the voice into text. The text conversion results are stored in a temporary database on the server, and the results are sent back to the device, which then presents the text analysis results to the user.

[1856] Check and correct text data

[1857] The user checks the displayed text analysis results, and if there are any errors, sends a correction request to the server via the terminal. The server receives the correction request and either performs a new analysis or accepts the user's manual correction.

[1858] Saving to a database

[1859] The final text and audio data, along with associated metadata, are then permanently stored in a database by the server, and the device notifies the user when the storage is complete.

[1860] Providing interactive learning tools

[1861] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[1862] Dialect reproduction and simulation

[1863] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology. The generated voice is provided to the user via the terminal, and a play button and a feedback form are displayed. The user can listen to the pronunciation and intonation of the reproduced dialect.

[1864] Dialect experience using VR and AR

[1865] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[1866] Multilingual Translation and Dialect Preservation

[1867] When a user sends a translation request, the device forwards it to the server. The server uses a multilingual translation AI model to translate the dialect data into another language. The translation results are presented to the user via the device, and correction requests are accepted as needed. The final data is stored in a database by the server.

[1868] Specific examples

[1869] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[1870] Prompt Sentence Examples

[1871] Example prompt sentence:

[1872] "Please record yourself saying 'The weather is nice' in the Niigata dialect."

[1873] Upload the recorded audio to the server.

[1874] Once the upload is complete, check the text conversion results.

[1875] "Make any necessary corrections."

[1876] Save the final result.

[1877] "Learn the Niigata dialect using the learning module."

[1878] In this way, the system of the present invention can consistently perform each process for effectively preserving and inheriting dialects.

[1879] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1880] Step 1:

[1881] The user launches the application and selects the "Dialect Recording" section. Next, they press the record button and speak the specified phrase. The device detects this and begins recording the audio using the built-in microphone. The recording is temporarily saved in local storage.

[1882] Input: Pressing the record button, user voice

[1883] Output: Audio data stored in local storage

[1884] Specific operation: The recording module detects the button press event and starts the recording process. After the recording is completed, the data is saved in a file format.

[1885] Step 2:

[1886] After the device has finished recording, it sends a request to upload the audio data to the server. The audio data is compressed and sent to the server as an HTTP POST request. The upload progress is displayed to the user. The server temporarily stores the received audio data.

[1887] Input: Audio data stored in local storage

[1888] Output: Audio data stored on the server

[1889] Specific operation: The device compresses the audio data and sends it to the server using the HTTP protocol. The server receives the data and stores it in temporary storage.

[1890] Step 3:

[1891] The server passes the uploaded voice data to an AI speech recognition module, which converts the voice into text, and the conversion results are stored in a temporary database on the server.

[1892] Input: Audio data stored on the server

[1893] Output: Conversion result in text format

[1894] Specific operation: The server calls the speech recognition API to convert the voice data into text format, which is then stored in the database.

[1895] Step 4:

[1896] The server notifies the terminal of the conversion results, and the terminal presents the text analysis results to the user.

[1897] Input: Conversion result in text format

[1898] Output: Text analysis results presented to the user

[1899] Specific operation: The server sends text data to the terminal, and the terminal displays it on the user interface.

[1900] Step 5:

[1901] The user checks the analysis results and, if necessary, sends a correction request to the server via the terminal. The server receives the correction request and performs a new analysis or approves manual correction.

[1902] Input: User correction request

[1903] Output: Corrected text data

[1904] Specific operation: The user makes corrections using a text editor, and the device sends the corrections to the server, which saves them in the database.

[1905] Step 6:

[1906] The server stores the final text data, audio data, and related metadata in a permanent storage database. Once the storage is complete, the device notifies the user.

[1907] Input: Final text and audio data

[1908] Output: Notification of save completion

[1909] Specific operation: The server saves the data permanently in the database and notifies the device of the status. The device notifies the user that the save is complete.

[1910] Step 7:

[1911] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the displayed content and receives feedback.

[1912] Input: User access to learning content

[1913] Output: User learning progress and feedback

[1914] Specific operation: The device retrieves learning content from the server and displays it to the user. Feedback is provided based on the user's interactions.

[1915] Step 8:

[1916] The server uses the stored dialect data to generate a reproduced voice using Text-to-Speech (TTS) technology, which is then provided to the user via their device, along with a playback button and a feedback form.

[1917] Input: Stored dialect data

[1918] Output: Reproduced audio provided to the user

[1919] Specific operation: The server generates speech using TTS technology, and the device displays a play button to the user, allowing the user to listen to the generated speech.

[1920] Step 9:

[1921] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device uses VR or AR technology to recreate specific dialect-using situations (such as the market, home, or school) in virtual reality. Users are immersed in dialect-using scenarios within the virtual environment, experiencing a realistic dialect experience.

[1922] Input: User access to the dialect experience module

[1923] Output: A virtual reality recreation of a dialect scene

[1924] Specific operation: The device uses VR / AR technology to create a virtual environment and fully render it so that users can experience the dialect within it.

[1925] Step 10:

[1926] The device receives translation requests from users and forwards them to the server. The server uses a multilingual translation AI model to translate the dialect data into other languages ​​and presents the results to the user via the device. The user can then submit correction requests as needed. The final data is stored in a database by the server.

[1927] Input: Translation request from user

[1928] Output: Data translated into other languages

[1929] Specific operation: The server translates the dialect data using the translation AI model, displays the translation results to the user through the terminal, and saves the final data in the database based on the user's feedback.

[1930] (Application example 1)

[1931] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1932] Conventional dialect learning systems simply record audio data and display the resulting text, lacking interactive feedback for learners and practical experience. Furthermore, they lack the ability to reuse saved dialect data or translate it into other languages, making it difficult to accommodate a wide range of users. Furthermore, they lack the ability to provide a learning environment that utilizes virtual reality or augmented reality technology.

[1933] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1934] In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for providing a variety of voice learning content using the saved dialect data, means for recreating situations in which the dialect is used using virtual reality or augmented reality technology, and means for translating the dialect into other languages, thereby enabling learners to learn the dialect through an interactive and realistic experience and to reuse the saved data for translation into other languages.

[1935] "Voice data" refers to digital data that electronically records the voice of a user speaking in a dialect or standard Japanese.

[1936] A "collection instrument" is a combination of hardware and software used to record and store a user's voice using a device.

[1937] The "means for uploading" is a mechanism for transferring collected voice data to a server via a network.

[1938] The "means for analyzing and converting to text" refers to algorithms and software for converting voice data into text information using voice recognition technology.

[1939] The "means for presenting to the user" is an interface for presenting the analysis results to the user visually or audibly.

[1940] The "means for storing in a database" refers to a storage system for temporarily or long-term storage of analyzed text data and audio data.

[1941] "Means for providing learning content" refers to a mechanism that provides interactive learning materials such as quizzes, games, and vocabulary practice to users to help them learn the dialect.

[1942] The "means for providing a variety of phonetic learning content" is a system for generating and providing various forms of phonetic learning content using stored dialect data.

[1943] "Virtual reality technology" or "augmented reality technology" refers to technology that allows users to experience virtual environments and information in real time, simulating specific scenarios or environments.

[1944] "Means for recreating situations in which dialects are used" refers to a system that uses virtual reality and augmented reality technology to recreate situations in which a specific dialect is actually used.

[1945] "Means for translating into other languages" refers to a multilingual translation system or algorithm for converting dialect data into other languages.

[1946] This invention relates to a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. This system includes a program that collects, uploads, analyzes, presents, and stores the speech data, and provides learning content. Specific embodiments of this system are described below.

[1947] 1. Collection of audio data

[1948] The user launches the application on a device such as a smartphone or tablet, selects the recording section, presses the record button, and speaks a dialect phrase. The device uses WebRTC to record the voice data and temporarily stores it in local storage.

[1949] 2. Uploading audio data

[1950] The device sends an HTTP POST request to upload the recorded audio data to the server. The progress is displayed on the screen to inform the user. The server receives the request using Flask and temporarily stores the audio data.

[1951] 3. Analysis of audio data

[1952] The server inputs the voice data into a speech recognition model using TensorFlow and converts it into text. The conversion results are temporarily stored in a MySQL database. The conversion results are then sent back to the device as an HTTP response, and the text result is presented to the user.

[1953] 4. Check and correct the analysis results

[1954] The user checks the text of the analysis result and sends a correction request if there is an error. The server then re-analyzes or manually corrects the text and stores it in the database again.

[1955] 5. Saving to the database

[1956] The server saves the final analysis results in a MySQL database and sends a notification to the terminal that the results have been saved.

[1957] 6. Providing interactive learning content

[1958] Users access the "Dialect Learning" module and use learning content such as quizzes and vocabulary practice. The server dynamically generates this content and provides it to users.

[1959] 7. Dialect Reproduction and Simulation

[1960] The server uses Text-to-Speech (TTS) technology to reproduce the speech of the dialect data stored and transmits it to the terminal. The user listens to the reproduced speech and learns pronunciation and intonation.

[1961] 8. Dialect experience using VR and AR

[1962] Users access the "Dialect Experience" module using a VR headset or AR-enabled device, and the server uses Unity or Unreal Engine to render the virtual environment and recreate specific dialect-using situations.

[1963] 9. Multilingual Translation

[1964] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database.

[1965] (Example)

[1966] For example, if a user follows the instruction "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect," the recorded audio data is uploaded to the server, analyzed, and converted into text. The results of this analysis are presented to the user, and any errors can be corrected. The data is then stored in a database and used as learning content for other users.

[1967] (Example of a prompt)

[1968] Audio collection prompt: "Please record yourself saying 'The weather is nice' in the Niigata dialect. Press the record button and start speaking."

[1969] Correction prompt: "Your voice data has been analyzed. The following text is the analysis result. Please correct any errors."

[1970] Learning Content Prompt: "Learn the following phrases. Check your pronunciation and take the quiz."

[1971] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1972] Step 1:

[1973] A user launches the application on a device such as a smartphone or tablet and selects the "Dialect Recording" section. The input is the user's actions and the dialect phrase to be recorded. Specifically, the user presses the record button and speaks a specific dialect phrase. The device uses WebRTC to record the voice data in real time and temporarily saves it in local storage. The output is the recorded voice data.

[1974] Step 2:

[1975] When the user finishes recording, the device uploads the audio data to the server. The input is the audio data recorded in step 1. The device sends an HTTP POST request to transfer the audio data to the server. The server receives the request using Flask and temporarily stores the audio data. The output is the audio data stored on the server.

[1976] Step 3:

[1977] To analyze the stored voice data, the server inputs the data into a voice recognition model using TensorFlow. The input is the voice data received in step 2. Specifically, the voice recognition model converts the voice data into text and generates analysis results in text format. The output is the converted text data.

[1978] Step 4:

[1979] The server temporarily stores the analysis results in a MySQL database and returns the text data to the terminal as an HTTP response. The input is the text data obtained in step 3. The terminal displays the analysis results to the user. The output is the text analysis results presented to the user.

[1980] Step 5:

[1981] The user checks the text of the analysis results and, if there are any errors, sends a correction request. The input is the displayed text data and the user's correction request. The terminal transfers the correction request to the server. The output is data including the correction request. The server re-analyzes or performs manual corrections and stores the data in the database again. Finally, corrected text data is generated.

[1982] Step 6:

[1983] The server stores the final analysis results in a MySQL database. The input is the data corrected in step 5. This stored data includes the audio data, associated text data, and metadata. The output is the final analysis results stored in the database.

[1984] Step 7:

[1985] A user accesses the "Dialect Learning" module. The input is dialect data stored in the database. The server dynamically generates learning content such as quizzes and vocabulary exercises using the stored dialect data and sends it to the terminal. The output is interactive learning content provided to the user.

[1986] Step 8:

[1987] The server uses Text-to-Speech (TTS) technology to convert the stored dialect data into speech and send it to the terminal. The input is text data from the database. The user listens to the reproduced speech and learns pronunciation and intonation. The output is the reproduced speech data.

[1988] Step 9:

[1989] A user accesses the "Dialect Experience" module using a VR headset or AR-enabled device. The input is the saved dialect data and the corresponding scenario settings. The server uses Unity or Unreal Engine to render the virtual environment and recreate a specific dialect usage scene. The output is the virtual environment provided to the user.

[1990] Step 10:

[1991] The server translates the dialect data into other languages ​​using a multilingual translation system such as Google Translate API and stores the results in a database. The input is the stored dialect data. The user checks and corrects the translation results as needed and uses the final data. The output is the translated data.

[1992] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1993] The system of the present invention collects, analyzes, and stores dialect speech data, provides interactive tools for learning the dialect, and combines it with an emotion engine that recognizes the user's emotions. The following describes in detail the embodiments of the present invention.

[1994] The system includes programs for collecting, uploading, analyzing, presenting, and saving voice data, providing learning content, and recognizing user emotions. Each process is realized through the cooperative operation of the user, the device, and the server.

[1995] 1. Audio data collection:

[1996] User: Launches the application and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[1997] Device: When the record button is pressed, audio is recorded and temporarily saved in local storage.

[1998] 2. Uploading audio data:

[1999] Terminal: Sends a request to upload the recorded audio data to the server. Displays the upload progress to the user.

[2000] Server: Processes and temporarily stores the received audio data.

[2001] 3. Analysis of audio data:

[2002] Server: Passes the uploaded voice data to the AI ​​speech recognition module and converts it into text. The conversion results are stored in a temporary database.

[2003] Terminal: The analysis results are returned and the converted text is presented to the user.

[2004] 4. Check and correct text data:

[2005] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no problems, presses the confirmation button.

[2006] Server: Receives the correction request and performs a new analysis or accepts the user's manual correction.

[2007] 5. Save to database:

[2008] Server: Stores the final text and audio data, as well as related metadata, in a database.

[2009] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[2010] 6. Providing interactive learning tools:

[2011] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[2012] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[2013] User: Interacts with the provided content and receives feedback on whether they got it right or wrong.

[2014] 7. Dialect Reproduction and Simulation:

[2015] Server: Using dialect data, reproduced speech is generated using Text-to-Speech (TTS) technology.

[2016] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[2017] Users: can listen to the pronunciation and intonation of the reproduced dialect.

[2018] 8. Dialect experience using VR and AR:

[2019] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[2020] Device: Renders virtual reality and augmented reality environments to recreate specific dialect-using situations (e.g., market, home, school).

[2021] User: Immerse yourself in a scenario using dialects in a virtual environment and experience the dialects in a realistic way.

[2022] 9. Multilingual Translation and Dialect Preservation:

[2023] Server: Uses a multilingual translation AI model to translate dialect data into other languages.

[2024] Terminal: Displays the translation results to the user and accepts correction requests as needed.

[2025] Server: Stores the translated data in the final database.

[2026] 10. Incorporating an Emotion Engine:

[2027] Device: Incorporates an emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[2028] Server: Stores the recognized emotion data and reflects it in analysis results and learning content.

[2029] 11. Emotion-based learning content adjustment:

[2030] Server: Dynamically adjusts the difficulty and content of learning content based on the user's emotions recognized by the emotion engine. For example, if the user is excited while learning, it provides more challenging questions, and if the user is stressed, it changes to easier questions.

[2031] 12. Emotional Data Storage:

[2032] Server: Receives user emotion data from the emotion engine and stores it in a database along with other dialect data.

[2033] As a concrete example, consider a case where a user uses a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. Finally, the data is stored in a database and used as learning content.

[2034] Furthermore, the emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and adjusts the learning content based on that data. For example, if the user expresses joy while learning, the system will present more difficult questions. On the other hand, if the system recognizes that the user is feeling stressed, it will switch to easier questions. Emotional data is also stored in a database and will be used to improve dialect learning in the future.

[2035] In this way, this system can effectively preserve and pass on dialects while providing an interactive learning experience that reflects the user's emotions.

[2036] The processing flow will be explained below.

[2037] Step 1:

[2038] User: Launches the application and selects the "Record Dialect" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[2039] Step 2:

[2040] User: Press the record button, speak the dialect phrase provided, and press the stop button when finished.

[2041] Step 3:

[2042] Device: Recorded audio data is temporarily stored in local storage.

[2043] Step 4:

[2044] Terminal: Sends a request to upload audio data to the server. Displays the upload progress to the user.

[2045] Step 5:

[2046] Server: Processes and temporarily stores the received voice data, then passes the voice data to the AI ​​voice recognition module.

[2047] Step 6:

[2048] Server: The AI ​​speech recognition module converts the voice data into text, which is then stored in a temporary database.

[2049] Step 7:

[2050] Server: Returns the text analysis results to the device.

[2051] Step 8:

[2052] Terminal: The analysis result text is presented to the user and confirmation is sought.

[2053] Step 9:

[2054] User: Checks the text of the analysis results, and if there are any errors, sends a correction request to the server via the terminal. If there are no errors, presses the confirmation button.

[2055] Step 10:

[2056] Server: Receives correction requests, performs reanalysis or accepts user manual corrections, and saves the final analysis results in the database.

[2057] Step 11:

[2058] On the device: Send a notification to the user that the save is complete, for example, displaying a message saying "Data saved successfully."

[2059] Step 12:

[2060] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[2061] Step 13:

[2062] Device: Provides learning content (e.g., quizzes, games, vocabulary practice) to help users learn dialects.

[2063] Step 14:

[2064] User: Interacts with the provided learning content and receives feedback on whether their answers are correct or incorrect.

[2065] Step 15:

[2066] Server: Based on the dialect data, the voice is reproduced using Text-to-Speech (TTS) technology.

[2067] Step 16:

[2068] Terminal: Provides the generated audio to the user and displays a play button and a feedback form.

[2069] Step 17:

[2070] User: Listen to the pronunciation and intonation of the dialect reproduced and provide feedback.

[2071] Step 18:

[2072] Users: Access the "Dialect Experience" module using a VR headset or AR-enabled device.

[2073] Step 19:

[2074] Device: Renders virtual reality and augmented reality environments and recreates specific dialect-based situations.

[2075] Step 20:

[2076] Users: Immerse themselves in a dialect-based scenario within a virtual environment for a realistic experience.

[2077] Step 21:

[2078] Server: Pass the dialect data to a multilingual translation AI model to translate it into other languages.

[2079] Step 22:

[2080] Terminal: Presents the translation results to the user and accepts correction requests as needed.

[2081] Step 23:

[2082] Server: Stores the translated data in the final database.

[2083] Step 24:

[2084] On the device: Activate the emotion engine to recognize emotions in real time from the user's facial expressions, tone of voice, and other inputs.

[2085] Step 25:

[2086] Server: Receives and temporarily stores the user's emotion data recognized by the emotion engine.

[2087] Step 26:

[2088] Server: Uses emotional data to adjust learning content, for example, making the content less difficult if the user is feeling stressed.

[2089] Step 27:

[2090] Device: Presents tailored learning content to users and supports them to keep them engaged.

[2091] Step 28:

[2092] Server: Stores the user's emotional data in a final database for future analysis and improvement.

[2093] Example 2

[2094] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2095] In recent years, the preservation and inheritance of dialects has become increasingly important from the perspective of cultural value. However, conventional dialect learning systems have limitations in collecting and analyzing speech data and providing learning content, and in particular lack the ability to optimize learning while taking into account the user's emotions. They are also inadequate in reproducing dialect data and providing an interactive learning environment using virtual reality technology. The present invention aims to solve these problems.

[2096] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for uploading the collected voice data, means for analyzing the uploaded voice data and converting it into text, means for presenting the analysis results to the user, means for the user to confirm and correct the text, means for saving the final analysis results in a database, means for providing learning content using the saved dialect data, means for incorporating an emotion engine that recognizes the user's emotions in real time, means for dynamically adjusting the learning content based on the user's emotions recognized by the emotion engine, and means for saving the user's emotion data in a database. This makes it possible to provide an interactive and optimized learning experience that takes the user's emotions into consideration.

[2097] "Audio data" refers to data that records sound in digital or analog format.

[2098] A "means of collection" is a device or software that provides the function of recording audio and storing it as data.

[2099] An "uploading means" is a device or software that provides the function of transmitting data stored in local storage to a remote server.

[2100] An "analyzing means" is a device or software that provides the functionality to process collected voice data and convert it into text.

[2101] "Means for converting to text" refers to software or a module that has the function of converting voice data into character data.

[2102] The "presentation means" refers to a device or software that has the function of allowing a server or terminal to display the processing results to the user visually or audibly.

[2103] A "verification and correction means" is a device or software that provides an interface or functionality for a user to review the presented information and correct it if necessary.

[2104] "Means for storage" refers to a database or storage device for permanently recording the final data.

[2105] "Learning Content" refers to the educational materials and interactive activities provided to users to help them learn a dialect.

[2106] An "emotion engine" is software or a module that analyzes a user's facial expressions, tone of voice, etc., and recognizes emotions in real time.

[2107] A "dynamic adjustment means" is a device or software that provides the functionality to change the difficulty or content of learning content in real time based on the user's emotional state.

[2108] The "means for converting to text" is software or hardware for converting voice data into text data.

[2109] A "reproduction means" is a device or software that provides the functionality to generate sounds and virtual scenes based on stored data.

[2110] "User" refers to an individual or end user who uses the System.

[2111] "Interactive" refers to the ability of a user to elicit a response from the system through their own actions.

[2112] "Virtual reality technology" refers to technology that allows users to experience computer-generated three-dimensional environments.

[2113] "Augmented reality technology" refers to technology that displays digital information overlaid on the real world.

[2114] The present invention is a system that collects, analyzes, and stores dialect speech data, and provides interactive tools for learning the dialect. In addition, by incorporating an emotion engine that recognizes the user's emotions, effective dialect learning is realized. The detailed processing of the system is described below.

[2115] Hardware and software used

[2116] The system uses the following hardware and software:

[2117] Mobile information terminals such as smartphones and tablets

[2118] Servers (including cloud-based storage and processing resources)

[2119] Voice Recording Module

[2120] AI voice recognition module

[2121] Text-to-Speech (TTS) software

[2122] Emotion Recognition Engine

[2123] Database Management Systems

[2124] Audio data collection

[2125] The user starts the application using a mobile information device such as a smartphone or tablet and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording instruction phrase (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect"). When the user presses the record button, the device's built-in microphone is used to record audio, which is then temporarily saved in local storage.

[2126] Uploading audio data

[2127] The recorded audio data is uploaded from the device to the server. The device sends a request to the server and displays the upload progress to the user. The server receives the audio data and stores it in temporary storage.

[2128] Analysis of audio data

[2129] The server passes the uploaded voice data to an AI speech recognition module, which converts it into text. This module analyzes the voice data and generates corresponding text data. The generated text data is stored in a temporary database. The analysis results are sent back from the server to the device, and the converted text is presented to the user.

[2130] Check and correct text data

[2131] The user checks the presented text data and, if there are any errors, sends a correction request to the server via their terminal. If there are no problems, they press the "Confirm" button to proceed. The server receives the correction request and either performs a new analysis or approves the user's manual correction.

[2132] Saving to a database

[2133] The server stores the final text and audio data, along with associated metadata, in a database. Once the data has been saved, the device displays a notification to the user stating, "Data saved successfully."

[2134] Providing interactive learning tools

[2135] The user accesses the "Dialect Learning" module of the application and begins the learning activity. The device provides learning content (e.g., quizzes, games, vocabulary practice) to help the user learn the dialect. The user interacts with the provided content and receives feedback on whether their answers are correct or incorrect.

[2136] Dialect reproduction and simulation

[2137] The server uses the stored dialect data to generate a reproduced speech using Text-to-Speech (TTS) technology. This speech is provided to the terminal, and the user can listen to the reproduced speech by pressing the play button. In addition, a feedback form can be used to provide a rating.

[2138] Dialect experience using VR and AR

[2139] Users access the "Dialect Experience" module using a VR headset or AR-enabled device. The device renders a virtual reality or augmented reality environment, recreating specific dialect-using situations (e.g., market, home, school). Within the virtual environment, users are immersed in dialect-based scenarios, providing a realistic dialect experience.

[2140] Multilingual Translation and Dialect Preservation

[2141] The server uses a multilingual translation AI model to translate the dialect data into other languages. The translation results are then sent back to the device, where the user can review them and send correction requests if necessary. The final translation data is stored in a database.

[2142] Incorporating an emotional engine and adjusting learning content

[2143] The device uses an emotion engine to recognize emotions in real time from the user's facial expressions and tone of voice. The recognized emotion data is sent to a server, which then dynamically adjusts the difficulty and content of the learning content. The emotion data is also stored in a database to help improve dialect learning in the future.

[2144] Examples and prompts

[2145] For example, a user might use a smartphone app to follow the instruction, "Please record yourself saying 'The weather is nice, isn't it?' in the Niigata dialect." The user presses the record button and speaks the phrase, and the device records the audio data and uploads it to the server. The server then uses AI speech recognition technology to analyze the audio and convert it into text format. The results are presented to the user, who can review and correct any errors. The data is finally stored in a database and used as learning content. Additionally, an emotion engine recognizes the user's emotions in real time from their facial expressions and voice, and the learning content is adjusted based on that data.

[2146] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2147] Step 1:

[2148] Audio data collection

[2149] User: Launches the smartphone app and selects the "Dialect Recording" section. The UI displays a record button, a stop button, and a recording prompt (e.g., "Please say 'The weather is nice, isn't it?' in the Niigata dialect").

[2150] Input: The user presses the record button.

[2151] Process: The device will record audio using the built-in microphone. When you finish recording, press the stop button.

[2152] Output: The recorded audio data is temporarily saved in the device's local storage.

[2153] Step 2:

[2154] Uploading audio data

[2155] Device: Sends a request to upload the recorded audio data to the server.

[2156] Input: Temporarily saved audio data.

[2157] Processing: The device uploads the audio data to the server and displays the upload progress to the user in real time.

[2158] Output: The audio data is saved in temporary storage on the server.

[2159] Step 3:

[2160] Analysis of audio data

[2161] Server: Passes the uploaded voice data to the AI ​​voice recognition module.

[2162] Input: Audio data uploaded to the server.

[2163] Processing: The AI ​​speech recognition module analyzes the voice data and converts it into text format.

[2164] Output: Save the converted text data in a temporary database.

[2165] Step 4:

[2166] User presentation of text data

[2167] Server: Sends the conversion results to the terminal.

[2168] Input: Text data stored in a temporary database.

[2169] Processing: The text data is returned to the terminal.

[2170] Output: Text data is displayed on the terminal.

[2171] Step 5:

[2172] Check and correct text data

[2173] User: Checks the presented text and sends a correction request to the server via the terminal if necessary. If there are no problems, the user presses the "Confirm" button.

[2174] Input: The displayed text data.

[2175] Action: The user confirms the error and submits a correction request, or presses the confirm button to proceed.

[2176] Output: The modification request is sent to the server or the confirm button is pressed.

[2177] Step 6:

[2178] Finalize text data

[2179] Server: Receives the correction request and performs a new analysis or approves manual corrections. After the corrections are confirmed, they are saved in the database.

[2180] Input: Correction request or user confirmation.

[2181] Processing: Performing a new analysis or manual correction is processed to approve and save it in the database.

[2182] Output: The final text and audio data and associated metadata are stored in a database.

[2183] Step 7:

[2184] Save completion notification

[2185] Server: Notifies completion of data saving.

[2186] Input: The final data is saved in the database.

[2187] Processing: Send a save completion notification to the terminal.

[2188] Output: The device displays the message "Data saved successfully" to the user.

[2189] Step 8:

[2190] Providing interactive learning content

[2191] User: Access the "Learn Dialects" module of the application and begin the learning activity.

[2192] Input: Access from the user.

[2193] Processing: The device provides learning content (e.g., quizzes, games, vocabulary practice). 【21...

Claims

1. means for collecting audio data; a means for uploading the collected audio data; A means of analyzing the uploaded voice data and converting it into text; means for presenting the analysis results to a user; A means of storing the final analysis results in a database; A system including a means for providing learning content using stored dialect data.

2. A method to reproduce pronunciation and intonation using AI from saved dialect data, 10. The system of claim 1, further comprising means for providing the reproduced audio to a user.

3. The system of claim 1 further comprising means for recreating dialect usage situations using virtual reality or augmented reality technology.

Images