Target language immersive hearing feedback system based on native language triggering

Abstract

The invention discloses a target language immersive hearing feedback system based on native language triggering. The system comprises a native language speech input module used for collecting native language speech input by a user and identifying a language corresponding to the native language speech; the semantic analysis module is used for performing sentence segmentation processing, keyword extraction, grammar structure analysis and semantic extraction on the native language speech to obtain an analysis result; the target language semantic mapping module is used for searching for an expression mode of semantic alignment in the target language according to the analysis result to obtain a target language text; the voice synthesis module is used for converting the target language text into a playable audio signal; the auditory feedback module is used for playing the audio signal to a user auditory channel in an ear return mode so as to realize immersive input feedback; and the self-adaptive regulation and control module is used for dynamically adjusting the speech speed, accent, vocabulary difficulty and feedback rhythm of the audio signal based on the user feedback data. The technical problem that the cognitive load is too high when a beginner learns languages is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of language learning and artificial intelligence interaction, and particularly relates to a real-time listening feedback system based on mother tongue voice input triggering target language output, which is used for realizing language transfer, immersive understanding training, cross-language semantic input control and other multi-scene language adaptation requirements. BACKGROUND

[0002] Existing language learning systems mostly use target language input, translation prompts, follow-up correction and other paths, and lack active identification and interaction response mechanisms for the user's mother tongue. Especially at the beginner stage, when the user has not enough understanding ability for the target language, forcibly inputting the target language is easy to cause frustration and high cognitive load, affecting learning persistence.

[0003] In the process of infant language acquisition, although the passive receiving mechanism of mother tongue as the main and foreign language as the auxiliary exists, the cognitive anchoring mechanism is more needed in the adult language transfer stage, that is, to obtain immersive listening information of the foreign language in the familiar language (mother tongue).

[0004] At present, no effective solution has been proposed for the above problems. SUMMARY

[0005] Embodiments of the present application provide a target language immersive listening feedback system triggered by a mother tongue, to at least solve the technical problem of high cognitive load when beginners learn a language.

[0006] According to an aspect of an embodiment of the present application, a target language immersive listening feedback system triggered by a mother tongue is provided, comprising: a mother tongue voice input module configured to collect a mother tongue voice input by a user and identify a language corresponding to the mother tongue voice; a semantic analysis module configured to perform sentence breaking processing, keyword extraction, grammar structure analysis and semantic extraction on the mother tongue voice to obtain an analysis result; a target language semantic mapping module configured to find a semantic alignment expression in a target language according to the analysis result to obtain a target language text; a voice synthesis module configured to convert the target language text into a playable audio signal; an auditory feedback module configured to play the audio signal to the user's auditory channel in a hear-back manner to realize immersive input feedback; and an adaptive regulation module configured to dynamically adjust a speech speed, an accent, a vocabulary difficulty and a feedback rhythm of the audio signal based on user feedback data.

[0007] In some embodiments, the semantic analysis module further comprises: a context perception sub-module configured to judge a user context and historical expression to improve semantic accuracy.

[0008] In some embodiments, the voice synthesis module is further configured to generate voice based on a localized or cloud pre-trained model, which supports multiple languages, adjustable speech speed, and multiple pronunciation styles.

[0009] In some embodiments, the auditory feedback module is further configured to provide multiple output forms, including at least one of Bluetooth earphones, wired earphones, bone conduction earphones, and smart speaker devices.

[0010] In some embodiments, the adaptive regulation module is further configured to construct a personalized parameter library based on user learning data, which is used for subsequent feedback precise matching.

[0011] In some embodiments, the system further comprises a teaching interface module for accessing external language teaching scripts or curriculum systems to realize task-based language immersion training.

[0012] In some embodiments, the system further comprises a data recording and tracking module for recording user input content, feedback history, system response logs, and learning path trajectories.

[0013] In some embodiments, the system is further configured to provide semantic mapping from any native language to at least one target language, and can be extended to a multi-to-multi language conversion structure.

[0014] In some embodiments, the system has a delay control mechanism to ensure real-time synchronization between ear return feedback and user voice input.

[0015] In some embodiments, the system is integrated with a semantic understanding system, an AI teacher agent, or a virtual dialogue engine to extend language cognitive interaction functions.

[0016] In the embodiments of the present application, the native language voice input module is configured to collect user input native language voice and identify the corresponding language of the native language voice; the semantic analysis module is configured to perform sentence breaking processing, keyword extraction, syntax structure analysis, and semantic extraction on the native language voice to obtain an analysis result; the target language semantic mapping module is configured to find a semantically aligned expression in a target language according to the analysis result to obtain a target language text; the voice synthesis module is configured to convert the target language text into a playable audio signal; the auditory feedback module is configured to play the audio signal to the user's auditory channel in an ear return manner to realize immersive input feedback; and the adaptive regulation module is configured to dynamically adjust the speech speed, accent, vocabulary difficulty, and feedback rhythm of the audio signal based on user feedback data. Through the above scheme, the technical problem of high cognitive load for beginners learning a language is solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0018] Figure 1 is a structural diagram of an optional mother-tongue-triggered target language immersive listening feedback system according to an embodiment of the application;

[0019] Figure 2 is an application scenario diagram of a mother-tongue-triggered target language immersive listening feedback system according to an embodiment of the application;

[0020] Figure 3 is a flowchart of an optional mother-tongue-triggered target language immersive listening feedback method according to an embodiment of the application;

[0021] Figure 4 shows a structural schematic diagram of a computer device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the protection scope of the present application.

[0023] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] The present application provides a reverse interactive language training system - the user uses his mother tongue to speak a sentence, and the system generates corresponding target language audio in real time after recognition, and completes the synchronous establishment of language transfer and semantic reflection through ear return mechanism.

[0025] The core of the present application is: collecting user's native language voice input; real-time semantic analysis and mapping; without changing the user's native language expression path, the target language result is fed back to the user's auditory channel; the target language conditioned reflex is established on the auditory level, so as to realize "no need to output the target language, only through the native language to stimulate the target language immersion understanding". The system allows the user to quickly form the target language listening comprehension ability without mastering the target language spelling and writing, and can be widely deployed in language teaching scenarios, AI language assistant terminals, personalized language rehabilitation systems and other occasions as the first stage training framework of language transfer.

[0026] Specifically, as shown in Figure 1 The system includes: native language voice input module 12, semantic analysis module 14, target language semantic mapping module 16, speech synthesis module 18, auditory feedback module 20, adaptive control module 22. The native language voice input module 12 is used to identify the user's native language and collect voice; the semantic analysis module 14 is used to segment the native language voice input, extract keywords, and map semantics; the target language semantic mapping module 16 is used to match the analysis result with the target language; the speech synthesis module 18 is used to call the target language corpus to generate semantic alignment target language voice; the auditory feedback module 20 is used to play the target language audio to the user's auditory end through earphones / speakers and other devices to form a synchronous feedback closed loop; the adaptive control module 22 is used to record the user's preferred speed, accent, and vocabulary range, and optimize the next round of feedback content. In some embodiments, a visual display module 24 can also be included for displaying bilingual example sentences.

[0027] Specifically, the native language voice input module 12 acquires voice input from the user through a microphone, earphone or mobile terminal conversation. Through language recognition mechanism, the input voice belonging to the native language category is determined. The supported languages include but are not limited to Chinese, English, Spanish, French, etc.

[0028] The semantic analysis module 14 segments the input voice, extracts keywords, analyzes the syntax structure and extracts semantic intent. A pre-trained language model is used in combination with a context perception algorithm to structure the sentence meaning. For example: input: "I want to go to the supermarket today." Analysis result: {subject: I; time: today; action: go; place: supermarket}.

[0029] The target language semantic mapping module 16 maps the semantic structure after analysis to the semantic equivalent expression in the target language, and calls the built-in or external target language corpus for matching. The system can use bilingual parallel corpus, cross-sentence nested structure, or self-built knowledge graph to realize semantic equivalent replacement.

[0030] The speech synthesis module 18 uses a TTS (Text-to-Speech) model to synthesize target language text content into speech audio in real time, and the synthesis model can be adjusted according to the user's set speech speed, tone, and accent. The module supports both local synthesis and cloud calling.

[0031] The auditory feedback module 20 is used to automatically push the target language audio to the user's auditory channel (earphones, bone conduction, sound boxes, etc.) through ear return after recognition and synthesis are completed. To ensure the interactive experience, a feedback delay threshold of 300 ms is set to form an immersive "speak and listen" experience.

[0032] The adaptive regulation module 22 records the user's behavior data such as speech speed, pronunciation stability, and vocabulary difficulty preference during use, establishes a personalized parameter library, and dynamically optimizes the audio generation logic and content selection strategy in subsequent interactions.

[0033] In some preferred embodiments, the system can also include a teaching interface module and / or a data recording and tracking module. The teaching interface module is used to interface with language courses or scripts, embed target language contexts, and complete systematic listening transfer tasks; the data recording and tracking module is used to generate user learning paths, error analysis, and behavior records for teaching evaluation; the system also uses a multi-language support framework to support mapping path configuration from any native language to multiple target languages, achieving cross-language adaptation.

[0034] The teaching interface module provides a standard API interface and can be integrated with language course scripts, AI teacher platforms, or virtual learning environments. Through task flow control, auxiliary training functions such as "scene dialogue" and "follow-up guidance" are realized.

[0035] The data recording and tracking module records all interaction process data, including speech input text, analysis results, target language output records, and user reaction time, for subsequent statistical analysis and teaching feedback. The data can be desensitized to meet privacy and compliance requirements.

[0036] The system can be deployed on mobile apps, web clients, smart hardware terminals, or integrated into existing language learning platforms; it supports both local reasoning and cloud processing; all modules can be called independently, and module cutting and lightweight deployment are supported.

[0037] Figure 2 This is the process of a user using the system in an actual scenario according to the embodiments of the present application. The user wears earphones and reads native language content in front of the screen, and the screen displays the corresponding target language translation at the same time; after the system recognizes the user's speech behavior, it plays the matching target language audio, realizing an immersive language input experience of "reading native language, looking at bilingual, and listening to target language".

[0038] In the embodiment of the application, after the user inputs in the native language, the system generates target language audio in real time and returns to the auditory channel, realizing the establishment of language transfer path without outputting the target language. The system supports personalized regulation and multi-language docking, is suitable for language teaching, language rehabilitation and multi-language interactive scenarios, and has the advantages of low cognitive load, high feedback efficiency, strong adaptability and the like.

[0039] Compared with the prior art, the application has the following beneficial effects: language immersion training can be directly started without the user mastering any pronunciation rules of the target language; cognitive pressure of beginners is reduced, and language learning confidence and efficiency are improved; the system can be widely used in language training, outbound preparation, children's language enlightenment, special language rehabilitation and the like; the system structure is modularized, has strong adaptability, and can be quickly integrated into a multi-language system and a mobile terminal; highly customized and personalized adjustment is supported, and the system adapts to user needs of different ages and cultural backgrounds; the system can be integrated into a language teaching platform or a semantic processing system through a standard interface to form a collaborative closed loop of language input and feedback; and the system can be evolved in cooperation with the Was semantic system to build a core input interface of a language understanding paradigm.

[0040] The system is not only a language training tool, but also a core component in the future "language understanding type human-computer interaction" paradigm, has broad development space in semantic modeling, intelligent response, cognitive transfer and the like, and can be continuously evolved as a basic component of the "immersion input layer" in the Was language civilization system.

[0041] According to the embodiment of the application, a method embodiment of target language immersion type hearing feedback triggered based on a native language is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0042] Figure 3 It is a flowchart of a method of target language immersion type hearing feedback triggered based on a native language according to the embodiment of the application, which shows the interaction process between the user and the system, including the key steps of displaying a bilingual sentence, the user reading, the system receiving the voice, judging the sentence, playing the target language audio, and rhythm matching. The arrow clearly shows the time sequence and response path of the event, reflecting the synchronous linkage behavior of the system.

[0043] As Figure 3 shown, the method comprises the following steps:

[0044] Step S302, display of a bilingual example sentence.

[0045] The bilingual example sentence is displayed based on a preset teaching corpus and a task-driven script. The corpus can be dynamically imported by a teaching interface module and an external language course system. For example, the example sentence can be selected from the corpus database according to the learning stage of the user, historical performance data in the personalized parameter library, and the current training task target. In some embodiments, the selection can be based on vocabulary coverage, grammatical complexity, and semantic scene. In this way, the example sentence can not only maintain high consistency with the semantic expression of the user's mother tongue, but also provide training materials with gradually increasing difficulty in the target language.

[0046] When displaying the bilingual example sentence, a split-screen mode can be used, that is, the complete sentence in the user's mother tongue is displayed on the left side, and the corresponding aligned sentence in the target language is displayed on the right side, with real-time highlighting of the corresponding word groups. For example, when the user is learning the "shopping scene", the mother tongue side displays "I want to go to the supermarket to buy fruits today", and the target language side displays its equivalent translation.

[0047] In some embodiments, a semantic mapping module can also be called to establish a parallel corpus index to ensure that the sentence in the target language is completely aligned in semantics and structure with the mother tongue sentence. In addition, speech tag embedding can be used when rendering the interface, so that the semantic weight of each word group in the sentence can be traced back when the speech speed is matched to achieve more precise rhythm control.

[0048] Step S304, the user reads the displayed mother tongue sentence.

[0049] When displaying the bilingual example sentence, environmental noise detection is triggered at the same time. The background sound is analyzed by spectrum through a local or external microphone, and when the environmental noise is detected to be higher than a threshold, the user is prompted to change the environment or the noise suppression filtering algorithm is automatically enabled. In some embodiments, a deep neural network spectrum subtraction model can be used for noise suppression, which can significantly reduce environmental interference without significantly affecting speech clarity.

[0050] The user's speech not only contains the speech signal itself, but also implies information such as pauses, intonation, and speech speed. Therefore, when the user reads, the user's acoustic features such as fundamental frequency (F0), formant distribution, and energy envelope are monitored in real time to form a set of preliminary reading feature vectors. These reading feature vectors can be combined with subsequent semantic analysis results to determine the segment boundary and generate a dynamic rhythm model.

[0051] Step S306, receiving and processing the user's input speech signal.

[0052] After the user finishes reading, the voice input module captures the voice data through the microphone array, and then performs language recognition and preprocessing. Specifically, first, it confirms which mother tongue the user input voice belongs to. In some embodiments, a language recognition model based on a deep convolutional neural network can be used to determine the language. In addition, endpoint detection can also be performed on the voice stream. During endpoint detection, an energy threshold and an adaptive silence detection method can be used. Through the above method, the starting point and the ending point of the voice can be effectively identified, thereby ensuring the accuracy of subsequent segmentation.

[0053] After completing language recognition, the semantic analysis module 14 processes the voice stream step by step. The voice stream is transcribed into a mother tongue text; then, sentence breaking, keyword extraction, syntactic structure analysis, and semantic extraction are performed. Unlike existing rule-based analysis, the embodiment of the present application uses a context-aware semantic analysis submodule that can combine user historical expressions and current task context to determine ambiguity. For example, when the user says "I want to go to the bank", it can determine whether "bank" should be mapped to "financial bank" or "river bank" according to whether the context task is "financial scenario" or "traffic scenario". The analysis result is stored in a structured data form, including one or more of the basic components of subject, predicate, object, time, place, etc., and a semantic intent label. Through the above structured analysis, semantic support is provided for subsequent segment division.

[0054] Step S308, the system determines the voice segment.

[0055] In the prior art, the division of voice segments usually relies on static pause detection or fixed time window segmentation, which is prone to problems of inaccurate boundaries or excessive delay.

[0056] To solve the above problems, the present application proposes a real-time voice segmentation mechanism based on a dynamic attention window. Specifically, after receiving the mother tongue voice input, acoustic features and semantic features are used to monitor the voice simultaneously. Acoustic features include energy mutation, fundamental frequency drop, and prosodic pause; semantic features include syntactic component completeness and semantic intent completeness.

[0057] When the acoustic features suggest that there may be a segment boundary, the dynamic attention window is automatically opened. The current semantic analysis result is input into the segment prediction model. The model uses a bidirectional long short-term memory network (Bi-LSTM) combined with an attention mechanism, which can dynamically calculate whether the voice segment forms a complete semantic unit. If the model determines that the segment has formed a complete semantic, it performs segment division; otherwise, the attention window is dynamically expanded to listen to the subsequent input. Through the above scheme, the error segmentation and delay problem can be significantly reduced, thereby ensuring the synchronization of the feedback audio and the user voice input.

[0058] In the segment prediction model, each candidate boundary point is regarded as a binary classification task, i.e., the position is either a valid segment boundary point or a non-boundary point. The probability value of the point belonging to the boundary is output by the segment prediction network. The result of the probability value is compared with the true result given in the labeled data. If a position is actually labeled as a boundary, the closer the model's predicted probability is to 1, the lower the loss of the point; otherwise, if the prediction deviates from the true label, the loss value increases significantly.

[0059] Finally, the average pause duration and speech rate distribution of the user's previous reading are obtained. The obtained data is input as a prior parameter into the segment prediction model, thereby improving the individual adaptability of the determination. In this way, while ensuring real-time, the accuracy of segment division can be significantly improved. The problem of cutting off half a sentence or waiting too long to play can be avoided in the prior art.

[0060] Step S310, the system performs speech rate matching.

[0061] In the speech rate matching method in the prior art, a simple global speech rate scaling is usually used. That is, the playback rate of the target language audio is adjusted according to the overall reading speech rate of the user. This scheme ignores the differences in local rhythm in the sentence. In addition, it is also easy to cause the feedback audio to be inconsistent with the actual speech of the user.

[0062] To solve the above problems, the present application proposes a speech rate prediction and matching method based on adaptive time series modeling. First, after receiving the segment division result, the acoustic features of the segment are time series modeled. Then, the instantaneous speech rate curve is extracted. The curve is used to reflect the speed change of the user's pronunciation at different positions in the sentence. For example, the beginning of the sentence may be slower, the middle part may be faster, and the end may have a longer pause. The speech rate curve is modeled using a Gated Recurrent Unit (GRU). In combination with the user's previous speech rate habits in the personalized parameter library, a predicted curve is generated. Then, the target language speech synthesis module 18 references the predicted curve when generating the audio. It dynamically adjusts the duration of each word group to achieve fine-grained speech rate matching.

[0063] In addition, the prosodic features of the user's native language segment can also be matched with the prosodic template of the target language audio through the Dynamic Time Warping (DTW) method. Through matching, the consistency of the key pause points and the stress positions of the two can be ensured, so that the user can feel that the target language feedback in the ear return is highly synchronized with their own reading rhythm, thereby establishing semantic conditioned reflex more quickly.

[0064] Step S312, the system plays the target language audio.

[0065] After the target language text is generated and processed for speech rate matching, a pre-trained model is invoked to generate the audio signal. The pre-trained model offers multilingual support, adjustable speech rate, and various pronunciation styles. Users can select their preferred accent and timbre in the personalized settings.

[0066] During audio synthesis, the system automatically selects whether to insert weakened cues based on the user's learning stage. For example, a slight emphasis is added before key grammar points. The generated audio signal is transmitted to the user's auditory channel via the auditory feedback module 20. The auditory feedback module 20 supports multiple output formats, which users can freely choose according to their usage scenario.

[0067] The present invention provides a native language-triggered immersive listening feedback system for target languages, which aims to enable users to achieve immersive listening comprehension and semantic transfer of target languages ​​through native language input without having to master the pronunciation rules of the target language.

[0068] Figure 4 A schematic diagram of a computer device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 4 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0069] like Figure 4 As shown, the computer device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage section 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0070] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0071] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A target language immersion listening feedback system based on native language triggering, characterized in that, include: The native language voice input module is used to collect the user's native language voice input and identify the language corresponding to the native language voice; The semantic parsing module is used to perform sentence segmentation, keyword extraction, grammatical structure analysis, and semantic extraction on the native language speech to obtain the parsing results; The target language semantic mapping module is used to find the semantically aligned expression in the target language based on the parsing results, and obtain the target language text; A speech synthesis module is used to convert the target language text into a playable audio signal; An auditory feedback module is used to play the audio signal to the user's auditory channel via an in-ear monitor to achieve immersive input feedback. An adaptive control module is used to dynamically adjust the speech rate, accent, vocabulary difficulty, and feedback rhythm of the audio signal based on user feedback data.

2. The system according to claim 1, characterized in that, The semantic parsing module further includes a context-aware submodule, used to determine the user's context and historical expressions to improve semantic accuracy.

3. The system according to claim 1, characterized in that, The speech synthesis module is also used to generate speech by adjusting localized or cloud-based pre-trained models, which support multiple languages, adjustable speech rate, and various pronunciation styles.

4. The system according to claim 1, characterized in that, The auditory feedback module is also used to provide multiple output formats, including at least one of the following: Bluetooth headphones, wired headphones, bone conduction headphones, and smart speaker devices.

5. The system according to claim 1, characterized in that, The adaptive control module is also used to build a personalized parameter library based on user learning data, and the personalized parameter library is used for accurate matching of subsequent feedback.

6. The system according to claim 1, characterized in that, The system also includes a teaching interface module for connecting to external language teaching scripts or curriculum systems to achieve task-based language immersion training.

7. The system according to claim 1, characterized in that, The system also includes a data recording and tracking module, which records user input, feedback history, system response logs, and learning path trajectory.

8. The system according to claim 1, characterized in that, The system is also used to provide semantic mapping from any native language to at least one target language, and can be expanded into a many-to-many language conversion structure.

9. The system according to claim 1, characterized in that, The system has a delay control mechanism to ensure real-time synchronization between ear feedback and user voice input.

10. The system according to claim 1, characterized in that, The system is integrated with semantic understanding systems, AI teacher agents, or virtual dialogue engines to expand language cognitive interaction functions.

Images