Multi-modal language learning auxiliary system and method based on artificial intelligence

Abstract

The invention relates to the technical field of artificial intelligence and language learning, in particular to a multi-modal language learning auxiliary system and method based on artificial intelligence, and the system comprises a multi-modal input module, a cross-modal feature fusion module, a dynamic adaptive learning module and an interactive feedback generation module. Wherein the multi-mode input module is used for receiving original text data, original image / video data and original voice data; the cross-modal feature fusion module is used for extracting a visual feature vector and a semantic coding vector and generating a cross-modal joint feature vector; the dynamic adaptive learning module is used for generating dynamic scene parameters; and the interactive feedback generation module is used for outputting a multi-mode feedback data packet through a prompt learning engine. According to the method, by fusing multi-modal feature alignment and dynamic context parameter modeling, linkage generation of grammar, culture and pronunciation feedback is achieved, and the context understanding ability and interaction feedback precision in the language learning process are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and language learning technology, and in particular to an artificial intelligence-based multimodal language learning assistance system and method. Background Art

[0002] In the language learning process, traditional teaching systems usually focus on single-modal information transmission, which makes it difficult to meet the understanding needs of multi-channel language input in real contexts; with the development of artificial intelligence technology, language learning assistance systems have gradually introduced modules such as speech recognition, image recognition and natural language processing, but most existing systems still use static text or speech as core input, lacking in-depth modeling of non-linguistic factors such as images, cultural symbols and grammatical structure changes. Especially when dealing with agglutinative languages ​​such as Tibetan and Mongolian, there are difficulties such as complex grammatical dependency relationships, variable use of honorifics, and symbol understanding being constrained by cultural context, making it difficult to achieve high-quality semantic adaptation and feedback output.

[0003] Existing technologies have yet to establish a unified approach that can integrate multimodal features of text, images, and speech, and incorporate linguistic and cultural context for dynamic semantic understanding and interactive feedback. Furthermore, the lack of a systematic response mechanism for addressing cultural symbolic differences, grammatical structure variations, and pronunciation fine-tuning requirements limits learning outcomes to the lexical level, failing to effectively convey the pragmatic logic and cultural connotations behind the language. Therefore, we continue to develop an artificial intelligence-based multimodal language learning assistance system and method to address these issues. Summary of the Invention

[0004] Based on the above objectives, the present invention provides a multimodal language learning assistance system and method based on artificial intelligence.

[0005] An artificial intelligence-based multimodal language learning assistance system includes a multimodal input module, a cross-modal feature fusion module, a dynamic adaptive learning module, and an interactive feedback generation module; wherein:

[0006] Multimodal input module: used to receive original text data, original image / video data and original voice data input by the user;

[0007] Cross-modal feature fusion module: used to extract visual feature vectors from original image / video data, phoneme sequence vectors from original speech data, and semantic encoding vectors from original text data, and generate a cross-modal joint feature vector through cross-modal attention mechanism alignment;

[0008] Dynamic adaptive learning module: used to input cross-modal joint feature vectors into the multimodal dynamic diffusion network, and generate dynamic scene parameters containing grammatical structure adaptation factors and cultural symbol interpretation identifiers in combination with the cultural marker database;

[0009] Interactive feedback generation module: Based on dynamic scene parameters and real-time user input, it outputs multimodal feedback data packets through the prompt learning engine.

[0010] Optionally, the multimodal input module includes a text receiving unit, a visual receiving unit, and a voice receiving unit; wherein:

[0011] Text receiving unit: used to obtain the original text data input by the user through the API interface or character input device, and perform coding standardization processing on the case markers of Tibetan and Mongolian;

[0012] Visual receiving unit: includes an image collector and a video stream parser, which is used to receive raw image / video data from a camera or uploaded files, and perform cultural taboo recognition operations through religious symbol filters;

[0013] Speech receiving unit: Integrates a microphone array and voiceprint separator to collect raw speech data in the environment and perform tone topology preprocessing operations for the Sino-Tibetan language family.

[0014] Optionally, the cross-modal feature fusion module includes a visual feature extraction unit, a speech feature extraction unit, a text semantic encoding unit, and a cross-modal alignment unit; wherein:

[0015] Visual feature extraction unit: This unit is used to divide the input visual data into two categories: static images and dynamic videos. It uses the Vision Transformer to extract static cultural symbol features and the 3D convolutional network to extract continuous gesture features, and then fuses them to generate a unified visual feature vector.

[0016] Speech feature extraction unit: Based on the Conformer model optimized for Sino-Tibetan languages, it processes speech data in two stages. First, the phoneme recognition layer obtains basic speech structure information, and second, the tone topology analysis layer extracts intonation curve variation characteristics. The final fusion generates a phoneme sequence vector that represents the pronunciation patterns of the language.

[0017] Text Semantic Encoding Unit: This unit uses a BERT variant model to process case-tagged text input. It first performs case-tagged-aware encoding through the embedding layer, then constructs a semantic dependency tree structure through the syntactic dependency parsing layer, and then outputs a semantic encoding vector.

[0018] Cross-modal alignment unit: This unit is used to execute a triple attention mechanism, calculating the cultural alignment weight between visual features and semantic encoding, and the tonal coupling between phoneme features and semantic encoding. It also automatically activates the cultural conflict correction process when cultural symbol matching deviations are found, and finally fuses the corrected features to generate a unified cross-modal semantic representation.

[0019] Optionally, the cross-modal alignment unit includes:

[0020] Visual-text attention subunit: used to calculate the cultural symbol alignment weight α between the visual feature vector and the text semantic encoding vector;

[0021] Speech-text attention subunit: used to calculate the tone-semantic coupling degree β between the phoneme sequence vector and the text semantic encoding vector;

[0022] Cultural conflict detection subunit: used to detect when the visual-text attention weight α is lower than the set threshold α th Conflict correction is triggered when , and the corrected semantic vector T is output * ;

[0023] Cross-modal fusion output subunit: used to fuse the corrected text semantic vector, original phoneme sequence vector, and original visual feature vector to generate the final cross-modal joint feature vector.

[0024] Optionally, the dynamic adaptive learning module includes a grammar diffusion unit, a culture diffusion unit, and a parameter generation unit; wherein:

[0025] Syntactic Diffusion Unit: This unit receives the text semantic components in the cross-modal joint feature vector, extracts grammatical dynamics using a time-varying graph convolutional network, and generates a grammatical structure adaptation factor based on the honorific level identifiers in the cultural marker database.

[0026] Cultural Diffusion Unit: This unit receives the visual component of the cross-modal joint feature vector, identifies cultural symbol primitives through the symbolic attention mechanism, and generates interpretation labels.

[0027] Parameter generation unit: used to integrate grammatical structure adaptation factors and cultural symbol interpretation identifiers to generate dynamic scene parameters.

[0028] Optionally, the grammar diffusion unit includes:

[0029] Graph construction subunit: This unit is used to extract text semantic components from the cross-modal joint feature vector and construct a semantic dependency graph structure. Each semantic segment is used as a graph node, and the grammatical dependency relationship is used as an edge to form a time-varying graph structure.

[0030] Trajectory extraction subunit: used to perform time-varying graph convolution operations on the graph structure and extract the grammatical dynamic trajectory vector H t ;

[0031] Honorific fusion subunit: used to extract the grammatical trajectory vector H t Query the cultural marker database to obtain the honorific level embedding vector H corresponding to the target language type honorific and concatenated with the semantic encoding vector to generate the grammatical structure adaptation factor F gram .

[0032] Optionally, the cultural diffusion unit includes:

[0033] The symbol extraction subunit receives the visual component in the cross-modal joint feature vector, applies the symbolic attention mechanism to the visual feature vector, calculates the matching correlation between each position and the predefined cultural symbol template, and obtains the symbolic attention weight;

[0034] Symbol recognition subunit: Based on the symbol attention weight, the symbol semantic vector set is weighted and summed to generate the preliminary cultural symbol semantic vector S raw ;

[0035] Semantic Confirmation Subunit: Based on the current language background and cultural environment, it queries the symbol polysemy mapping relationship in the cultural marker database and outputs the cultural symbol interpretation ID with confidence culture .

[0036] Optionally, the parameter generation unit includes:

[0037] Vector splicing subunit: used to receive the grammatical structure adaptation factor F output by the grammar diffusion unit gram Cultural symbol interpretation ID output by cultural diffusion unit culture , and perform vector concatenation operations to construct the fused input vector X fusion :

[0038] Feature compression subunit: used to convert the fused high-dimensional vector X fusion Input into the multi-layer perceptron structure, and extract the dynamic scene parameters P through nonlinear transformation and weight mapping scene .

[0039] Optionally, the multimodal feedback data packet specifically includes a target language grammar tree, a cultural background explanation diagram, and a pronunciation correction parameter matrix.

[0040] An artificial intelligence-based multimodal language learning assistance method, implemented by the artificial intelligence-based multimodal language learning assistance system described above, comprises the following steps:

[0041] S1: Collect user input text, image / video, and voice data to form corresponding multimodal input data streams;

[0042] S2: Extract the semantic encoding vector of text, the visual feature vector of image / video, and the phoneme sequence vector of speech as the input of the unified semantic space;

[0043] S3: Executes a triple attention mechanism to determine the cultural alignment between text and image, as well as the tonal coupling between text and speech, and performs feature correction when cultural alignment is abnormal.

[0044] S4: Based on the joint feature vector, the grammatical structure change features are extracted and combined with the honorific level to generate the grammatical adaptation factor. At the same time, the cultural symbols in the image are identified and the interpretation mark is generated.

[0045] S5: Integrate grammatical adaptation factors with cultural interpretation markers to construct dynamic scenario parameters to drive feedback generation;

[0046] S6: Generate grammatical tree structure, cultural diagram interpretation and pronunciation correction parameters based on dynamic scenario parameters, and push feedback results simultaneously.

[0047] Beneficial effects of the present invention:

[0048] The present invention, by introducing multimodal feature extraction and triple attention alignment mechanism, can simultaneously identify semantic dependencies in text, cultural symbol information in images, and tone change characteristics in speech, and realize deep correlation analysis of grammar, speech and visual content; based on the grammatical structure adaptation factors and cultural interpretation identifiers generated by the time-varying graph convolutional network and symbolic attention mechanism, it can dynamically construct scene parameters that reflect the context of language behavior, effectively improving the situational understanding ability of language input.

[0049] This invention provides users with structured, visual, and multi-dimensional learning feedback by integrating syntax tree construction, cultural diagram generation, and pronunciation correction parameter modeling. It overcomes the problems of existing systems with single feedback content and poor context adaptability. It is suitable for multilingual learning, especially for learning assistance scenarios of agglutinative languages, and significantly improves the learning system's ability to handle complex grammar and cultural differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 Schematic diagram of a multimodal language learning assistance system according to an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of a multimodal language learning assistance method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0054] It should be noted that references in the specification to "one embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not every embodiment necessarily includes such specific features, structures, or characteristics. In addition, when specific features, structures, or characteristics are described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement such features, structures, or characteristics in conjunction with other embodiments (whether or not explicitly described).

[0055] In general, terms can be understood, at least in part, from their use in context. For example, depending at least in part on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending at least in part on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0056] like Figure 1 As shown, a multimodal language learning assistance system based on artificial intelligence includes a multimodal input module, a cross-modal feature fusion module, a dynamic adaptive learning module, and an interactive feedback generation module; wherein:

[0057] Multimodal input module: used to receive original text data, original image / video data and original voice data input by the user;

[0058] Cross-modal feature fusion module: used to extract visual feature vectors from original image / video data, phoneme sequence vectors from original speech data, and semantic encoding vectors from original text data, and generate a cross-modal joint feature vector through cross-modal attention mechanism alignment;

[0059] Dynamic adaptive learning module: used to input cross-modal joint feature vectors into the multimodal dynamic diffusion network, and generate dynamic scene parameters containing grammatical structure adaptation factors and cultural symbol interpretation identifiers in combination with the cultural marker database;

[0060] Interactive feedback generation module: Based on dynamic scene parameters and real-time user input, it outputs multimodal feedback data packets through the prompt learning engine.

[0061] The multimodal input module includes a text receiving unit, a visual receiving unit, and a voice receiving unit; wherein:

[0062] Text receiving unit: used to obtain the original text data input by the user through the API interface or character input device, and perform encoding standardization processing on the case markers of agglutinative languages ​​such as Tibetan and Mongolian. The mapping function expression of the processing method is: T e =f norm (T r , M g ), where T e is the standardized text data stream; T r The agglutinative text data originally input by the user; M g is the lattice mark encoding mapping table; f norm It is a coding standard function for the unification of agglutinative case marking;

[0063] Visual receiving unit: includes image collector and video stream parser, which is used to receive the original image / video data uploaded by camera or file, and perform cultural taboo recognition operation through religious symbol filter. The image filtering process satisfies the discriminant function: I f =δ(I r , S c ), where I f is the filtered visual data; I r is the original input image or video frame; S c is a set of cultural taboo symbols (such as specific gestures in West Asian culture); δ() is a filtering function based on the symbol recognition model;

[0064] Speech receiving unit: This unit integrates a microphone array and a voiceprint separator to collect raw speech data from the environment and perform tone topology preprocessing operations unique to the Sino-Tibetan language family. It uses the following tone extraction functions: Among them, A p is the normalized tone value curve; V r is the original speech waveform frame sequence; F t It is a set of four-tone feature templates; It is the tone pattern matching and curve fitting function.

[0065] The cross-modal feature fusion module includes a visual feature extraction unit, a speech feature extraction unit, a text semantic encoding unit, and a cross-modal alignment unit; wherein:

[0066] Visual feature extraction unit: This unit is used to classify the input visual data into two categories: static images and dynamic videos. It uses the Vision Transformer to extract static cultural symbol features, and the 3D convolutional network to extract continuous gesture features. These features are then fused to generate a unified visual feature vector to express the visual semantics of the user's cultural context.

[0067] Speech feature extraction unit: Based on the Conformer model optimized for Sino-Tibetan languages, it processes speech data in two stages. First, the phoneme recognition layer obtains basic speech structure information, and second, the tone topology analysis layer extracts intonation curve variation characteristics. The final fusion generates a phoneme sequence vector that represents the pronunciation patterns of the language.

[0068] Text Semantic Encoding Unit: This unit uses a BERT variant model to process case-tagged text input. It first performs case-tagged-aware encoding via an embedding layer, then constructs a semantic dependency tree structure via a syntactic dependency parsing layer. The unit then outputs a semantic encoding vector that reflects the language structure and semantic dependencies.

[0069] Cross-modal alignment unit: This unit is used to execute a triple attention mechanism, calculating the cultural alignment weights between visual features and semantic encoding, and the tonal coupling between phoneme features and semantic encoding. It also automatically activates the cultural conflict correction process when cultural symbol matching deviations are detected, and finally fuses the corrected features to generate a unified cross-modal semantic representation. This unit achieves deep semantic fusion of cross-modal content by triple-aligning static vision, dynamic gestures, phoneme structure, and text semantics and introducing a cultural symbol conflict detection mechanism, effectively improving the system's accuracy in recognizing language behavior in the agglutinative language cultural context.

[0070] The cross-modal alignment unit includes:

[0071] The visual-textual attention subunit is used to calculate the cultural symbol alignment weight α between the visual feature vector and the textual semantic encoding vector. It uses the additive attention mechanism and performs the following operations:

[0072] α=Softmax(W1·tanh(W2V+W3T)), where V is the visual feature vector; T is the text semantic encoding vector; W1, W2, and W3 are the trained weight matrices; α is the cultural symbol alignment weight of visual and text;

[0073] The speech-text attention subunit is used to calculate the tone-semantic coupling β between the phoneme sequence vector and the text semantic encoding vector. It uses the scaled dot product attention mechanism and performs the following operations:

[0074] Where P is the phoneme sequence vector; T is the text semantic encoding vector; d is the vector dimension; β is the coupling weight between tone and semantics;

[0075] Cultural conflict detection subunit: used to detect when the visual-text attention weight α is lower than the set threshold α th When the conflict correction is triggered, the following judgment is performed: Ifα<α th , then activate in, is the cultural correction function; M c is the cultural tag database; T is the original semantic encoding vector; the output is the corrected semantic vector T * ;

[0076] Cross-modal fusion output subunit: used to fuse the corrected text semantic vector, the original phoneme sequence vector, and the original visual feature vector to generate the final cross-modal joint feature vector, which is expressed as: F = Ω(V, P, T * ), where Ω() is the feature fusion function; F is the final cross-modal semantic representation vector; by introducing a triple attention mechanism and nesting cultural conflict detection and feature correction processes, the above sub-unit can achieve high-precision alignment between cross-modal semantics, avoid information misunderstandings caused by inconsistent cultural symbols, and enhance the semantic understanding ability of the language learning system in multicultural scenarios.

[0077] The dynamic adaptive learning module includes a grammar diffusion unit, a culture diffusion unit, and a parameter generation unit; wherein:

[0078] Syntactic Diffusion Unit: This unit receives the text semantic components in the cross-modal joint feature vector, extracts grammatical dynamic trajectories based on a time-varying graph convolutional network (T-GCN), and generates a grammatical structure adaptation factor based on the honorific level identifiers in the cultural marker database.

[0079] Cultural Diffusion Unit: This unit receives the visual component of the cross-modal joint feature vector, identifies cultural symbol primitives through the symbolic attention mechanism, and generates interpretation labels.

[0080] Parameter generation unit: used to integrate grammatical structure adaptation factors and cultural symbol interpretation identifiers to generate dynamic scenario parameters; the above unit introduces a grammatical diffusion mechanism based on graph neural networks and a cultural diffusion mechanism based on symbolic attention into the dynamic adaptive learning module. It can automatically construct adaptive parameters according to the dynamic changes of input language and cultural characteristics, improve the matching degree of subsequent feedback content with the user's language background and cultural environment, and thus improve the accuracy and personalization of language learning.

[0081] Syntactic diffusion units include:

[0082] Graph construction subunit: It is used to extract text semantic components from the cross-modal joint feature vector and construct a semantic dependency graph structure. Each semantic segment is used as a graph node and the grammatical dependency relationship is used as an edge to form a time-varying graph structure. Its expression is: in, is the grammatical dependency graph at time t; is a set of semantic nodes; t is the set of dependency edges between nodes;

[0083] Trajectory extraction subunit: used to perform time-varying graph convolution operations on the graph structure and extract the grammatical dynamic trajectory vector H t , graph convolution expression: H t =σ(A t ·V t W g ), where H t A is the extracted graph representation grammar trajectory vector; t is the graph adjacency matrix at time t, from ε t ; V t is the node feature matrix; W g is the graph convolutional network weight matrix; σ() is the nonlinear activation function;

[0084] Honorific fusion subunit: used to extract the grammatical trajectory vector H t Query the cultural marker database to obtain the honorific level embedding vector H corresponding to the target language type honorific and concatenated with the semantic encoding vector to generate the grammatical structure adaptation factor F gram , the formula is: F gram =σ(W f ·[H t ||H honorific ]), where F gram is the grammatical structure adaptation factor; W f is the fusion weight matrix; [·||·]: represents the vector concatenation operation; the above sub-unit introduces a time-varying graph convolution mechanism based on the semantic dependency graph during the grammatical diffusion process, and combines it with the honorific level features for fusion correction. This unit can dynamically identify the tense-honorificate interaction pattern in agglutinative languages, effectively improving the grammatical matching and cultural adaptability of subsequent feedback content.

[0085] The cultural diffusion unit includes:

[0086] The symbol extraction subunit receives the visual component in the cross-modal joint feature vector, applies the symbol attention mechanism to the visual feature vector, calculates the matching correlation between each position and the predefined cultural symbol template, and obtains the symbol attention weight. The formula is: Among them, γ iThe attention weight corresponding to the i-th symbol primitive; W q ,W k are the query matrix and key matrix of the symbolic attention mechanism; V is the input visual feature vector sequence;

[0087] Symbol recognition subunit: Based on the symbol attention weight, the symbol semantic vector set is weighted and summed to generate the preliminary cultural symbol semantic vector S raw , the formula is Among them, S raw is the initially generated symbolic semantic vector; S i is the semantic representation vector of the i-th cultural symbol primitive; γ i is the attention weight corresponding to the i-th symbol primitive; n is the number of candidate symbol primitives;

[0088] Semantic Confirmation Subunit: Based on the current language background and cultural environment, it queries the symbol polysemy mapping relationship in the cultural marker database and outputs the cultural symbol interpretation ID with confidence culture : The above-mentioned sub-unit performs a symbolic attention mechanism on visual information and introduces context-driven ambiguous semantic confirmation processing. The unit can efficiently identify the differences in visual symbols in multicultural scenarios and output semantically clear interpretation results, thereby improving the system's cultural sensitivity and expression accuracy in the language learning process.

[0089] The parameter generation unit includes:

[0090] Vector splicing subunit: used to receive the grammatical structure adaptation factor F output by the grammar diffusion unit gram Cultural symbol interpretation ID output by cultural diffusion unit culture , and perform vector concatenation operations to construct the fused input vector X fusion :

[0091] Feature compression subunit: used to convert the fused high-dimensional vector X fusion Input into the multi-layer perceptron (MLP) structure, and extract the dynamic scene parameters P through nonlinear transformation and weight mapping scene , the calculation expression is:

[0092] P scene =σ(W2·σ(W1·X fusion +b1)+b2), where P scene is the final generated dynamic scene parameter vector; W1, W2 are the weight matrices of each layer in MLP; b1, b2 are the corresponding bias terms; σ() is the activation function (such as ReLU or GELU).

[0093] The multimodal feedback data package specifically includes the target language grammar tree, cultural background explanation diagram and pronunciation correction parameter matrix.

[0094] The steps to generate the syntax tree structure of user input in the target language are as follows:

[0095] 1. Extract the grammatical factors in the dynamic scene parameters;

[0096] 2. Call the target language grammar rule library and generate a dependency tree based on the semantic tags and syntactic position encoding;

[0097] 3. Output a syntax tree structure, whose nodes and edges represent the dependency relationships between words.

[0098] To generate a graphical interpretation of the cultural background based on the cultural symbol interpretation identifier, perform the following steps:

[0099] 1. Call the cultural graphic template library and select a graphic template based on the symbol components with significant weights in the interpretation vector;

[0100] 2. Sort the diagram contents by confidence weight and add brief annotations to the diagram;

[0101] 3. Output cultural background explanation diagram for presentation on a display terminal.

[0102] Generate speech correction prompt information based on user voice data and dynamic scene parameters, and perform the following steps:

[0103] 1. Extract the tone information and honorific identifiers from the dynamic scene parameters, and analyze the user's actual pronunciation parameters P in conjunction with the user's voice user ;

[0104] 2. Compare the user's pronunciation parameters with the standard pronunciation model and calculate the correction difference;

[0105] 3. Output pronunciation correction parameter matrix, the expression is: C phonetic =P ref -P user , where: P ref is the standard pronunciation parameter; C phonetic is the correction difference matrix that needs to be fed back.

[0106] like Figure 2 As shown, an artificial intelligence-based multimodal language learning assistance method is implemented by the artificial intelligence-based multimodal language learning assistance system described above, comprising the following steps:

[0107] S1: Collect user input text, image / video, and voice data to form corresponding multimodal input data streams;

[0108] S2: Extract the semantic encoding vector of text, the visual feature vector of image / video, and the phoneme sequence vector of speech as the input of the unified semantic space;

[0109] S3: Executes a triple attention mechanism to determine the cultural alignment between text and image, as well as the tonal coupling between text and speech, and performs feature correction when cultural alignment is abnormal.

[0110] S4: Based on the joint feature vector, the grammatical structure change features are extracted and combined with the honorific level to generate the grammatical adaptation factor. At the same time, the cultural symbols in the image are identified and the interpretation mark is generated.

[0111] S5: Integrate grammatical adaptation factors with cultural interpretation markers to construct dynamic scenario parameters to drive feedback generation;

[0112] S6: Generate grammatical tree structure, cultural diagram interpretation and pronunciation correction parameters based on dynamic scenario parameters, and push feedback results simultaneously.

[0113] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0114] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A multimodal language learning assistance system based on artificial intelligence, characterized in that: It includes a multimodal input module, a cross-modal feature fusion module, a dynamic adaptive learning module, and an interactive feedback generation module; among which: Multimodal input module: used to receive original text data, original image / video data and original voice data input by the user; Cross-modal feature fusion module: used to extract visual feature vectors from original image / video data, phoneme sequence vectors from original speech data, and semantic encoding vectors from original text data, and generate a cross-modal joint feature vector through cross-modal attention mechanism alignment; Dynamic adaptive learning module: used to input cross-modal joint feature vectors into the multimodal dynamic diffusion network, and generate dynamic scene parameters containing grammatical structure adaptation factors and cultural symbol interpretation identifiers in combination with the cultural marker database; Interactive feedback generation module: Based on dynamic scene parameters and real-time user input, it outputs multimodal feedback data packets through the prompt learning engine.

2. The multimodal language learning assistance system based on artificial intelligence according to claim 1, characterized in that: The multimodal input module includes a text receiving unit, a visual receiving unit and a voice receiving unit; wherein: Text receiving unit: used to obtain the original text data input by the user through the API interface or character input device, and perform coding standardization processing on the case markers of Tibetan and Mongolian; Visual receiving unit: includes an image collector and a video stream parser, which is used to receive raw image / video data from a camera or uploaded files, and perform cultural taboo recognition operations through religious symbol filters; Speech receiving unit: Integrates a microphone array and voiceprint separator to collect raw speech data in the environment and perform tone topology preprocessing operations for the Sino-Tibetan language family.

3. The multimodal language learning assistance system based on artificial intelligence according to claim 1, characterized in that: The cross-modal feature fusion module includes a visual feature extraction unit, a speech feature extraction unit, a text semantic encoding unit, and a cross-modal alignment unit; wherein: Visual feature extraction unit: This unit is used to divide the input visual data into two categories: static images and dynamic videos. It uses the Vision Transformer to extract static cultural symbol features and the 3D convolutional network to extract continuous gesture features, and then fuses them to generate a unified visual feature vector. Speech feature extraction unit: Based on the Conformer model optimized for Sino-Tibetan languages, it processes speech data in two stages. First, the phoneme recognition layer obtains basic speech structure information, and second, the tone topology analysis layer extracts intonation curve variation characteristics. The final fusion generates a phoneme sequence vector that represents the pronunciation patterns of the language. Text Semantic Encoding Unit: This unit uses a BERT variant model to process case-tagged text input. It first performs case-tagged-aware encoding through the embedding layer, then constructs a semantic dependency tree structure through the syntactic dependency parsing layer, and then outputs a semantic encoding vector. Cross-modal alignment unit: This unit is used to execute a triple attention mechanism, calculating the cultural alignment weight between visual features and semantic encoding, and the tonal coupling between phoneme features and semantic encoding. It also automatically activates the cultural conflict correction process when cultural symbol matching deviations are found, and finally fuses the corrected features to generate a unified cross-modal semantic representation.

4. The multimodal language learning assistance system based on artificial intelligence according to claim 3, characterized in that: The cross-modal alignment unit includes: Visual-text attention subunit: used to calculate the cultural symbol alignment weight α between the visual feature vector and the text semantic encoding vector; Speech-text attention subunit: used to calculate the tone-semantic coupling degree β between the phoneme sequence vector and the text semantic encoding vector; Cultural conflict detection subunit: used to detect when the visual-text attention weight α is lower than the set threshold α th Conflict correction is triggered when , and the corrected semantic vector T is output * ; Cross-modal fusion output subunit: used to fuse the corrected text semantic vector, original phoneme sequence vector, and original visual feature vector to generate the final cross-modal joint feature vector.

5. The multimodal language learning assistance system based on artificial intelligence according to claim 1, characterized in that: The dynamic adaptive learning module includes a grammar diffusion unit, a culture diffusion unit and a parameter generation unit; wherein: Syntactic Diffusion Unit: This unit receives the text semantic components in the cross-modal joint feature vector, extracts the grammatical dynamics trajectory based on a time-varying graph convolutional network, and generates a grammatical structure adaptation factor based on the honorific level identifiers in the cultural marker database. Cultural Diffusion Unit: This unit receives the visual component of the cross-modal joint feature vector, identifies cultural symbol primitives through the symbolic attention mechanism, and generates interpretation labels. Parameter generation unit: used to integrate grammatical structure adaptation factors and cultural symbol interpretation identifiers to generate dynamic scene parameters.

6. The multimodal language learning assistance system based on artificial intelligence according to claim 5, characterized in that: The grammar diffusion unit includes: Graph construction subunit: This unit is used to extract text semantic components from the cross-modal joint feature vector and construct a semantic dependency graph structure. Each semantic segment is used as a graph node, and the grammatical dependency relationship is used as an edge to form a time-varying graph structure. Trajectory extraction subunit: used to perform time-varying graph convolution operations on the graph structure and extract the grammatical dynamic trajectory vector H t ; Honorific fusion subunit: used to extract the grammatical trajectory vector H t Query the cultural marker database to obtain the honorific level embedding vector H corresponding to the target language type honorific and concatenated with the semantic encoding vector to generate the grammatical structure adaptation factor F gram .

7. The multimodal language learning assistance system based on artificial intelligence according to claim 6, characterized in that: The cultural diffusion unit includes: The symbol extraction subunit receives the visual component in the cross-modal joint feature vector, applies the symbolic attention mechanism to the visual feature vector, calculates the matching correlation between each position and the predefined cultural symbol template, and obtains the symbolic attention weight; Symbol recognition subunit: Based on the symbol attention weight, the symbol semantic vector set is weighted and summed to generate the preliminary cultural symbol semantic vector S raw ; Semantic Confirmation Subunit: Based on the current language background and cultural environment, it queries the symbol polysemy mapping relationship in the cultural marker database and outputs the cultural symbol interpretation ID with confidence culture .

8. The multimodal language learning assistance system based on artificial intelligence according to claim 7, characterized in that: The parameter generation unit includes: Vector splicing subunit: used to receive the grammatical structure adaptation factor F output by the grammar diffusion unit gram Cultural symbol interpretation ID output by cultural diffusion unit culture , and perform vector concatenation operations to construct the fused input vector X fusion : Feature compression subunit: used to convert the fused high-dimensional vector X fusion Input into the multi-layer perceptron structure, and extract the dynamic scene parameters P through nonlinear transformation and weight mapping scene .

9. The multimodal language learning assistance system based on artificial intelligence according to claim 1, characterized in that: The multimodal feedback data package specifically includes a target language grammar tree, a cultural background explanation diagram, and a pronunciation correction parameter matrix.

10. An artificial intelligence-based multimodal language learning assistance method, implemented by an artificial intelligence-based multimodal language learning assistance system according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1: Collect user input text, image / video, and voice data to form corresponding multimodal input data streams; S2: Extract the semantic encoding vector of text, the visual feature vector of image / video, and the phoneme sequence vector of speech as the input of the unified semantic space; S3: Executes a triple attention mechanism to determine the cultural alignment between text and image, as well as the tonal coupling between text and speech, and performs feature correction when cultural alignment is abnormal. S4: Based on the joint feature vector, the grammatical structure change features are extracted and combined with the honorific level to generate the grammatical adaptation factor. At the same time, the cultural symbols in the image are identified and the interpretation mark is generated. S5: Integrate grammatical adaptation factors with cultural interpretation markers to construct dynamic scenario parameters to drive feedback generation; S6: Generate grammatical tree structure, cultural diagram interpretation and pronunciation correction parameters based on dynamic scenario parameters, and push feedback results simultaneously.

Images