A child foreign language pronunciation correction system based on multi-modal learning

Abstract

The application discloses a kind of based on multimodal learning's children foreign language pronunciation correction system, it is related to speech analysis technical field, including, window detection module, based on target pronunciation timing skeleton, the data of children follow reading acquisition is normalized mapping, obtains children follow reading predicted window sequence, and carries out cross-modal change detection, obtains boundary candidate point set;Pronunciation segment reconstruction module, boundary candidate point set and target pronunciation timing skeleton are reconstructed with skeleton constraint competition, generate children candidate pronunciation sequence;Alignment calibration module, based on children candidate pronunciation sequence and target pronunciation timing skeleton, skeleton constraint alignment calibration is executed, and pronunciation parameter contrast structure is constructed;Attribution correction module, using leading bias chain decoupling attribution algorithm, pronunciation bias cause decomposition is executed to pronunciation parameter contrast structure, generates children foreign language pronunciation correction information.The application improves the training effectiveness of children foreign language pronunciation correction under speech recognition condition by target pronunciation timing skeleton constraint.

Description

Technical Field

[0001] This invention relates to the field of speech analysis technology, and in particular to a children's foreign language pronunciation correction system based on multimodal learning. Background Technology

[0002] With the development of intelligent speech processing, computer-aided language learning, speech recognition, and multimodal perception technologies, pronunciation evaluation and correction technologies for foreign language learning scenarios have evolved from early single acoustic scoring modes based on template matching and hidden Markov models to comprehensive analysis modes combining deep neural networks, temporal modeling, speech recognition, and visual perception. Current technologies typically collect learners' speech signals and analyze pronunciation duration, rhythm, spectral similarity, or segment boundaries, combining this with localization results from speech recognition and visual information such as children's lip movements and lip trajectories to improve detection stability and pronunciation analysis accuracy in complex noisy environments. Therefore, temporal skeleton construction, window-constrained detection, pronunciation segment reconstruction, audiovisual co-analysis, and speech recognition-assisted pronunciation correction related to children's pronunciation have become important and continuously evolving technological directions.

[0003] However, existing methods still have the following two shortcomings: First, existing schemes mostly focus on whole-sentence scoring, coarse-grained segment matching, and comparison of results after speech recognition, lacking a skeleton constraint mapping and reconstruction mechanism for children's local pronunciation fluctuation features. This makes it difficult to stably form a pronunciation segment sequence that matches the standard pronunciation in cases of connected speech, misalignment, boundary overlap, and local rhythm abnormalities, thus reducing the precision of pronunciation analysis under speech recognition conditions. Second, existing pronunciation correction methods mostly treat deviations as isolated error terms, usually only providing single-point scores, recognition deviation prompts, or simple error correction information, lacking analysis of the temporal transmission relationship between pronunciation deviation features and the dominant deviation chain. Therefore, it is difficult to generate targeted parameter-level correction information, and both interpretability and correction effectiveness are limited. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a children's foreign language pronunciation correction system based on multimodal learning to solve the problems of inaccurate pronunciation segment reconstruction and unclear attribution of deviations.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a children's foreign language pronunciation correction system based on multimodal learning, comprising: a skeleton construction module, which collects multimodal standard reference data and children's follow-up reading data, and performs multimodal temporal collaborative analysis to construct a target pronunciation temporal skeleton; a window detection module, which, based on the target pronunciation temporal skeleton, normalizes and maps the children's follow-up reading data to obtain the expected window sequence of children's follow-up reading, and performs cross-modal change detection to obtain a set of boundary candidate points; a pronunciation segment reconstruction module, which performs skeleton constraint competition reconstruction of the boundary candidate point set and the target pronunciation temporal skeleton to generate children's candidate pronunciation sequences; an alignment calibration module, which, based on children's candidate pronunciation sequences and the target pronunciation temporal skeleton, performs skeleton constraint alignment calibration to construct a pronunciation parameter comparison structure; and an attribution correction module, which uses a dominant deviation chain decoupling attribution algorithm to perform pronunciation deviation causal decomposition on the pronunciation parameter comparison structure to generate children's foreign language pronunciation correction information.

[0007] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the multimodal standard reference data includes target text, standard pronunciation audio, and standard lip-syncing video. The data collected from children's reading aloud includes children's real-time reading aloud and videos of children's lip movements.

[0008] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the specific steps for constructing the target pronunciation temporal skeleton are as follows: The target text is transcribed into phonetics to obtain the target pronunciation sequence. Then, the time sequence correspondence analysis is performed by combining standard pronunciation audio and standard lip-sync video to establish the audio-visual time sequence correspondence relationship. The target pronunciation sequence is collaboratively associated with the audiovisual temporal correspondence to construct the target pronunciation temporal skeleton.

[0009] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the step of normalizing and mapping the children's follow-up reading data based on the target pronunciation temporal skeleton to obtain the children's expected follow-up reading window sequence is as follows: Based on the target pronunciation temporal skeleton, the overall duration of the children's reading aloud data is scaled to form an initial mapping window sequence; Local pronunciation fluctuations are located in the data collected from children's reading aloud, forming a local fluctuation position sequence. Based on the local fluctuation position sequence, the initial mapping window sequence is calibrated to obtain the expected window sequence for children's reading aloud.

[0010] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the specific steps for obtaining the boundary candidate point set are as follows: Based on the expected window sequence of children's follow-up reading, speech change detection is performed on children's real-time follow-up reading speech to obtain speech boundary candidate points; Detect changes in lip movements in children's lip-reading videos and obtain candidate points for lip-reading boundaries; The temporal correspondence between speech boundary candidate points and lip shape boundary candidate points is extracted, and a collaborative screening algorithm is used to obtain a set of boundary candidate points.

[0011] As a preferred embodiment of the multimodal learning-based children's foreign language pronunciation correction system of the present invention, the specific steps of reconstructing the boundary candidate point set and the target pronunciation temporal skeleton through skeleton constraint competition to generate a children's candidate pronunciation sequence are as follows: Extract the target pronunciation sequence and temporal relationship from the target pronunciation temporal skeleton, perform sequential anchoring and pairing on the set of boundary candidate points, and generate candidate boundary pairs; A competitive resolution algorithm is used to calculate the interval assignment priority value, and based on the interval assignment priority value, the candidate boundary pairs are locked for the dominant interval and the adjacent intervals are reconstructed to determine the target pronunciation interval. Based on the target pronunciation range, the child's real-time pronunciation is continuously segmented to generate a candidate pronunciation sequence for the child.

[0012] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the specific steps of performing skeleton constraint alignment calibration based on the children's candidate pronunciation sequence and the target pronunciation temporal skeleton are as follows: The target pronunciation temporal skeleton is located and expanded to obtain the target pronunciation range; The candidate pronunciation sequences of children are expanded to obtain pronunciation localization sequences; The target articulation range and articulation localization sequence are subjected to fall area correction and cross area correction to form a calibration correlation group.

[0013] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the pronunciation parameter comparison structure is constructed by performing misalignment correction on the calibration association group and using the interval adsorption recombination algorithm to calculate the interval adsorption intensity for adsorption recombination.

[0014] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the method of employing a dominant deviation chain decoupling attribution algorithm to perform pronunciation deviation cause decomposition on the pronunciation parameter comparison structure includes the following specific steps: The deviation expansion analysis of the pronunciation parameter comparison structure is performed to obtain pronunciation offset feature terms; Adjacency correlation analysis is performed on the pronunciation offset feature terms to establish temporal adjacency relationships, and conduction tests are performed on adjacent pronunciation offset feature terms to establish influence transmission relationships; Based on temporal adjacency and influence transmission, pronunciation deviation feature terms are chained together to form deviation chain candidate groups; Based on the candidate groups of deviation chains, the dominant screening value is calculated by the dominant deviation chain competitive screening algorithm, and the candidate groups of deviation chains are screened according to the dominant screening value to generate deviation chain attribution features.

[0015] As a preferred embodiment of the children's foreign language pronunciation correction system based on multimodal learning described in this invention, the specific steps for generating children's foreign language pronunciation correction information are as follows: Decoupling attribution processing is performed on the attribution features of the deviation chain to construct the deviation attribution structure, and back-projection reconstruction of the pronunciation parameters is performed to generate a set of correction parameters; The pronunciation correction trajectory reconstruction is performed on the correction parameter group to generate children's foreign language pronunciation correction information.

[0016] The beneficial effects of this invention are as follows: by constraining the target pronunciation temporal skeleton, competitive resolution of boundary conflicts, boundary overlaps and local misalignments in continuous children's pronunciation is achieved, generating children's candidate pronunciation sequences. Then, chain-like attribution and dominant screening are performed on the pronunciation deviation feature items in the pronunciation parameter comparison structure to form a deviation chain with causal hierarchy and complete the reconstruction of correction parameters. This not only improves the accuracy of children's pronunciation localization and segmentation, but also enhances the pertinence, interpretability and training effectiveness of children's foreign language pronunciation correction under speech recognition conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a children's foreign language pronunciation correction system based on multimodal learning.

[0019] Figure 2 A flowchart for constructing the target pronunciation timing skeleton.

[0020] Figure 3 The flowchart for skeleton constraint competition reconstruction.

[0021] Figure 4 A flowchart for decoupling and attribution of the dominant deviation chain.

[0022] Figure 5A comparison chart of success rates in resolving border conflicts.

[0023] Figure 6 This is a comparison chart of boundary overlap rate and local misalignment rate. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Reference Figures 1-6 This is one embodiment of the present invention, which provides a children's foreign language pronunciation correction system based on multimodal learning, including the following steps: The skeleton construction module collects multimodal standard reference data and children's reading aloud data, and performs multimodal temporal collaborative analysis to construct the target pronunciation temporal skeleton.

[0028] Multimodal standard reference data includes target text, standard pronunciation audio, and standard lip-syncing video.

[0029] Specifically, the system receives the standard writing content corresponding to the current children's foreign language pronunciation correction task and enters the standard writing content into a continuous text according to the actual reading order; when the standard writing content contains multiple words, phrases or sentences, they are arranged in the order of actual reading to form the target text.

[0030] The target text is read aloud continuously by a reader with standard pronunciation, and all sound content is recorded simultaneously during the reading process. After the reading is completed, the recorded sound content is saved as a complete reading process to form a standard pronunciation audio corresponding to the target text.

[0031] During the same reading process recorded with standard pronunciation audio, the reader's lip area is continuously videotaped; after the recording is completed, the lip movements during the same reading process are saved continuously in chronological order to form a standard lip-shape video corresponding to the standard pronunciation audio.

[0032] The data collected from children's reading aloud includes real-time audio recordings of children reading aloud and videos of children's lip movements.

[0033] Specifically, the system outputs the target text to the child and records the continuous speech produced by the child when the child begins to repeat it. After the child completes a full repetition, the recorded speech is saved continuously according to the repetition process to form the child's real-time repetition speech.

[0034] During the same reading process, the child's lip area is continuously recorded. After the child completes a full reading, the lip movements during the same reading process are saved sequentially in time to form a video of the child's lip movements corresponding to the child's real-time reading.

[0035] The target text is transcribed into phonetics to obtain the target pronunciation sequence. Then, a temporal correspondence analysis is performed by combining standard pronunciation audio and standard lip-sync video to establish an audio-visual temporal correspondence relationship.

[0036] Specifically, the text content in the target text is read segment by segment in the actual reading order, and each segment of the target text is converted into corresponding pronunciation content. Then, all the pronunciation content is arranged in the order of the target text to form the target pronunciation sequence.

[0037] The sound content in the standard pronunciation audio is continuously read along the time direction of the standard pronunciation audio, and the sound change process in the standard pronunciation audio is compared with the previous and next steps in the target pronunciation sequence to determine the start and end positions of each pronunciation content in the target pronunciation sequence in the standard pronunciation audio.

[0038] The lip movements in the standard lip-sync video are continuously read along the time direction of the standard lip-sync video. Then, the lip-sync changes in the standard lip-sync video are compared sequentially according to the order of the target pronunciation sequence to determine the start and end positions of each pronunciation in the target pronunciation sequence in the standard lip-sync video.

[0039] The sound start position, sound end position, mouth shape start position, and mouth shape end position corresponding to the same pronunciation content are arranged one by one according to the order in the target pronunciation sequence to establish the audio-visual temporal correspondence.

[0040] The target pronunciation sequence is collaboratively associated with the audiovisual temporal correspondence to construct the target pronunciation temporal skeleton.

[0041] Specifically, each pronunciation content is read in the order of the target pronunciation sequence. The start position of the sound, the end position of the sound, the start position of the mouth shape, and the end position of the mouth shape are retrieved one by one from the audio-visual temporal correspondence. Each pronunciation content is then combined with its corresponding start position of the sound, the end position of the sound, the start position of the mouth shape, and the end position of the mouth shape to form the temporal association content corresponding to each pronunciation content.

[0042] All temporally related content is arranged sequentially according to the order in the target pronunciation sequence, so that the sequential relationship between adjacent pronunciation content is consistent with the target pronunciation sequence, and the start and end positions of the sound and mouth shape corresponding to each pronunciation content are consistent with the temporal relationship of the audiovisual sequence; all temporally related content obtained in order is organized as a whole to construct the temporal skeleton of the target pronunciation.

[0043] It should be noted that the target pronunciation temporal skeleton is a temporal pronunciation organization structure formed by associating and organizing each pronunciation content in the target pronunciation sequence with its corresponding sound start and end positions and mouth shape start and end positions in a unified order. It is used to characterize the sequential arrangement of each pronunciation content in the standard pronunciation process and the corresponding time position of each pronunciation content in the standard pronunciation audio and standard mouth shape video.

[0044] The window detection module, based on the target pronunciation temporal skeleton, normalizes and maps the data collected from children's reading aloud, obtains the expected window sequence of children's reading aloud, and performs cross-modal change detection to obtain a set of boundary candidate points.

[0045] Based on the target pronunciation temporal skeleton, the overall duration of the children's reading aloud data is scaled to form an initial mapping window sequence.

[0046] Specifically, the corresponding start and end positions of sounds are read one by one along the arrangement order of the target pronunciation timing skeleton. The first sound start position in the target pronunciation timing skeleton is determined as the starting position of the overall standard pronunciation time range, and the last sound end position in the target pronunciation timing skeleton is determined as the ending position of the overall standard pronunciation time range, thus obtaining the entire sentence repetition time range.

[0047] The starting and ending positions of the child's real-time pronunciation are read from the entire sentence-following time range and determined as the overall follow-up time range; the corresponding ratio between the overall standard pronunciation time range and the overall follow-up time range is used as the scaling ratio; while keeping the order of each pronunciation content unchanged, the time boundaries corresponding to each pronunciation content are mapped to the entire sentence-following time range according to the scaling ratio.

[0048] The time boundaries corresponding to each scaled pronunciation content are continuously arranged according to the order of the target pronunciation temporal skeleton to form an initial mapping window sequence.

[0049] Local pronunciation fluctuations are located in the data collected from children's reading aloud, forming a local fluctuation position sequence. Based on the local fluctuation position sequence, the initial mapping window sequence is calibrated to obtain the expected window sequence for children's reading aloud.

[0050] Specifically, each initial time range in the initial mapping window sequence is sequentially mapped to the child's real-time pronunciation; the duration of speech, the duration of pauses, and the position of adjacent pronunciation transitions are continuously recorded at adjacent time positions. When the duration of speech at adjacent time positions increases or decreases, the duration of pauses lengthens or shortens, or the position of adjacent pronunciation transitions moves forward or backward, the corresponding speech change position is recorded.

[0051] Each initial time range is mapped to the same time period in the child's lip-reading video. Within the corresponding time period, the duration of lip opening and closing, the duration of lip-reading, and the position of lip-reading pause are continuously recorded at adjacent time positions. When the duration of lip opening and closing increases or decreases, the duration of lip-reading is prolonged or shortened, or the position of lip-reading pause is moved forward or backward, the corresponding position of lip-reading change is recorded.

[0052] Based on the correspondence between the speech change positions and the mouth shape change positions within the same initial time range, the local speech fluctuation positions are determined. These speech change positions and mouth shape change positions, whose time positions match the order of arrangement, are then organized to form a sequence of local fluctuation positions. The initial time range is read item by item according to the order in the initial mapping window sequence, and the corresponding local fluctuation positions are found (e.g., positions of speech duration changes, pause duration changes, adjacent speech transition changes, lip opening / closing duration changes, mouth shape holding time changes, and mouth shape pause changes). When a corresponding local fluctuation position exists, the start and end positions of the initial time range are shifted forward or backward. When no corresponding local fluctuation position exists, the corresponding initial time range remains unchanged.

[0053] Organize all adjusted time ranges sequentially according to the initial mapping window sequence to obtain the expected window sequence for children's follow-up reading.

[0054] Based on the expected window sequence of children's follow-up reading, speech change detection is performed on children's real-time follow-up reading speech to obtain speech boundary candidate points.

[0055] Specifically, the audio content is continuously read along the time direction of the child's real-time reading aloud, and each expected time range is sequentially matched according to the arrangement order in the expected window sequence of the child's reading aloud. From the beginning to the end of the expected time range, the audio duration, pauses and pronunciation transitions at adjacent positions are continuously compared. When the audio duration changes from long to short or from short to long, the pause duration changes from short to long or from long to short, or the adjacent pronunciation transition position moves forward or backward, the corresponding time position is recorded.

[0056] All recorded time positions are arranged in chronological order to form a set of speech change positions corresponding to the current expected time range; the sets of speech change positions corresponding to each expected time range are then arranged continuously according to the order in the expected window sequence for children's reading aloud to obtain candidate speech boundary points.

[0057] Detect changes in lip movements in children's lip-sync videos and obtain candidate points for lip-sync boundaries.

[0058] Specifically, the lip movements in the video are continuously read along the timeline of the child's lip movements, and each predicted time range is sequentially assigned according to the order in the predicted window sequence for the child's follow-along reading. The opening and closing of the lips, the holding of the lip shape, and the pausing of the lip shape are continuously compared from the beginning to the end of the predicted time range. When the duration of the opening and closing of the lips changes from long to short or from short to long, the holding of the lip shape changes from long to short or from short to long, or the pausing position moves forward or backward between adjacent positions, the corresponding time position is recorded.

[0059] All recorded time positions are arranged in chronological order to form a set of mouth shape change positions corresponding to the expected time range; the sets of mouth shape change positions corresponding to each expected time range are then arranged continuously according to the order in the expected window sequence for children's reading aloud to obtain candidate points for mouth shape boundaries.

[0060] The temporal correspondence between speech boundary candidate points and lip shape boundary candidate points is extracted, and a collaborative screening algorithm is used to obtain a set of boundary candidate points.

[0061] Specifically, the speech boundary candidate points and mouth shape boundary candidate points are read one by one in the order of the expected window sequence of the child's reading. Starting from the first speech boundary candidate point, they are compared with each time position in the mouth shape boundary candidate point. When the speech boundary candidate point and the mouth shape boundary candidate point are in the same order and have the same time position, the corresponding speech boundary candidate point and mouth shape boundary candidate point are organized into a set of corresponding positions to form a temporal correspondence between the speech boundary candidate point and the mouth shape boundary candidate point.

[0062] Within the expected time range, positions that have both speech boundary candidate points and lip shape boundary candidate points are retained. Positions that only have speech boundary candidate points or only have lip shape boundary candidate points and intersect with adjacent expected time ranges are removed. All the retained and sorted corresponding positions are organized continuously according to the order in the expected window sequence of children's reading aloud to obtain the boundary candidate point set.

[0063] It should be noted that the audio-visual collaborative boundary fusion algorithm is a processing method that jointly filters and fuses candidate speech boundary points and candidate lip shape boundary points. Its function is to extract boundary positions from speech change locations and lip shape movement change locations. The principle of the audio-visual collaborative boundary fusion algorithm is to match the positions of candidate speech boundary points and candidate lip shape boundary points within the same expected time range according to the expected window sequence of the child's reading aloud. Then, it retains the corresponding positions that simultaneously exhibit both speech and lip shape movement changes, eliminates positions with only unilateral changes that overlap with adjacent expected time ranges, and merges and organizes corresponding positions with close time positions.

[0064] The pronunciation segment reconstruction module performs skeleton constraint competition reconstruction on the boundary candidate point set and the target pronunciation temporal skeleton to generate a child candidate pronunciation sequence.

[0065] Extract the target pronunciation sequence and temporal relationship from the target pronunciation temporal skeleton, perform sequential anchoring and pairing on the set of boundary candidate points, and generate candidate boundary pairs.

[0066] Specifically, the preceding and following positions and corresponding time positions of each pronunciation are read in sequence, and the target pronunciation sequence and time sequence are formed according to the reading order.

[0067] The positions of each boundary candidate point are read sequentially along the arrangement order of the boundary candidate point set, and the order and time interval relationships between adjacent boundary candidate points are compared one by one. When the order of adjacent boundary candidate points is consistent with the target pronunciation order, and the time interval between adjacent boundary candidate points falls within the time range of the corresponding pronunciation content, the boundary candidate point located at the start time position is determined as the start boundary candidate point, and the boundary candidate point located at the end time position is determined as the end boundary candidate point. The start boundary candidate point and the end boundary candidate point are then organized into a set of sequential anchoring pairs.

[0068] Record the sequential pairings of all candidate boundary points, and then arrange all the formed sequential anchoring pairs in the order of the candidate boundary point set to generate candidate boundary pairs.

[0069] A competitive resolution algorithm is used to calculate the interval assignment priority value. Based on the interval assignment priority value, the candidate boundary pairs are locked with dominant intervals and reconstructed by adjacent intervals to determine the target pronunciation interval.

[0070] Specifically, the time difference between the time range of the candidate boundary pairs and the time position of the corresponding pronunciation content in the target pronunciation time sequence skeleton is compared as the proximity of the time positions; the order of the candidate boundary pairs in the set of candidate boundary points is compared with the order of the corresponding pronunciation content in the target pronunciation time sequence skeleton to obtain the degree of order consistency (e.g., the degree of order consistency is 1 when the order is consistent and 0 when the order is inconsistent), and a competition resolution algorithm is used to calculate the interval assignment priority value for each candidate boundary pair, expressed as: ; in, This is the priority value for interval assignment. This represents the time difference between the time range of the candidate boundary pair and the time position of the corresponding pronunciation content. For the degree of consistency in sequence, It is a natural constant.

[0071] The time ranges between adjacent candidate boundary pairs are compared sequentially along the arrangement order. When there is no overlap between the time ranges of adjacent candidate boundary pairs, the time ranges corresponding to the adjacent candidate boundary pairs are directly retained. When there is an overlap between the time ranges of adjacent candidate boundary pairs, the interval assignment priority values ​​of the adjacent candidate boundary pairs are compared, and the time range corresponding to the candidate boundary pair with the higher interval assignment priority value is determined as the dominant interval. The start or end boundary candidate points of the candidate boundary pairs with the lower interval assignment priority value are adjusted, and the adjusted candidate boundary pairs are compared with the adjacent target pronunciation intervals in sequence to determine the start position difference, end position difference, and overlap length between the adjusted candidate boundary pairs and the previous and next target pronunciation intervals, respectively. The adjacent target pronunciation intervals with close position differences and larger overlap lengths are determined as the reabsorption intervals, and the time ranges corresponding to the adjusted candidate boundary pairs are re-merged into the reabsorption intervals to complete the reabsorption reconstruction of adjacent intervals.

[0072] The candidate boundaries reconstructed through the reabsorption of adjacent intervals are rearranged sequentially according to the target pronunciation order to determine the target pronunciation interval.

[0073] It should be noted that the competition resolution algorithm is a method for comparing, filtering, and eliminating interval assignment conflicts between candidate boundary pairs. Its function is to retain the more suitable boundary range from multiple candidate boundary pairs corresponding to the same target pronunciation interval, and to exclude overlapping or conflicting boundary combinations. The principle is to calculate the interval assignment priority value for each candidate boundary pair, and then compare adjacent candidate boundary pairs one by one according to their order of arrangement to see if there is any overlap in time ranges. When there is overlap, the candidate boundary pair with the higher interval assignment priority value is retained, while the candidate boundary pairs with the lower interval assignment priority value are deleted or have their boundaries adjusted.

[0074] Based on the target pronunciation range, the child's real-time pronunciation is continuously segmented to generate a candidate pronunciation sequence for the child.

[0075] Specifically, the starting position corresponding to the target pronunciation interval is used as the starting point of the current pronunciation segmentation, and the ending position corresponding to the target pronunciation interval is used as the ending point of the current pronunciation segmentation. The continuous speech content between the starting point and the ending point of the current pronunciation segmentation is extracted from the child's real-time follow-up speech to form candidate pronunciation content corresponding to the current target pronunciation interval.

[0076] The candidate pronunciation content corresponding to all target pronunciation intervals is counted and recorded, and then arranged in the order of the target pronunciation intervals to generate a candidate pronunciation sequence for children.

[0077] Example Figure 5 The comparison results of the success rate of boundary conflict resolution are shown, with the three curves corresponding to Control Group 1, Control Group 2, and the experimental group, respectively. Control Group 1 uses a coarse-grained segment matching method, that is, it only segments the pronunciation segments according to the whole sentence duration mapping and the position of the speech pause; Control Group 2 uses the target pronunciation temporal skeleton and audiovisual collaborative boundary fusion method, that is, first constructing the target pronunciation temporal skeleton, and then, within the expected window sequence of the child's reading, matching the speech boundary candidate points and mouth shape boundary candidate points in time position, retaining the corresponding positions with consistent time order and close time position, eliminating the positions with only unilateral changes and intersection with adjacent expected time ranges, and merging and organizing the corresponding positions with close time positions; The experimental group uses the complete method of this invention, that is, after obtaining the candidate boundary pairs, it further calculates the interval assignment priority value, performs dominant interval locking and adjacent interval retraction reconstruction on the candidate boundary pairs with overlapping time ranges, determines the target pronunciation interval, and generates the child's candidate pronunciation sequence. The curves of the experimental group in the figure are higher than those of the two control groups in each round, indicating that the present invention can more effectively eliminate boundary conflicts, boundary overlaps and local misalignments in continuous children's pronunciation, thereby improving the accuracy of children's pronunciation positioning and segmentation.

[0078] The alignment calibration module performs skeleton constraint alignment calibration based on the child's candidate pronunciation sequence and the target pronunciation temporal skeleton, and constructs a pronunciation parameter comparison structure.

[0079] The target pronunciation temporal skeleton is located and expanded to obtain the target pronunciation range.

[0080] Specifically, the start and end positions of each pronunciation are read sequentially along the arrangement order of the target pronunciation timing skeleton, and the arrangement position of each pronunciation in the target pronunciation timing skeleton is matched with the corresponding start and end positions of the sound to form the positioning content corresponding to each pronunciation.

[0081] According to the sequential arrangement in the target pronunciation time sequence skeleton, all the localized content is continuously expanded to ensure that the time range corresponding to each pronunciation content and the sequential connection relationship between adjacent pronunciation content are consistent with the target pronunciation time sequence skeleton, thereby obtaining the target pronunciation interval.

[0082] The candidate pronunciation sequences of children are expanded to obtain pronunciation localization sequences.

[0083] Specifically, each candidate pronunciation is read sequentially according to the order of the child's candidate pronunciation sequence. Then, the start and end positions of each candidate pronunciation in the child's real-time follow-up speech are determined according to the order in the child's candidate pronunciation sequence. The start and end positions of each candidate pronunciation are matched one by one to form the pronunciation position range corresponding to each candidate pronunciation.

[0084] According to the sequential arrangement of the child's candidate pronunciation sequence, all pronunciation position ranges are continuously expanded to ensure that the pronunciation position ranges corresponding to the previous and subsequent candidate pronunciation contents, as well as the sequential connection relationships between adjacent candidate pronunciation contents, remain consistent with the child's candidate pronunciation sequence. All pronunciation position ranges are then arranged sequentially to obtain the pronunciation positioning sequence.

[0085] The target articulation range and articulation localization sequence are subjected to fall area correction and cross area correction to form a calibration correlation group.

[0086] Specifically, the positional range of the candidate pronunciation content is compared with the time range of each target pronunciation interval item by item. When the positional range of each candidate pronunciation content falls completely within the corresponding target pronunciation interval, the candidate pronunciation content remains unchanged. When the positional range of each candidate pronunciation content is before or after the corresponding target pronunciation interval, the positional proximity between the positional range of each candidate pronunciation content and the adjacent target pronunciation interval is compared, and the target pronunciation interval with the closest position is redefined as the corresponding interval, thus completing the area correction.

[0087] Compare whether the pronunciation position range corresponding to each candidate pronunciation content simultaneously crosses two adjacent target pronunciation intervals. When the pronunciation position range corresponding to each candidate pronunciation content falls into two adjacent target pronunciation intervals, determine the coverage length of the pronunciation position range corresponding to each candidate pronunciation content in the two adjacent target pronunciation intervals, and determine the target pronunciation interval with the largest coverage length as the corresponding interval.

[0088] Each candidate pronunciation content is matched with the corrected target pronunciation interval according to the order of the target pronunciation interval and pronunciation positioning sequence to form a calibration association group, thus completing the cross-regional correction.

[0089] The calibration correlation group was misaligned and adjusted, and the interval adsorption intensity was calculated using the interval adsorption recombination algorithm to construct a sound parameter comparison structure.

[0090] Specifically, according to the order of the calibration association groups, the correspondence between each candidate pronunciation content and the corresponding target pronunciation interval is read sequentially. Then, the start position relationship, end position relationship, and coverage relationship between the pronunciation position range corresponding to each candidate pronunciation content and the time range corresponding to the target pronunciation interval are compared item by item. The correspondences where the start position is before the start position of the target pronunciation interval, the start position is after the start position of the target pronunciation interval, the end position is before the end position of the target pronunciation interval, the end position is after the end position of the target pronunciation interval, and the coverage overlaps are recorded item by item to form a misalignment position record.

[0091] According to the order of the calibration association groups, the starting position of the pronunciation position range corresponding to each candidate pronunciation content is made close to the starting position of the corresponding target pronunciation interval, so that the ending position of the pronunciation position range corresponding to each candidate pronunciation content is made close to the ending position of the corresponding target pronunciation interval.

[0092] According to the order of the calibration association groups, the pronunciation position range of each candidate pronunciation content after being corrected and the corresponding time range of the target pronunciation interval are read sequentially. Then, the interval adsorption recombination algorithm is used to calculate the interval length, start position difference, end position difference, coverage length, and overlap length between the corrected pronunciation position range of each candidate pronunciation content and the corresponding target pronunciation interval. Finally, the interval adsorption strength is calculated, expressed as: ; in, The adsorption strength is the region. The overlapping length, The length of the target pronunciation interval. For the coverage length, The difference between the starting positions. This is the difference between the ending positions.

[0093] The candidate pronunciation content with the highest adsorption strength in a given interval is adsorbed into the corresponding target pronunciation interval. The candidate pronunciation content with the lowest adsorption strength in a given interval and the smallest difference between the start position and the end position of the adjacent target pronunciation interval is adjusted into the adjacent target pronunciation interval, thus completing the adsorption recombination. All the recombined candidate pronunciation content and the corresponding target pronunciation interval are then continuously arranged according to the order of the target pronunciation interval and the pronunciation positioning sequence to construct a pronunciation parameter comparison structure.

[0094] It should be noted that the interval adsorption and recombination algorithm is a processing method for re-correcting and recombinating the correspondence between candidate pronunciation content and target pronunciation intervals. Its function is to re-merge misaligned, overlapping, or intersecting candidate pronunciation content into a more matching target pronunciation interval. The principle of the interval adsorption and recombination algorithm is to extract the start position difference, end position difference, coverage length, and overlap length between the pronunciation position range corresponding to the candidate pronunciation content and the corresponding time range of the target pronunciation interval. Then, based on the start position difference, end position difference, coverage length, and overlap length, the interval adsorption strength is calculated. Adsorption and recombination are completed according to the magnitude of the interval adsorption strength, thereby stabilizing the correspondence between candidate pronunciation content and target pronunciation intervals.

[0095] Example Figure 6 The comparison results of lower boundary overlap rate and local misalignment rate for different processing groups are shown. Direct sequential correspondence processing, skeleton constraint alignment calibration processing, and the complete processing of this invention are employed, respectively. Direct sequential correspondence processing first performs coarse-grained segmentation of the child's pronunciation based on the entire sentence duration and speech pause positions, then matches the resulting pronunciation segments with the target pronunciation intervals item by item according to the arrangement order, without performing subsequent position correction. Skeleton constraint alignment calibration processing expands the target pronunciation temporal skeleton and the child's candidate pronunciation sequence, retaining candidate pronunciation content that completely falls into the target pronunciation interval in its original position, reassigning positions before and after the interval to the nearest target pronunciation interval, and assigning candidate pronunciation content that crosses adjacent intervals to the interval with the longest coverage length, forming a calibration association group. The complete processing of this invention records the misalignment relationship of the start position, end position, and coverage range based on the calibration association group, then determines the interval adsorption strength based on the overlap length, coverage length, and start-end position difference, re-merging candidate pronunciation content into the corresponding or adjacent target pronunciation intervals, constructing a pronunciation parameter comparison structure. The figure shows that the two indicators for the complete processing group of this invention are the lowest, indicating that it can effectively reduce the cross-correspondence and local misalignment of pronunciation intervals.

[0096] The attribution correction module uses a dominant deviation chain decoupling attribution algorithm to perform pronunciation deviation cause decomposition on the pronunciation parameter comparison structure and generate children's foreign language pronunciation correction information.

[0097] The pronunciation parameter comparison structure is analyzed by deviation expansion to obtain pronunciation offset feature terms.

[0098] Specifically, the pronunciation position range corresponding to the candidate pronunciation content in the pronunciation parameter comparison structure and the time range corresponding to the target pronunciation interval are read one by one. By comparing the positional relationship between the start position, the positional relationship between the end position, the coverage relationship between the pronunciation position range and the time range, and the connection relationship between adjacent candidate pronunciation content, when the start position is inconsistent, the end position is inconsistent, the coverage range is inconsistent, or the connection position between adjacent candidate pronunciation content is inconsistent, the corresponding inconsistent content is recorded one by one.

[0099] All recorded content is continuously organized according to the arrangement order of the pronunciation parameter comparison structure, so that the adjacent recorded content is consistent with the pronunciation parameter comparison structure, and the pronunciation offset feature items are obtained.

[0100] Adjacency correlation analysis is performed on the pronunciation offset feature terms to establish temporal adjacency relationships, and conduction tests are performed on adjacent pronunciation offset feature terms to establish influence transmission relationships.

[0101] Specifically, each pronunciation offset feature item is compared to see if they correspond to adjacent target pronunciation intervals, whether they maintain a continuous sequential arrangement, and whether there are no other pronunciation offset features between the end position of the time range corresponding to the previous pronunciation offset feature item and the beginning position of the time range corresponding to the next pronunciation offset feature item. When adjacent pronunciation offset features simultaneously satisfy the conditions of adjacent target pronunciation intervals, continuous sequential arrangement, and adjacent time ranges, the correspondence between adjacent pronunciation offset features items is recorded item by item to form a temporally adjacent relationship.

[0102] Extract the temporal adjacency order and read the preceding and following pronunciation offset feature items sequentially. Then, compare the changes in inconsistencies corresponding to the preceding pronunciation offset feature item with the corresponding changes in the start position relationship, end position relationship, coverage relationship, or connection relationship of the following pronunciation offset feature item. If the change in inconsistency corresponding to the preceding pronunciation offset feature item occurs before the change in the start position relationship, end position relationship, coverage relationship, or connection relationship of the following pronunciation offset feature item occurs after the change, and the target pronunciation intervals corresponding to the two pronunciation offset feature items remain adjacent, record the correspondence between the preceding and following pronunciation offset feature items item by item to form the influence transmission relationship.

[0103] It should be noted that adjacent pronunciation offset feature items refer to two pronunciation offset feature items that are adjacent to each other in the order of the target pronunciation intervals and whose corresponding time ranges are adjacent in the pronunciation parameter comparison structure. They are used to characterize the adjacency state of the offset relationship between consecutive pronunciation contents. Adjacent pronunciation offset feature items not only reflect the connection relationship between adjacent target pronunciation intervals, but also serve as the basis for establishing temporal adjacency relationships and influence transmission relationships.

[0104] The influence transmission relationship refers to the correlation formed when a change in the preceding pronunciation offset feature term results in a corresponding change in the following pronunciation offset feature term.

[0105] Based on temporal adjacency and influence transmission, the pronunciation offset feature terms are chained together to form a candidate group of deviation chains.

[0106] Specifically, the inconsistencies in the start position, end position, coverage, and connection position of each pronunciation offset feature item are extracted. The corresponding relationships of adjacent pronunciation offset feature items are read one by one according to the temporal adjacency relationship. The preceding and following pronunciation offset feature items with temporal adjacency relationships are connected in the order of their predecessors and successors to form an initial connection chain.

[0107] According to the order of influence transmission relationships, the transmission correspondence between the previous pronunciation offset feature item and the next pronunciation offset feature item is read item by item, and the next pronunciation offset feature item with influence transmission relationship is incorporated into the corresponding initial connection chain, so that each pronunciation offset feature item in the same connection chain maintains both temporal adjacency relationship and influence transmission relationship.

[0108] All pronunciation deviation features are arranged sequentially according to their order, and each connection chain formed by the continuous connection of multiple pronunciation deviation features is further arranged to form a candidate group of deviation chains.

[0109] Based on the candidate groups of deviation chains, the dominant screening value is calculated by the dominant deviation chain competitive screening algorithm, and the candidate groups of deviation chains are screened according to the dominant screening value to generate deviation chain attribution features.

[0110] Specifically, the number of pronunciation offset feature terms, the number of corresponding target pronunciation intervals, and the connection length between the first and last pronunciation offset feature terms in each deviation chain candidate group are read sequentially. The dominant screening value for each deviation chain candidate group is then calculated, expressed as: ; in, These are the dominant screening values ​​corresponding to each candidate group in the deviation chain. The number of pronunciation offset feature terms. The corresponding number of target pronunciation intervals, This is the normalized value of the connection length between the first and last pronunciation offset feature terms.

[0111] The dominant screening values ​​of each candidate deviation chain are compared one by one according to the order of the candidate deviation chains. The candidate deviation chain with the highest dominant screening value is determined as the dominant deviation chain. The candidate deviation chain with a temporal adjacent relationship or influence transmission relationship with the dominant deviation chain and a dominant screening value lower than that of the dominant deviation chain is determined as the associated deviation chain.

[0112] The dominant deviation chains and associated deviation chains are continuously sorted according to the order of the candidate deviation chain groups to generate deviation chain attribution features.

[0113] It should be noted that the dominant deviation chain competitive screening algorithm is a comparative screening method used to determine the dominance of each deviation chain candidate group. Its function is to identify the dominant deviation chain with a stronger impact on overall pronunciation abnormalities from multiple deviation chain candidate groups, and to distinguish the associated deviation chains related to the dominant deviation chain. The principle of the dominant deviation chain competitive screening algorithm is to first extract the number of pronunciation deviation feature items, the number of corresponding target pronunciation intervals, and the connection length between the first and last pronunciation deviation feature items in each deviation chain candidate group, and then calculate the dominant screening value. Based on the magnitude of the dominant screening value, each deviation chain candidate group is sorted and screened.

[0114] Deviation chain attribution features are used to characterize the primary and secondary attribution relationships between the dominant deviation chain and the associated deviation chain, which helps to transform scattered pronunciation deviations into a corrective basis with causal hierarchy.

[0115] Decoupling attribution processing is performed on the attribution features of the deviation chain to construct the deviation attribution structure, and back-projection reconstruction of the pronunciation parameters is performed to generate a set of correction parameters.

[0116] Specifically, each pronunciation deviation feature in the dominant deviation chain is compared with each pronunciation deviation feature in the associated deviation chain. Based on the influence transmission relationship, the associated pronunciation deviation features that have a forward-backward transmission relationship with the dominant deviation chain are organized into the associated pronunciation deviation features formed by transmission. The pronunciation deviation features that do not have an influence transmission relationship with the dominant deviation chain and only correspond to the local target pronunciation interval are organized into the independent pronunciation deviation features. The dominant deviation chain, associated deviation chain, associated pronunciation deviation features formed by transmission, and independent pronunciation deviation features are organized continuously according to the order of deviation chain attribution features to construct the deviation attribution structure.

[0117] Following the order of the deviation attribution structure, the changes in start position, end position, coverage, and transition position corresponding to each pronunciation deviation feature item are read sequentially. The changes in start position are mapped to start position adjustment content, the changes in end position to end position adjustment content, the changes in coverage to coverage adjustment content, and the changes in transition position to transition position adjustment content. Then, the changes in start position, end position, coverage, and transition position are mapped to the target pronunciation interval item by item, and continuously organized according to the order of the target pronunciation intervals to generate a set of correction parameters.

[0118] The pronunciation correction trajectory reconstruction is performed on the correction parameter group to generate children's foreign language pronunciation correction information.

[0119] Specifically, according to the order of the target pronunciation intervals, the start position adjustment content, end position adjustment content, and coverage adjustment content corresponding to the previous target pronunciation interval, the start position adjustment content, end position adjustment content, and coverage adjustment content corresponding to the next target pronunciation interval, and the connection position adjustment content between adjacent target pronunciation intervals are connected one by one to form a pronunciation correction trajectory that continuously corresponds to all target pronunciation intervals.

[0120] The starting position correction content, ending position correction content, coverage correction content, and connection correction content between adjacent target pronunciation intervals are read sequentially along the pronunciation correction trajectory. The starting position correction content, ending position correction content, coverage correction content, and connection correction content between adjacent target pronunciation intervals are then continuously organized according to the arrangement order of the target pronunciation intervals to generate children's foreign language pronunciation correction information.

[0121] It should be noted that the dominant deviation chain decoupling attribution algorithm is a method for chain analysis and primary / secondary attribution of pronunciation deviations. Its function is to connect the scattered pronunciation deviation feature items in the pronunciation parameter comparison structure according to their temporal and propagation relationships to form a candidate deviation chain group. Then, it filters out the dominant deviation chain that has the greatest impact on the overall pronunciation abnormality from the candidate deviation chain group, and distinguishes the associated deviation chains that are linked to the dominant deviation chain. The dominant deviation chain decoupling attribution algorithm first determines the adjacent pronunciation deviation feature items through temporal adjacency, then judges whether the deviation has spread in a cascading manner through the influence propagation relationship, separating and organizing the associated pronunciation deviation feature items formed by propagation from the independent pronunciation deviation feature items, and constructing a deviation attribution structure.

[0122] In summary, this invention, through the constraint of the target pronunciation temporal skeleton, competitively resolves boundary conflicts, boundary overlaps, and local misalignments in continuous children's pronunciation repetition, generating candidate pronunciation sequences for children. Then, it performs chain-like attribution and dominant screening on the pronunciation deviation feature items in the pronunciation parameter comparison structure, forming a deviation chain with causal hierarchical relationship and completing the reconstruction of correction parameters. This not only improves the accuracy of children's pronunciation localization and segmentation, but also enhances the pertinence, interpretability, and training effectiveness of children's foreign language pronunciation correction under speech recognition conditions.

[0123] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A children's foreign language pronunciation correction system based on multimodal learning, characterized in that, include: The skeleton construction module collects multimodal standard reference data and children's reading aloud data, and performs multimodal temporal collaborative analysis to construct the target pronunciation temporal skeleton; The window detection module, based on the target pronunciation temporal skeleton, normalizes and maps the data collected from children's repetition, obtains the expected window sequence of children's repetition, and performs cross-modal change detection to obtain a set of boundary candidate points; The pronunciation segment reconstruction module performs skeleton constraint competition reconstruction on the boundary candidate point set and the target pronunciation temporal skeleton to generate a child candidate pronunciation sequence; The alignment calibration module performs skeleton constraint alignment calibration based on the child's candidate pronunciation sequence and the target pronunciation temporal skeleton, and constructs a pronunciation parameter comparison structure. The attribution correction module uses a dominant deviation chain decoupling attribution algorithm to perform pronunciation deviation cause decomposition on the pronunciation parameter comparison structure and generate children's foreign language pronunciation correction information.

2. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 1, characterized in that, The multimodal standard reference data includes target text, standard pronunciation audio, and standard lip-syncing video; The data collected from children's reading aloud includes children's real-time reading aloud and videos of children's lip movements.

3. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 1, characterized in that, The specific steps for constructing the target articulation temporal skeleton are as follows: The target text is transcribed into phonetics to obtain the target pronunciation sequence. Then, the time sequence correspondence analysis is performed by combining standard pronunciation audio and standard lip-sync video to establish the audio-visual time sequence correspondence relationship. The target pronunciation sequence is collaboratively associated with the audiovisual temporal correspondence to construct the target pronunciation temporal skeleton.

4. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 3, characterized in that, The process of normalizing and mapping the children's follow-up reading data based on the target pronunciation temporal skeleton to obtain the expected window sequence of children's follow-up reading is as follows: Based on the target pronunciation temporal skeleton, the overall duration of the children's reading aloud data is scaled to form an initial mapping window sequence; Local pronunciation fluctuations are located in the data collected from children's reading aloud, forming a local fluctuation position sequence. Based on the local fluctuation position sequence, the initial mapping window sequence is calibrated to obtain the expected window sequence for children's reading aloud.

5. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 1 or 4, characterized in that, The specific steps for obtaining the set of candidate boundary points are as follows: Based on the expected window sequence of children's follow-up reading, speech change detection is performed on children's real-time follow-up reading speech to obtain speech boundary candidate points; Detect changes in lip movements in children's lip-reading videos and obtain candidate points for lip-reading boundaries; The temporal correspondence between speech boundary candidate points and lip shape boundary candidate points is extracted, and a collaborative screening algorithm is used to obtain a set of boundary candidate points.

6. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 5, characterized in that, The specific steps for reconstructing the child candidate pronunciation sequence by performing skeleton constraint competition on the boundary candidate point set and the target pronunciation temporal skeleton are as follows: Extract the target pronunciation sequence and temporal relationship from the target pronunciation temporal skeleton, perform sequential anchoring and pairing on the set of boundary candidate points, and generate candidate boundary pairs; A competitive resolution algorithm is used to calculate the interval assignment priority value, and based on the interval assignment priority value, the candidate boundary pairs are locked for the dominant interval and the adjacent intervals are reconstructed to determine the target pronunciation interval. Based on the target pronunciation range, the child's real-time pronunciation is continuously segmented to generate a candidate pronunciation sequence for the child.

7. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 6, characterized in that, The skeleton constraint alignment calibration based on the child's candidate pronunciation sequence and the target pronunciation temporal skeleton is performed in the following steps: The target pronunciation temporal skeleton is located and expanded to obtain the target pronunciation range; The candidate pronunciation sequences of children are expanded to obtain pronunciation localization sequences; The target articulation range and articulation localization sequence are subjected to fall area correction and cross area correction to form a calibration correlation group.

8. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 1, characterized in that, The pronunciation parameter comparison structure is constructed by performing misalignment correction on the calibration association group and using the interval adsorption recombination algorithm to calculate the interval adsorption intensity for adsorption recombination.

9. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 8, characterized in that, The dominant deviation chain decoupling attribution algorithm is used to perform a decomposition of the causes of pronunciation deviations on the pronunciation parameter comparison structure. The specific steps are as follows: The deviation expansion analysis of the pronunciation parameter comparison structure is performed to obtain pronunciation offset feature terms; Adjacency correlation analysis is performed on the pronunciation offset feature terms to establish temporal adjacency relationships, and conduction tests are performed on adjacent pronunciation offset feature terms to establish influence transmission relationships; Based on temporal adjacency and influence transmission, pronunciation deviation feature terms are chained together to form deviation chain candidate groups; Based on the candidate groups of deviation chains, the dominant screening value is calculated by the dominant deviation chain competitive screening algorithm, and the candidate groups of deviation chains are screened according to the dominant screening value to generate deviation chain attribution features.

10. The children's foreign language pronunciation correction system based on multimodal learning as described in claim 9, characterized in that, The specific steps for generating children's foreign language pronunciation correction information are as follows: Decoupling attribution processing is performed on the attribution features of the deviation chain to construct the deviation attribution structure, and back-projection reconstruction of the pronunciation parameters is performed to generate a set of correction parameters; The pronunciation correction trajectory reconstruction is performed on the correction parameter group to generate children's foreign language pronunciation correction information.

Images