A multi-modal fusion-based digital keyboard interaction system

Abstract

The application discloses a kind of based on multimodal fusion digital keyboard interaction system, it is related to digital keyboard interaction technical field, including backtracking mark module, influence propagation module, mapping determination module, mapping reconstruction module and dynamic control module: mapping determination module, based on the position mapping relationship of backtracking influence range, the position mapping relationship of digital keyboard interaction input in intermediate representation sequence is associated relationship analysis, by determining the structural association degree between digital keyboard interaction input position and backtracking influence range, whether the position mapping relationship of digital keyboard interaction input in intermediate representation sequence needs to be dynamically adjusted with backtracking change is judged.The application solves the problem that the position of digital keyboard interaction input is dislocated due to voice input backtracking update, realizes dynamic determination based on backtracking influence range and self-adapting adjustment of position mapping relationship, so as to ensure the consistency of multimodal fusion input in structure and semantics.

Description

Technical Field

[0001] This invention relates to the field of digital keyboard interaction technology, and more specifically to a multimodal fusion digital keyboard interaction system. Background Technology

[0002] Multimodal fusion-based digital keyboard interaction refers to the introduction of multiple input modalities such as voice, touch, gesture, vision, or sensors into the traditional digital keyboard input system. Through a unified data processing and fusion mechanism, user input information from different channels is collaboratively analyzed and decided upon, thereby achieving a more efficient, intelligent, and natural human-computer interaction method. Existing technologies are typically implemented in a system architecture. The overall process first includes a multi-source input acquisition stage, where user operation signals are acquired in real time through touchscreens, microphones, cameras, and various sensors. Next, a preprocessing and feature extraction stage is performed, standardizing the data from different modalities and extracting key feature information. Finally, in the fusion decision-making stage, through… This technology employs fixed rules or simple fusion strategies to synchronize, weight, and resolve conflicts of multimodal inputs to determine the final input intent. It then proceeds to the keyboard mapping and interaction execution stage, converting the fused result into specific key-value inputs or control commands, which are then applied to the digital keyboard interface. Finally, the interaction result is returned to the user through feedback mechanisms (such as visual cues, vibration, or voice feedback), thus forming a complete closed-loop interaction process. This technology typically encompasses multiple key stages, including input perception, data processing, modal fusion, intent recognition, keyboard response, and feedback optimization. Its core lies in improving input accuracy, reducing operational complexity, and enhancing the system's adaptability to complex scenarios through the collaborative utilization of multimodal information.

[0003] The existing technology has the following shortcomings: In multimodal fusion-based digital keyboard interaction, when voice input is mapped to a traceable intermediate representation structure that supports retrospective correction of existing content, and the digital keyboard interaction input has already been applied to a certain position in this intermediate representation structure, and the voice input subsequently updates the content before that position, the intermediate representation structure will be adjusted as a whole due to the context-based dynamic retrospective correction characteristic of the voice input during multimodal fusion. This causes a relative positional shift in the inserted digital keyboard interaction input. However, existing multimodal fusion-based digital keyboard interaction technologies cannot determine whether the positional mapping relationship of the digital keyboard interaction input in the intermediate representation needs to be dynamically adjusted with the retrospective change based on the retrospective impact range when the voice input is updated. Instead, they maintain its original position, resulting in a misalignment between the digital keyboard interaction input and the updated voice structure. This leads to inconsistencies in the structure and semantics of the fusion result, affecting input accuracy and interaction reliability.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion digital keyboard interaction system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion digital keyboard interaction system, comprising a backtracking marking module, an influence propagation module, a mapping determination module, a mapping reconstruction module, and a dynamic control module: The backtracking labeling module forms an intermediate representation sequence for the speech input and constructs a backtracking state labeling sequence based on the generation dependency relationship between semantic units. It determines whether the speech input has undergone backtracking update based on the label changes in the backtracking state labeling sequence. The influence propagation module, when it is determined that the voice input has undergone a backtracking update, takes the changed mark position in the backtracking state mark sequence as the starting point, establishes the backtracking influence propagation path along the dependency relationship of the intermediate representation sequence, and determines the backtracking influence range based on the backtracking influence propagation path and influence intensity characterization parameters. The mapping determination module analyzes the correlation between the position mapping of the numeric keypad interactive input in the intermediate representation sequence based on the backtracking influence range. By determining the degree of structural correlation between the position of the numeric keypad interactive input and the backtracking influence range, it determines whether the position mapping of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted with the backtracking changes. The mapping reconstruction module, when it is determined that dynamic adjustment is required, reconstructs the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence according to the scope of backtracking influence, and performs position rearrangement processing on the intermediate representation sequence. The dynamic control module continuously updates the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence based on the results of the reconstruction of the backtracking influence range and position mapping relationship.

[0007] Preferably, the implementation of the backtracking marker module is as follows: The speech input is parsed into semantic units arranged in chronological order, and then serialized and organized according to the generation order of the semantic units to form an intermediate representation sequence. Based on the generation dependency relationship between semantic units in the intermediate representation sequence, the dependency association between semantic units is established, and backtracking state tags are configured for each semantic unit according to the arrangement order of the intermediate representation sequence, forming a backtracking state tag sequence that corresponds one-to-one with the intermediate representation sequence. The sequence of backtracking state markers corresponding to different time points is compared position by position. When the backtracking state marker corresponding to the non-end position in the middle of the sequence changes, it is determined that the voice input has been updated backtracking.

[0008] Preferably, the implementation of the influence propagation module is as follows: When it is determined that the voice input has undergone a backtracking update, the starting position is determined based on the changed position of the marker in the backtracking state marker sequence, and the starting position is mapped to the corresponding semantic unit position in the intermediate representation sequence. Based on the semantic unit corresponding to the starting position, the dependencies of the intermediate representation sequence are expanded step by step. Semantic units that have a generation dependency relationship with the starting position are traversed in turn. The backtracking influence propagation path is established according to the connection order of the dependency relationship, and the dependency level of each semantic unit in the backtracking influence propagation path is recorded. Based on the established backtracking impact propagation path, the impact intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path, and the propagation path is bounded according to the impact intensity characterization parameters to determine the scope of backtracking impact.

[0009] Preferably, based on establishing the backtracking impact propagation path, the impact intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path, and the propagation path is boundary-limited according to the impact intensity characterization parameters to determine the backtracking impact range, specifically: Based on the established path of retrospective influence propagation, each semantic unit is hierarchically divided according to its dependency level relative to the starting point, and each dependency level is assigned a corresponding influence intensity characterization parameter value. Based on the dependency level corresponding to each semantic unit, the influence intensity characterization parameter is assigned a value that decreases step by step in the direction of increasing dependency level, forming a monotonically decreasing sequence of influence intensity characterization parameters along the propagation path, and establishing a one-to-one correspondence between the influence intensity characterization parameter values ​​and each semantic unit in the backtracking influence propagation path. The backtracking influence propagation path is bounded based on the influence intensity characterization parameter value. The backtracking influence propagation path is truncated according to the preset influence intensity characterization parameter truncation position. The semantic unit range before the truncation position corresponding to the influence intensity characterization parameter value is determined as the backtracking influence range.

[0010] Preferably, the mapping determination module is implemented as follows: Based on the backtracking influence range, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is analyzed. This includes determining the position index of the numeric keypad interactive input in the intermediate representation sequence, and judging whether the position of the numeric keypad interactive input is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range based on the position index, and establishing the association relationship between the position mapping relationship and the backtracking influence range. Based on the analysis of the correlation, the degree of structural correlation between the numeric keypad interactive input position and the backtracking influence range is determined by comparing the hierarchical distance between the numeric keypad interactive input position and the starting and ending positions of the backtracking influence range, and the degree of structural correlation is graded and labeled according to the hierarchical distance. Based on the degree of structural association, the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is determined. When the hierarchical identifier corresponding to the degree of structural association changes, it is determined that the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted as the backtracking changes.

[0011] Preferably, based on the position index, it is determined whether the numeric keypad interactive input position is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range, and an association relationship between the position mapping relationship and the backtracking influence range is established, specifically as follows: Based on the position index, obtain the position index value of the numeric keypad interactive input in the middle representation sequence, and obtain the starting position index value and ending position index value corresponding to the backtracking influence range; Based on the interval comparison between the position index value and the start position index value and the end position index value, when the position index value is between the start position index value and the end position index value, it is determined that the numeric keypad interactive input position is within the backtracking influence range, and the distance relationship between the position index value and the start position index value and the end position index value is further determined. Based on the distance relationship between the location index value and the boundary of the retrospective influence range, the boundary location of the digital keyboard interactive input position is classified, and the classification result is associated with the location mapping relationship to establish the association between the location mapping relationship and the retrospective influence range.

[0012] Preferably, based on the completion of the correlation analysis, the structural correlation between the numeric keypad input position and the backtracking influence range is determined by comparing the hierarchical distance between the numeric keypad input position and the start and end positions of the backtracking influence range. The structural correlation is then graded and labeled according to the hierarchical distance, specifically as follows: Based on the completed relationship parsing, the hierarchical position corresponding to the numeric keypad interactive input position is obtained, and the hierarchical position corresponding to the start position and the end position of the backtracking influence range are obtained respectively. The difference between the hierarchical position corresponding to the numeric keypad interactive input position and the hierarchical position corresponding to the start and end positions of the backtracking influence range is calculated to obtain the hierarchical distance between the numeric keypad interactive input position and the boundary of the backtracking influence range. The degree of structural association is classified and labeled according to the size range of the hierarchical distance. The range corresponding to different hierarchical distances is mapped to different levels of structural association, thus completing the determination of the degree of structural association between the digital keyboard interactive input position and the backtracking influence range.

[0013] Preferably, the implementation of the mapping reconstruction module is as follows: If dynamic adjustment is required, the position range of the backtracking influence range in the intermediate representation sequence is determined based on the backtracking influence range, and the position index corresponding to the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is obtained. At the same time, the target position index is determined based on the structural correlation between the numeric keypad interactive input position and the backtracking influence range. Based on the target position index, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is reconstructed, including removing the position mapping relationship between the numeric keypad interactive input and the original position index, and establishing the position mapping relationship between the numeric keypad interactive input and the target position index. Based on the reconstruction of the position mapping relationship, the intermediate representation sequence is rearranged. The semantic units in the intermediate representation sequence are arranged according to the target position index, and the position indexes of the adjusted semantic units are reassigned to ensure that the position order of the intermediate representation sequence is consistent with the reconstructed position mapping relationship.

[0014] Preferably, the dynamic control module is implemented as follows: Based on the changes in the backtracking influence range, the position interval change results of the backtracking influence range in the intermediate representation sequence are obtained, and the position mapping relationship change sequence corresponding to the position mapping relationship reconstruction result is obtained simultaneously; Based on the changes in the retrospective influence range and the reconstruction results of the position mapping relationship, the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence is continuously updated. This includes synchronously adjusting the position index corresponding to the position mapping relationship according to the changes in the retrospective influence range, and updating the position correspondence between the digital keyboard interactive input and the intermediate representation sequence according to the reconstruction results of the position mapping relationship. Based on the continuous updating of the position mapping relationship, consistency constraint processing is applied to the position mapping relationship in the intermediate representation sequence. The position index of the semantic unit in the intermediate representation sequence is continuously corrected according to the continuously updated position mapping relationship, so that the position mapping relationship remains continuous and consistent as the scope of backtracking changes.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention introduces a collaborative processing flow of backtracking markers, influence propagation, mapping determination, mapping reconstruction, and dynamic adjustment. When backtracking updates occur in voice input, it can accurately model and limit the affected semantic structure in the intermediate representation sequence, thereby achieving adaptive determination and adjustment of the mapping relationship between the digital keyboard interactive input position. Specifically, by constructing a backtracking state marker sequence and combining it with the dependency propagation path, the starting point of the backtracking update and its propagation range in the semantic structure can be accurately identified, avoiding misjudgment problems caused by coarse-grained processing based solely on local changes. Furthermore, by introducing structural correlation, the relationship between the digital keyboard interactive input position and the backtracking influence range is quantified, providing a clear basis for determining whether the position mapping relationship needs adjustment. This solves the problem in existing technologies where dynamic judgment based on the backtracking influence range is impossible, achieving structural coordination and consistency between voice input and digital keyboard input during multimodal fusion.

[0016] 2. This invention enables digital keyboard interactive input to undergo structural alignment adjustments during the voice backtracking update process through position mapping relationship reconstruction and synchronous rearrangement of intermediate representation sequences, avoiding position misalignment issues caused by voice content correction. Simultaneously, by dynamically controlling the changes in the backtracking influence range and the position mapping relationship reconstruction results, and combining consistency constraint processing to continuously correct the intermediate representation sequence, the stability and continuity of the position mapping relationship can be maintained in multiple backtracking update scenarios, preventing the accumulation of positional offsets and structural disorder. Therefore, this technical solution not only improves the structural and semantic consistency of the input results but also enhances the robustness and reliability of the system in complex interaction scenarios, thereby improving the accuracy and user experience of multimodal fusion digital keyboard interaction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a schematic diagram of a module of a multimodal fusion digital keyboard interaction system according to the present invention. Detailed Implementation

[0019] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0020] This invention provides, for example Figure 1 The illustrated multimodal fusion-based digital keyboard interaction system includes a backtracking marking module, an influence propagation module, a mapping determination module, a mapping reconstruction module, and a dynamic control module. The backtracking labeling module forms an intermediate representation sequence for the speech input and constructs a backtracking state labeling sequence based on the generation dependency relationship between semantic units. It determines whether the speech input has undergone backtracking update based on the label changes in the backtracking state labeling sequence. In this embodiment, the backtracking marker module is implemented as follows: The speech input is parsed into semantic units arranged in chronological order, and then serialized and organized according to the generation order of the semantic units to form an intermediate representation sequence. The process first performs continuous signal parsing on the speech input, dividing the original speech stream into several continuous speech segments along the time axis. Each speech segment is then converted into a corresponding text representation unit using a speech recognition model. Subsequently, based on semantic integrity, the text representation units are further subdivided to form semantic units with independent semantic meanings. On this basis, the semantic units are linearly arranged according to the generation time order of each semantic unit during the speech input process. Combined with the context prediction relationship during the speech recognition process, the generation sequence relationship between semantic units is determined. For example, when the speech content is "open the application and then input content," it can be divided into two semantic units: "open the application" and "input content," and organized according to their generation order. Then, each semantic unit is uniformly encoded according to both the time order and the generation order, giving each semantic unit a unique positional identifier in the sequence while maintaining the semantic association between preceding and following segments. This forms an intermediate representation sequence that reflects the speech generation process. This processing can transform the original speech input into structured sequence data, providing a stable foundation for subsequent retrospective update analysis.

[0021] Voice input refers to continuous speech signals collected through a microphone, which are essentially acoustic data that change continuously over time. A semantic unit is the smallest semantic expression unit extracted from the voice input. Each semantic unit corresponds to a text segment with complete semantics, used to express the user's intent. Sequencing organization refers to arranging discrete semantic units according to time order and generation logic, and establishing dependencies to form a data sequence with a sequential structure. The intermediate representation sequence is the result of structured processing of the voice input. This sequence not only contains the semantic units themselves, but also implicitly contains the generation order and relationships between semantic units. It is used to describe the generation process and evolution of the voice input. Through the intermediate representation sequence, changes in voice content can be tracked and located, thus providing a unified data foundation for subsequent backtracking, updating recognition, and multimodal fusion processing.

[0022] Based on the generation dependency relationship between semantic units in the intermediate representation sequence, the dependency association between semantic units is established, and backtracking state tags are configured for each semantic unit according to the arrangement order of the intermediate representation sequence, forming a backtracking state tag sequence that corresponds one-to-one with the intermediate representation sequence. The process first analyzes the generation relationships between semantic units based on the intermediate representation sequence. By analyzing the context prediction or decoding paths of semantic units in the speech recognition process, it determines whether each semantic unit depends on the generation of the previous semantic unit. For example, in continuous speech expression, the generation of subsequent semantic units is usually based on the contextual information of the preceding semantic units, thus establishing a dependency relationship between semantic units. After establishing the dependency relationship, a backtracking state marker is introduced for each semantic unit, and the marker value is configured one by one according to the arrangement order of the intermediate representation sequence. This marker value can use discrete state encoding to indicate whether the semantic unit participates in backtracking correction. For example, the initial state marker is unchanged. If a semantic unit is corrected during the speech recognition process, the corresponding marker changes. Furthermore, by traversing the intermediate representation sequence, the backtracking state markers of all semantic units are arranged in order to form a backtracking state marker sequence that corresponds one-to-one with the position of the intermediate representation sequence. For example, when the semantic unit at a certain intermediate position changes after the speech input is corrected, the marker at the corresponding position is updated. In this way, the modification trajectory of the speech input in the time dimension can be accurately recorded, providing a basis for subsequent backtracking update recognition.

[0023] Generative dependencies between semantic units refer to the situation in speech recognition or semantic parsing where the generation of a semantic unit depends on the contextual information provided by preceding semantic units. This dependency is manifested as a contextual structure in semantic generation. Dependency associations between semantic units are explicit expressions of these generative dependencies. By establishing associations, the logical connections between each semantic unit in the overall sequence can be clarified. Backtracking state markers are state identifiers attached to each semantic unit, used to characterize whether the semantic unit has been modified or participated in backtracking correction during speech processing. Its state is updated as the speech input changes. The backtracking state marker sequence is a sequence structure formed by arranging the backtracking state markers of all semantic units according to the order of the intermediate representation sequence. This sequence maintains a positional correspondence with the intermediate representation sequence and is used to describe the changes in speech input throughout the processing. Through this sequence, the specific location of the backtracking modification and its scope of influence can be located.

[0024] The sequence of backtracking state markers corresponding to different time points is compared position by position. When the backtracking state marker corresponding to the non-end position in the middle of the sequence changes, it is determined that the voice input has been updated backtracking.

[0025] This process is achieved by comparing the backtracking state marker sequences generated at consecutive time points position by position. Specifically, during speech input processing, backtracking state marker sequences corresponding to multiple time points are continuously recorded, and the marker values ​​at the same position at different time points are compared one by one according to the positional order in the intermediate representation sequence. In practice, a new backtracking state marker sequence can be generated after each speech recognition result update and aligned with the backtracking state marker sequence of the previous time point. Then, starting from the beginning of the sequence, the changes in marker values ​​at corresponding positions are compared one by one. When a change in the marker value at a certain position is detected, and that position is located within the internal region of the intermediate representation sequence, it can be determined that the change originates from a correction of existing semantic content rather than new content. For example, if a semantic sequence is first generated for the speech input, and then the semantic unit at the middle position is modified, the backtracking state marker corresponding to that position will change. This change can be identified by comparing position by position, thus determining that a backtracking update has occurred in the speech input. This processing can distinguish between new input and historical content correction, ensuring the accuracy of backtracking update recognition.

[0026] Position-by-position comparison refers to comparing the label values ​​at the same position in the backtracking state label sequence at different time points according to the arrangement order of semantic units in the intermediate representation sequence. This process ensures that the comparison results can accurately reflect the state changes at specific positions. The backtracking state label corresponding to non-terminal positions refers to the label value corresponding to any position in the intermediate representation sequence other than the end position. These positions usually correspond to semantic content that has been generated and may be subsequently modified, and their label changes have important judgment significance. Backtracking update of speech input refers to the behavior of modifying or replacing the generated semantic units during the speech input process. This behavior is different from simple sequence appending. Its characteristic is that the state of the position within the sequence changes. By detecting the label changes of non-terminal positions, the backtracking correction behavior in speech input can be accurately identified, thereby providing a reliable basis for subsequent multimodal fusion processing.

[0027] The influence propagation module, when it is determined that the voice input has undergone a backtracking update, takes the changed mark position in the backtracking state mark sequence as the starting point, establishes the backtracking influence propagation path along the dependency relationship of the intermediate representation sequence, and determines the backtracking influence range based on the backtracking influence propagation path and influence intensity characterization parameters. In this embodiment, the implementation of the influence propagation module is as follows: When it is determined that the voice input has undergone a backtracking update, the starting position is determined based on the changed position of the marker in the backtracking state marker sequence, and the starting position is mapped to the corresponding semantic unit position in the intermediate representation sequence. After determining that a backtracking update has occurred in the speech input, the positions where the markers in the backtracking state marker sequence have changed are located. By aligning the backtracking state marker sequences at consecutive time points, the changed marker positions are extracted into a change index set. Then, the earliest changed position is selected from the change index set as the starting position. After determining the starting position, based on the one-to-one correspondence between the backtracking state marker sequence and the intermediate representation sequence, the starting position is directly mapped to the corresponding index position in the intermediate representation sequence, thereby determining the starting semantic unit. For example, in the intermediate representation sequence formed by the speech input, if the backtracking state marker corresponding to the third semantic unit changes, the third position is determined as the starting position, and this position is mapped to the third semantic unit in the intermediate representation sequence. Through this mapping process based on position index consistency, the backtracking change can be accurately transmitted from the marker space to the semantic unit space, providing a clear starting node for establishing a propagation path along the dependency relationship.

[0028] The starting position refers to the index of the location in the backtracking state marker sequence where the marker first changes. This position is used to identify the starting point of the backtracking update, and its corresponding semantic unit is the source point for the subsequent propagation of influence. Mapping the starting position to the corresponding semantic unit position in the intermediate representation sequence means using the sequential correspondence between the backtracking state marker sequence and the intermediate representation sequence to directly convert the position index in the marker sequence into the semantic unit index in the intermediate representation sequence, so that the two sequences are consistent in the position dimension. This mapping relationship relies on the one-to-one correspondence structure between semantic units and backtracking state markers, so that each marker position can uniquely determine a semantic unit position, thereby achieving precise positioning from marker change to semantic content. This process ensures that the starting point of the backtracking update can be accurately identified at the semantic level, providing a foundation for the construction of the subsequent backtracking influence propagation path.

[0029] Based on the semantic unit corresponding to the starting position, the dependencies of the intermediate representation sequence are expanded step by step. Semantic units that have a generation dependency relationship with the starting position are traversed in turn. The backtracking influence propagation path is established according to the connection order of the dependency relationship, and the dependency level of each semantic unit in the backtracking influence propagation path is recorded. Starting with the semantic unit corresponding to the starting position as the initial node, the propagation sequence is expanded level by level according to the generation dependency relationship between semantic units. In specific implementation, a dependency connection table or dependency link set between semantic units can be constructed first, in which each semantic unit records the preceding or subsequent semantic units with which it has a generation relationship. Then, starting from the starting semantic unit, all directly dependent semantic units are traversed in sequence according to the dependency connection relationship and added to the propagation path. After completing the first level of traversal, the semantic units that have been added to the path continue to the next level of dependency traversal, expanding level by level until there are no more new dependent nodes, thus forming a complete backtracking influence propagation path. For example, in the semantic sequence "Enter account, enter password, click login", if "Enter password" undergoes a backtracking change, the semantic unit is used as the starting point, and the traversal along the dependency relationship is continued to "Click login", and the existence of subsequent dependencies is further determined, thus constructing a propagation path. During the traversal, each level of expanded semantic unit is assigned a level number, with the starting semantic unit as the initial level, the directly dependent semantic units as the next level, and so on, to characterize the propagation distance and influence depth.

[0030] Expanding step-by-step along the dependencies of the intermediate representation sequence means taking the starting semantic unit as the initial node and expanding the traversal range outward according to the generation dependencies between semantic units, following the hierarchical structure. This ensures that the propagation process follows semantic generation logic rather than a simple positional order. Sequential traversal means visiting each layer of semantic units sequentially according to the connection order of dependencies, ensuring that each semantic unit is visited only once, thus forming a complete and non-repeating propagation path. The backtracking influence propagation path refers to the set of semantic units connected according to dependencies starting from the starting semantic unit. This path is used to describe the propagation range of backtracking updates in the semantic structure. The dependency hierarchy of each semantic unit in the backtracking influence propagation path refers to the hierarchical distance of each semantic unit relative to the starting semantic unit. This is used to represent the positional level of each node during the propagation process. This hierarchical information can reflect the depth and scope of influence propagation and provide a basis for limiting the subsequent influence range.

[0031] Based on the established backtracking impact propagation path, the impact intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path, and the propagation path is bounded according to the impact intensity characterization parameters to determine the scope of backtracking impact.

[0032] Based on establishing the backtracking influence propagation path, influence strength characterization parameters are configured according to the dependency hierarchy corresponding to each semantic unit in the propagation path. The propagation path is then boundary-limited based on these influence strength characterization parameters to determine the backtracking influence range. This is because when speech input undergoes backtracking updates, its influence does not spread uniformly throughout the entire intermediate representation sequence, but rather exhibits a propagation characteristic that gradually weakens from near to far along semantic dependencies. If the range is determined solely based on the propagation path, it is impossible to distinguish the differences in the degree of influence of different semantic units, easily including weakly related or even unrelated semantic units within the influence range, thus introducing redundant interference. By introducing dependency hierarchy and configuring influence... The intensity characterization parameter can quantify the degree of influence in the propagation path in a hierarchical manner, so that semantic units closer to the starting position correspond to higher influence intensity characterization parameter values, while semantic units farther from the starting position correspond to lower values. Based on this, the propagation path is bounded according to the influence intensity characterization parameter values. This allows for precise structural extraction of the range of semantic units actually affected by the backtracking update. As a result, the backtracking influence range includes both the key affected areas and avoids irrelevant expansion. This processing provides a clear and quantifiable basis for subsequent position mapping relationship adjustments based on the backtracking influence range, ensuring that position adjustments during multimodal fusion only affect the truly affected areas.

[0033] In this embodiment, based on establishing the backtracking impact propagation path, the influence intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path, and the propagation path is boundary-limited according to the influence intensity characterization parameters to determine the backtracking impact range, specifically: Based on the established path of retrospective influence propagation, each semantic unit is hierarchically divided according to its dependency level relative to the starting point, and each dependency level is assigned a corresponding influence intensity characterization parameter value. Given that the propagation path of influence has been established, the semantic units in the propagation path are first grouped according to their dependency distance from the starting point, using the starting point as the benchmark. Semantic units that are directly dependent on the starting point are classified into the first level, semantic units that are indirectly dependent on the starting point through an intermediate semantic unit are classified into the second level, and so on, increasing the hierarchy. After the hierarchy is completed, a corresponding influence strength characterization parameter is assigned to each dependency level. This value can be preset according to the hierarchy distance; for example, the closer the hierarchy distance, the larger the value, and the farther the hierarchy distance, the smaller the value. This value is then distributed to all semantic units within that level. For example, if the starting semantic unit is "enter password", the directly related "point" would be... "Click to log in" can be classified as the first level and assigned a higher value, while subsequent semantic units with more distant connections are classified as higher levels and assigned lower values. By classifying semantic units according to their dependency levels and configuring the values ​​of the influence intensity characterization parameters, the distribution of the strength of the backtracking influence in the propagation path can be quantitatively expressed. Here, the hierarchical classification refers to organizing semantic units in layers according to the dependency distance between the semantic unit and the starting position, so that the propagation structure has a hierarchical nature. The influence intensity characterization parameter values ​​are numerical identifiers used to characterize the degree of influence of the backtracking update on semantic units at each level. By establishing a correspondence between the levels and the values, an orderly description of the influence intensity in the propagation path can be achieved, providing a clear basis for subsequent boundary constraints based on values.

[0034] Based on the dependency level corresponding to each semantic unit, the influence intensity characterization parameter is assigned a value that decreases step by step in the direction of increasing dependency level, forming a monotonically decreasing sequence of influence intensity characterization parameters along the propagation path, and establishing a one-to-one correspondence between the influence intensity characterization parameter values ​​and each semantic unit in the backtracking influence propagation path. Based on the obtained dependency levels for each semantic unit, the influence intensity characterization parameters are assigned values ​​in ascending order of dependency level. The starting level corresponds to the maximum influence intensity characterization parameter value. As the dependency level increases, the influence intensity characterization parameters for subsequent levels are assigned values ​​progressively lower according to a preset decreasing rule. The values ​​corresponding to each level are then distributed to the semantic units at the corresponding levels in the propagation path. For example, when the starting semantic unit is at the first level, its influence intensity characterization parameter value is set to the highest value; the semantic units directly dependent on it are at the second level and assigned a lower value; and the values ​​for further levels decrease accordingly. Through this progressively decreasing assignment process in the direction of increasing dependency level, a gradually weakening influence intensity characterization parameter is formed along the propagation path from the starting point outwards. The influence intensity characterization parameter sequence is used, and each value is matched one-to-one with the corresponding semantic unit to ensure that each semantic unit has a unique influence intensity characterization parameter value. Among them, the dependency hierarchy increasing direction refers to the direction in which the hierarchy number gradually increases as it expands outward from the starting semantic unit. The progressively decreasing value assignment refers to the distribution process of the influence intensity characterization parameter value decreasing layer by layer as the hierarchy increases. The monotonically decreasing influence intensity characterization parameter sequence along the propagation path refers to the sequence structure in which the values ​​corresponding to each semantic unit in the propagation path show a continuous decreasing relationship. The one-to-one correspondence means that each semantic unit corresponds to a unique influence intensity characterization parameter value. This relationship ensures that the influence intensity can be accurately mapped to the specific semantic location, providing a quantifiable basis for subsequent boundary constraints.

[0035] The backtracking influence propagation path is bounded based on the influence intensity characterization parameter value. The backtracking influence propagation path is truncated according to the preset influence intensity characterization parameter truncation position. The semantic unit range before the truncation position corresponding to the influence intensity characterization parameter value is determined as the backtracking influence range.

[0036] After obtaining the numerical sequence of influence intensity characterization parameters that monotonically decrease along the propagation path, the truncation position of the influence intensity characterization parameters is first determined according to a pre-set truncation rule. This truncation position can be achieved by sequentially scanning the numerical sequence of influence intensity characterization parameters. That is, starting from the starting semantic unit, the influence intensity characterization parameter values ​​of the corresponding semantic units are read one by one along the propagation path. When the value decreases to the sequence position corresponding to the preset truncation benchmark, this position is marked as the influence intensity characterization parameter truncation position. Then, using this truncation position as the boundary, the propagation path is truncated, retaining only the semantic units between the starting position and the truncation position, and discarding the semantic units after the truncation position. For example, in the propagation path, the starting semantic unit corresponds to the highest value, and the values ​​of subsequent semantic units gradually decrease. When the value at a certain position corresponds to the truncation benchmark, the values ​​after that position are removed. Semantic units are no longer included in the propagation range. In this way, the range of semantic units corresponding to the influence intensity characterization parameter value before the truncation position is determined as the retrospective influence range. Here, boundary limitation refers to restricting the propagation range by determining the effective termination position in the propagation path. The preset truncation position of the influence intensity characterization parameter is a termination judgment position predefined according to the trend of the change of the influence intensity characterization parameter value. Truncation processing refers to the operation process of segmenting, retaining and eliminating segments of the propagation path. The range of semantic units corresponding to the influence intensity characterization parameter value before the truncation position is a continuous interval composed of all semantic units from the starting point to the truncation position. The retrospective influence range is the set of semantic units actually affected during the retrospective update process. This range is determined by a numerically driven boundary limitation method, so that the retrospective influence has a clear range boundary in structure.

[0037] The mapping determination module analyzes the correlation between the position mapping of the numeric keypad interactive input in the intermediate representation sequence based on the backtracking influence range. By determining the degree of structural correlation between the position of the numeric keypad interactive input and the backtracking influence range, it determines whether the position mapping of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted with the backtracking changes. In this embodiment, the mapping determination module is implemented as follows: Based on the backtracking influence range, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is analyzed. This includes determining the position index of the numeric keypad interactive input in the intermediate representation sequence, and judging whether the position of the numeric keypad interactive input is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range based on the position index, and establishing the association relationship between the position mapping relationship and the backtracking influence range. This process, based on the backtracking influence range as a constraint, performs a refined analysis of the positional mapping relationship of numeric keypad input in the intermediate representation sequence. Specifically, it first records the sequence position corresponding to the numeric keypad input insertion, obtaining its position index in the intermediate representation sequence. This position index is then aligned with the start and end position indices corresponding to the backtracking influence range. Interval comparison is used to determine whether the position falls within the backtracking influence range. Furthermore, the relative positional relationship between the position index and the boundary of the backtracking influence range is calculated, such as being inside the range, near the start boundary, or near the end boundary. Based on this, the interval relationship between the position index and the backtracking influence range is mapped and encoded, forming an association between the positional mapping relationship and the backtracking influence range. For example, when the numeric keypad input position is located at... When tracing back to the area within the influence range, it is marked as an affected associated position; when it is outside the range, it is marked as a non-associated position. In this way, a direct link can be established between the positional state of the numeric keypad interactive input and the tracing influence range, thus providing a basis for subsequent structural association calculation. Among them, the position mapping relationship refers to the correspondence between the specific position of the numeric keypad interactive input in the intermediate representation sequence and the semantic unit, which is used to describe the structural position of the input in the sequence. Association relationship parsing refers to parsing the spatial relationship between the position mapping relationship and the tracing influence range and establishing mapping rules to form a computable association structure between the two. The position index refers to the sequence number used to identify the position of the numeric keypad interactive input in the intermediate representation sequence. Through the position index, the precise location of the input position and the determination of its relationship with the tracing influence range can be achieved.

[0038] Based on the analysis of the correlation, the degree of structural correlation between the numeric keypad interactive input position and the backtracking influence range is determined by comparing the hierarchical distance between the numeric keypad interactive input position and the starting and ending positions of the backtracking influence range, and the degree of structural correlation is graded and labeled according to the hierarchical distance. Based on the degree of structural association, the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is determined. When the hierarchical identifier corresponding to the degree of structural association changes, it is determined that the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted as the backtracking changes.

[0039] This process is achieved by monitoring and comparing the dynamic changes in the degree of structural correlation. Specifically, the degree of structural correlation corresponding to the numeric keypad input position is first recorded at different time points, and a corresponding hierarchical identifier is assigned to each degree of structural correlation. For example, high, medium, and low levels obtained based on hierarchical distance correspond to different hierarchical identifiers. Subsequently, after a retrospective update of the voice input, the degree of structural correlation corresponding to the numeric keypad input position is recalculated, and the current hierarchical identifier is compared with the hierarchical identifier before the update. When the hierarchical identifiers obtained from the two calculations are inconsistent, it indicates that the relative structural position of the numeric keypad input position within the retrospective influence range has changed. For example, a position that was originally located outside the boundary of the retrospective influence range may enter the retrospective influence range after the update. Within the influence range, the corresponding structural correlation level changes from low to medium or high. At this point, the positional mapping relationship of the numeric keypad input in the intermediate representation sequence needs to be dynamically adjusted. By judging based on the changes in the hierarchical identifier corresponding to the degree of structural correlation, the adjustment decision of the positional mapping relationship can be based on the objective basis of structural changes. Among them, the hierarchical identifier corresponding to the degree of structural correlation is a hierarchical representation of the strength of the relationship between the numeric keypad input position and the backtracking influence range. When the hierarchical identifier changes, it means that the relative position of the input position in the dependency structure has changed. Therefore, it is necessary to redetermine its positional mapping relationship in the intermediate representation sequence to ensure that the positional relationship and semantic structure remain consistent during the multimodal fusion process.

[0040] In this embodiment, the location index is used to determine whether the numeric keypad interactive input position is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range. A relationship is then established between the position mapping and the backtracking influence range: Based on the position index, obtain the position index value of the numeric keypad interactive input in the middle representation sequence, and obtain the starting position index value and ending position index value corresponding to the backtracking influence range; This process is achieved by extracting position indices and locating intervals in the intermediate representation sequence. Specifically, when an intermediate representation sequence is inserted via numeric keypad input, its corresponding sequence position number is recorded and stored as a position index value. Then, after determining the backtracking influence range, the semantic unit intervals within the backtracking influence range are identified. The sequence position number corresponding to the starting semantic unit of each interval is extracted as the starting position index value, and the sequence position number corresponding to the ending semantic unit of each interval is extracted as the ending position index value. For example, in the intermediate representation sequence, if the backtracking influence range covers the 2nd to 5th semantic units, the starting position index value is 2, and the ending position index value is 5. If the numeric keypad input is located at the 3rd position, then... Its position index value is 3. By obtaining the position index value and the starting and ending position index values ​​corresponding to the backtracking influence range, discrete semantic units can be mapped to continuous numerical intervals, thereby enabling subsequent interval-based comparison and relationship judgment. Among them, the position index value is a numerical number used to identify the specific position of the numeric keyboard interactive input in the intermediate representation sequence. The starting and ending position index values ​​corresponding to the backtracking influence range represent the starting and ending boundaries of the backtracking influence range in the intermediate representation sequence, respectively. Through these two boundary indices, a continuous interval can be constructed to describe the coverage of the backtracking influence range in the sequence, thus providing a basis for subsequent judgment of the relationship between the numeric keyboard interactive input position and the backtracking influence range.

[0041] Based on the interval comparison between the position index value and the start position index value and the end position index value, when the position index value is between the start position index value and the end position index value, it is determined that the numeric keypad interactive input position is within the backtracking influence range, and the distance relationship between the position index value and the start position index value and the end position index value is further determined. This process is implemented by performing interval determination and distance calculation on the position indices in the intermediate representation sequence. Specifically, it first compares the position index value corresponding to the numeric keypad input with the starting and ending position indices corresponding to the backtracking influence range. By determining whether the position index value falls within the interval formed by the starting and ending position indices, it is determined whether the numeric keypad input position falls within the backtracking influence range. For example, if the starting position index value is 2 and the ending position index value is 6, and the position index value is 4, then the position is determined to be within the backtracking influence range. After completing the interval determination, the relative distances between the position index value and the starting and ending position indices are further calculated to determine the position's location within the backtracking influence range. The specific positional relationship within the range, for example, when the position index value is 4, the distance from the starting position index value is 2, and the distance from the ending position index value is 2. Based on this, it can be determined that the position is located in the middle area of ​​the retrospective influence range. By combining this interval comparison and distance calculation, it is possible not only to determine whether the position is affected, but also to characterize its relative distribution position within the influence range. Among them, interval comparison refers to determining whether a position belongs to a certain range by judging the magnitude relationship between the position index value and the interval boundary index value. The distance relationship between the position index value and the starting and ending position index values ​​refers to the numerical interval from the position to the two ends of the interval, which is used to characterize the relative positional characteristics of the position within the retrospective influence range. This relationship provides a basis for the subsequent division of the degree of structural association.

[0042] Based on the distance relationship between the location index value and the boundary of the retrospective influence range, the boundary location of the digital keyboard interactive input position is classified, and the classification result is associated with the location mapping relationship to establish the association between the location mapping relationship and the retrospective influence range.

[0043] This process is achieved by classifying and mapping the distance relationship between the position index value and the boundary of the backtracking influence range. Specifically, the calculated distance relationship between the position index value and the starting and ending position index values ​​is used to subdivide the numeric keypad interactive input position within the interval. For example, when the position index value is closer to the starting position index value, the position is classified as the region near the starting boundary; when the position index value is closer to the ending position index value, it is classified as the region near the ending boundary; and when the position index value is close to both side boundaries, it is classified as the middle region. After classifying the boundary positions, corresponding classification labels are assigned to different categories, and these classification labels are bound to the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence, forming a position mapping relationship label with classification attributes. For example, when the position index value is 3 and the backtracking influence range is 2 to 6, the position can be marked as the region near the starting boundary based on the distance relationship. The process begins by categorizing boundary locations and associating these categories with the location mapping relationship. This distance-based boundary location classification and associative labeling extends the location mapping relationship from a single location description to a structural description that includes boundary attributes, thereby establishing a connection between the location mapping relationship and the retrospective influence range. Boundary location classification involves classifying locations based on the distance difference between the location index value and the boundary of the retrospective influence range, giving the location boundary attribute characteristics. Associative labeling involves attaching the classification results to the location mapping relationship in the form of identifiers to represent the specific relationship between the location and the retrospective influence range. The connection between the location mapping relationship and the retrospective influence range is established by using classification labels to establish a corresponding link between the location mapping relationship and the retrospective influence range, so that the location not only has sequential location attributes but also structural attributes relative to the retrospective influence range, providing a basis for subsequent determination of the degree of structural association.

[0044] In this embodiment, based on the completion of the association relationship analysis, the structural association degree between the numeric keypad interactive input position and the backtracking influence range is determined by comparing the hierarchical distance between the numeric keypad interactive input position and the start and end positions of the backtracking influence range. The structural association degree is then graded and labeled according to the hierarchical distance, specifically as follows: Based on the completed relationship parsing, the hierarchical position corresponding to the numeric keypad interactive input position is obtained, and the hierarchical position corresponding to the start position and the end position of the backtracking influence range are obtained respectively. In specific implementation, firstly, during the backtracking influence propagation path construction phase, dependency level numbers have already been assigned to each semantic unit. Based on this, according to the position index of the numeric keypad input in the intermediate representation sequence, the level number of the semantic unit corresponding to that position is found, and this number is used as the level position corresponding to the numeric keypad input position. Simultaneously, based on the interval boundaries of the backtracking influence range in the intermediate representation sequence, the corresponding semantic units are located using the start and end position indices, and their respective level numbers are read to obtain the level positions corresponding to the start and end positions of the backtracking influence range. For example, in the propagation path, if a semantic unit's level number is 1, its subsequent dependent unit is level 2, and then it expands outward to level 3, when the numeric keypad input is located at level 2, and the backtracking influence range covers… When covering levels 1 to 3, the input position level and range boundary level can be clearly defined. In this way, the spatial relationship based on sequence position can be transformed into a structural relationship based on dependency level, thus providing a unified hierarchical coordinate system for subsequent calculation of structural association degree based on hierarchical distance. Among them, the hierarchical position corresponding to the numeric keypad interactive input position refers to the dependency level number of the semantic unit corresponding to the input in the intermediate representation sequence, which is used to characterize its hierarchical position in the propagation structure. The hierarchical positions corresponding to the starting position and ending position of the backtracking influence range respectively represent the starting and ending levels of the backtracking influence range in the dependency hierarchical structure. These two levels together limit the range of the backtracking influence at the structural level, so that subsequent analysis no longer depends on simple positional order, but unfolds based on semantic dependency level.

[0045] The difference between the hierarchical position corresponding to the numeric keypad interactive input position and the hierarchical position corresponding to the start and end positions of the backtracking influence range is calculated to obtain the hierarchical distance between the numeric keypad interactive input position and the boundary of the backtracking influence range. In practice, the hierarchical position number corresponding to the numeric keypad input position is first obtained, along with the hierarchical position numbers corresponding to the start and end positions of the backtracking influence range. Then, the differences between the hierarchical position number corresponding to the numeric keypad input position and the hierarchical position number corresponding to the start and end positions are calculated to obtain the hierarchical distance values ​​in both directions. These two hierarchical distance values ​​represent the distance between the numeric keypad input position and the start and end boundaries of the backtracking influence range within the dependency hierarchy. For example, when the hierarchical position corresponding to the numeric keypad input position is the second level, the start position of the backtracking influence range corresponds to the first level, and the end position corresponds to the fourth level, the hierarchical distance relative to the start boundary can be obtained. The distance is one layer, and the hierarchical distance from the termination boundary is two layers, thus clarifying the relative position distribution of the input location within the entire influence range structure. This difference calculation method transforms the original position- or interval-based relationship into a distance expression based on hierarchical structure, making the structural proximity of different locations quantifiable and providing a basis for subsequent structural correlation classification. Specifically, difference calculation refers to numerically calculating the difference between two hierarchical position numbers to reflect the size of the hierarchical interval. The hierarchical distance between the numeric keypad interactive input location and the boundary of the backtracking influence range refers to the hierarchical interval between the input location and the starting or ending boundary in the dependency hierarchical structure. This hierarchical distance characterizes the relative positional features of the location in the backtracking influence propagation structure and serves as an important basis for subsequent classification determination.

[0046] The degree of structural association is classified and labeled according to the size range of the hierarchical distance. The range corresponding to different hierarchical distances is mapped to different levels of structural association, thus completing the determination of the degree of structural association between the digital keyboard interactive input position and the backtracking influence range.

[0047] This process is achieved by dividing the hierarchical distance into intervals and mapping it to different levels. Specifically, the calculated hierarchical distance between the numeric keypad input position and the boundary of the backtracking influence range is first divided into several continuous intervals. For example, smaller hierarchical distances are divided into close intervals, medium hierarchical distances into medium intervals, and larger hierarchical distances into far intervals. Then, a corresponding structural correlation level label is pre-defined for each interval, and a mapping relationship is established between the interval to which the hierarchical distance belongs and the corresponding level. For example, a high-level structural correlation is assigned when the hierarchical distance is in the close interval, a medium-level structural correlation is assigned when the hierarchical distance is in the medium interval, and a low-level structural correlation is assigned when the hierarchical distance is in the far interval. This process is achieved by dividing the hierarchical distance into intervals and mapping it to different levels. By matching distances with intervals, the structural correlation level corresponding to the numeric keypad input position can be determined. This process transforms continuously varying hierarchical distances into discrete hierarchical expressions, giving the structural correlation a clear hierarchical characteristic, thus facilitating subsequent unified processing based on the level. Here, hierarchical identification refers to the process of marking the structural correlation level corresponding to different intervals in the form of a level, so that each level distance corresponds to a clear level. The structural correlation level refers to the strength of the correlation between the numeric keypad input position and the backtracking influence range in the dependency hierarchy. Different levels of structural correlation level are used to distinguish the differences in the strength of the backtracking influence on the input position. By completing the determination of the structural correlation level through interval mapping, the correlation relationship can be transformed from a continuous numerical value into a discrete hierarchical structure that can be classified and processed.

[0048] The mapping reconstruction module, when it is determined that dynamic adjustment is required, reconstructs the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence according to the scope of backtracking influence, and performs position rearrangement processing on the intermediate representation sequence. In this embodiment, the mapping reconstruction module is implemented as follows: If dynamic adjustment is required, the position range of the backtracking influence range in the intermediate representation sequence is determined based on the backtracking influence range, and the position index corresponding to the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is obtained. At the same time, the target position index is determined based on the structural correlation between the numeric keypad interactive input position and the backtracking influence range. After determining that dynamic adjustment is needed, the location is first determined based on the position interval of the backtracking influence range in the intermediate representation sequence. Specifically, the starting and ending position indices corresponding to the backtracking influence range are obtained, and this interval is mapped to a continuous position range in the intermediate representation sequence. Then, the position index corresponding to the position mapping relationship of the numeric keyboard interactive input in the intermediate representation sequence is obtained, that is, the sequence position number corresponding to the current input is determined. On this basis, combined with the hierarchical results of the structural correlation degree, the numeric keyboard interactive input position is re-recalculated. For example, when the position is located in the middle of the backtracking influence range and the structural correlation degree is high, the target position index can be adjusted to a position closer to the center of the backtracking influence range. When the position is located near the boundary of the backtracking influence range and the structural correlation degree is low, the target position index can be adjusted to a position closer to the boundary. In this way, the position index is redistributed according to the structural correlation degree. For example, in the intermediate representation sequence, the backtracking influence range covers positions 2 to 6, and the original position of the numeric keyboard interactive input is position 5 with a high structural correlation degree. Then, the target position index can be adjusted to position 3 or 4, so that the input position is consistent with the semantic structure after backtracking.

[0049] Furthermore, when determining the target location index based on the structural correlation between the numeric keypad input position and the backtracking influence range, the target location index is adjusted according to the hierarchical identifier corresponding to the structural correlation. Specifically: when the structural correlation corresponds to a high level, the numeric keypad input position is determined to be located within the backtracking influence range and the hierarchical distance between it and the center area of ​​the backtracking influence range is less than a preset high-level hierarchical distance threshold. In this case, the target location index is determined to be the location index corresponding to the center position of the backtracking influence range. When the structural correlation corresponds to a medium level, the numeric keypad input position is determined to be located within the boundary area of ​​the backtracking influence range and the hierarchical distance between it and the center area of ​​the backtracking influence range is greater than the preset high-level hierarchical distance threshold and less than the preset medium-level hierarchical distance threshold. In this case, the target location index is determined to be the location index corresponding to the center position of the backtracking influence range. The intermediate position index between the position indices corresponding to the center position; when the structural correlation level corresponds to a low level, it is determined that the numeric keypad interactive input position is located outside the boundary of the backtracking influence range or the hierarchical distance between it and the center area of ​​the backtracking influence range is greater than the preset medium-level hierarchical distance threshold. In this case, the original position index remains unchanged; among them, the position index corresponding to the center position of the backtracking influence range is calculated and determined based on the starting position index value and the ending position index value corresponding to the backtracking influence range. Specifically, the starting position index value and the ending position index value are summed and divided by two to obtain the position index corresponding to the center position; the intermediate position index between the original position index and the position index corresponding to the center position of the backtracking influence range is calculated and determined based on the difference between the original position index and the position index corresponding to the center position. Specifically, the original position index and the position index corresponding to the center position are summed and divided by two to obtain the intermediate position index.

[0050] For example, in the intermediate representation sequence, if the backtracking influence range covers positions 2 to 6, then the starting position index value corresponding to the backtracking influence range is 2, and the ending position index value is 6. By summing the starting and ending position index values ​​and dividing by two, we obtain the position index corresponding to the center position of the backtracking influence range, which is the 4th position. When the original position of the numeric keypad input is the 5th position and the structural correlation level corresponds to a high level, the target position index is directly determined as the 4th position. When the original position of the numeric keypad input is the 5th position and the structural correlation level corresponds to a medium level, we sum the original position index 5 and the center position index 4 and divide by two to obtain the intermediate position index as the 4th position, and then determine the 4th position as the target position index. When the original position of the numeric keypad input is the 5th position and the structural correlation level corresponds to a low level, we keep the 5th position unchanged. This ensures that different structural correlation levels correspond to clear position mapping adjustment results and guarantees that the position mapping relationship of the numeric keypad input in the intermediate representation sequence is consistent with the semantic dependency structure after the backtracking update.

[0051] The position range of the backtracking influence in the intermediate representation sequence refers to the continuous position range defined by the start position index and the end position index, used to describe the coverage area of ​​the backtracking update in the sequence; the position index corresponding to the position mapping relationship of the numeric keyboard interactive input in the intermediate representation sequence refers to the specific position number of the input in the sequence, which reflects its correspondence with the semantic unit; the target position index refers to the sequence position number re-determined for the numeric keyboard interactive input during the dynamic adjustment process based on the degree of structural correlation, used to replace the original position index; the determination of the target position index depends on the combined effect of the backtracking influence range and the degree of structural correlation, so that the position adjustment is based not only on the spatial range, but also on the semantic dependency structure, thereby ensuring the consistency of the position mapping relationship in the intermediate representation sequence after the backtracking update.

[0052] Based on the target position index, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is reconstructed, including removing the position mapping relationship between the numeric keypad interactive input and the original position index, and establishing the position mapping relationship between the numeric keypad interactive input and the target position index. After determining the target position index, the position mapping relationship of the numeric keypad input in the intermediate representation sequence is reconstructed. Specifically, firstly, based on the original position index of the numeric keypad input, its original insertion position is located in the intermediate representation sequence, and the mapping relationship between this position and the corresponding semantic unit is removed, i.e., the numeric keypad input is removed from its original position index. Then, based on the target position index, a new insertion position is located in the intermediate representation sequence, and the numeric keypad input is re-inserted into this position, while updating its mapping relationship with the corresponding semantic unit. For example, when the original position index of the numeric keypad input is position 5, and the target position index is adjusted to position 3, the input is first removed from position 5 and then inserted into position 3, thus completing the reconstruction of the position mapping relationship. Through this "removal-relocation-rebinding" process, the position of the numeric keypad input in the intermediate representation sequence can be kept consistent with the semantic structure after backtracking and updating, avoiding position misalignment problems.

[0053] De-mapping the positional mapping between the numeric keypad input and its original position index means separating the numeric keypad input from the semantic unit position corresponding to its original position index, so that the position is no longer bound to the input. The original position index refers to the sequence position number initially corresponding to the numeric keypad input in the intermediate representation sequence, used to identify its initial insertion position. The positional mapping between the numeric keypad input and the target position index means re-binding the input to a new sequence position number during the reconstruction process, establishing a new positional correspondence between it and the semantic unit corresponding to the target position. By de-mapping the original mapping and establishing a new mapping, the positional mapping can be updated, ensuring that the position of the numeric keypad input in the intermediate representation sequence is consistent with the structure after the backtracking influence range adjustment, thereby guaranteeing the correctness and consistency of the positional relationships during multimodal fusion.

[0054] Based on the reconstruction of the position mapping relationship, the intermediate representation sequence is rearranged. The semantic units in the intermediate representation sequence are arranged according to the target position index, and the position indexes of the adjusted semantic units are reassigned to ensure that the position order of the intermediate representation sequence is consistent with the reconstructed position mapping relationship.

[0055] After reconstructing the position mapping relationship of the numeric keypad interactive input, the intermediate representation sequence is rearranged to ensure that the sequence structure is consistent with the new position mapping relationship. Specifically, the insertion position of the numeric keypad interactive input in the sequence is first determined based on the target position index, and the intermediate representation sequence is locally rearranged based on this position, that is, the positions of the target position index and its adjacent semantic units are adjusted. Then, the affected semantic units are reorganized according to the new arrangement order. For example, when the numeric keypad interactive input moves from the original 5th position to the 3rd position, the semantic units in the original 3rd to 4th positions need to be moved one position to the right to make room for the target position, while maintaining their original relative order. After the order adjustment is completed, the entire intermediate representation sequence is traversed once, and the position index is reassigned according to the new arrangement order so that each semantic unit and numeric keypad interactive input corresponds to the updated consecutive position number. In this way, the intermediate representation sequence can be kept consistent with the reconstructed position mapping relationship in structure, avoiding sequence chaos caused by position changes.

[0056] Position rearrangement refers to the process of adjusting the order of semantic units in the intermediate representation sequence according to the new position mapping relationship. This process involves not only changing a single position, but also coordinating the overall order of related semantic units to maintain the continuity and consistency of the sequence. Adjusting the order of semantic units in the intermediate representation sequence according to the target position index means moving related semantic units in the sequence forward or backward based on the target position index, so that the numeric keypad input can be accurately inserted into the target position. Reassigning position indices to the adjusted semantic units means assigning a new position number to each semantic unit according to the new arrangement order, so that the position index can truly reflect the current sequence structure. Through position rearrangement, the changes in the position mapping relationship can be synchronized to the intermediate representation sequence structure, keeping the sequence order, position index, and mapping relationship consistent, thereby providing a stable data foundation for subsequent multimodal fusion processing.

[0057] The dynamic control module continuously updates the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence based on the results of the reconstruction of the backtracking influence range and position mapping relationship.

[0058] In this embodiment, the dynamic control module is implemented as follows: Based on the changes in the backtracking influence range, the position interval change results of the backtracking influence range in the intermediate representation sequence are obtained, and the position mapping relationship change sequence corresponding to the position mapping relationship reconstruction result is obtained simultaneously; This process is achieved by synchronously collecting and serializing the changes in the retrospective influence range and position mapping relationship over continuous time. Specifically, during the continuous update of voice input, the update result of each retrospective influence range is recorded. The start and end position indices of the retrospective influence range in the intermediate representation sequence are continuously collected, and the interval boundaries of adjacent time points are compared to obtain the changes in the position interval. For example, when the retrospective influence range expands from position 2 to 5 to position 2 to 6, it can be recorded as a change in the end boundary. At the same time, after the position mapping relationship is reconstructed, the position mapping relationship corresponding to the numeric keypad interactive input is synchronously recorded. Each reconstructed position index change is serialized and stored to form a sequence of position mapping relationship changes that evolve over time. For example, when the numeric keypad interactive input is adjusted from position 5 to position 3, and then from position 3 to position 4, a continuous change sequence is formed. By synchronously acquiring the changes in the retrospective influence range and the changes in the position mapping relationship, a dynamic correspondence between the two can be established, providing input basis for subsequent continuous updates.

[0059] The position interval change result refers to the change of the coverage area of ​​the retrospective influence range in the intermediate representation sequence at different time points, specifically manifested as the change of the starting position index or the ending position index, used to describe the expansion, contraction, or movement of the retrospective influence range in the sequence; the position mapping relationship reconstruction result refers to the new position mapping relationship of the numeric keyboard interactive input in the intermediate representation sequence formed during the mapping reconstruction process, which is essentially the re-established position correspondence between the input and the semantic unit; the position mapping relationship change sequence refers to the sequence structure formed by arranging the changes of the position mapping relationship in a continuous time process in chronological order. This sequence is used to record the evolution trajectory of the numeric keyboard interactive input position in multiple reconstruction processes. By synchronously acquiring the position interval change result and the position mapping relationship change sequence, a dynamic correlation description between the changes in the retrospective influence range and the position adjustment behavior can be realized, thus providing a data foundation for the continuous updating of the position mapping relationship based on change-driven changes.

[0060] Based on the changes in the retrospective influence range and the reconstruction results of the position mapping relationship, the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence is continuously updated. This includes synchronously adjusting the position index corresponding to the position mapping relationship according to the changes in the retrospective influence range, and updating the position correspondence between the digital keyboard interactive input and the intermediate representation sequence according to the reconstruction results of the position mapping relationship. This process achieves continuous updates to the position mapping relationship by jointly driving changes in the retrospective influence range and the reconstruction results of the position mapping relationship. Specifically, it first monitors the changes in the retrospective influence range in the intermediate representation sequence over a continuous time period and obtains the position interval boundary change information corresponding to each change. Simultaneously, it combines the target position index changes recorded in the position mapping relationship reconstruction results to synchronously correct the current position index of the numeric keypad interactive input. That is, when the retrospective influence range expands or contracts, the position index is adjusted forward or backward according to the direction of change. Then, based on the latest position mapping relationship reconstruction results, the adjusted position index is re-established with the corresponding semantic unit to establish a position correspondence. For example, when the retrospective influence range expands from positions 2 to 5 to positions 2 to 7, and the numeric keypad interactive input is adjusted from position 4 to position 3 in the reconstruction results, the position index can be further corrected to position 4 according to the range expansion trend and rebound to the corresponding semantic unit, thereby achieving continuous updates with retrospective changes. By jointly driving the interval changes with the reconstruction results, it is ensured that the position mapping relationship remains consistent with the semantic structure during dynamic changes.

[0061] Continuous updating based on the changes in the backtracking influence range and the reconstruction results of the position mapping relationship refers to continuously correcting the position mapping relationship of the numeric keypad interactive input by using the interval changes of the backtracking influence range in the intermediate representation sequence as the driving factor, and combining the position adjustment results formed during the position mapping relationship reconstruction process. Synchronous adjustment of the position index corresponding to the position mapping relationship according to the changes in the backtracking influence range means that when the boundary of the backtracking influence range changes, the position index of the numeric keypad interactive input is adjusted in the same direction as the change to keep its relative position stable. Updating the position correspondence between the numeric keypad interactive input and the intermediate representation sequence based on the position mapping relationship reconstruction results means establishing a correspondence between the numeric keypad interactive input and the new semantic unit position based on the reconstructed target position index. Through this continuous updating process, the position mapping relationship can remain continuous and consistent during multiple backtracking changes, avoiding the problem of accumulated position offsets caused by multiple adjustments.

[0062] Based on the continuous updating of the position mapping relationship, consistency constraint processing is applied to the position mapping relationship in the intermediate representation sequence. The position index of the semantic unit in the intermediate representation sequence is continuously corrected according to the continuously updated position mapping relationship, so that the position mapping relationship remains continuous and consistent as the scope of backtracking changes.

[0063] After continuously updating the position mapping relationship, this process achieves consistency constraint processing by uniformly correcting the position indices and position mapping relationships in the intermediate representation sequence. Specifically, it first performs a comprehensive scan of the continuously updated numeric keypad interactive input position indices and the arrangement order of each semantic unit in the intermediate representation sequence to identify any discontinuous or conflicting indices, such as jumps or repetitions in the position indices of some semantic units due to multiple backtracking updates and position adjustments. Then, based on the current position mapping relationship, the semantic units in the intermediate representation sequence are renumbered according to their actual arrangement order. Taking the starting position of the sequence as a reference, new position indices are assigned incrementally, and the position indices corresponding to the numeric keypad interactive input are updated synchronously to ensure consistency with the new sequence numbering. For example, semantic units with original position indices of 3, 5, and 6 in the adjusted sequence are corrected to 1, 2, and 3 by renumbering, and the position indices of the numeric keypad interactive input are synchronously mapped to the corresponding new numbered positions, thereby eliminating discontinuities. Through this continuous correction process, the structural order and position mapping relationship of the intermediate representation sequence can be guaranteed to remain consistent.

[0064] Consistency constraint processing refers to applying unified rules to the correspondence between position mapping relationships and intermediate representation sequences, ensuring that the position indexes, semantic unit arrangement order, and positional relationships of numeric keypad interactive input remain consistent. This processing avoids index conflicts or structural misalignments by constraining the continuity and order of position indexes. Continuous correction of semantic unit position indexes in the intermediate representation sequence based on continuously updated position mapping relationships means readjusting the position numbers of semantic units in the sequence based on the currently valid position mapping relationships, ensuring that the position indexes continuously increase in the order of arrangement. Maintaining continuity and consistency in position mapping relationships as the scope of backtracking changes means ensuring that each position adjustment result matches the sequence structure during multiple backtracking updates, thereby avoiding the accumulation of position offsets or structural disorder, and guaranteeing the stability and consistency of positional relationships during multimodal fusion.

[0065] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0066] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0067] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0068] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal fusion digital keyboard interaction system, characterized in that, It includes a backtracking and marking module, an impact propagation module, a mapping determination module, a mapping reconstruction module, and a dynamic control module: The backtracking labeling module forms an intermediate representation sequence for the speech input and constructs a backtracking state labeling sequence based on the generation dependency relationship between semantic units. It determines whether the speech input has undergone backtracking update based on the label changes in the backtracking state labeling sequence. The influence propagation module, when it is determined that the voice input has undergone a backtracking update, takes the changed mark position in the backtracking state mark sequence as the starting point, establishes the backtracking influence propagation path along the dependency relationship of the intermediate representation sequence, and determines the backtracking influence range based on the backtracking influence propagation path and influence intensity characterization parameters. The mapping determination module analyzes the correlation between the position mapping of the numeric keypad interactive input in the intermediate representation sequence based on the backtracking influence range. By determining the degree of structural correlation between the position of the numeric keypad interactive input and the backtracking influence range, it determines whether the position mapping of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted with the backtracking changes. The mapping reconstruction module, when it is determined that dynamic adjustment is required, reconstructs the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence according to the scope of backtracking influence, and performs position rearrangement processing on the intermediate representation sequence. The dynamic control module continuously updates the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence based on the results of the reconstruction of the backtracking influence range and position mapping relationship.

2. The multimodal fusion digital keyboard interaction system according to claim 1, characterized in that, The implementation of the backtracking marker module is as follows: The speech input is parsed into semantic units arranged in chronological order, and then serialized and organized according to the generation order of the semantic units to form an intermediate representation sequence. Based on the generation dependency relationship between semantic units in the intermediate representation sequence, the dependency association between semantic units is established, and backtracking state tags are configured for each semantic unit according to the arrangement order of the intermediate representation sequence, forming a backtracking state tag sequence that corresponds one-to-one with the intermediate representation sequence. The sequence of backtracking state markers corresponding to different time points is compared position by position. When the backtracking state marker corresponding to the non-end position in the middle of the sequence changes, it is determined that the voice input has been updated backtracking.

3. The multimodal fusion digital keyboard interaction system according to claim 1, characterized in that, The specific implementation details of the propagation module are as follows: When it is determined that the voice input has undergone a backtracking update, the starting position is determined based on the changed position of the marker in the backtracking state marker sequence, and the starting position is mapped to the corresponding semantic unit position in the intermediate representation sequence. Based on the semantic unit corresponding to the starting position, the dependencies of the intermediate representation sequence are expanded step by step. Semantic units that have a generation dependency relationship with the starting position are traversed in turn. The backtracking influence propagation path is established according to the connection order of the dependency relationship, and the dependency level of each semantic unit in the backtracking influence propagation path is recorded. Based on the established backtracking impact propagation path, the impact intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path, and the propagation path is bounded according to the impact intensity characterization parameters to determine the scope of backtracking impact.

4. The multimodal fusion digital keyboard interaction system according to claim 3, characterized in that, Based on establishing the backtracking impact propagation path, influence intensity characterization parameters are configured according to the dependency level corresponding to each semantic unit in the propagation path. The propagation path is then boundary-limited based on these influence intensity characterization parameters to determine the scope of the backtracking impact. Specifically: Based on the established path of retrospective influence propagation, each semantic unit is hierarchically divided according to its dependency level relative to the starting point, and each dependency level is assigned a corresponding influence intensity characterization parameter value. Based on the dependency level corresponding to each semantic unit, the influence intensity characterization parameter is assigned a value that decreases step by step in the direction of increasing dependency level, forming a monotonically decreasing sequence of influence intensity characterization parameters along the propagation path, and establishing a one-to-one correspondence between the influence intensity characterization parameter values ​​and each semantic unit in the backtracking influence propagation path. The backtracking influence propagation path is bounded based on the influence intensity characterization parameter value. The backtracking influence propagation path is truncated according to the preset influence intensity characterization parameter truncation position. The semantic unit range before the truncation position corresponding to the influence intensity characterization parameter value is determined as the backtracking influence range.

5. The multimodal fusion digital keyboard interaction system according to claim 1, characterized in that, The mapping determination module is implemented as follows: Based on the backtracking influence range, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is analyzed. This includes determining the position index of the numeric keypad interactive input in the intermediate representation sequence, and judging whether the position of the numeric keypad interactive input is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range based on the position index, and establishing the association relationship between the position mapping relationship and the backtracking influence range. Based on the analysis of the correlation, the degree of structural correlation between the numeric keypad interactive input position and the backtracking influence range is determined by comparing the hierarchical distance between the numeric keypad interactive input position and the starting and ending positions of the backtracking influence range, and the degree of structural correlation is graded and labeled according to the hierarchical distance. Based on the degree of structural association, the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is determined. When the hierarchical identifier corresponding to the degree of structural association changes, it is determined that the positional mapping relationship of the numeric keypad interactive input in the intermediate representation sequence needs to be dynamically adjusted as the backtracking changes.

6. A multimodal fusion digital keyboard interaction system according to claim 5, characterized in that, Based on the position index, determine whether the numeric keypad input position is within the backtracking influence range and its positional relationship relative to the boundary of the backtracking influence range. Establish the correlation between the position mapping relationship and the backtracking influence range, specifically: Based on the position index, obtain the position index value of the numeric keypad interactive input in the middle representation sequence, and obtain the starting position index value and ending position index value corresponding to the backtracking influence range; Based on the interval comparison between the position index value and the start position index value and the end position index value, when the position index value is between the start position index value and the end position index value, it is determined that the numeric keypad interactive input position is within the backtracking influence range, and the distance relationship between the position index value and the start position index value and the end position index value is further determined. Based on the distance relationship between the location index value and the boundary of the retrospective influence range, the boundary location of the digital keyboard interactive input position is classified, and the classification result is associated with the location mapping relationship to establish the association between the location mapping relationship and the retrospective influence range.

7. A multimodal fusion digital keyboard interaction system according to claim 5, characterized in that, Based on the completed relational analysis, the structural correlation between the numeric keypad input position and the start and end positions of the backtracking influence range is determined by comparing the hierarchical distance between them. The degree of structural correlation is then graded and labeled according to the hierarchical distance, specifically as follows: Based on the completed relationship parsing, the hierarchical position corresponding to the numeric keypad interactive input position is obtained, and the hierarchical position corresponding to the start position and the end position of the backtracking influence range are obtained respectively. The difference between the hierarchical position corresponding to the numeric keypad interactive input position and the hierarchical position corresponding to the start and end positions of the backtracking influence range is calculated to obtain the hierarchical distance between the numeric keypad interactive input position and the boundary of the backtracking influence range. The degree of structural association is classified and labeled according to the size range of the hierarchical distance. The range corresponding to different hierarchical distances is mapped to different levels of structural association, thus completing the determination of the degree of structural association between the digital keyboard interactive input position and the backtracking influence range.

8. A multimodal fusion digital keyboard interaction system according to claim 1, characterized in that, The implementation of the mapping reconstruction module is as follows: If dynamic adjustment is required, the position range of the backtracking influence range in the intermediate representation sequence is determined based on the backtracking influence range, and the position index corresponding to the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is obtained. At the same time, the target position index is determined based on the structural correlation between the numeric keypad interactive input position and the backtracking influence range. Based on the target position index, the position mapping relationship of the numeric keypad interactive input in the intermediate representation sequence is reconstructed, including removing the position mapping relationship between the numeric keypad interactive input and the original position index, and establishing the position mapping relationship between the numeric keypad interactive input and the target position index. Based on the reconstruction of the position mapping relationship, the intermediate representation sequence is rearranged. The semantic units in the intermediate representation sequence are arranged according to the target position index, and the position indexes of the adjusted semantic units are reassigned to ensure that the position order of the intermediate representation sequence is consistent with the reconstructed position mapping relationship.

9. A multimodal fusion digital keyboard interaction system according to claim 1, characterized in that, The implementation of the dynamic control module is as follows: Based on the changes in the backtracking influence range, the position interval change results of the backtracking influence range in the intermediate representation sequence are obtained, and the position mapping relationship change sequence corresponding to the position mapping relationship reconstruction result is obtained simultaneously; Based on the changes in the retrospective influence range and the reconstruction results of the position mapping relationship, the position mapping relationship of the digital keyboard interactive input in the intermediate representation sequence is continuously updated. This includes synchronously adjusting the position index corresponding to the position mapping relationship according to the changes in the retrospective influence range, and updating the position correspondence between the digital keyboard interactive input and the intermediate representation sequence according to the reconstruction results of the position mapping relationship. Based on the continuous updating of the position mapping relationship, the position mapping relationship in the intermediate representation sequence is subjected to consistency constraint processing. The position index of the semantic unit in the intermediate representation sequence is continuously corrected according to the continuously updated position mapping relationship, so that the position mapping relationship remains continuous and consistent as the scope of backtracking changes.

Images