Multi-modal information fusion control system and method

Abstract

The invention relates to the technical field of information fusion, in particular to a multi-modal information fusion control system and method, and the system comprises a target difference evaluation module, a modal replacement judgment module, a modal lag recognition module, a modal weight reconstruction module and a control instruction output module. According to the method, dynamic adjustment and optimization of the control system are achieved by comprehensively evaluating the difference of the execution parameters in the control period, tiny changes in task execution can be reflected in detail by calculating the path coordinate difference value, the category variation frequency and the offset variation amplitude, and then the control strategy is accurately adjusted; the adaptive capacity and decision precision of the system to the environment change are improved, the control mode more suitable for the current task environment can be switched to in time by evaluating the output error and the matching score between the modes, and the resource allocation is optimized and the response speed and the processing efficiency are improved by identifying the abnormal response mode and reallocating the mode weight.

Description

Technical Field

[0001] The present invention relates to the technical field of information fusion, and in particular to a multimodal information fusion control system and method. Background Art

[0002] The field of information fusion encompasses the technical aspects of jointly processing and analyzing information from multiple sensory sources. It primarily studies how to collaboratively process and fuse heterogeneous multimodal information (such as images, speech, text, motion signals, and physiological parameters) from diverse sources. This technical field focuses on intermodal feature representation, temporal and spatial alignment mechanisms, intermodal weight distribution strategies, and fusion algorithms. These include, but are not limited to, deep learning-based feature-level fusion, decision-level fusion, attention mechanisms, cross-modal alignment networks, and graph neural networks. Its core technology lies in enhancing information complementarity, strengthening system robustness, and reducing modal redundancy, thereby enabling more efficient system responses and precise decision-making in scenarios such as intelligent control, situational recognition, and human-computer interaction.

[0003] A multimodal information fusion control system is a system designed for multi-source input control tasks. Its primary purpose is to use multiple modal data as input, perform collaborative analysis using a fusion algorithm, and then output control signals to drive or schedule execution units. This system can be applied to fields such as intelligent manufacturing, autonomous driving, and intelligent robotics, improving the control system's environmental perception capabilities, state recognition accuracy, and action decision-making efficiency.

[0004] Traditional control systems face the problem of insufficient synchronization and fusion between modalities when dealing with complex and dynamically changing environments. The use of fixed fusion strategies makes it difficult to cope with rapidly changing environmental conditions, resulting in the system's response being less flexible or accurate when handling real-time tasks. For example, in autonomous driving or intelligent manufacturing scenarios, a single or fixed-weight modal fusion strategy may not be able to promptly reflect rapid changes in the environment, such as emergencies or operational errors. This deficiency can lead to delayed or incorrect decision-making, affecting the safety and efficiency of the overall system. In addition, traditional systems lack an effective real-time evaluation and adjustment mechanism for the dynamic adjustment of modal weights, which limits the system's optimal performance in a changing environment and fails to fully utilize the complementary advantages of each modal data, affecting the overall performance and reliability of the system. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a multimodal information fusion control system and method.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a multimodal information fusion control system, the system comprising: The target difference evaluation module obtains the control execution parameters of the multimodal task in adjacent control cycles, calculates the path coordinate difference, category variation frequency and offset change amplitude, and uses the path, category and feedback difference thresholds for judgment to generate the task difference change value; The modality replacement judgment module compares the output error of the main control mode with the task difference change value to see whether it is greater than the error response threshold and whether the modality matching score is lower than the modality optimization critical value. If both conditions are met, the main control mode is marked as a pending switching state and a modality replacement judgment result is generated. The modal hysteresis identification module calculates the linkage ratio between the modal response time and the state change amplitude based on the modal replacement judgment result, determines whether the linkage ratio exceeds the modal disturbance judgment threshold, and obtains abnormal response modal identification information; The modal weight reconstruction module calls the feature participation ratio of the corresponding modality in the fusion feature generation based on the abnormal response modal identification information, calculates the product between the time interval deviation value and the feature participation ratio, sets a weight adjustment factor for the modality whose product result is greater than the time contribution threshold, and distributes new weights according to the adjustment factor to obtain the modal participation weight reconstruction value.

[0007] The present invention has the following improvements: the task difference change value is specifically a path offset judgment value, a category mutation state value and a feedback change amplitude value; the modal replacement judgment result includes a modal switching trigger mark, a main control mode degradation mark and an alternative mode preference mark; the abnormal response mode identification information is specifically a state disturbance index, a response delay level and a signal validity label; the modal participation weight reconstruction value includes a participation proportion adjustment coefficient, a feature contribution attenuation factor and a time penalty allocation value.

[0008] The present invention is improved in that the target difference evaluation module includes: The path deviation calculation submodule obtains the control execution parameters of the multimodal task in adjacent control cycles, extracts the coordinate vector values ​​of the path starting point and ending point coordinates, calculates the Euclidean distance difference between the coordinate vectors of the starting and ending points of the two cycles, and compares the absolute value difference between the difference result and the set path deviation threshold to generate a path deviation judgment value; The category mutation detection submodule calls the target category number in the two previous and next control cycles based on the path offset judgment value, extracts whether the category number state has switched, counts the number of category numbers that have switched and performs cycle normalization processing, and compares the result with the category state mutation frequency threshold to generate a category mutation state value; The offset amplitude calculation submodule calls the two-cycle feedback offset value based on the category mutation state value, obtains the maximum amplitude difference of the offset in the time series, and compares it with the feedback change amplitude threshold using the formula: ; Calculate the offset normalized amplitude value, combine it with the first three judgment values ​​and add them together to obtain the task difference change value; Where D represents the offset normalized amplitude value, P end Indicates the coordinate vector value of the end point of the execution path in the current control cycle, P start represents the coordinate vector value of the execution path starting point in the previous control cycle, C represents the category mutation state value, Fi represents the feedback offset value at the i-th time point, and F i-1 represents the feedback offset value at the i-1th time point, and n represents the total number of data points in the feedback offset sequence.

[0009] The present invention is improved in that the modality replacement judgment module includes: The error extraction submodule obtains the output response sequence of the main control modal signal and the alternative modal signal in each task segment of the current control cycle according to the task difference variation value, extracts the average error value of the corresponding feedback signal in each sequence, and records the signal time window position corresponding to the error value. The absolute value difference judgment is performed based on the error value and the corresponding error response threshold to obtain the modal error deviation value; The matching score judgment submodule calls the matching score value corresponding to the control target of each modal signal in the current control cycle according to the modal error deviation value, extracts the task relevance index, response stability coefficient and structural coupling consistency index of the score value, and constructs a matching composite evaluation expression using the formula: ; The composite score error value is obtained by calculation, and the modal score deviation state is determined based on the size of the composite score error value and the modal optimization critical value, thereby generating a modal score difference value; Among them, R represents the composite scoring error value, E m represents the task relevance index corresponding to the modal signal, S j A represents the response stability coefficient of the jth modal signal in the current control cycle, er Indicates the structural coupling consistency index between the modal signal and the current task, M er Indicates the score value of the current modal signal, T er represents the target expected mean of the modal score in the current control period, Cer represents the adjustment constant of the matching score evaluation, and K represents the total number of items in the stability coefficient sequence; The switching state marking submodule determines whether the main control modal signal satisfies both the error deviation value greater than the error response threshold and the score difference value less than the preferred critical value based on the modal score difference value. If both conditions are met, the main control modal signal is marked as a state to be switched, and a candidate list of modal candidates to be selected is established to obtain the modal replacement judgment result.

[0010] The present invention is improved in that the modality replacement judgment module includes: The timestamp extraction submodule uses the modal replacement determination result to obtain the response timestamp sequence of the image modality, voice modality, and motion modality in the current control cycle, performs signal statistical processing on each type of modal signal within a fixed time window, extracts the first response time and the end response time, and calculates the average response interval within the unit time window to generate the modal response interval value; The state variation calculation submodule calls the state change sequence of the three types of modes within the response time window according to the modal response interval value, and extracts the state change frequency, the number of overlapping change directions, and the change amplitude value respectively, using the formula: ; Calculate and obtain the modal linkage offset value; Where L represents the modal linkage offset value, W el Indicates the total duration of the statistical period. Represents the difference of the k-th state variation, R el Indicates the variation value of the modal response direction vector, D el Indicates the consistency index of modal change direction, F avg Represents the mean value of the change amplitude of the modal state value, T m represents the modal response interval value, and N represents the total number of state variation events; The linkage ratio judgment submodule compares the modal linkage offset value with the set modal disturbance judgment threshold. If it is greater than the judgment threshold, the corresponding mode is recorded as an unstable signal mode, and the response label is set to an abnormal state to obtain abnormal response mode identification information.

[0011] The present invention is improved in that the modal weight reconstruction module includes: The interval extraction submodule obtains the abnormal response modal identification information, extracts the first response time and feedback completion time of each type of modal signal based on the control feedback timestamp of each type of modal signal in the previous control cycle, calculates the action execution duration in the current cycle, and performs difference processing with the duration of the previous cycle to obtain the modal interval difference value; The contribution product calculation submodule calls the feature participation ratio data of each type of modality in the fusion feature generation process according to the modal interval difference value, using the formula: ; The modal product evaluation value is obtained by calculating the product evaluation value of the modal interval deviation and the characteristic participation ratio, and the modal product evaluation value is obtained; Among them, W ce represents the modal product evaluation value, T d Indicates the modal feedback end time of the current control cycle, T p Indicates the modal feedback start time of the previous control cycle, B zIndicates the single feature contribution value corresponding to the z-th mode, N ce Indicates the total number of modal types in which the feature participates, P r Indicates the proportion of mode participation in the feature generation process, V c Indicates the stability deviation of the control feedback process in the current cycle; The weight reset generation submodule judges the size relationship between each type of modal evaluation value and the time contribution threshold based on the modal product evaluation value, filters out the modes above the threshold and sets their weight adjustment factors, and replaces and updates the original modal weights according to the adjustment factor ratio to obtain the modal participation weight reconstruction value.

[0012] The present invention is improved in that the system further comprises: The control instruction output module calls the modal fusion output vector in the current cycle according to the modal participation weight reconstruction value, performs weighted combination on the fusion components of each modal dimension according to the updated weight, inputs the weighted fusion vector into the instruction mapping matrix, generates the fusion instruction coding set of the task scenario in the current cycle, and obtains the control instruction coding information of the current cycle; The current cycle control instruction encoding information specifically refers to the fusion vector mapping number, the task control instruction group code and the signal channel activation index.

[0013] The present invention is improved in that the control instruction output module includes: The weight mapping submodule obtains the modal participation weight reconstruction value, calls the modal fusion output vector in the current control cycle, performs a positioning operation on the fusion component of each modal dimension, assigns the corresponding modal participation weight to the fusion component, and forms a weighted result matrix to obtain the modal weighted combination value; The fusion coding construction submodule constructs the instruction output vector within the task cycle according to the modal weighted combination value, and combines all modal weighted component values ​​with the scene label matching index, structural information coupling factor, and instruction mapping template index using the formula: ; Calculate and obtain the fusion output vector value; Among them, F dp Represents the fusion output vector value, A x Represents the x-th dimension fusion component value, W x represents the participation weight of the x-th dimension fusion component, S x represents the structural coupling factor of the x-th mapping template, L x Indicates the x-th modal tag matching index, T x represents the target value of the x-th scene task label, M dp Indicates the number of dimensions of the fusion output vector; The task instruction generation submodule is based on the fusion output vector value, inputs it into the control instruction mapping matrix by column, performs matrix channel matching and index replacement operations according to the instruction path configuration rules, obtains the mapping result set of each task in the instruction channel, and establishes the current cycle control instruction encoding information.

[0014] A multimodal information fusion control method is implemented based on the multimodal information fusion control system, and includes the following steps: S1: Obtain the control execution parameters of the multimodal task in adjacent control cycles, calculate the path coordinate difference, category variation frequency and offset change amplitude, call the path, category and feedback three types of difference thresholds for judgment, and generate the task difference change value; S2: Based on the task difference change value, compare whether the output error of the main control mode is greater than the error response threshold, and whether the modality matching score is lower than the modality optimization critical value. If both conditions are met, mark the main control mode as a pending switching state and generate a modality replacement judgment result; S3: Based on the modal replacement determination result, calculating the linkage ratio between the modal response time and the state change amplitude, determining whether the linkage ratio exceeds the modal disturbance determination threshold, and obtaining abnormal response modal identification information; S4: Calculate the product of the time interval deviation value and the feature participation ratio based on the abnormal response modal identification information, set a weight adjustment factor for the modal whose product result is greater than the time contribution threshold, and perform new weight distribution according to the adjustment factor to obtain a modal participation weight reconstruction value; S5: According to the modal participation weight reconstruction value, the fusion components of each modal dimension are weightedly combined according to the updated weight, and the weighted fusion vector is input into the instruction mapping matrix to generate a fusion instruction coding set for the task scenario in the current cycle, and obtain the current cycle control instruction coding information.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, dynamic adjustment and optimization of the control system are achieved by comprehensively evaluating the execution parameter differences within the control cycle. By calculating the path coordinate difference, category variation frequency and offset change amplitude, it is possible to carefully reflect the slight changes in task execution, and then accurately adjust the control strategy, thereby improving the system's adaptability to environmental changes and decision-making accuracy. By evaluating the output error and matching score between modes, it is possible to switch to a control mode that is more suitable for the current task environment in a timely manner, reducing erroneous decisions caused by modal mismatch and enhancing the stability and reliability of the system. By identifying abnormal response modes and reallocating modal weights, resource allocation is optimized, reaction speed and processing efficiency are improved, and the performance of the multimodal information fusion control system in complex environments is improved, ensuring higher operating accuracy and response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a system flow chart of the present invention; Figure 2 A flow chart of a target difference evaluation module of the present invention; Figure 3 This is a flow chart of the modal replacement judgment module of the present invention; Figure 4 is a flow chart of the modal hysteresis identification module of the present invention; Figure 5 This is a flow chart of the modal weight reconstruction module of the present invention; Figure 6 This is a flow chart of the control instruction output module of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore should not be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0019] See also Figure 1 The present invention provides a technical solution: a multimodal information fusion control system, the system comprising: The target difference assessment module obtains the control execution parameters of the multimodal task in adjacent control cycles, including the execution path parameters, target classification number and feedback offset value, calculates the path coordinate difference, category variation frequency and offset change amplitude, and uses the path, category and feedback difference thresholds to make single judgments. The judgment results are accumulated to generate the task difference change value. The modal replacement judgment module calls the historical output error value and modal matching score corresponding to the main control modal signal and the alternative modal signal in the current control cycle according to the task difference change value, and compares whether the output error of the main control modal is greater than the error response threshold and whether the modal matching score is lower than the modal optimization critical value. If both conditions are met, the main control modal is marked as the waiting state and the modal replacement judgment result is generated; Based on the modal replacement judgment result, the modal hysteresis recognition module calls the response timestamp sequence of the image modality, voice modality, and motion modality received in the current cycle, extracts the state value change frequency, change direction overlap ratio, and change amplitude mean of each type of modal signal within a fixed time window, calculates the linkage ratio between the modal response time and the state change amplitude, and determines whether the linkage ratio exceeds the modal disturbance judgment threshold to obtain abnormal response modal recognition information; The modal weight reconstruction module uses the abnormal response modal identification information to call the control feedback timestamp corresponding to each modal signal in the previous control cycle, extract the first reaction time and feedback completion time, calculate the difference between the action execution duration interval and the time interval of the previous cycle, call the feature participation ratio of the corresponding modality in the fusion feature generation, calculate the product between the time interval deviation value and the feature participation ratio, set the weight adjustment factor for the modality whose product result is greater than the time contribution threshold, and distribute the new weight according to the adjustment factor to obtain the modal participation weight reconstruction value; The control instruction output module reconstructs the modal participation weight value, calls the modal fusion output vector in the current cycle, performs a weighted combination of the fusion components of each modal dimension according to the updated weight, inputs the weighted fusion vector into the instruction mapping matrix, generates the fusion instruction encoding set of the task scenario in the current cycle, and obtains the control instruction encoding information of the current cycle; The task difference change value specifically includes the path offset judgment value, the category mutation state value and the feedback change amplitude value. The modal replacement judgment result includes the modal switching trigger mark, the main control mode degradation mark and the alternative mode preference mark. The abnormal response mode identification information specifically includes the state disturbance index, the response delay level and the signal validity label. The modal participation weight reconstruction value includes the participation ratio adjustment coefficient, the feature contribution attenuation factor and the time penalty allocation value. The current cycle control instruction encoding information specifically refers to the fusion vector mapping number, the task control instruction group code and the signal channel activation index.

[0020] See also Figure 2 , the target difference assessment module includes: The path deviation calculation submodule obtains the control execution parameters of the multimodal task in adjacent control cycles, extracts the coordinate vector values ​​of the path starting point and ending point coordinates, calculates the Euclidean distance difference between the coordinate vectors of the starting and ending points of the two cycles, and compares the absolute value difference between the difference result and the set path deviation threshold to generate a path deviation judgment value; The path offset calculation submodule obtains the control execution parameters of the multimodal task in the adjacent control cycle, and reads the starting point coordinate vector and the ending point coordinate vector of the task path in the previous control cycle and the current control cycle in turn. In the example, the starting point coordinates of the previous cycle are (2, 3, 5), and the ending point coordinates are (8, 6, 10). The starting point coordinates of the current cycle are (2.5, 3.2, 5.1), and the ending point coordinates are (8.5, 6.1, 10.2). Comparing the starting and ending coordinates of the previous and next cycle paths, the Euclidean distance of the vectors is calculated respectively. The path length of the previous cycle is , the current cycle path length is The difference between the two cycle paths is , and the difference is compared with the path offset threshold of 0.05. The offset threshold is set based on the maximum allowable change of the path history of 5%. The maximum length of the historical path is set to 10 units, and the threshold is set to 5%×10=0.5. Since the unit normalization is set to 0.05, it is judged whether the difference of 0.026 exceeds the threshold. The result is that it does not exceed the threshold, so the path offset judgment value is 0.

[0021] The category mutation detection submodule calls the target category number in the two previous and next control cycles based on the path offset judgment value, extracts whether the category number status has switched, counts the number of category numbers that have switched and performs cycle normalization processing, and compares the result with the category status mutation frequency threshold to generate the category mutation status value; The category variation detection submodule compares the task classification numbers of the previous and next control cycles based on the path offset judgment value. The classification number of the previous cycle is read as C1 and the current cycle is C2. It determines whether there is a change. If the classification numbers are inconsistent, it is determined to be a switch and the number of switches is recorded. It is assumed that a total of 10 tasks are recorded in one sampling cycle, of which 3 tasks undergo classification switching, and the switching ratio is 3 / 10=0.3. The category state mutation frequency threshold is compared with 0.2. The mutation frequency threshold is set based on the maximum mutation rate allowed by the system. The upper limit of the mutation rate in the historical operation data is collected and set to 20%, that is, 0.2. The comparison result is 0.3>0.2, which determines that a significant category mutation has occurred in the current cycle and generates a category mutation state value of 1.

[0022] The offset amplitude calculation submodule calls the two-cycle feedback offset value based on the category mutation state value, obtains the maximum amplitude difference of the offset in the time series, and compares it with the feedback change amplitude threshold using the formula: ; The normalized offset amplitude value is obtained by calculation, and it is combined and summed with the first three judgment values ​​to obtain the task difference change value; Where D represents the offset normalized amplitude value, P end Indicates the coordinate vector value of the end point of the execution path in the current control cycle, P startrepresents the coordinate vector value of the execution path starting point in the previous control cycle, C represents the category mutation state value, Fi represents the feedback offset value at the i-th time point, and F i-1 represents the feedback offset value at the i-1th time point, and n represents the total number of data points in the feedback offset sequence; The offset amplitude calculation submodule extracts the two-cycle feedback offset data sequence based on the category mutation state value. The example feedback sequence is: the feedback sequence of the previous cycle is {1.0, 1.2, 1.1}, the feedback sequence of the current cycle is {1.3, 1.5, 1.6}, and the merged full sequence is {1.0, 1.2, 1.1, 1.3, 1.5, 1.6}. The maximum value is 1.6, the minimum value is 1.0, and the maximum amplitude difference is 1.6-1.0=0.6. The feedback change amplitude threshold is 0.5, which is set according to the maximum fluctuation amplitude allowed by the feedback control system. The maximum change in feedback in the historical data is 0.5, so it is set as the feedback change amplitude threshold. Since 0.6>0.5, the offset amplitude calculation is entered and substituted into the formula: ; Set the coordinate vector of the end point of the current cycle to (8.5, 6.1, 10.2), and the square sum is , the coordinates of the starting point of the previous cycle are (2,3,5), and the sum of squares is C=1,n=6, the sequence differences are 0.2, 0.1, 0.2, 0.2, 0.1, and the sum is 0.8. ; ; ; The offset normalization amplitude value D is approximately 97.5. Adding D to the path offset judgment value of 0 and the category mutation state value of 1, the task difference change degree is 97.5+0+1=98.5. This result indicates that the current task status has changed significantly compared with the previous cycle and dynamic adjustment is required.

[0023] See also Figure 3 , the modal replacement judgment module includes: The error extraction submodule obtains the output response sequence of the main control modal signal and the alternative modal signal in each task segment of the current control cycle according to the task difference variation value, extracts the average error value of the corresponding feedback signal in each sequence, and records the signal time window position corresponding to the error value. The absolute value difference between the error value and the corresponding error response threshold is judged to obtain the modal error deviation value; The error extraction submodule obtains the output response sequence of the main control modal signal and the alternative modal signal in each task segment in the current control cycle according to the task difference change value. For each task segment, the main control modal signal feedback sequence and the actual response sequence are extracted, and the difference between the two sequences at the corresponding time points in the task segment is calculated and averaged. In the example, the feedback sequence of task segment 1 is {1.2, 1.3, 1.4}, the actual response sequence is {1.1, 1.2, 1.3}, the calculated difference sequence is {0.1, 0.1, 0.1}, and the average error value is 0.1. The error value and the task segment are recorded. The corresponding relationship of time window 1, repeat the operation to obtain the error value of task segment 2 0.15, compare the error value with the set error response threshold, the threshold is set to 0.12, the threshold is set according to the task type and response tolerance range, the standard is 80% of the maximum allowable error range of the task, the maximum allowable error of the example task is 0.15, then the threshold is 0.12, perform absolute value difference judgment, judge 0.1 < 0.12, no deviation, 0.15 > 0.12, deviation, record the deviating task segment and deviation value 0.15, and finally obtain the modal error deviation value sequence {0.0, 0.03}.

[0024] The matching score judgment submodule calls the matching score value corresponding to the control target of each modal signal in the current control cycle according to the modal error deviation value, extracts the task relevance index, response stability coefficient and structural coupling consistency index of the score value, and constructs a matching composite evaluation expression using the formula: ; The composite score error value is obtained by calculation, and the modal score deviation state is determined based on the size of the composite score error value and the modal optimization critical value, thereby generating a modal score difference value; Among them, R represents the composite scoring error value, E m represents the task relevance index corresponding to the modal signal, S j A represents the response stability coefficient of the jth modal signal in the current control cycle, er Indicates the structural coupling consistency index between the modal signal and the current task, M er Indicates the score value of the current modal signal, T er represents the target expected mean of the modal score in the current control period, C er represents the adjustment constant for the matching score evaluation, K represents the total number of items in the stability coefficient sequence; The matching score judgment submodule reads the matching score values ​​of the main control and alternative modal signals in the current control cycle in turn according to the modal error deviation value. The score value includes the task relevance index E m , response stability coefficient sequence S j Coupling consistency index A erIn this example, the task relevance index of the master mode is 0.85, the response stability coefficient sequence is {0.9, 0.8, 0.85}, the coupling consistency index is 0.75, and the current mode score value is called , target expected mean , matching score adjustment constant ,The adjustment constant is set according to the system control sensitivity parameter. It is obtained by multiplying the upper limit of the system control error range by the adjustment factor 1.1. The setting range is 1.0 to 1.2. The example is 1.1. Substitute it into the formula: ; ; ; ; ; ; ; Composite scoring error value ,This value is compared with the modal preferred critical value of 5.000. The preferred critical value is set as the 95% percentile of the task history score, which is 5.000 in this example. Since R<5.000, it is determined that the score deviation exists and a modal score difference value of 1 is generated.

[0025] The switching state marking submodule determines whether the main control modal signal satisfies both the error deviation value greater than the error response threshold and the score difference value less than the optimal critical value based on the modal score difference value. If both conditions are met, the main control modal signal is marked as a state to be switched, and a candidate list of modal candidates is established to obtain the modal replacement judgment result. The switching state marking submodule is based on the modal score difference value. It first determines whether the main control modal signal meets two conditions. Condition one is that the error deviation value is greater than the error response threshold. In the example, the maximum deviation value is 0.15 and the threshold is 0.12, which meets condition one. Condition two is that the score difference value is less than the preferred critical value. In the example, it is 1<5.000, which also meets condition two. When both conditions are met, the main control mode is marked as the state to be switched. Then, the current alternative modal signal list is obtained, and the alternative modes with a score greater than 5.000 are screened. In the example, the alternative mode B1 scores 5.5 and B2 scores 4.8. Only B1 meets the conditions, and a candidate list {B1} is established. Finally, the modal replacement judgment result is generated. The current main control mode needs to enter the switching process, and the alternative mode is B1.

[0026] See also Figure 4 , the modal replacement judgment module includes: The timestamp extraction submodule determines the modal replacement result and obtains the response timestamp sequence of the image mode, voice mode, and motion mode in the current control cycle. It performs signal statistical processing on each type of modal signal within a fixed time window, extracts the first response time and the end response time, and calculates the average response interval within the unit time window to generate the modal response interval value. The timestamp extraction submodule operates on the modal replacement judgment results and obtains the response timestamp sequences of the image modality, speech modality, and motion modality within the current control cycle. For each modal signal, signal statistical processing is performed within a fixed time window. For example, the response timestamps of the image modality within the fixed time window are {100ms, 150ms, 200ms}. The first response time of 100ms and the last response time of 200ms of this sequence are extracted, and the average response interval within the unit time window is calculated as (200ms-100ms) / 2=50ms. This process is repeated for the speech and motion modalities. The first and last response times of the speech modality range from 120ms to 220ms, with an interval of 50ms. The first and last response times of the motion modality range from 130ms to 230ms, with an interval of 50ms. The final response interval values ​​generated for each modality are 50ms, 50ms, and 50ms, respectively. These values ​​provide a quantitative indicator for evaluating the time sensitivity and response efficiency of each modality.

[0027] The state variation calculation submodule calls the state change sequence of the three types of modes within the response time window according to the modal response interval value, and extracts the state change frequency, the number of coincidences in the change direction, and the change amplitude value respectively, using the formula: ; ; set up ; ; The linkage offset value of the image modality obtained by the operation is 0.226. This calculation process is repeated and similar operations are performed on the voice and action modalities.

[0028] The linkage ratio judgment submodule compares the modal linkage offset value with the set modal disturbance judgment threshold. If it is greater than the judgment threshold, the corresponding mode is recorded as an unstable signal mode, and the response label is set to an abnormal state to obtain abnormal response mode identification information; The linkage ratio judgment submodule compares the modal linkage offset value with the set modal disturbance judgment threshold. For example, if the threshold is set to 0.2 and the modal linkage offset value of 0.226 is greater than 0.2, the image mode is recorded as an unstable signal mode and the response label is set to an abnormal state, thereby obtaining abnormal response modal identification information. This judgment sets the threshold based on the actual dynamic characteristics of the modal response and the expected stability requirements to ensure the system's response accuracy and stable operation. The comparison result of the modal linkage offset value directly affects the modal application decision and the subsequent adjustment of the control strategy.

[0029] See also Figure 5 , the modal weight reconstruction module includes: The interval extraction submodule obtains abnormal response modal identification information, extracts the first response time and feedback completion time of each type of modal signal based on the control feedback timestamp of each type of modal signal in the previous control cycle, calculates the action execution duration in the current cycle, and performs difference processing with the duration of the previous cycle to obtain the modal interval difference value; After obtaining abnormal response modal identification information, the interval extraction submodule sequentially obtains the control feedback timestamps of the image modality, voice modality, and motion modality in the previous control cycle. It then extracts the first response time and feedback completion time of each modality in the previous cycle. For example, the first response time of the image modality in the previous cycle was 120ms, the feedback completion time was 320ms, and the action execution duration was 320ms-120ms=200ms. In the current control cycle, the image modality response time is 130ms, the feedback completion time is 350ms, and the duration is 220ms. The difference between the current cycle and the previous cycle is calculated as 220ms-200ms=20ms. This difference is used as the image modality interval difference value. Similarly, the difference values ​​for the voice modality and motion modality are calculated to be 15ms and 25ms, respectively. This operation quantifies the fluctuation of the modal execution time and provides input for subsequent evaluation.

[0030] The contribution product calculation submodule calls the feature participation ratio data of each type of modality in the fusion feature generation process according to the modal interval difference value, using the formula: ; The modal product evaluation value is obtained by calculating the product evaluation value of the modal interval deviation and the characteristic participation ratio, and the modal product evaluation value is obtained; Among them, W ce represents the modal product evaluation value, T d Indicates the modal feedback end time of the current control cycle, T p Indicates the modal feedback start time of the previous control cycle, Bz represents the single feature contribution value corresponding to the z-th mode, N ce Indicates the total number of modal types in which the feature participates, P r Indicates the proportion of mode participation in the feature generation process, V c Indicates the stability deviation of the control feedback process in the current cycle;

[0031] The contribution product calculation submodule calls the feature participation ratio data in the fusion feature generation process according to the modal interval difference value, and sets the current feedback end time of the image modality , the start time of the previous cycle feedback , the sequence of single feature contribution values ​​is {0.3, 0.2}, the total number of modal types , feature participation ratio , stability offset , substitute into the formula: Using the formula: ; ; ; The product evaluation value of the image modality is 132251.25. The voice modality and motion modality are calculated according to the same logic, and the example results are 122000.0 and 138000.5 respectively. The modal product evaluation value reflects the combined impact of the time interval deviation and feature contribution of each modality.

[0032] The weight resetting submodule determines the relationship between each modal evaluation value and the time contribution threshold based on the modal product evaluation value, selects the modes above the threshold and sets their weight adjustment factors. The original modal weights are replaced and updated according to the adjustment factor ratio to obtain the modal participation weight reconstruction value. The weight reset generation submodule compares the product evaluation value of each modality with the time contribution threshold and sets the time contribution threshold to 130000.0. The threshold setting is based on the minimum contribution value required for the fusion feature generation process multiplied by the maximum stable offset tolerance required by the periodic performance. The minimum contribution value is set to 100000.0 and the stable offset tolerance is 1.3, so the threshold is 130000.0. The image modality evaluation value is 132251.25> the threshold, which meets the adjustment conditions. The speech modality is 122000. .0<threshold, not satisfied, action mode 138000.5>threshold, satisfied the conditions, set the weight adjustment factor 1.1 for the image mode, set the factor 1.2 for the action mode, the original weights are 0.3 for image and 0.4 for action, the adjusted image weight is 0.3×1.1=0.33, the action mode weight is 0.4×1.2=0.48, perform the replacement update operation, and obtain the new modality participation weight reconstruction value image 0.33, speech remains unchanged at 0.3, and action is 0.48.

[0033] See also Figure 6 , the control instruction output module includes: The weight mapping submodule obtains the modal participation weight reconstruction value, calls the modal fusion output vector in the current control cycle, performs positioning operations on the fusion components of each modal dimension, assigns the corresponding modal participation weight to the fusion component, and forms a weighted result matrix to obtain the modal weighted combination value; After the weight mapping submodule obtains the modal participation weight reconstruction value, it first calls the fusion output vector generated by the image modality, speech modality and action modality in the current control cycle, and locates the components according to the modal dimension. For example, the fusion output vector is a three-dimensional vector , the corresponding image modality is the first dimension, the speech modality is the second dimension, and the action modality is the third dimension. Combined with the reconstructed modality participation weights, the image modality is 0.33, the speech modality is 0.30, and the action modality is 0.48. The mapping operation between the components and the corresponding weights is performed in sequence, and each dimension fusion component is multiplied by its weight coefficient. The weighted component of the image modality is , the speech mode is , the action mode is , combine the weighted components into a matrix , and record it in the modal weighted result matrix, combined with the control task identifier, to complete the generation of the modal weighted combination value of the corresponding control cycle.

[0034] The fusion coding construction submodule constructs the instruction output vector within the task cycle according to the modal weighted combination value, and combines all modal weighted component values ​​with the scene label matching index, structural information coupling factor, and instruction mapping template index using the formula: ; Calculate and obtain the fusion output vector value; Among them, F dp Represents the fusion output vector value, A x Represents the x-th dimension fusion component value, W x represents the participation weight of the x-th dimension fusion component, S x represents the structural coupling factor of the x-th mapping template, L x Indicates the x-th modal tag matching index, T x represents the target value of the x-th scene task label, M dp Indicates the number of dimensions of the fusion output vector; The fusion coding construction submodule performs fusion processing on the modal weighted combination value by dimension within the task cycle. Let the first dimension weighted component of the image modality be , weight , structural coupling factor , modal matching index , task label target value , substitute into the formula: ; ; ; ; ; ; ; ; The fused output vector value is 0.34, which is used for subsequent instruction generation.

[0035] The task instruction generation submodule inputs the fused output vector value into the control instruction mapping matrix column by column, performs matrix channel matching and index replacement operations according to the instruction path configuration rules, obtains the mapping result set of each task in the instruction channel, and establishes the control instruction encoding information of the current cycle; The task instruction generation submodule inputs the fusion output vector value 0.34 as the instruction parameter of the current cycle into the control instruction mapping matrix. Multiple preset instruction path mapping channels are arranged in columns in the matrix, and the matrix channel matching process is performed. Suppose the mapping matrix index segment is 0.30 to 0.40, then 0.34 is located in this interval, and the matching index number is channel 5. The instruction path replacement operation is performed, and the task control instruction template under index channel 5 is loaded. The output coding content is task identifier A001, the corresponding instruction is "MOVE-35", and it is matched with the task cycle identifier to complete the generation of control instruction coding information within the control cycle.

[0036] Table 1 Modal dimension mapping and fusion operation example table

[0037] Table 1 lists the participating terms and calculation results of each modal component in the fusion operation, combined with the fusion code construction steps. The results show that the mapping index corresponding to the fusion output vector value 0.34 matches channel 5, and thus derives that the task instruction code for the current control cycle is "MOVE-35".

[0038] A multimodal information fusion control method comprises the following steps: S1: Obtain the control execution parameters of the multimodal task in adjacent control cycles, calculate the path coordinate difference, category variation frequency and offset change amplitude, call the path, category and feedback three types of difference thresholds for judgment, and generate the task difference change value; S2: Based on the task difference change value, compare whether the output error of the main control mode is greater than the error response threshold, and whether the modality matching score is lower than the modality optimization critical value. If both conditions are met, mark the main control mode as the waiting state and generate the modality replacement judgment result; S3: Based on the modal replacement judgment result, the linkage ratio between the modal response time and the state change amplitude is calculated, and it is determined whether the linkage ratio exceeds the modal disturbance judgment threshold to obtain abnormal response modal identification information; S4: Based on the abnormal response modal identification information, the feature participation ratio of the corresponding modality in the fusion feature generation is called, the product between the time interval deviation value and the feature participation ratio is calculated, and a weight adjustment factor is set for the modality whose product result is greater than the time contribution threshold. A new weight is distributed according to the adjustment factor to obtain the modal participation weight reconstruction value; S5: Based on the modal participation weight reconstruction value, the fusion components of each modal dimension are weighted and combined according to the updated weight, and the weighted fusion vector is input into the instruction mapping matrix to generate the fusion instruction encoding set of the task scenario in the current cycle, and obtain the control instruction encoding information of the current cycle.

[0039] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multimodal information fusion control system, characterized in that: The system comprises: The target difference evaluation module obtains the control execution parameters of the multimodal task in adjacent control cycles, calculates the path coordinate difference, category variation frequency and offset change amplitude, and uses the path, category and feedback difference thresholds for judgment to generate the task difference change value; The modality replacement judgment module compares the output error of the main control mode with the task difference change value to see whether it is greater than the error response threshold and whether the modality matching score is lower than the modality optimization critical value. If both conditions are met, the main control mode is marked as a pending switching state and a modality replacement judgment result is generated. The modal hysteresis identification module calculates the linkage ratio between the modal response time and the state change amplitude based on the modal replacement judgment result, determines whether the linkage ratio exceeds the modal disturbance judgment threshold, and obtains abnormal response modal identification information; The modal weight reconstruction module calls the feature participation ratio of the corresponding modality in the fusion feature generation based on the abnormal response modal identification information, calculates the product between the time interval deviation value and the feature participation ratio, sets a weight adjustment factor for the modality whose product result is greater than the time contribution threshold, and distributes new weights according to the adjustment factor to obtain the modal participation weight reconstruction value.

2. The multimodal information fusion control system according to claim 1, characterized in that: The task difference change value is specifically the path deviation judgment value, the category mutation state value and the feedback change amplitude value; the modal replacement judgment result includes the modal switching trigger mark, the main control mode degradation mark and the alternative mode preference mark; the abnormal response mode identification information is specifically the state disturbance index, the response delay level and the signal validity label; the modal participation weight reconstruction value includes the participation proportion adjustment coefficient, the feature contribution attenuation factor and the time penalty allocation value.

3. The multimodal information fusion control system according to claim 1, characterized in that: The target difference assessment module includes: The path deviation calculation submodule obtains the control execution parameters of the multimodal task in adjacent control cycles, extracts the coordinate vector values ​​of the path starting point and ending point coordinates, calculates the Euclidean distance difference between the coordinate vectors of the starting and ending points of the two cycles, and compares the absolute value difference between the difference result and the set path deviation threshold to generate a path deviation judgment value; The category mutation detection submodule calls the target category number in the two previous and next control cycles based on the path offset judgment value, extracts whether the category number state has switched, counts the number of category numbers that have switched and performs cycle normalization processing, and compares the result with the category state mutation frequency threshold to generate a category mutation state value; The offset amplitude calculation submodule calls the two-cycle feedback offset value based on the category mutation state value, obtains the maximum amplitude difference of the offset in the time series, and compares it with the feedback change amplitude threshold using the formula: ; The normalized offset amplitude value is obtained by calculation, and it is combined and summed with the first three judgment values ​​to obtain the task difference change value; Where D represents the offset normalized amplitude value, P end Indicates the coordinate vector value of the end point of the execution path in the current control cycle, P start represents the coordinate vector value of the execution path starting point in the previous control cycle, C represents the category mutation state value, F i Indicates the feedback offset value at the i-th time point, F i-1 Indicates the i-1 The feedback offset value at each time point, n represents the total number of data points in the feedback offset sequence.

4. The multimodal information fusion control system according to claim 1, characterized in that: The modality replacement judgment module includes: The error extraction submodule obtains the output response sequence of the main control modal signal and the alternative modal signal in each task segment of the current control cycle according to the task difference variation value, extracts the average error value of the corresponding feedback signal in each sequence, and records the signal time window position corresponding to the error value. The absolute value difference judgment is performed based on the error value and the corresponding error response threshold to obtain the modal error deviation value; The matching score judgment submodule calls the matching score value corresponding to the control target of each modal signal in the current control cycle according to the modal error deviation value, extracts the task relevance index, response stability coefficient and structural coupling consistency index of the score value, and constructs a matching composite evaluation expression using the formula: ; Obtain the composite score error value by operation, compare the composite score error value with the modal optimization critical value, judge the modal score deviation state, and generate the modal score difference value; Among them, R represents the composite scoring error value, E m represents the task relevance index corresponding to the modal signal, S j A represents the response stability coefficient of the jth modal signal in the current control cycle, er Indicates the structural coupling consistency index between the modal signal and the current task, M er Indicates the score value of the current modal signal, T er represents the target expected mean of the modal score in the current control period, C er represents the adjustment constant for the matching score evaluation, K represents the total number of items in the stability coefficient sequence; The switching state marking submodule determines whether the main control modal signal satisfies both the error deviation value greater than the error response threshold and the score difference value less than the preferred critical value based on the modal score difference value. If both conditions are met, the main control modal signal is marked as a state to be switched, and a candidate list of modal candidates to be selected is established to obtain the modal replacement judgment result.

5. The multimodal information fusion control system according to claim 1, characterized in that: The modality replacement judgment module includes: The timestamp extraction submodule uses the modal replacement determination result to obtain the response timestamp sequence of the image modality, voice modality, and motion modality in the current control cycle, performs signal statistical processing on each type of modal signal within a fixed time window, extracts the first response time and the end response time, and calculates the average response interval within the unit time window to generate the modal response interval value; The state variation calculation submodule calls the state change sequence of the three types of modes within the response time window according to the modal response interval value, and extracts the state change frequency, the number of overlapping change directions, and the change amplitude value respectively, using the formula: ; Calculate the modal linkage offset value; where L represents the modal linkage offset value, W el Indicates the total duration of the statistical period. Represents the difference of the k-th state variation, R el Indicates the variation value of the modal response direction vector, D el Indicates the consistency index of modal change direction, F avg Represents the mean value of the change amplitude of the modal state value, T m represents the modal response interval value, and N represents the total number of state variation events; The linkage ratio judgment submodule compares the modal linkage offset value with the set modal disturbance judgment threshold. If it is greater than the judgment threshold, the corresponding mode is recorded as an unstable signal mode, and the response label is set to an abnormal state to obtain abnormal response mode identification information.

6. The multimodal information fusion control system according to claim 1, characterized in that: The modal weight reconstruction module includes: The interval extraction submodule obtains the abnormal response modal identification information, extracts the first response time and feedback completion time of each type of modal signal based on the control feedback timestamp of each type of modal signal in the previous control cycle, calculates the action execution duration in the current cycle, and performs difference processing with the duration of the previous cycle to obtain the modal interval difference value; The contribution product calculation submodule calls the feature participation ratio data of each type of modality in the fusion feature generation process according to the modal interval difference value, using the formula: ; The modal product evaluation value is obtained by calculating the product evaluation value of the modal interval deviation and the characteristic participation ratio, and the modal product evaluation value is obtained; wherein, W ce represents the modal product evaluation value, T d Indicates the modal feedback end time of the current control cycle, T p Indicates the modal feedback start time of the previous control cycle, B z Indicates the single feature contribution value corresponding to the z-th mode, N ce Indicates the total number of modal types in which the feature participates, P r Indicates the proportion of mode participation in the feature generation process, V c Indicates the stability deviation of the control feedback process in the current cycle; The weight reset generation submodule judges the size relationship between each type of modal evaluation value and the time contribution threshold based on the modal product evaluation value, filters out the modes above the threshold and sets their weight adjustment factors, and replaces and updates the original modal weights according to the adjustment factor ratio to obtain the modal participation weight reconstruction value.

7. The multimodal information fusion control system according to claim 1, characterized in that: The system further comprises: The control instruction output module calls the modal fusion output vector in the current cycle according to the modal participation weight reconstruction value, performs weighted combination on the fusion components of each modal dimension according to the updated weight, inputs the weighted fusion vector into the instruction mapping matrix, generates the fusion instruction coding set of the task scenario in the current cycle, and obtains the control instruction coding information of the current cycle; The current cycle control instruction encoding information specifically refers to the fusion vector mapping number, the task control instruction group code and the signal channel activation index.

8. The multimodal information fusion control system according to claim 7, characterized in that: The control instruction output module includes: The weight mapping submodule obtains the modal participation weight reconstruction value, calls the modal fusion output vector in the current control cycle, performs a positioning operation on the fusion component of each modal dimension, assigns the corresponding modal participation weight to the fusion component, and forms a weighted result matrix to obtain the modal weighted combination value; The fusion coding construction submodule constructs the instruction output vector within the task cycle according to the modal weighted combination value, and combines all modal weighted component values ​​with the scene label matching index, structural information coupling factor, and instruction mapping template index using the formula: ;Operation to obtain the fusion output vector value; Among them, F dp Represents the fusion output vector value, A x Represents the x-th dimension fusion component value, W x represents the participation weight of the x-th fusion component, Sx represents the structural coupling factor of the x-th mapping template, Lx represents the x-th modal label matching index, T x represents the target value of the x-th scene task label, M dp Indicates the number of dimensions of the fusion output vector; The task instruction generation submodule is based on the fusion output vector value, inputs it into the control instruction mapping matrix by column, performs matrix channel matching and index replacement operations according to the instruction path configuration rules, obtains the mapping result set of each task in the instruction channel, and establishes the current cycle control instruction encoding information.

9. A multimodal information fusion control method, characterized in that: The multimodal information fusion control system according to any one of claims 1 to 8 comprises the following steps: S1: Obtain the control execution parameters of the multimodal task in adjacent control cycles, calculate the path coordinate difference, category variation frequency and offset change amplitude, call the path, category and feedback three types of difference thresholds for judgment, and generate the task difference change value; S2: Based on the task difference change value, compare whether the output error of the main control mode is greater than the error response threshold, and whether the modality matching score is lower than the modality optimization critical value. If both conditions are met, mark the main control mode as a pending switching state and generate a modality replacement judgment result; S3: Based on the modal replacement determination result, calculating the linkage ratio between the modal response time and the state change amplitude, determining whether the linkage ratio exceeds the modal disturbance determination threshold, and obtaining abnormal response modal identification information; S4: Calculate the product of the time interval deviation value and the feature participation ratio based on the abnormal response modal identification information, set a weight adjustment factor for the modal whose product result is greater than the time contribution threshold, and perform new weight distribution according to the adjustment factor to obtain a modal participation weight reconstruction value; S5: According to the modal participation weight reconstruction value, the fusion components of each modal dimension are weightedly combined according to the updated weight, and the weighted fusion vector is input into the instruction mapping matrix to generate a fusion instruction coding set for the task scenario in the current cycle, and obtain the current cycle control instruction coding information.

Images