An adaptive AI control method based on multimodal perception and fusion of heterogeneous data

Abstract

The present invention discloses an adaptive AI control method based on multimodal perception and fusion of heterogeneous data, which specifically relates to the field of multimodal perception technology. The method comprises the following steps: collecting multimodal data and constructing a unified time series state map, extracting modal activity index and variation index to characterize the response and change trend of each modal; constructing a multipath control intention candidate map based on two types of indexes, and dynamically selecting the optimal path to execute control instructions; the control instructions include action parameters and modal feedback calibration parameters, the latter being used to adjust the priority of subsequent modal acquisition; the present invention realizes time alignment and state fusion of heterogeneous data by constructing a multimodal state transition map, utilizes modal activity index and variation index to improve the system's ability to recognize environmental changes, generates control paths and feedback parameters based on dynamic indicators, establishes a closed-loop control mechanism that links perception, decision-making and execution, and significantly enhances the system's adaptability and stability.

Description

Technical Field

[0001] The present invention relates to the field of multimodal perception technology, and more specifically, to an adaptive AI control method based on multimodal perception and fusion of heterogeneous data. Background Art

[0002] With the rapid development of artificial intelligence perception and control technologies, intelligent systems are increasingly being used in transportation, security, manufacturing, healthcare, human-computer interaction, and other fields. To cope with environmental complexity and task diversity, modern AI control systems generally introduce multimodal perception mechanisms. By introducing heterogeneous perception modalities such as images, audio, inertia, temperature and humidity, and lidar, the system's perception granularity and robustness of the scene are improved. However, existing control methods based on multimodal input still have the following significant problems and technical bottlenecks in practical applications:

[0003] Most existing methods directly perform fusion encoding or classification operations on modal features, and fail to explore the state transition pattern of the modality itself in the time dimension. Especially in non-static scenarios, the modal state has a nonlinear evolution process, and it is difficult to accurately capture the system evolution path based on instantaneous features alone. Most current control methods based on deep learning adopt the "perception-control" end-to-end black box model, which makes it difficult to explain the control path generation process and the control intention difficult to analyze. In terms of strategy selection, they often rely on fixed paths or heuristic rules, and are unable to dynamically adjust the control path according to the modal state, and it is also difficult to support multi-path competition and optimization. In traditional perception systems, the selection of sampling frequency or sensor channels is often statically preset, and there is a lack of a mechanism for active adjustment based on the current modal state feedback. In particular, when the modal performance degrades or the modal conflicts, it is impossible to adaptively adjust the priority, resulting in redundant data collection or information distortion. Therefore, the present invention proposes an adaptive AI control method based on multimodal perception and fusion of heterogeneous data to solve the above problems. Summary of the Invention

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] An adaptive AI control method based on multimodal perception and fusion of heterogeneous data includes the following steps:

[0006] Step 1: Collect raw data from multiple heterogeneous sensing modalities, construct a modal data state sequence covering a fixed time window, construct the state trajectory of each modality through a unified time series reference, and uniformly map the state trajectories of multiple modalities to a multimodal state transition map to express the dynamic changes and mutual correlations of each modality in the time dimension;

[0007] Step 2: In each sampling period, the modal activity index is calculated based on the connection density between continuous state nodes in the modal state transition graph, and the modal variation index is calculated based on the nonlinear amplitude of the change in the state label of the graph node. The modal activity index is used to measure the information response density of the corresponding mode in the current time window, and the modal variation index is used to measure the instability or mutation trend of the internal state of the corresponding mode.

[0008] Step 3: Based on the combination of the modal activity index and the modal variation index, a multi-path strategy space for control intent generation is constructed. This multi-path strategy space is represented as a control intent candidate atlas. The atlas contains multiple control path branches, each corresponding to a set of candidate control instructions and modal mapping rules. After the control intent candidate atlas is constructed, a control path is selected based on the dynamic structure and evolution trend of the modal state transition graph, and other paths are pruned.

[0009] Step 4: Convert the control intention structure contained in the final selected control path into control execution instructions. The control execution instructions contain action parameters for executing the target task and modal feedback calibration parameters. The modal feedback calibration parameters are generated in real time based on the joint trend of the current modal activity index and the modal variation index, and the data acquisition priority of each modality is adjusted in the subsequent time window.

[0010] In a preferred embodiment, a unified timing reference is constructed based on a global time synchronization signal, which has a one-millisecond resolution. The original data of multiple heterogeneous perception modalities are timestamp checked and sequence rearranged through this time synchronization signal before entering the modal data state sequence. After the modal data state sequence is constructed, the state trajectory of each modality is recorded at equal intervals under a unified time axis, and the state trajectory is continuously updated at fixed time intervals. All modal state trajectories are strictly aligned on the time axis, which facilitates subsequent unified mapping to a multimodal state transition map.

[0011] In a preferred embodiment, during the mapping process, the state trajectory of each modality is assigned a modal category label and a timestamp number. The construction of the nodes in the graph is based on the joint identification of the modal category and the time window index. The edge connection rule is the directed edge connection between the states of the same modality in continuous time slices, and the bidirectional edge connection between the states of different modalities at the same time point. The multimodal state transition graph thus constructed contains the complete temporal association relationship between modalities and the state transfer path.

[0012] In a preferred embodiment, the calculation process of the modal activity index in step 2 is based on the directed edge structure constructed by continuous time slices between the same modal categories in the multimodal state transition graph, and is obtained by iterative activity calculation, specifically comprising the following steps:

[0013] In the time window corresponding to the current sampling period, all state nodes matching the target modal category are extracted from the multimodal state transition graph, and only the directed edges formed between consecutive time points of the modality are retained to form the time evolution subgraph of the modality;

[0014] Initialize an activity weight for each node in the modal time evolution subgraph with an initial value of 1.0. In each round of propagation, each node evenly distributes its current activity to all target nodes connected by outgoing edges, receives the distribution value from the incoming edge nodes, and updates the current node value. After completing three rounds of iteration, the final activity of all nodes is summed up to obtain the total activity value H;

[0015] The total number of state nodes T of the mode in the current time window is counted. The modal activity index is defined as the quotient of H divided by T, that is, modal activity index = H / T; this index reflects the overall propagation activity of the modal state under the directed propagation network structure, and the calculation process is entirely based on the time series evolution path within the mode.

[0016] In a preferred embodiment, the calculation of the modal variation index in step 2 is based on the time evolution structure of the mode in the current time window and is calculated using sliding fluctuation entropy, specifically including the following steps:

[0017] Extract all state nodes of the current modal category and their connected time-directed edges from the multimodal state transition graph to form a time-ordered node sequence, and sort them by timestamp to form a state label sequence S. The labels in the state label sequence S are the discrete coded values ​​of the modal perception data after state extraction and identification in the current sampling period.

[0018] Use a sliding window with a window length of m to scan the state label sequence S step by step, calculate the standard deviation of the state code value in each window, and record the standard deviation value set of all sliding windows;

[0019] The average value of the set of standard deviation values ​​is defined as the average fluctuation W. The difference between the maximum and minimum values ​​encoded in the state label sequence is then counted to obtain the encoding range R. The modal variation index is defined as W divided by R, that is, modal variation index = W / R; this index characterizes the amplitude and frequency of the state fluctuations of the mode in the current time evolution path, and specifically measures the state evolution trend of the mode itself. The calculation process is entirely based on the time evolution directed edge path within the mode and does not involve cross-modal interference edges.

[0020] In a preferred embodiment, based on the combination of the modal activity index and the modal variation index, in the process of constructing a multi-path strategy space for control intention generation, the activity index is predefined as high, medium and low, and the variation index is predefined as severe, medium and stable. The two types of indexes are cross-combined to generate nine types of unique joint type numbers, corresponding to nine control path branches respectively. Each control path branch corresponds to the initialization of a path branch node. The path branch node is bound to three components, including the control intention structure, candidate control instructions and modal mapping rules. The path branch nodes are connected by number to form a control intention candidate graph set, which serves as the structural basis for control path screening and pruning.

[0021] The control intention structure represents the control objectives that the system is expected to achieve under the current modal state trend, and is expressed in a structured manner; the candidate control instructions represent a set of executable control actions that match the control intention structure, which are used to drive the corresponding control module to complete the target task; the modal mapping rule represents the mapping relationship between the type of perception modality on which the control path depends and its corresponding state information, and is used to limit the adaptability of the control path under input conditions.

[0022] In a preferred embodiment, after the control intention candidate atlas is constructed, a path fitness evaluation mechanism is used to score the paths in the process of selecting a control path and pruning other paths based on the dynamic structure and evolution trend of the modal state transition map. The path scoring includes the following steps:

[0023] Extract the target mode involved in the modal mapping rule in each control path branch node, and extract the state node sets of the mode in the current time window and the previous time window in the modal state transition graph, which are recorded as N1 and N0 respectively. The node evolution rate is defined as: (N1-N0) / (N0+C), which is used to represent the growth rate of the number of modal nodes in the time window, and C is a preset non-zero constant;

[0024] In the current time window, the number of directed edges between all state nodes of the modality is extracted, recorded as E, and the number of nodes is T. The edge connection density is defined as: E / (T×(T-1)), which is used to reflect the tightness of the internal state connection of the current modality;

[0025] After normalizing the node evolution rate and edge connection density, they are weighted and added according to a preset weight ratio, such as 2:1, to obtain a fitness score. A higher score indicates that the modal state involved in the path is more active and the structure is more concentrated in the current window, which means it is more consistent with the system control objectives.

[0026] The pruning process sorts all control path branches according to the score value, retains only the path with the highest score, and eliminates the remaining path branches. The scoring and pruning process is updated once in each control cycle.

[0027] In a preferred embodiment, in the process of converting the control intention structure contained in the finally selected control path into control execution instructions, the action parameters are a set of specific instructions in the corresponding candidate control instructions in the control path, which contain the action direction, action amplitude and action duration information required to execute the target task. After the control intention structure is determined, the action parameters are directly called from the control path and are used to drive the execution module to complete the system operation behavior during the current control cycle. The setting of the action parameters maintains a one-to-one correspondence with the control intention structure in the path, and is refreshed synchronously when the control path is updated.

[0028] In a preferred embodiment, in the process of adjusting the acquisition priority of each modal data, a fuzzy logic device is used to perform a combined judgment on three inputs: modal type, modal activity index trend, and modal variation index trend. The fuzzy logic device is a modal priority scheduling module constructed based on a rule-based reasoning mechanism. The three input parameters are respectively represented as: modal type, modal activity index trend indicating the direction of change of the current modal activity index in two consecutive control cycles, and modal variation index trend indicating the fluctuation trend of the modal variation index between adjacent cycles;

[0029] The fuzzy logic controller matches the above three inputs to specific acquisition priority action outputs based on the preset fuzzy rule base. The acquisition priority actions include four discrete strategies: increasing the acquisition frequency, reducing the acquisition frequency, maintaining the original sampling rate, and temporarily stopping the acquisition.

[0030] The modal feedback calibration parameters are a mapping set of sampling priority actions in the modal space, where each sensing modality corresponds to a scheduling output, which is used to guide the data acquisition configuration strategy of the corresponding modality in the next time window.

[0031] The technical effects and advantages of the present invention are as follows:

[0032] The present invention collects the raw data of multiple heterogeneous perception modalities and constructs their respective state trajectories under the constraints of a unified time window, thereby mapping the state information from different sources to a unified multimodal state transition map. This map not only retains the evolution law of each modality on its own time axis, but also constructs a cross-connection relationship of temporal consistency between modalities, enabling the system to achieve consistent alignment based on semantics and timing before cross-modal information fusion. This method effectively avoids the problem of inter-modal mismatch caused by differences in sampling frequency or inconsistent information update rates, and provides a semantic expression space with temporal coordination and complete structure for subsequent processing processes such as modal index calculation and control path selection, thereby improving the data consistency and fusion accuracy of the entire control chain.

[0033] In each sampling cycle, the present invention calculates the modal activity index and modal variation index based on the connection density and label change amplitude of the state nodes in the multimodal state transition graph, thereby establishing a dual measurement system that reflects the modal response capability and fluctuation trend. The modal activity index reflects the strength and frequency of the mode's participation in the connection in the graph within the current time window, indirectly measuring the actual role weight of the mode in the current scenario; while the modal variation index measures the nonlinear amplitude of the state label change to identify whether the mode has undergone mutations, anomalies, or drastic fluctuations. The two are used in conjunction, enabling the system to dynamically judge the stability and reference value of each mode at the current moment based on quantitative parameters, enhancing the perception layer's sensitive response capability to environmental complexity, and providing a dynamically adaptable input basis for the control intention path selection.

[0034] Based on the calculation of activity index and variation index, the present invention constructs a multi-path strategy space for control intention generation, which is organized into a control intention candidate atlas containing multiple candidate control paths. Each path in the atlas is associated with a set of control instructions and modal mapping rules. After performing path evaluation in combination with the evolution trend of the atlas, the system performs path selection and prunes the remaining paths, effectively simplifying the control decision-making process and improving response efficiency. At the same time, the control intention in the selected path is converted into a control execution instruction in real time. The instruction not only contains the action parameters required for the current task, but also dynamically generates modal feedback calibration parameters. The parameters are generated in real time according to the joint trend of activity and variation, and act on the data acquisition priority adjustment process in the subsequent time window. This closed-loop mechanism enables the system to self-perceive, autonomously adjust, and adaptively execute, significantly improving the control chain's ability to respond to environmental changes, the feedback efficiency of information processing, and the overall robustness and controllability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0036] Figure 1 This is a schematic diagram of an adaptive AI control method based on multimodal perception and fusion of heterogeneous data in the present invention. DETAILED DESCRIPTION

[0037] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] Reference Figure 1 The following examples were obtained:

[0039] Example 1: An adaptive AI control method based on multimodal perception and heterogeneous data fusion includes the following steps: Step 1: Collecting raw data from multiple heterogeneous perception modalities, constructing a modal data state sequence covering a fixed time window, constructing a state trajectory for each modality using a unified temporal reference, and uniformly mapping the state trajectories of multiple modalities to a multimodal state transition map to express the dynamic changes and interdependencies of each modality in the temporal dimension. This step, as the foundational perception step of the overall method, aims to synchronously acquire raw data with diverse structures from multiple heterogeneous perception channels (such as image sensors, audio collectors, and environmental monitoring devices) and organize them into a standardized modal data state sequence within a controllable time window. The unified temporal reference ensures alignment of the modalities on the time axis, preventing offsets or conflicts in the perception results caused by inconsistent sampling times during the fusion process. Furthermore, through the state trajectory construction operation, the state evolution process of each modality is represented as a continuous temporal structure, and the state trajectories of all modalities are further uniformly mapped to the same multimodal state transition map. This map not only records the changing trend of each mode in the time dimension, but also depicts the state interactions, synchronous changes or contradictory relationships that may exist between them at the same moment. It is the core foundation for subsequent multimodal joint modeling, index extraction and path screening.

[0040] Step 2: Within each sampling period, the modal activity index is calculated based on the density of connections between consecutive state nodes in the modal state transition graph. The modal variation index is calculated based on the nonlinear amplitude of changes in the graph node state labels. The modal activity index measures the information response density of the corresponding modality within the current time window, while the modal variation index measures the instability or mutation trend of the corresponding modality's internal state. This step provides a structured measure of the state activity and stability of different modalities within the current time window. The calculation of the modal activity index relies on the density of directed edge connections between consecutive state nodes in the graph structure, reflecting the frequency and density of state transitions of the modality during its temporal evolution. The modal variation index analyzes the amplitude of changes in state labels during temporal evolution, capturing the state value jumps or fluctuations, and thus indicating whether the modality exhibits strong unstable characteristics or sudden behavior. Both indices rely on the established modal state transition graph in their calculation process, extracting key behavioral information from both structural and semantic dimensions, respectively, to form a quantifiable assessment of the state trends of each modality. This serves as the core judgment basis for subsequent strategy generation and modal scheduling.

[0041] Step 3: Based on the combination of the modal activity index and the modal variation index, a multi-path strategy space for control intent generation is constructed. This multi-path strategy space is represented as a control intent candidate atlas. The atlas contains multiple control path branches, each corresponding to a set of candidate control instructions and modal mapping rules. After the control intent candidate atlas is constructed, a control path is selected based on the dynamic structure and evolution trend of the modal state transition graph, and other paths are pruned. This step is based on the two behavioral indices extracted in step 2 and combines them with preset multimodal control strategy rules to organize the control intents corresponding to different combination types into a multi-path structure. The control intent candidate atlas is a graph-structured strategy expression. Each control path branch in the atlas is associated with a set of specific and feasible candidate control instructions and corresponding modal mapping rules to indicate under what perception conditions the path should perform a specific action. The candidate atlas construction mechanism allows multiple paths to coexist, providing the system with alternative solutions when there are ambiguous, conflicting, or uncertain inputs. By analyzing the evolution trend of the current modal state transition graph (such as structural changes such as state aggregation and transfer acceleration), the system dynamically selects the optimal path to execute, and discards non-optimal paths through a pruning mechanism, achieving real-time compression and convergence of the control strategy space.

[0042] Step 4: Convert the control intent structure contained in the finalized control path into control execution instructions. These control execution instructions contain action parameters for executing the target task and modal feedback calibration parameters. These modal feedback calibration parameters are generated in real time based on the joint trend of the current modal activity index and modal variation index, and are used to adjust the acquisition priority of each modal data in the subsequent time window. This step extracts the corresponding control intent structure based on the single path selected in step 3 and concretizes it into two major parts of the control execution instruction: the action parameters of the target task, which directly drive the system actuators to generate specific actions; and the modal feedback calibration parameters, which are used to negatively influence the behavioral strategy of the perception module. The modal feedback calibration parameters are generated based on the joint trend of the modal activity index and modal variation index in the current cycle. Using an inference mechanism (such as a fuzzy logic controller), they are mapped into executable configurations such as acquisition frequency adjustment, start-stop control, and power consumption status. These parameters are used to dynamically adjust the acquisition priority of each perception modality in the next time window, completing a complete closed loop from perception, evaluation, control, and perception strategy feedback.

[0043] A unified timing reference is constructed based on a global time synchronization signal with a one-millisecond resolution. The time synchronization signal is used to perform timestamp verification and sequence rearrangement on the original data of multiple heterogeneous perception modalities before entering the modal data state sequence. After the modal data state sequence is constructed, the state trajectory of each modality is recorded at equal intervals on a unified time axis, and the state trajectory is continuously updated at fixed time intervals. All modal state trajectories are strictly aligned on the time axis, which facilitates subsequent unified mapping to the multimodal state transition map.

[0044] In actual system deployment, different perception modalities usually have their own independent sampling mechanisms and triggering timings, which can easily lead to temporal offsets and structural misalignments of the data. In order to solve this problem, the present invention introduces a high-precision time synchronization mechanism and sets a unified timing reference to ensure that all perception data can be forced to be organized under the same time base. The global time synchronization signal is generated by the main control module or introduced through an external synchronization device. It has a time resolution of at least one millisecond and can meet the synchronization accuracy requirements of multi-source high-frequency data. After the data acquisition is completed, the data of all modalities must undergo a timestamp verification process based on the synchronization signal. If a deviation in the timestamp is detected, the sequence will be rearranged according to the global reference time, so that the original data has completed preliminary time standardization before entering the state sequence construction stage. The construction of the state sequence is based on a fixed time window and is divided according to equal time intervals, so that the state trajectory of each modality can be updated with equal steps on a unified time axis.

[0045] For example, if the time interval is set to 20 milliseconds, the system will collect state update data from each modality every 20 milliseconds and record it in the trajectory structure. In this way, even if there are differences in data update frequency between modalities, they can be fully aligned on a unified time axis, establishing a stable and synchronized timing foundation for subsequent trajectory mapping and cross-modal interaction. This alignment is not only consistent in time, but also in structure. Only when the multimodal state trajectories share the same time anchor point can we ensure that the connection relationship between states is real and valid when constructing the multimodal state transition graph, and there will be no distortion of edge connections or abnormal graph structure due to time misalignment. This also directly affects the reliable calculation of the subsequent modal activity index and modal variation index, and further affects the accuracy of strategy path selection and execution instruction generation.

[0046] For example, in actual applications, the visual modality uses a camera with a frame rate of 50 frames per second, the audio modality uses a microphone with a sampling rate of 10kHz, and the environmental modality uses a sensor that samples once per second. When the system starts a unified timing reference, the system records a synchronization signal every millisecond. After each sampling, each modality is assigned a timestamp and aligned according to the timestamp in a unified timing buffer. Ultimately, every 20 milliseconds is used as a time segment, and the three modalities are rearranged into a set of inputs to form corresponding state update points. Through this mechanism, even if there are differences in sampling frequency and physical characteristics between modalities, they can be effectively coordinated through the timestamp mechanism, so that all state data are strictly synchronized in the time dimension, ensuring the stability and accuracy of the subsequent unified mapping of state trajectories and graph fusion process.

[0047] During the mapping process, the state trajectory of each modality is assigned a modal category label and a timestamp number. The construction of the nodes in the graph is based on the joint identification of the modal category and the time window index. The edge connection rule is the directed edge connection between the states of the same modality in continuous time slices, and the bidirectional edge connection between the states of different modalities at the same time point. The multimodal state transition graph constructed in this way contains the complete temporal correlation relationship and state transition path between modalities. With the support of the aforementioned unified temporal reference mechanism, the system has obtained the state trajectories of multiple modalities and ensured that they are strictly aligned on the time axis. On this basis, this step uniformly maps all modal state trajectories into a unified structure, namely the multimodal state transition graph. During the mapping process, the state of each time point in each trajectory needs to be converted into a node in the graph.

[0048] To ensure identifiability and structural uniqueness of nodes in the graph, each node is associated with two types of labels upon generation: a modality category label, which indicates the perceptual channel (e.g., visual, audio, or environmental) from which the node originates; and a timestamp number, which indicates the specific window index within the timeline in which the state node resides. The primary key of a node consists of the modality category and the time window number, ensuring redundancy or confusion within the graph structure. Edges in the graph represent temporal relationships and state transfer pathways between nodes. Edge construction follows two rules: Directed edges are established between state nodes of the same modality between consecutive time slices, representing the natural evolution of the modality's state over time. These edges form a time evolution subgraph for each modality, supporting the calculation of temporal metrics such as the modal activity index and the modal variation index. Bidirectional edges are established between state nodes of different modalities at the same time point, indicating the possibility of information interaction, event co-occurrence, or state coordination between the modalities at a given moment. This structure captures the system's synchronous response relationships along the "horizontal perceptual dimension." The graph constructed in the above way not only retains the inherent evolutionary structure of a single modality, but also introduces the state coupling relationship between modalities in the time dimension, thereby forming a joint state evolution network covering the time axis and modal axis at the structural level.

[0049] The calculation process of the modal activity index in step 2 is based on the directed edge structure constructed between the same modal category through continuous time slices in the multimodal state transition graph, and is obtained by iterative activity calculation. Specifically, it includes the following steps: within the time window corresponding to the current sampling period, all state nodes matching the target modal category are extracted from the multimodal state transition graph, and only the directed edges formed between consecutive time points of the modality are retained to form the time evolution subgraph of the modality; an activity weight is initialized for each node in the modal time evolution subgraph, with the initial value set to 1.0. In each round of propagation, each node distributes its current activity evenly to all target nodes connected by outgoing edges, receives the distribution value from the incoming edge node, updates the current node value, and after completing three rounds of iteration, sums up the final activity of all nodes to obtain the total activity value H;

[0050] The total number of state nodes T of the modality in the current time window is counted, and the modal activity index is defined as the quotient of H divided by T, that is, the modal activity index = H / T; this index reflects the overall propagation activity of the modal state under the directed propagation network structure, and the calculation process is completely based on the time series evolution path within the modality. The above calculation process is specifically used to measure the state propagation characteristics of each modality in the current sampling period, focusing on the structural activity in the direction of its time evolution. The prerequisite is that the state trajectories of all modes in the graph have been strictly aligned according to a unified time axis, and each state node has a modal category label and a time window index, so the system can accurately extract the subgraph structure of a single modal category in a complete time window.

[0051] During iterative propagation, the initial activity of each node is set to a constant 1.0 to simulate the initial state of information diffusion. Each round of propagation simulates the flow of state information from front to back in time using the principle of equal distribution. This rule ensures that the propagation direction remains consistent with temporal logic. Since propagation occurs only along directed edges, it fully adheres to the evolutionary path within the modality and does not involve cross-modal influences. The number of iterations is set to three, ensuring that the propagation depth can cover multi-step transmission effects across time while also keeping the computational complexity manageable. After all nodes have completed propagation, the system sums the current activity value of each node to obtain the total activity value H for the entire modality in the current cycle. By dividing this value by the total number of state nodes T, a normalized modality activity index is obtained, making it comparable across different modalities and time windows.

[0052] Assume a modality is the "visual modality" and that within the current time window (e.g., 500 milliseconds), it contains 10 state nodes, originating from five time slices, with two nodes per slice. There are 12 directed edges between the nodes. Initially, each node has an activity of 1.0. During propagation, each round of propagation allocates activity to the next node based on the number of outgoing edges, forming a diffusion structure as shown in the figure. After three rounds of propagation, the activity of each node accumulates and transfers, ultimately summing the activity of the 10 nodes to H = 27.4. The modality activity index is 27.4 / 10 = 2.74. This value can be compared with the visual modality indices of other modalities or historical time windows to assess their response density in the current period.

[0053] The calculation of the modal variation index in step 2 is based on the time evolution structure of the mode in the current time window and is calculated using the sliding fluctuation entropy. Specifically, the following steps are included: all state nodes of the current modal category and their connected time-directed edges are extracted from the multimodal state transition graph to form a time-ordered node sequence, and the nodes are sorted according to the timestamps to form a state label sequence S. The labels in the state label sequence S are the discrete coding values ​​of the modal perception data after state extraction and identification in the current sampling period; a sliding window with a window length of m is used to scan the state label sequence S step by step, and the standard deviation of the state coding value in each window is calculated, and the standard deviation value set of all sliding windows is recorded;

[0054] The average value of the set of standard deviations is defined as the average volatility W. The difference between the maximum and minimum values ​​encoded in the state label sequence is then calculated to obtain the encoding range R. The modal variation index is defined as W divided by R, that is, modal variation index = W / R. This index characterizes the amplitude and frequency of state fluctuations of the modal in the current time evolution path. It specifically measures the state evolution trend of the modal itself. The calculation process is entirely based on the time evolution directed edge path within the modal and does not involve cross-modal interference edges. This calculation process constitutes the core module of modal stability analysis, focusing on identifying whether the current modal perception state exhibits sudden changes, violent fluctuations, or complex change trends. Unlike the modal activity index, which emphasizes propagation density, the modal variation index focuses more on the temporal fluctuation degree and discrete change structure of the state itself.

[0055] At the implementation level, the system first extracts all state nodes and their forward-connecting edges within the current time window from the constructed multimodal state transition graph, based on modality category. This operation ensures that the selected node sequence truly reflects the state evolution trajectory of the modality itself, avoiding analytical bias caused by cross-modal connections. Subsequently, all nodes are sorted by timestamp, and a corresponding state label sequence S is generated. These labels are discrete representations of the modal raw data after being identified by the state extraction module. Specific implementation methods include cluster labeling, semantic identification numbering, and numerical interval mapping. The resulting sequence S encodes the evolution path of the modality within the sampling period. A sliding window is introduced to detect local dynamic fluctuations. The window length m can be set based on the modality type and control period (for example, 5 or 7), and it slides across the entire sequence S with a step size of 1. Each window scan extracts a state subsequence and calculates its standard deviation to reflect local volatility. The system records the standard deviation values ​​obtained for all sliding windows to form a volatility set.

[0056] The average volatility W represents the overall volatility intensity of the mode in the current window. At the same time, in order to avoid interference caused by changes in the encoding range of the state label itself, the difference between the maximum and minimum values ​​in the label sequence is further counted to obtain the encoding range R. The modal variation index is normalized and calculated in the form of W / R, so that the index has universality for cross-modal and cross-time comparisons. The larger the value of the final modal variation index, the more drastic and unstable the change in the modal state is, and vice versa, the state is more stable and the trend is smoother. This index is completely based on the directed edge path calculation of the internal time evolution of the mode, without using any cross-modal data or external disturbance factors, so it can truly reflect the evolution characteristics of the mode itself.

[0057] For example, in a pressure-sensing modality, the system extracts 12 state nodes within a sampling period. After state extraction, the state label sequence is: S = [2, 2, 3, 5, 4, 4, 3, 2, 1, 1, 2, 3]. Assuming a sliding window m = 4, the system performs nine window slides, calculating the standard deviation of each window, resulting in a set of standard deviations such as: [0.5, 1.25, 0.96, 0.82, 1.03, 0.87, 0.67, 0.43, 0.76]. The average fluctuation W = the average of all standard deviations ≈ 0.81; the range R = the maximum state label 5 − the minimum label 1 = 4; and the modal variation index = 0.81 / 4 ≈ 0.2025. This index reflects a certain degree of state variation within the modality within the period, but the overall fluctuation is controllable, making it suitable as a mild response indicator when adjusting the sampling strategy.

[0058] Based on the combination of modal activity index and modal variation index, in the process of constructing a multi-path strategy space for control intention generation, the activity index is predefined as high, medium and low, and the variation index is predefined as severe, medium and stable. The two types of indexes are cross-combined to generate nine unique joint type numbers, corresponding to nine control path branches respectively. Each control path branch corresponds to the initialization of a path branch node. The path branch node is bound to three components, including the control intention structure, candidate control instructions and modal mapping rules. The path branch nodes are connected according to the number to form a control intention candidate graph set, which serves as the structural basis for control path screening and pruning.

[0059] In the present invention, the control intention generation strategy is not a static call template, but a dynamic construction based on the state characteristics presented by the perception system in the current sampling period. Among them, the modal activity index is used to reflect the density of the perception response, and the modal variation index is used to characterize the stability trend of the state evolution. The two together determine the information complexity and uncertainty of the system in the current period. In order to map the above information to the control strategy layer, the system presets a grading strategy for the activity index and the variation index, and divides the two into three levels, namely high, medium, low and severe, medium, and stable. Through pairwise combination, a total of nine unique combination types are formed (for example: high + severe, medium + medium, low + stable, etc.), and each combination represents a typical perception state characteristic pattern. The system assigns a number to each combination type, and constructs nine control path branches accordingly. Each path branch corresponds to a path branch node as a structural unit in the control strategy atlas. Each node is bound to three core components during initialization:

[0060] ① Control Intent Structure: This structure expresses the control objectives that the system aims to achieve under the current modal state trends. It has clear directionality and purpose, and is typically represented in a structured manner, such as a tree-like instruction set, a state transition target set, or a multi-stage task vector. Its definition is not derived from a static set of commands, but rather is generated by the system during path construction based on the current modal state trends. Objectives may include, for example, "stabilizing system output fluctuations," "enhancing forward motion response," and "reducing energy consumption through behavioral intervention." Within the same numbered path, the control intent structure maintains a one-to-one correspondence with the modal combination pattern.

[0061] ② Candidate control instructions: Candidate control instructions are a set of executable actions that match the semantic hierarchy of the control intent structure and are used to drive downstream control modules (such as motors, servos, and camera steering) to complete specific tasks. This instruction group typically consists of multiple optional actions, such as direction instructions, amplitude settings, and duration parameters, to support subsequent strategy refinement. Candidate control instructions have a direct operational mapping relationship with the control intent structure and are initialized and matched based on the system's policy space and historical training data.

[0062] ③ Modal Mapping Rules: Modal mapping rules describe the perceptual input structure that a control path relies on, including the specific perceptual modality, the source location and format of state information, and other details. For example, a control path might rely solely on the combination of "audio modality + visual modality" or on the mapping of "temperature + humidity state vectors in the environmental modality." These rules constrain whether the system has sufficient perceptual support conditions when executing that path. Modal mapping rules can be expressed in a structured graph using methods such as vector hash tables and mask structures.

[0063] The nine path branch nodes are not just static lists in the atlas. Instead, connections between nodes are constructed based on the similarity and evolutionary trends between their numbers (i.e., modal combination type numbers). For example, if the evolution from a medium + stable state to a high + medium state is considered a natural transition, a connecting edge from that node to that node can be established in the graph to express the strategy evolution path. After all path branch nodes are integrated, a complete candidate atlas of control intent is formed. This atlas serves as the input basis for subsequent path scoring and pruning operations and is a structured representation of the entire control strategy space. The candidate atlas enables the system to flexibly select under complex state combinations without being restricted to fixed templates.

[0064] Assume that in the current sampling period, the modal activity index of the visual modality is "high" and the modal variation index is "medium". This combination corresponds to category A3 of the nine predefined categories. When constructing the multipath strategy space, the system initializes a control path branch node with number A3 and binds the following three components to it:

[0065] Example of a control intent structure: The control intent structure bound to this control path branch is "Achieve dynamic direction adjustment with high response accuracy." This structure indicates that, in a scenario with highly dense but moderately fluctuating visual information, the system desires to rapidly fine-tune the target direction to adapt to environmental changes while avoiding overreaction or system instability. The control intent structure is structured and contains fields such as the target type (e.g., direction fine-tuning), the control target priority (e.g., accuracy priority), and constraints (e.g., obstacle avoidance restrictions). These fields guide the scope of subsequent control strategies.

[0066] Example of candidate control instructions: The candidate control instructions corresponding to the above control intent structure include the following set of instructions: Instruction 1: Move direction +30°, speed 30%, action duration 0.6 seconds; Instruction 2: Move direction +15°, speed 25%, action duration 0.8 seconds; Instruction 3: Maintain current position, maintain observation state, action duration 1.2 seconds. These control instructions are generated by the system's historical strategy model or manually preset, and all have highly executable and clearly structured parameter expression formats. The final control execution module selects one of these as the final action output based on the path score, and binds it to the path number A3.

[0067] Example of a modal mapping rule: The modal mapping rule for this control path node is defined as follows: "The dependent modal set includes the visual modality and the inertial modality, both of which must be in a state that can be decodable; the visual modality must provide three consecutive state labels within the current cycle, and the inertial modality must provide an encoding of the angular velocity vector change trend." This rule limits the prerequisites for sensory input during the execution of this path, while excluding path branches that are unavailable when certain modal states are abnormal or sampling fails. The system can use this to determine whether the path can participate in path scoring and pruning. After path number A3 is activated, the system includes it in the control intention candidate atlas and forms a policy space structure together with the other eight path branch nodes. If the current modal state evolution trend supports the highest score for path A3, the system will directly extract the optimal item from the control intention structure and instruction set under this path, and use this to generate control execution instructions, ultimately executing the action parameters for the target task to complete the operation.

[0068] It should be emphasized that each modality (such as vision, audio, inertial, etc.) has its own modal activity index and modal variation index; the system will collect the indices of all modalities and obtain a representative activity-variation joint gear (for example: high + intense) based on the preset joint trend analysis mechanism (such as weighting, dominant mode, confidence fusion and other existing technologies); this joint gear corresponds to one of the 9 predefined strategy categories (A1-A9 in total), rather than generating 9 paths for each modality.

[0069] For example, assume that in this perception system, the visual modality has high activity and medium variation, the inertial modality has medium activity and medium variation, and the audio modality has low activity and stability. Using the joint trend strategy, the system can determine that the system is in a high activity and medium variation state, thus falling into the A3 category. This then loads the A3 path branch node for path evaluation and pruning.

[0070] After the control intention candidate atlas is constructed, the system needs to select the path that best matches the current system state from the nine control paths as the control output. To this end, the system introduces a path fitness evaluation mechanism to score each control path branch node and perform pruning operations based on the scoring results. The calculation process of the path fitness score combines the dynamic evolution characteristics of the modal state in the atlas and the density of the connection structure to reflect the fit between the path and the current system state. The specific scoring process includes the following steps:

[0071] First, extract the target mode involved in the modal mapping rule in each control path branch node, and locate the state node set corresponding to the modal category in the current multimodal state transition map. Obtain all state nodes of the modality in the current time window, denoted as N1; obtain the state nodes of the corresponding modality in the previous time window, denoted as N0. The system compares the number of nodes in these two time periods and calculates the node evolution rate of the modality, which is defined as: (N1-N0) / (N0+C), where C is a preset non-zero constant to avoid the situation where the denominator is zero. The node evolution rate is used to measure the growth trend of the number of modal states in the time dimension, which can reflect whether the modality is in an active development period at the current stage.

[0072] Next, we further count the number of directed edges between all state nodes within the modal category within the current time window, denoted as E, and the corresponding number of nodes, T. In graph theory, the edge connection density of the modal category within the current time period is defined as: E / (T × (T-1)). This measures the closeness of connections between nodes and reflects the structural complexity and connectivity of the modal's internal state. A higher connection density indicates a more concentrated state transition network within the current window, potentially supporting richer control mechanisms.

[0073] The node evolution rate and edge connection density are then normalized separately to unify their numerical ranges and prevent scale differences from affecting the scoring results. The system then uses a preset weighting strategy to perform a fusion calculation on the two normalized indicators. For example, a weight ratio of 2:1 is set, meaning the node evolution rate accounts for 2 and the edge connection density accounts for 1. The weighted sum of the two gives the final fitness score. A higher value indicates that the mode on which the path depends is more active and more concentrated in the current time window, indicating a higher fitness for the system's control objectives.

[0074] After scoring all paths, the system ranks the nine control paths from highest to lowest suitability, retaining only the path with the highest score. The remaining paths are pruned for the current control cycle, temporarily removing them from control decision-making. This ensures unique control command output, avoids multi-path conflicts, and enhances the stability and directionality of control decisions through structural selection. The pruning and scoring process is updated within each control cycle, ensuring the system can dynamically adjust the control strategy structure in real time based on the evolution of the perceived state.

[0075] For example, in a typical application scenario, the number of nodes in the previous window of the current visual modality is 40 (N0=40), and the number of nodes in the current window is 56 (N1=56). Assuming the preset constant C=10, the evolution rate is (56-40) / (40+10)=0.32. At the same time, the number of nodes in the current window visual modality is 56, and the number of connecting edges is 620, so the connection density is 620 / (56×55)≈0.20. After normalization and weighted calculation, the system may determine that this path is the most suitable path and activate its corresponding control intention structure and control instructions for execution.

[0076] When converting the control intent structure contained in the finalized control path into control execution instructions, the system uses the control intent structure as a core reference, calls a set of specific instructions from the candidate control instructions corresponding to the control path as the final action parameters, and uses these action parameters to drive the execution module to complete the task response. The specific content of the action parameters includes three key dimensions required for the execution of the target task: action direction, action amplitude, and action duration information.

[0077] During system operation, the determination of a control intent structure indicates that the system has clearly defined the functional objective to be achieved at the current moment, such as performing a fine directional adjustment, executing a rapid obstacle avoidance maneuver, or achieving a stable parking response. Each control intent structure is bound to a set of candidate control instructions during the construction phase and embedded as a complete path element during control path construction. Once path fitness scoring and pruning are complete, the control intent structure in the final selected path becomes the core expression of the current control objective.

[0078] Based on this, the system automatically calls a set of candidate control instructions attached to the path and selects specific execution items that correspond to the control intent structure. These action parameters are then generated from the control execution instructions. These action parameters are directly inherited from the preset or dynamically generated control instruction templates within the candidate instructions, preserving their structural integrity and matching the action intent, ensuring that the action output is logically consistent with the intended task. For example, the set of candidate control instructions bound to the control path includes the following three preset actions: a forward offset of +20°, a speed of 40%, and an action duration of 0.8 seconds; a 15° rightward rotation at a medium speed (30%) for 1.0 seconds; and a 2.0-second hold position with a reduced sensor sampling frequency. Based on the target type expressed by the current control intent structure (e.g., "fine-tune execution direction"), the system selects the second instruction that strongly matches the intent. The generated action parameters are: a 15° rightward rotation, a 30% speed, and a 1.0-second action duration. The action parameters are packaged and injected into the control execution instruction format, and sent to the execution module within the current control cycle to control specific mechanical components, actuators or algorithm modules to produce corresponding behaviors. It is worth noting that the action parameters are refreshed synchronously when the control path is updated. In other words, after each control cycle is completed, the system will re-evaluate the modal state map and control path structure. If the new control path is different from the previous cycle, the new path will correspond to the new control intent structure and candidate control instruction set. The system will re-select new action parameters for execution to ensure that the control strategy and system state always maintain dynamic matching and closed-loop adaptation.

[0079] When adjusting the data acquisition priorities for each modality, the system incorporates a fuzzy logic controller as the decision-making hub. This controller dynamically generates modal feedback calibration parameters for perception scheduling based on current modal state trends. This fuzzy logic controller is a modal priority scheduling module built on a rule-based inference mechanism. Its core function is to classify and judge the perception behaviors of different modalities based on multiple input variables and output corresponding sampling configuration instructions.

[0080] The input of the fuzzy logic controller consists of three items: modal type, modal activity index trend, and modal variation index trend. The modal type indicates the category of heterogeneous perception channels used in the perception system, such as visual modality, audio modality, inertial modality, etc., which is used to distinguish the perception module of the input source. The modal activity index trend indicates the direction of change of the modal activity index of the modality in two consecutive control cycles, and determines whether its information response density is increasing, decreasing, or remaining unchanged. The modal variation index trend is used to evaluate the fluctuation trend of the internal state of the modality in adjacent sampling cycles, and measure whether its state change tends to be stable, increasing, or drastically mutated. After the above three inputs are input into the fuzzy logic device, the system performs inference matching based on the pre-built fuzzy rule library. The fuzzy rule library contains multiple sets of correspondences from "input combinations" to "sampling strategy outputs". For example: Rule 1: If the modality is vision, the activity trend is increasing, and the variation trend is stable, then the acquisition frequency is increased; Rule 2: If the modality is audio, the activity trend is decreasing, and the variation trend is drastic, then the acquisition frequency is reduced; Rule 3: If the modality is inertia, the activity trend is fluctuating, and the variation trend is stable, then the original sampling rate is maintained; Rule 4: If the modality is thermal imaging, and both the activity and variation are at a low trend, then the acquisition is temporarily stopped.

[0081] The fuzzy logic controller runs once during each control cycle, judging the results based on the above rules and outputting the corresponding acquisition priority action. Acquisition priority actions are limited to four discrete strategies: increasing the acquisition frequency, decreasing the acquisition frequency, maintaining the original sampling rate, and temporarily stopping acquisition. Each strategy corresponds to a specific sampling interval or data acquisition control instruction. The acquisition priority action is ultimately encoded and encapsulated as modal feedback calibration parameters, which form a set of mappings in the modal space, meaning that each sensing modality has an independent scheduling output. Based on this scheduling output, the system dynamically configures the acquisition behavior of the corresponding modality within the next time window.

[0082] For example, within a given control cycle, the activity index of the visual modality shows a rapid upward trend, while the variation index remains stable. After evaluation by the fuzzy logic controller, an "increase acquisition frequency" strategy is output, and the corresponding modal feedback calibration parameter is set to "shorten the sampling interval to 60% of the current baseline value." However, if the activity trend of the audio modality is stable and the variation index fluctuates violently, a "reduce acquisition frequency" strategy is output, with the corresponding parameter set to "increase the sampling interval to 150% of the original value." These parameters are injected into the modal scheduling engine for execution in the next sampling cycle, thereby enabling adaptive regulation of the data acquisition rhythm of the multimodal perception system. This mechanism ensures that the system has state perception and resource optimization capabilities at the perception input layer. It can improve the perception rate of key modalities when the system state changes dramatically, reduce redundant sampling in inefficient modes, enhance control response efficiency, and reduce energy consumption.

[0083] Example 2: Further Optimization - This invention does not directly explain the use scenarios of "bidirectional edges", which can easily give the impression that its construction purpose is out of touch with reality. Therefore, the following reasonable supplementary points are introduced to give bidirectional edges a practical functional logic foundation in the overall method. The supplementary technical uses are as follows:

[0084] Supplementary factors in path pruning scoring: When calculating path fitness, "node bidirectional adjacency" can be introduced as a weighting factor reflecting modal collaboration density. For example, if a modality forms bidirectional edges with multiple other modalities at the current time point, indicating that its state has an influence on other modalities, its path score weight can be appropriately increased. That is, after normalizing the node evolution rate and edge connection density, the edge connection density is first multiplied by the weighting factor, and then weighted addition is performed according to a preset weight ratio, such as 2:1, to obtain the fitness score value. The weighting factor of modal collaboration density, namely "node bidirectional adjacency", is obtained by normalizing the total number of bidirectional edge connections of the corresponding modal type in the current period.

[0085] Priority correction mechanism for modal feedback calibration parameters: When the fuzzy logic judgment result has fuzzy edges (multiple strategy scores are close), the system can refer to the bidirectional connection density between the modality and other modalities in the current time window as an auxiliary basis for determining the acquisition priority. That is, on the basis of the fuzzy logic input consisting of three items, another auxiliary item is added, namely, the "node bidirectional adjacency" value obtained by normalizing the total number of bidirectional edge connections of the corresponding modal type in the current period.

[0086] Branch filtering for candidate screening of control paths: If the modal nodes involved in the control path do not have any bidirectional edge connections with other modalities in the graph structure, they can be regarded as "isolated modalities" and the corresponding paths can be preferentially eliminated in specific tasks.

[0087] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0088] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0089] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0090] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0091] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An adaptive AI control method based on multimodal perception and fusion of heterogeneous data, characterized in that: The following steps are involved: Step 1: Collect raw data from multiple heterogeneous sensing modalities, construct a modal data state sequence covering a fixed time window, construct the state trajectory of each modality through a unified time series reference, and uniformly map the state trajectories of multiple modalities to a multimodal state transition map; Step 2: In each sampling period, extract the state nodes within the target modal time window, construct a directed edge structure, initialize the activity to 1.0, and perform three rounds of iterative propagation. In each round, the activity is evenly distributed among the outgoing edges and the incoming values ​​are received. Finally, the sum is calculated and divided by the number of nodes to obtain the modal activity index. Extract the state label sequence of the modality, calculate the standard deviation set using a sliding window, and calculate its average value as the average volatility. Combined with the coding range, the ratio of the two is the modal variation index. Step 3: Based on the combination of the modal activity index and the modal variation index, a control intention candidate atlas is constructed. The atlas contains multiple control path branches, each of which corresponds to a set of candidate control instructions and modal mapping rules. After the construction is completed, a control path is selected based on the evolution trend of the modal state transition graph, and other paths are pruned. Step 4: Convert the control intention structure in the final selected control path into a control execution instruction. The control execution instruction contains action parameters and modal feedback calibration parameters. The modal feedback calibration parameters are generated in real time based on the joint trend of the two indices, and the data acquisition priority of each modal is adjusted in the subsequent time window.

2. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 1 is characterized in that: A unified timing reference is constructed based on a global time synchronization signal, through which the raw data of multiple heterogeneous perception modalities are timestamp-checked and sequence-rearranged before entering the modal data state sequence. After the modal data state sequence is constructed, the state trajectory of each modality is recorded at equal intervals on a unified time axis, the state trajectory is continuously updated at fixed time intervals, and all modal state trajectories are strictly aligned on the time axis.

3. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 2 is characterized in that: During the mapping process, the state trajectory of each modality is assigned a modality category label and a timestamp number. The construction of the nodes in the graph is based on the joint identification of the modality category and the time window index. The edge connection rule is the directed edge connection between the states of the same modality in continuous time slices, and the bidirectional edge connection between the states of different modalities at the same time point.

4. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 3 is characterized in that: The calculation process of the modal activity index in step 2 is based on the directed edge structure constructed by continuous time slices between the same modal categories in the multimodal state transition graph, and is obtained by iterative activity calculation. Specifically, it includes the following steps: In the time window corresponding to the current sampling period, all state nodes matching the target modal category are extracted from the multimodal state transition graph, and only the directed edges formed between consecutive time points of the modality are retained to form the time evolution subgraph of the modality; Initialize an activity weight for each node in the modal time evolution subgraph with an initial value of 1.

0. In each round of propagation, each node evenly distributes its current activity to all target nodes connected by outgoing edges, receives the distribution value from the incoming edge nodes, and updates the current node value. After completing three rounds of iteration, the final activity of all nodes is summed up to obtain the total activity value H; The total number of state nodes T of the mode in the current time window is counted. The modal activity index is defined as the quotient of H divided by T, that is, modal activity index = H / T.

5. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 4 is characterized in that: The calculation of the modal variation index in step 2 is based on the time evolution structure of the mode in the current time window and is calculated using the sliding fluctuation entropy. Specifically, it includes the following steps: Extract all state nodes of the current modal category and their connected time-directed edges from the multimodal state transition graph to form a time-ordered node sequence, and sort them by timestamp to form a state label sequence S. The labels in the state label sequence S are the discrete coded values ​​of the modal perception data after state extraction and identification in the current sampling period. Use a sliding window with a window length of m to scan the state label sequence S step by step, calculate the standard deviation of the state code value in each window, and record the standard deviation value set of all sliding windows; The average value of the standard deviation value set is defined as the average volatility W. Then, the difference between the maximum and minimum values ​​encoded in the state label sequence is counted to obtain the encoding range R. The modal variation index is defined as W divided by R, that is, modal variation index = W / R.

6. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 5 is characterized in that: Based on the combination of modal activity index and modal variation index, in the process of constructing a multi-path strategy space for control intention generation, the activity index is predefined as high, medium and low, and the variation index is predefined as severe, medium and stable. The two types of indexes are cross-combined to generate nine unique joint type numbers, corresponding to nine control path branches respectively. Each control path branch corresponds to the initialization of a path branch node. The path branch node is bound to three components, including the control intention structure, candidate control instructions and modal mapping rules. The path branch nodes are connected according to the number to form a control intention candidate graph set, which serves as the structural basis for control path screening and pruning.

7. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 6 is characterized in that: The control intention structure represents the control objectives that the system is expected to achieve under the current modal state trend, and is expressed in a structured manner; the candidate control instructions represent a set of executable control actions that match the control intention structure, which are used to drive the corresponding control module to complete the target task; the modal mapping rule represents the mapping relationship between the type of perception modality on which the control path depends and its corresponding state information, and is used to limit the adaptability of the control path under input conditions.

8. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 7 is characterized in that: After the control intention candidate atlas is constructed, a control path is selected based on the dynamic structure and evolution trend of the modal state transition map and other paths are pruned. The path fitness evaluation mechanism is used to score the paths. The path scoring includes the following steps: Extract the target mode involved in the modal mapping rule in each control path branch node, and extract the state node sets of the mode in the current time window and the previous time window in the modal state transition graph, which are recorded as N1 and N0 respectively. The node evolution rate is defined as: (N1-N0) / (N0+C), which is used to represent the growth rate of the number of modal nodes in the time window, and C is a preset non-zero constant; In the current time window, the number of directed edges between all state nodes of the modality is extracted, recorded as E, and the number of nodes is T. The edge connection density is defined as: E / (T×(T-1)), which is used to reflect the tightness of the internal state connection of the current modality; After normalizing the node evolution rate and edge connection density, they are weighted and added according to the preset weight ratio to obtain the fitness score. The pruning process sorts all control path branches according to the score value, retains only the path with the highest score, and eliminates the remaining path branches. The scoring and pruning process is updated once in each control cycle.

9. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 8 is characterized in that: In the process of converting the control intention structure contained in the finally selected control path into control execution instructions, the action parameters are a set of specific instructions in the corresponding candidate control instructions in the control path, which contain the action direction, action amplitude and action duration information required to execute the target task. After the control intention structure is determined, the action parameters are directly called from the control path and are used to drive the execution module to complete the system operation behavior in the current control cycle. The setting of the action parameters maintains a one-to-one correspondence with the control intention structure in the path, and is refreshed synchronously when the control path is updated.

10. The adaptive AI control method based on multimodal perception and fusion of heterogeneous data according to claim 9 is characterized in that: In the process of adjusting the acquisition priority of each modal data, a fuzzy logic controller is used to make a combined judgment on three inputs: modal type, modal activity index trend, and modal variation index trend. The fuzzy logic controller is a modal priority scheduling module built based on a rule-based reasoning mechanism. The three input parameters are represented as: modal type, modal activity index trend indicating the direction of change of the current modal activity index in two consecutive control cycles, and modal variation index trend indicating the fluctuation trend of the modal variation index between adjacent cycles. The fuzzy logic controller matches the above three inputs to specific acquisition priority action outputs based on the preset fuzzy rule base. The acquisition priority actions include four discrete strategies: increasing the acquisition frequency, reducing the acquisition frequency, maintaining the original sampling rate, and temporarily stopping the acquisition. The modal feedback calibration parameters are a mapping set of sampling priority actions in the modal space, where each sensing modality corresponds to a scheduling output, which is used to guide the data acquisition configuration strategy of the corresponding modality in the next time window.

Images