AI interactive data processing system based on multi-modal perception and dynamic decision

Abstract

The invention relates to the technical field of AI interaction data processing, and discloses an AI interaction data processing system based on multi-modal perception and dynamic decision, comprising the following modules: a multi-modal perception module used for collecting environment data through a multi-source sensor; the data fusion module is used for generating fused feature data; the causal decision-making module is used for generating a decision-making action to cope with the change of the environment; the decision security module is used for identifying potential safety hazards in the high-risk scene and generating alternative decision or early warning information; the sensing calibration module is used for optimizing a sensing strategy in a changing environment; and the adaptive optimization module continuously optimizes the perception and decision strategy. According to the invention, the multi-modal sensing module is combined with a cross-modal consistency learning mechanism and a noise robustness enhancement technology, and the data fusion module introduces a context sensing attention mechanism and a multi-level feature alignment network, so that the sensing ability of the system to complex environment information and the comprehensiveness and accuracy of feature representation are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AI interactive data processing, and specifically to an AI interactive data processing system based on multimodal perception and dynamic decision-making. Background Art

[0002] With the rapid development of artificial intelligence (AI), an increasing number of intelligent systems are being applied in various real-world scenarios, particularly in areas such as autonomous driving, intelligent robotics, security monitoring, and smart homes. Traditional intelligent systems typically rely on a single type of sensor (such as a camera, microphone, or pressure sensor) to perceive their environment. These systems analyze sensor data, execute appropriate decisions, and complete pre-defined tasks.

[0003] In current AI applications, systems typically rely on single-modal sensor data for environmental perception, such as using only a camera to acquire visual information or only a microphone to capture sound. For processing multimodal information, a common approach is to perform simple feature concatenation or fusion of data from different modalities before inputting it into downstream decision-making models. For example, in robotic navigation applications, lidar might be used alone for positioning, or visual images might be initially fused with inertial measurement unit data to estimate pose. In intelligent voice assistants, the primary focus is on recognizing and understanding the semantics of voice signals.

[0004] However, existing AI systems, such as single-modal perception or simple multimodal fusion, struggle to cope with information loss or noise interference, leading to partiality and inaccuracy in perception results. They also suffer from insufficient robustness in unexpected situations, leading to unpredictable risks. This poses a serious threat, particularly in safety-critical applications. They also have limited generalization capabilities in new environments or tasks and a strong reliance on large amounts of labeled data, resulting in high deployment costs and poor adaptability. Therefore, the present invention provides an AI interactive data processing system based on multimodal perception and dynamic decision-making to address these shortcomings in the existing technology. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides an AI interactive data processing system based on multimodal perception and dynamic decision-making, which solves the problems of insufficient data processing efficiency, decision-making accuracy and system adaptability of the existing technology in complex interactive environments.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A first aspect of the present invention provides an AI interactive data processing system based on multimodal perception and dynamic decision-making, comprising:

[0008] Multimodal perception module: used to obtain environmental data of different modalities through multiple independent sensors (such as image acquisition sensors, audio acquisition sensors, pressure sensors or contact sensors). After data acquisition, pre-processing operations are performed on the original data, including but not limited to denoising, normalization, and feature extraction. In order to enhance data quality and consistency, this module further combines cross-modal consistency learning mechanism and noise robustness enhancement technology. Among them, the cross-modal consistency learning mechanism aims to ensure the alignment and coordination of the processing results of each modality at the semantic level by learning the inherent correlation between different modal data; the noise robustness enhancement technology effectively suppresses or eliminates the noise introduced in the acquisition process through specific algorithm processing, improves the accuracy of subsequent data processing, and outputs optimized data of each modality.

[0009] Data fusion module: used to receive processed data from each modality. Deeply integrate heterogeneous data from different modalities to form a unified perceptual feature representation with deep semantic associations. Specifically, efficient and high-quality fusion is achieved by introducing a context-aware attention mechanism and a multi-level feature alignment network. The context-aware attention mechanism can dynamically evaluate the importance of different modal features in the current context and assign different weights accordingly, so that the fusion process can focus on key information; the multi-level feature alignment network performs structured alignment of modal features at different levels of abstraction, effectively capturing and fusing potential associations between modalities. The fused feature data serves as the input of the subsequent decision module.

[0010] Causal Decision Module: Based on the fused feature data output by the Data Fusion Module, it generates decision actions to respond to environmental changes. Decisions are made by constructing an environment-action causal graph and employing a graph neural network (GNN) combined with a causal inference algorithm. This graph explicitly models environmental states, system actions, and their causal relationships, making the decision process explainable. Graph neural networks can efficiently process graph-structured data, learning and capturing complex dependencies from causal graphs. Causal inference algorithms, building on GNN learning, infer the potential impact of different action sequences on environmental states, thereby generating optimal decision actions with clear causal evidence.

[0011] The Decision Safety Module intervenes after the causal decision module generates a decision action, focusing on assessing the potential risks of the decision and ensuring the safety of system operation. This module constructs a simulation environment and applies formal verification methods to pre-evaluate the consequences of potential decisions. The simulation environment allows for virtual simulation testing of the potential impact of decisions before the system actually executes them. Formal verification methods, based on rigorous mathematical and logical reasoning, formally check the correctness and safety of decisions, identifying high-risk scenarios that could lead to dangerous or unstable conditions under specific conditions. Once safety hazards are identified, alternative decisions or warnings are generated to prevent the system from executing potentially dangerous actions, thereby ensuring system robustness and safety.

[0012] Perception Calibration Module: This module is used to intelligently adjust and optimize perception strategies based on the execution effects of the decision actions generated by the causal decision module in the environment and the real-time environmental feedback re-collected by the system. This module uses cross-domain transfer learning and domain adaptation technologies to achieve dynamic adjustment of perception parameters. Cross-domain transfer learning enables the system to transfer knowledge learned from the source domain to a new, unknown target domain, reducing dependence on large amounts of newly annotated data; domain adaptation technology enables the system to adapt to different or changing environmental conditions by adjusting the feature mapping relationship of the perception model. By intelligently adjusting the perception area, the gain of the sensor, or enhancing the sampling frequency of the perception channel, the system's perception strategy in unknown or changing environments is optimized, improving the system's ability to adapt to environmental changes.

[0013] Adaptive Optimization Module: Based on the decision-making performance of the causal decision-making module and the optimization results of the perception calibration module, it learns from historical tasks to rapidly adapt to new environments or task requirements and continuously optimizes the perception and decision-making strategies of the multimodal perception module and the causal decision-making module. It summarizes the perception-decision mapping patterns for different task categories and presets the optimal parameter structure when initializing new tasks, enabling rapid adaptation. During task execution, it monitors system performance in real time and, based on feedback, makes small adjustments to the perception and decision-making strategies online, either through meta-learning or through online fine-tuning, to ensure continuous improvement and long-term stability.

[0014] A second aspect of the present invention provides an AI interactive data processing method based on multimodal perception and dynamic decision-making, which is applied to the above-mentioned system and includes the following steps:

[0015] S1. Collect raw data such as images, audio, and touch in the environment through multiple different types of sensors (including image acquisition sensors, audio acquisition sensors, pressure sensors, or contact sensors); perform preprocessing operations such as denoising, illumination normalization, spectrum analysis, and structure extraction on the collected environmental data; combine cross-modal consistency learning mechanism and noise robustness enhancement technology to generate standardized data for each modality.

[0016] S2. Based on the standardized data of each modality, feature dimension alignment and time synchronization are performed; by introducing a context-aware attention mechanism and a multi-level feature alignment network, the features of different modalities are deeply integrated and semantically associated to form a unified perceptual feature representation data with deep semantic associations.

[0017] S3. Based on the semantically associated perceptual feature representation, an environment-action causal graph is constructed, which explicitly models the causal relationship between the environmental state and the system action. Subsequently, a graph neural network combined with a causal reasoning algorithm is used to reason on the graph to generate optimal and explainable decision actions in response to environmental changes.

[0018] S4. Based on the generated decision action, a simulation environment is constructed to simulate the environmental changes and system responses that may be caused by the decision; formal verification methods are used to pre-evaluate the potential consequences of the decision action and identify safety hazards in high-risk scenarios; if risks exist, alternative decisions or warning information are generated.

[0019] S5. Evaluate the applicability of the current perception strategy under different environmental conditions based on the execution effect of the decision-making action in the environment and the current environmental state reflected by the real-time environmental data re-collected by the multimodal perception module; use cross-domain transfer learning and domain adaptation technology to intelligently adjust the perception area, sensor gain or enhance the sampling frequency of the perception channel to optimize the system's perception strategy in unknown or changing environments.

[0020] S6. Based on the decision-making performance of the causal decision-making module and the optimization results of the perception calibration module, by learning from historical tasks, summarizing the task categories and their perception-decision mapping rules, so as to quickly adapt to new environments or task requirements, and continuously optimize the perception and decision-making strategies of the multimodal perception module and the causal decision-making module, for example, by performing small, low-cost online fine-tuning of the strategy parameters of the perception module and the causal decision-making module or rapid adaptive adjustment based on meta-learning.

[0021] The present invention provides an AI interactive data processing system based on multimodal perception and dynamic decision-making. It has the following beneficial effects:

[0022] 1. This invention combines a multimodal perception module with a cross-modal consistency learning mechanism and noise robustness enhancement technology, and introduces a context-aware attention mechanism and a multi-level feature alignment network into the data fusion module, effectively improving the system's ability to perceive complex environmental information and the comprehensiveness and accuracy of feature representation. It can overcome the limitations of single-modal data and noise interference, construct unified perceptual features with deep semantic associations, provide a more reliable and richer data foundation, and reduce the rate of decision-making errors caused by insufficient or biased perceptual information.

[0023] 2. The present invention constructs an environment-action causal graph through a causal decision-making module and adopts a graph neural network combined with a causal reasoning algorithm to achieve the explainability and high precision of the decision-making process. By explicitly modeling the causal relationship between action and environmental state, the system decision logic is clear and traceable, and combined with formal verification methods to pre-evaluate potential decision consequences, effectively identify and avoid high-risk scenarios, thereby significantly enhancing the robustness and security of system decisions and reducing the risk of unpredictable behaviors in critical applications.

[0024] 3. This invention utilizes cross-domain transfer learning and domain adaptation techniques in the perception calibration module, and the adaptive optimization module continuously optimizes perception and decision-making strategies through historical task learning, thereby improving the system's adaptability and generalization performance in unknown or changing environments. By intelligently adjusting perception parameters to respond to environmental changes, it reduces reliance on large amounts of annotated data, and rapidly adapts to new tasks through inductive learning based on historical experience. This enables rapid deployment and stable operation in a variety of practical application scenarios, reducing the debugging and maintenance costs required to maintain system performance in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the AI ​​interactive data processing system architecture based on multimodal perception and dynamic decision-making of the present invention;

[0026] Figure 2 Schematic diagram of the structure of the multimodal perception module of the present invention;

[0027] Figure 3 Schematic diagram of the data fusion module structure of the present invention;

[0028] Figure 4 This is a schematic diagram of the causal decision module structure of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of the decision-making security module of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of the perception calibration module of the present invention;

[0031] Figure 7 Schematic diagram of the structure of the adaptive optimization module of the present invention;

[0032] Figure 8 This is a workflow diagram of the AI ​​interactive data processing system based on multimodal perception and dynamic decision-making of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] Refer to the attached Figure 1 , Figure 1 The figure is a schematic diagram of the architecture of an AI interactive data processing system based on multimodal perception and dynamic decision-making according to one embodiment of the present invention. The present invention provides an AI interactive data processing system based on multimodal perception and dynamic decision-making, comprising: a multimodal perception module, a data fusion module, a causal decision module, a decision security module, a perception calibration module, and an adaptive optimization module.

[0035] The multimodal perception module collects environmental data from multiple sensors. It preprocesses the raw data and, in combination with a cross-modal consistency learning mechanism and noise robustness enhancement technology, generates processed data for each modality. The output of the multimodal perception module serves as input to the data fusion module.

[0036] The data fusion module receives the processed data from each modality output by the multimodal perception module. By introducing a context-aware attention mechanism and a multi-level feature alignment network, it deeply integrates the data from each modality to construct a unified perceptual feature representation with semantic associations. The fused feature data generated by the data fusion module serves as input to the causal decision module.

[0037] The causal decision module constructs an environment-action causal graph based on the fused feature data output by the data fusion module. It then uses a graph neural network combined with a causal inference algorithm to generate decision actions to respond to environmental changes. The decision actions generated by this causal decision module serve as input to the decision safety module and the perception calibration module.

[0038] The Decision Safety Module receives the decision actions generated by the Causal Decision Module. It constructs a simulation environment and uses formal verification methods to pre-evaluate the potential consequences of these decisions, identifying potential high-risk scenarios and safety hazards. The output of the Decision Safety Module includes alternative decisions or warning information.

[0039] The perception calibration module uses cross-domain transfer learning and domain adaptation techniques to intelligently adjust the perception area, sensor gain, or enhance the sampling frequency of the perception channel based on the environmental performance of the decision actions generated by the causal decision module and the current environmental state reflected by the real-time environmental data re-collected by the multimodal perception module. The adjustment results of the perception calibration module serve as input to the adaptive optimization module.

[0040] The adaptive optimization module receives the decision-making performance of the causal decision module and the adjustment results of the perception calibration module. By learning from historical tasks, it can quickly adapt to new environments or task requirements and continuously optimize the perception and decision-making strategies of the multimodal perception module and the causal decision module.

[0041] Refer to the attached Figure 2 , Figure 2 2 is a schematic diagram of the structure of a multimodal perception module according to an embodiment of the present invention.

[0042] The multimodal perception module includes an image perception unit, a sound perception unit, and a tactile perception unit. These units independently but collaboratively acquire and process raw data of different modalities from the environment.

[0043] The image perception unit uses an image acquisition sensor to capture static images or dynamic videos of the surrounding environment. This unit performs a series of preprocessing operations on the captured image or video data, including edge detection, illumination normalization, and spatial feature extraction. Edge detection identifies object boundaries within the image, illumination normalization eliminates the effects of uneven illumination on the image data, and spatial feature extraction captures local and global features such as texture and shape. The processed image feature data is then transmitted to subsequent modules.

[0044] The sound perception unit acquires ambient sound or voice input through an audio sensor. It performs noise suppression on the audio data to reduce the impact of ambient noise on speech or sound recognition. It also performs voice activity detection to identify portions of the audio stream containing valid speech. It also performs spectrum analysis to extract acoustic features such as frequency and energy. The processed sound feature data is then transmitted to subsequent modules.

[0045] The tactile sensing unit acquires interactive contact data from pressure sensors or contact sensors. From the raw sensor output, it extracts contact intensity features, such as the magnitude of the contact force, and contact rate of change features, such as the rate at which the contact force changes over time. The extracted tactile feature data is transmitted to subsequent modules.

[0046] The multimodal perception module combines a cross-modal consistency learning mechanism and noise robustness enhancement technology in the process of integrating the output data of each perception unit. The cross-modal consistency learning mechanism ensures that the features learned between different modalities remain consistent at the semantic level by designing methods such as shared representation space or adversarial learning, thereby improving the synergy of multimodal information. Noise robustness enhancement technology further suppresses various noises present in the data of each modality by applying methods such as filtering, adaptive threshold processing, or deep learning denoising networks, thereby improving the purity and reliability of the perception results. Ultimately, the multimodal perception module generates standardized and enhanced data for each modality, providing high-quality input for the data fusion module.

[0047] Refer to the attached Figure 3 , Figure 3 FIG. 4 is a structural diagram of a data fusion module according to an embodiment of the present invention.

[0048] The data fusion module receives the processed data of each modality output by the multimodal perception module and integrates it into a fused feature representation with semantic associations. This module includes a feature alignment unit, a weight learning unit, and a fused feature generation unit.

[0049] The feature alignment unit is used to receive the processed data of each modality from the multimodal perception module, such as the image feature F image , sound characteristics F audio and tactile features F tactile This unit performs feature dimension unification, time synchronization alignment, and spatial mapping reconstruction. Feature dimension unification projects the feature vectors of different modalities into a common feature space through linear transformation or nonlinear mapping, for example:

[0050] F′ mod =W mod F mod +b mod ;

[0051] Where, F mod represents the original modal features, W mod and b mod denote the modality-specific weight matrix and bias term, F′ mod Represents the modal features after unifying the dimensions. Time synchronization ensures that data collected by different sensors at different sampling rates are aligned on the time axis, for example through interpolation or resampling techniques. Spatial mapping reconstruction addresses the differences in spatial dimensions of the data from different modalities and maps them to a unified spatial coordinate system.

[0052] The weight learning unit is used to model the importance of different modalities and assign dynamic weights to each modality. This unit is implemented by training the context-aware attention mechanism or the dynamic modality confidence assessment module. The context-aware attention mechanism dynamically calculates the attention weights of each modality based on the current task or environmental context. For example, the attention weight A mod It can be calculated as follows:

[0053] A mod =Softmax(W a ReLU(W h F′ mod +b h )+b a );

[0054] Where A modis the modal weight output by the weight learning unit, W a ,W h ,b a ,b h is a learnable parameter, F′ mod is the aligned modal feature output by the feature alignment unit, and ReLU(·) is the rectified linear unit activation function. These weights reflect the importance of each modal information in the current context for generating the final fused features. The dynamic modal confidence assessment module evaluates the confidence of each modal data in real time based on indicators such as quality, reliability, or information entropy, and converts it into modal weights.

[0055] The fusion feature generation unit performs feature splicing, adaptive feature weighting or nonlinear feature transformation based on the feature alignment results and modality weights to construct a unified fusion feature representation F fused Feature concatenation is to directly connect all aligned modal features to form a longer feature vector. Adaptive feature weighting is to multiply the aligned modal features with their corresponding dynamic weights and then sum them, for example:

[0056] F fused =∑ mod A mod ·F′ mod ;

[0057] Where A mod is the modal weight output by the weight learning unit, F′ mod The aligned modal features are output by the feature alignment unit. Nonlinear feature transformation uses structures such as multilayer perceptrons or convolutional neural networks to perform nonlinear processing on the weighted or concatenated features to extract more abstract and discriminative fused features. Ultimately, this unit outputs semantically correlated fused feature data, which serves as input to the subsequent causal decision module.

[0058] Refer to the attached Figure 4 , Figure 4 FIG. 4 is a schematic diagram of the structure of a causal decision module according to an embodiment of the present invention.

[0059] The causal decision module receives the fused feature data output by the data fusion module and generates decision actions to respond to environmental changes. This module includes an environment representation unit, a causal graph construction unit, a policy reasoning unit, and a decision execution unit.

[0060] The environment representation unit is used to extract and construct a structured environment representation including environment entities, attributes, states, and relationships based on the generated fused feature data. This unit processes the fused features through a neural network or feature extraction algorithm to identify the objects in the environment, their spatial location, current state (e.g., stationary or moving), and the relationships between objects. This information is encoded into a structured data format, such as an attribute graph or tensor representation, as input to the causal graph construction unit.

[0061] The causal graph construction unit dynamically constructs a causal graph that reflects the causal relationship between actions and environmental states based on the structured environmental representation constructed by the environmental representation unit and combined with a preset knowledge graph. This causal graph is represented as a directed acyclic graph G = (V, E), where the node set V represents environmental state variables and possible actions, and the edge set E represents the causal relationship between them. For example, causal connections can be identified and established using rules based on expert knowledge, machine learning methods (such as Bayesian network learning), or a combination of both. For each node v∈V, its parent node Pa(v) represents the variable that directly causes v.

[0062] The policy reasoning unit uses the causal graph generated by the causal graph construction unit and combines it with the current task objectives to use the graph neural network (GNN) and causal reasoning algorithm to infer and generate an interpretable optimal decision action sequence. GNN is used to learn the high-dimensional representation of nodes and edges in the causal graph and capture complex causal dependencies. For example, the representation of node v is The k+1th layer of GNN can be updated as:

[0063]

[0064] Where N(v) represents the set of neighbor nodes of node v, c vu is the normalization constant, W (k) is a learnable weight matrix, ReLU(·) is the rectified linear unit activation function, Represents the feature representation of node u at the kth layer of the GNN. A causal inference algorithm is used to evaluate the impact of different actions on future environmental states, thereby selecting an action sequence that achieves the task goal. Given an intervention do(A=a), forcing variable A to have a value a, its causal effect P(Y|do(A=a)) on other variables Y can be calculated. This unit generates one or more candidate decision-action sequences and their corresponding causal explanations.

[0065] The decision execution unit converts the decision action sequence generated by the policy reasoning unit into executable system response behaviors or interaction instructions, driving the execution of system actions or human-computer interaction processes. For example, if the decision action is to grasp an object, the decision execution unit converts this into a sequence of motion instructions for the robotic arm; if the decision action is to respond with a voice message, the decision execution unit generates the corresponding text or voice output. This unit ensures that decision actions are accurately and effectively converted into system responses at the physical or information level.

[0066] Refer to the attached Figure 5 , Figure 5 FIG. 4 is a schematic diagram of the structure of a decision security module according to an embodiment of the present invention.

[0067] The decision safety module receives the decision actions generated by the causal decision module and is used to evaluate their potential consequences and identify safety hazards. This module includes a simulation environment construction unit and a formal verification unit.

[0068] The simulation environment construction unit is responsible for constructing a virtual simulation environment based on the generated decision action and the current environment state. This simulation environment replicates or approximates the physical laws, object interaction characteristics, and the system's own dynamic model of the real world. For example, if the decision action involves the movement of a robotic arm, the simulation environment will include a 3D model of the robotic arm, target objects, obstacles, and physical laws such as gravity and collision. The construction of the simulation environment ensures that the physical or logical consequences of the decision action can be simulated without actually executing it.

[0069] The formal verification unit uses formal verification methods to pre-evaluate the potential consequences of decision-making actions in the simulated environment to identify safety hazards in high-risk scenarios. Formal verification involves constructing a system model M and a safety property φ. Model M can be a state transition system that describes the simulated environment and decision-making behavior. Safety property φ can be that no action will cause a collision with an obstacle or that key parameters always remain within safety thresholds. For example, safety property φ can be expressed as a linear temporal logic (LTL) formula, such as:

[0070]

[0071] Where G represents global ly and Collision represents the collision state.

[0072] The verification process includes reachability analysis or model checking algorithms to explore all reachable states of the simulation environment after executing the decision action and check whether these states violate predefined safety properties. For example, a simple high risk metric R can be expressed as:

[0073]

[0074] Where Sunsafe represents the set of all unsafe states, and P(s|do(A)) represents the probability or likelihood of reaching state s after executing decision action A. When a decision action is detected that could lead to a state that violates safety properties, the formal verification unit identifies this high-risk scenario and outputs safety hazard information to the system. Based on the identified safety hazards, the decision safety module can generate alternative decisions, recommending a safer action plan, or issue a warning message to the system operator, alerting them to the risk of the current decision, thereby preventing the system from executing potentially dangerous actions and ensuring the overall safety and robustness of the system.

[0075] Refer to the attached Figure 6 , Figure 6 FIG. 4 is a schematic diagram of the structure of a perception calibration module according to an embodiment of the present invention.

[0076] The perception calibration module is used to dynamically adjust and optimize the perception strategy of the multimodal perception module based on the execution results of the decision actions generated by the causal decision module and real-time environmental feedback. This module includes an adaptability evaluation unit and a domain adaptation unit.

[0077] The adaptability evaluation unit is used to evaluate the applicability of the current perception strategy under different environmental conditions based on the actual execution effect of the generated decision action in the environment, combined with the environmental data collected in real time by the multimodal perception module. This evaluation process aims to identify perception blind spots or performance degradation areas. For example, after the system performs an action, if the real-time perception data shows a large deviation from expectations, or the perception accuracy drops significantly under specific lighting and noise environments, the adaptability evaluation unit will identify these areas where perception performance is insufficient. If the perception accuracy drops significantly under lighting and noise environments, the adaptability evaluation unit will identify these areas where perception performance is insufficient. The evaluation can be quantified as the perception performance error E percep :

[0078] E percep =Loss(P actual ,P expected );

[0079] Where, P actual Indicates the actual perception result, P expected represents the ideal or expected perception result, and the Loss function is used to measure the difference between the two. If E percep Exceeding a preset threshold indicates the need for perception calibration.

[0080] Based on the evaluation results of the adaptability evaluation unit, the domain adaptation unit uses cross-domain transfer learning and domain adaptation techniques to adjust and optimize the perception strategy to adapt to the new environmental domain or task scenario. Cross-domain transfer learning allows the system to effectively transfer and apply knowledge learned from a labeled source domain (for example, a perception model trained in a laboratory environment) to a new, data-scarce target domain (for example, a complex outdoor environment actually deployed). This reduces the need to collect and label large amounts of data from scratch in a new environment. Domain adaptation technology makes the model more robust to the data distribution of the target domain by adjusting the feature mapping relationship of the perception model. For example, an adversarial domain adaptation method can be used to enable the perception model to learn domain-invariant feature representations by minimizing the distance between the feature distributions of the source domain and the target domain. This can be achieved through the following loss function:

[0081] L total =L task +λL domain ;

[0082] Where, L task is the task-specific loss function (e.g., classification loss), L domain is the domain adversarial loss that aims to reduce the difference in feature distribution between the source and target domains (e.g., by maximizing the error of the domain classifier), and λ is a balancing factor.

[0083] Specific adjustments to the domain adaptation unit include: intelligently adjusting the perception area, such as dynamically changing the camera's field of view or the microphone's pickup direction to focus on key information sources; adjusting sensor gain, such as increasing the image sensor's sensitivity to adapt to low-light environments or reducing the audio sensor's gain to avoid oversaturation; or enhancing the sampling frequency of the perception channel, such as increasing the lidar's scanning frequency in scenarios requiring higher temporal resolution. These adjustments are intended to optimize the multimodal perception module's data acquisition strategy in changing environments, ensuring it can provide high-quality, reliable environmental perception data under varying or unknown conditions, thereby supporting the accuracy of subsequent decision-making.

[0084] Refer to the attached Figure 7 , Figure 7 FIG. 4 is a schematic diagram of the structure of an adaptive optimization module according to an embodiment of the present invention.

[0085] The adaptive optimization module is responsible for the system's long-term learning and performance optimization. Based on the decision-making performance of the causal decision module and the optimization results of the perception calibration module, this module continuously optimizes the perception and decision-making strategies of the multimodal perception module and the causal decision module, and quickly adapts to new environments or task requirements. This module includes a task induction unit, a model initialization unit, and a policy fine-tuning unit.

[0086] The task induction unit is used to learn historical tasks and summarize the perception-decision mapping rules under different task categories. This unit analyzes the correspondence between the perception output of the multimodal perception module, the decision action of the causal decision module, and the environmental feedback when the system performs historical tasks. Through clustering, pattern recognition or meta-learning and other technologies, the typical perception features, optimal decision strategies and corresponding environmental constraints corresponding to different task types (for example, navigation, grasping, interactive question and answer) are identified. For example, a task knowledge base can be constructed, which contains task type T i and its corresponding perception strategy parameter θ percept,i and decision strategy parameter φ decision,i .

[0087] The model initialization unit presets the optimal parameter structure when initializing a new task based on the task categories and perception-decision mapping patterns summarized by the task induction unit. When the system faces a new or partially unfamiliar task, the model initialization unit searches the summarized task knowledge base for the most similar task type and loads the corresponding preset perception and decision model parameters. This significantly shortens the system's cold start time in a new task environment and avoids the inefficiency of starting from scratch.

[0088] The policy fine-tuning unit is responsible for real-time monitoring of system performance during task execution and for making small or meta-learning-based online fine-tuning of the perception and decision-making strategies based on feedback. The unit continuously receives decision performance indicators (e.g., task completion rate, decision accuracy) from the causal decision module and calibration feedback (e.g., perception error, calibration parameters) from the perception calibration module. Based on these real-time feedbacks, the policy fine-tuning unit applies online learning algorithms, reinforcement learning algorithms, or meta-learning methods to adjust the perception model parameters θ currently in use. percept and decision model parameter φ decision Perform iterative updates.

[0089] The AI ​​interactive data processing method based on multimodal perception and dynamic decision-making described below and the AI ​​interactive data processing system based on multimodal perception and dynamic decision-making described above can be referenced to each other.

[0090] Refer to the attached Figure 8 , Figure 8 This is a schematic diagram of the workflow of an AI interactive data processing system based on multimodal perception and dynamic decision-making according to an embodiment of the present invention.

[0091] The workflow of the AI ​​interactive data processing system of the present invention begins with the multimodal perception module. During system operation, the multimodal perception module continuously collects raw data from multiple sources, including images, sounds, and touch, from the environment. This raw data is preprocessed within each perception unit, including operations such as noise suppression and feature extraction. It is then optimized using a cross-modal consistency learning mechanism and noise robustness enhancement technology to generate processed data for each modality.

[0092] The processed data from each modality output by the multimodal perception module is transmitted to the data fusion module. This module receives this data and uses a context-aware attention mechanism and a multi-level feature alignment network to deeply integrate and align features from different modalities, ultimately constructing a unified fused feature representation with semantic associations.

[0093] The fused feature representation serves as input to the causal decision module. Based on the received fused features, the causal decision module first constructs a structured environment representation using the environment representation unit. Next, the causal graph construction unit dynamically constructs a causal graph reflecting the causal relationship between actions and environment states. The policy reasoning unit utilizes a graph neural network and a causal reasoning algorithm to generate one or more candidate decision-action sequences based on the causal graph and output these decision-action sequences.

[0094] The decision actions generated by the causal decision module are transmitted to the decision safety module for preliminary evaluation. The decision safety module constructs a virtual simulation environment within the simulation environment construction unit and uses formal verification methods to evaluate the potential consequences of the decision actions in the simulation environment to identify high-risk scenarios or safety hazards. If a potential risk is detected, the decision safety module outputs an alternative decision or a warning message.

[0095] The decision actions generated by the causal decision module are also transmitted to the perception calibration module. Based on the actual execution results of the decision actions and the environmental state reflected by the real-time environmental data re-collected by the multimodal perception module, the perception calibration module uses the adaptability evaluation unit to evaluate the suitability of the current perception strategy. Based on the evaluation results, the domain adaptation unit uses cross-domain transfer learning and domain adaptation techniques to intelligently adjust the multimodal perception module's perception area, sensor gain, or sampling frequency to optimize its perception performance in the current environment.

[0096] The evaluation results of the decision safety module and the adjustment results of the perception calibration module are transmitted to the adaptive optimization module. The adaptive optimization module uses the task induction unit to learn the perception-decision mapping patterns of historical tasks and presets the optimal parameters at the beginning of a new task through the model initialization unit. During system operation, the policy fine-tuning unit continuously fine-tunes the perception and decision-making strategies of the multimodal perception module and the causal decision module based on real-time decision performance and calibration feedback. This iterative process enables the system to continuously adapt to environmental changes and task requirements, thereby maintaining long-term stable performance.

[0097] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An AI interactive data processing system based on multimodal perception and dynamic decision-making, characterized by: Includes the following modules: A multimodal perception module is used to collect environmental data through multi-source sensors, pre-process the environmental data, and generate processed data for each modality by combining a cross-modal consistency learning mechanism and noise robustness enhancement technology; The data fusion module is used to construct a perceptual feature representation with semantic associations based on the processed data of each modality by introducing a context-aware attention mechanism and a multi-level feature alignment network to generate fused feature data; The causal decision module is used to build an environment-action causal graph based on fused feature data, and uses a graph neural network combined with a causal reasoning algorithm to generate decision actions to respond to environmental changes; The decision safety module is used to build a simulation environment based on the generated decision actions, and use formal verification methods to pre-evaluate potential decision consequences, identify safety hazards in high-risk scenarios, and generate alternative decisions or early warning information; The perception calibration module uses cross-domain transfer learning and domain adaptation technologies to intelligently adjust the perception area, sensor gain, or enhance the sampling frequency of the perception channel based on decision actions and real-time environmental feedback, thereby optimizing the perception strategy in a changing environment. The adaptive optimization module is used to quickly adapt to new environments or task requirements and continuously optimize perception and decision-making strategies based on the adjustment results of the causal decision module and the perception calibration module by learning from historical tasks.

2. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 1 is characterized in that: The multi-source sensor includes an image acquisition sensor, an audio acquisition sensor, and a pressure sensor or a contact sensor, and the multimodal perception module includes: An image perception unit, which is used to obtain static images or dynamic videos of the surrounding environment through an image acquisition sensor and perform edge detection, illumination normalization, and spatial feature extraction; The sound perception unit is used to obtain ambient sound or voice input through audio acquisition sensors and perform noise suppression, voice activity detection and spectrum analysis; The tactile sensing unit is used to obtain interactive contact data through a pressure sensor or a contact sensor, and extract the contact intensity and change rate as tactile features.

3. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 1 is characterized in that: The data fusion module includes: The feature alignment unit is used to receive the processed data of each modality and reconstruct it through feature dimension unification, time synchronization alignment and spatial mapping; A weight learning unit is used to model the importance of different modalities and assign weights to each modality by training a context-aware attention mechanism or a dynamic modality confidence assessment module; The fusion feature generation unit performs feature splicing, adaptive feature weighting or nonlinear feature transformation based on the feature alignment results and modality weights to construct a fusion feature representation with semantic association.

4. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 3 is characterized in that: The constructing of a fusion feature representation with semantic association includes: Based on the data after multi-level feature alignment, the contextual semantic information of each modal feature in the time series and spatial structure is extracted to identify the potential semantic connections between different modalities; Establish a contextual dependency structure between modalities and construct a dynamic cross-modal attention map, which is used to adjust the interaction path between modal features; The modal features enhanced with contextual semantics are input into the fusion operation process and work together with the dynamic modal weights to output a fusion feature representation that includes cross-modal semantic linkage relationships.

5. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 1 is characterized in that: The causal decision module includes: Environmental representation unit: used to extract and construct a structured environmental representation including environmental entities, attributes, states and relationships based on the generated fusion feature data; A causal graph construction unit is used to dynamically construct a causal graph that can reflect the causal relationship between actions and environmental states based on the constructed structured environment representation and combined with the preset knowledge graph; The policy reasoning unit is used to infer and generate an interpretable optimal decision action sequence based on the generated causal graph and the current task objectives, using graph neural networks and causal reasoning algorithms; The decision execution unit is used to convert the generated decision action sequence into system-executable response behaviors or interaction instructions, and drive the execution of system actions or human-computer interaction processes.

6. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 5 is characterized in that: The construction of a causal graph capable of reflecting the causal relationship between actions and environmental states includes: Extract the information content of the environmental structure from the fused feature data, including spatial orientation, object type, and motion trend, and perform structured coding and causal relationship modeling to form a preliminary description of the environmental state; Identify the current task type and key target parameters, map the task information into constraints or priority dimensions in the state vector, and establish a causal dependency relationship between the task and the environment state; The description of the environment state and the causal dependency between the task and the environment state are integrated to construct a multi-dimensional state representation with causal relationships and input it into the policy network.

7. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 1 is characterized in that: The perception calibration module includes: The adaptability evaluation unit is used to evaluate the applicability of the current perception strategy under different environmental conditions based on the execution effect of the generated decision actions in the environment and combined with the real-time environmental data collected by the multimodal perception module to identify perception blind spots or performance degradation areas; The domain adaptation unit is used to construct and update the feature mapping relationship of the cross-modal perception model based on the evaluation results using the domain adaptation algorithm to adapt to the new environmental domain or task scenario.

8. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 1 is characterized in that: The adaptive optimization module includes: The task induction unit is used to model the system performance in multiple tasks based on the adjustment results of the causal decision module and the perception calibration module, and summarize the task categories and perception-decision mapping rules; The model initialization unit presets the initial parameter structure of the multimodal perception module and dynamic decision module according to the summarized task category and perception-decision mapping rules, and quickly initializes them to adapt to new tasks; The policy fine-tuning unit performs meta-learning adjustments to the parameter structure based on the current task feedback at the beginning of the new task.

9. The AI ​​interactive data processing system based on multimodal perception and dynamic decision-making according to claim 8 is characterized in that: The meta-learning adjustment of the parameter structure according to the current task feedback includes: Real-time monitoring of system performance indicators during the current task execution process, including perception accuracy, response latency, and task completion rate, and evaluation of decision explainability and safety indicators; Compare and analyze performance monitoring results with standard performance in historical tasks to determine whether there are performance degradation or abnormal response issues, which will serve as a basis for determining whether to optimize the strategy; If the judgment result shows that the system performance does not reach the set threshold, the strategy parameters of the multimodal perception module and the causal decision module are quickly adaptively adjusted through meta-learning.

10. An AI interactive data processing method based on multimodal perception and dynamic decision-making, applied to an AI interactive data processing system based on multimodal perception and dynamic decision-making according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Collect environmental data through multi-source sensors, perform denoising, normalization, and structure extraction preprocessing on the collected environmental data, and combine cross-modal consistency learning mechanism and noise robustness enhancement technology to generate standardized data for each modality; S2. Based on the standardized data of each modality, feature dimension alignment and time synchronization are performed. By introducing a context-aware attention mechanism and a multi-level feature alignment network, a perceptual feature representation with semantic association is constructed. S3. Based on the semantically related perceptual feature representation, we construct an environment-action causal graph and use a graph neural network combined with a causal reasoning algorithm to generate decision-making actions to respond to environmental changes. S4. Based on the generated decision actions, a simulation environment is constructed and formal verification methods are used to pre-evaluate the potential consequences of the decision actions, identify safety hazards in high-risk scenarios, and generate alternative decisions or warning information; S5. Based on decision-making actions and collected real-time environmental data, cross-domain transfer learning and domain adaptation technologies are used to intelligently adjust the perception area, sensor gain, or enhance the sampling frequency of the perception channel to optimize the perception strategy in a changing environment. S6. Based on the adjustment results of the causal decision module and the perception calibration module, the system quickly adapts to new environments or task requirements by learning from historical tasks, and continuously optimizes perception and decision-making strategies.

Images