Intelligent agent autonomous decision control method based on multi-modal data fusion

Abstract

The invention discloses an agent autonomous decision control method based on multi-modal data fusion. The method comprises the following steps: S1, synchronously collecting multi-source heterogeneous data; s2, dynamic weight adaptive fusion is carried out; s3, generating a task-driven decision; and S4, performing autonomous decision closed-loop optimization. According to the method, through dynamic weight distribution and space-time correlation modeling, the problems of heterogeneity and environment adaptation in multi-modal data fusion are solved; furthermore, a risk-sensitive reinforcement learning framework and a closed-loop feedback mechanism are combined, so that full-link cooperative control from data fusion, strategy generation to optimization execution is realized. In the mechanism level, the method breaks through the limitations of static fusion, single-target optimization and offline training, can adapt to a dynamic environment, ensures that the intelligent agent is in a complex scene such as noise interference, illumination abrupt change and task emergency switching, and meets the requirements of decision-making efficiency, safety and environment robustness at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous decision-making, and more specifically, relates to an intelligent agent autonomous decision-making control method based on multimodal data fusion. Background Art

[0002] Current autonomous decision-making systems for intelligent agents often rely on single-modal data, such as vision or speech, and face the following technical challenges at the mechanism level in complex dynamic environments:

[0003] Lack of synergy between multimodal heterogeneous data: Differences in the spatiotemporal characteristics of visual, speech, and text data lead to synchronization difficulties. Static fusion strategies struggle to adapt to environmental disturbances such as sudden changes in illumination and noise interference, and the fused semantic representations fail to accurately reflect the spatiotemporal correlations across modalities.

[0004] Insufficient dynamic adaptation between scenarios and modalities: Traditional methods use fixed weight allocation rules and are unable to dynamically adjust the contribution of each modality based on scenario attributes such as task urgency and environmental interference intensity. This results in low-confidence data interfering with decision-making in high-noise scenarios.

[0005] Lack of decision-making risk modeling: Existing reinforcement learning models typically focus on maximizing expected returns, without explicitly modeling environmental risks such as trajectory deviation and resource overruns. Consequently, the strategy generation process may overlook key safety hazards.

[0006] Closed-loop optimization lag: The architecture that separates offline training and online execution makes it difficult for feedback signals to act on weight allocation and model parameter adjustment in a timely manner. This makes it difficult for the system to respond to sudden environmental changes such as sensor failure or dynamic switching of mission objectives.

[0007] Based on the above, we propose an agent autonomous decision-making control method based on multimodal data fusion. Summary of the Invention

[0008] The purpose of the present invention is to address the shortcomings of the existing technology and propose an intelligent agent autonomous decision-making control method based on multimodal data fusion. Through dynamic weight allocation and spatiotemporal correlation modeling, it solves the problems of heterogeneity and environmental adaptation in multimodal data fusion; further combined with the risk-sensitive reinforcement learning framework and closed-loop feedback mechanism, it realizes full-link collaborative control from data fusion, strategy generation to execution optimization.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] An intelligent agent autonomous decision-making control method based on multimodal data fusion includes the following steps:

[0011] S1. Synchronous acquisition of multi-source heterogeneous data. Through the sensor group and communication module configured by the intelligent agent, real-time collection of visual modal data for environmental perception, voice modal data, and contextual text modal data for the target task is carried out. The data of each modality is then timestamp aligned and denoised.

[0012] S2, dynamic weighted adaptive fusion, dynamically assigns fusion weight coefficients to each modality based on the confidence index of multimodal data and the decision-making requirements of the current scenario, and generates a joint semantic representation vector containing spatiotemporal correlation features;

[0013] S3, task-driven decision generation, calculates the action value function of candidate decisions based on the joint semantic representation vector and the preset task goal through a multi-level reinforcement learning model, and selects the decision strategy with the maximum comprehensive expected benefit;

[0014] S4, autonomous decision-making closed-loop optimization, after executing the selected strategy, combined with the goal achievement feedback and environmental state changes, dynamically adjust the weight distribution rules in step S2 and the reward function parameters of the reinforcement learning model in step S3.

[0015] Preferably, in step S2, the confidence index includes an image clarity score of the visual modality, a signal-to-noise ratio of the speech modality, and a semantic integrity coefficient of the text modality;

[0016] During the dynamic weight allocation process, a scene classifier is introduced to identify the light intensity, ambient sound pressure level, and task urgency labels in the current environmental attributes. Based on this, a weight coefficient calculation matrix is constructed. The matrix nonlinearly combines the confidence index with the scene attributes to generate weight coefficients for each modality that adapt to the environment. The semantic representation vector is combined with the graph convolutional network to model the cross-modal association of multimodal features.

[0017] Preferably, the weight coefficient calculation matrix in step S2 is defined as:

[0018] ,in, is the fusion weight of the mth mode, For the The confidence index of the modal, is the mth modality adaptation score output by the scene classifier, α is the learnable temperature coefficient, is the confidence index of the nth mode, n is used as the denominator to traverse all modes, The scene adaptation score of the nth modality is dynamically calculated by the scene classifier, reflecting the applicability of the modality to the current environmental conditions;

[0019] The joint semantic representation vector encodes cross-modal correlation information through the following formula:

[0020] ;in, is a unimodal eigenvector, is the adjacency matrix of the graph convolutional network, Represents tensor multiplication;

[0021] The above-mentioned exponential weight distribution enhances inter-modal complementarity, and graph convolution fusion preserves cross-modal topological relationships, significantly improving the accuracy of semantic representation in dynamic environments.

[0022] Preferably, in step S3, the action value function is defined as:

[0023] ;

[0024] Where, It represents the immediate reward value obtained by the agent after executing action a in state s, which is used to directly feedback the short-term benefits of the current decision; γ represents the discount factor, which is used to balance the contribution weight of the current reward and future long-term benefits; is the estimated value of environmental risk; λ is the risk sensitivity coefficient, which is adaptively optimized through the reinforcement learning model to adjust the impact of environmental risk on long-term value assessment; The Q value output by the target network is calculated by the target network of the multi-level reinforcement learning model to avoid overestimation of the Q value; stands for expected value, an operator used in probability theory to calculate the average value of a random variable. P( ) represents the calculation condition of expected value;

[0025] Preferably, the triggering condition for the policy re-planning is:

[0026] ,in, is the characteristic vector of the actual target trajectory, The theoretical expected trajectory set for the task, The deviation tolerance threshold is set. When the actual trajectory deviation exceeds this threshold, the strategy is triggered to re-plan to correct the action candidate pool.

[0027] Formula, introduction The environmental risk estimation is embedded in the long-term benefit calculation of Q value, so that the intelligent agent can actively avoid high-risk scenarios while pursuing high rewards. The trigger mechanism β constraint based on trajectory deviation is used to realize real-time update of the policy space. By adjusting γ and λ, the intelligent agent can flexibly adapt to the needs of different scenarios.

[0028] Preferably, in step S3, the multi-level reinforcement learning model includes a strategy planning layer and a behavior execution layer;

[0029] The strategy planning layer uses the deep Q-network (DQN) algorithm to calculate the long-term benefit values of different decision sequences. The behavior execution layer generates executable action instructions based on the proximal policy optimization (PPO) algorithm. The calculation of the action value function is synchronously integrated with the real-time environmental risk estimation value. When it is detected that the target trajectory deviates from the expected threshold, the strategy is triggered to re-plan and update the action candidate pool.

[0030] Preferably, in step S4, the goal achievement feedback includes task completion time deviation, resource consumption rate and decision safety score, and the environmental state change is achieved by real-time monitoring of the frequency and intensity of external interference events;

[0031] Adjusting the weight distribution rule includes the following process: if the confidence of the speech modality is continuously lower than the preset threshold and the ambient sound pressure level exceeds the threshold, its weight coefficient is reduced and the visual modality redundancy verification mechanism is activated; the adjustment of the reward function parameters is based on an incremental gradient descent algorithm to ensure that the model maintains policy stability in a dynamic environment.

[0032] Preferably, steps S1 to S4 further include the following enhanced functions:

[0033] In the fusion process of step S2, an adversarial training mechanism is introduced. By generating adversarial networks (GANs) to simulate abnormal feature samples in scenarios of sensor failure or data pollution, the robustness of the joint semantic representation vector to noise is improved. In the decision generation stage of step S3, a real-time causal reasoning module is embedded to analyze the causal relationship chain between the decision sequence and the task goal, thereby eliminating candidate strategies with logical conflicts.

[0034] Preferably, the method further sets an emergency decision-making mode, and when step S4 detects that the environmental mutation index exceeds a critical value, the following process is started:

[0035] The dynamic weight allocation of step S2 is frozen and switched to the preset modal priority. At the same time, a Monte Carlo tree search algorithm is used to generate a rapid response strategy in step S3. The execution result of the strategy is fed back to step S4 to evaluate the effectiveness of the emergency mode, and the weight allocation rules are iteratively optimized based on historical emergency data.

[0036] Technical effects and advantages of the present invention: Compared with the prior art, the present invention provides an intelligent agent autonomous decision-making control method based on multimodal data fusion, which has the following effects:

[0037] Dynamic modeling capabilities for cross-modal spatiotemporal associations. Through the fusion of timestamp alignment and graph convolutional networks, the joint semantic representation vector can preserve the spatiotemporal topological relationships of multimodal data, resolving semantic distortion issues caused by temporal misalignment or modal fragmentation in traditional methods.

[0038] Scenario-driven weight adaptability, combined with confidence metrics and a dynamic weight allocation mechanism for scene classifiers, enables the system to prioritize voice modality in low-light conditions and enhance text command parsing when tasks are urgent, achieving on-demand optimization of modality resources.

[0039] A risk-sensitive benefit assessment mechanism introduces an objective function for environmental risk estimation and a trajectory deviation trigger condition, embedding risk aversion into the long-term benefit prediction of reinforcement learning to avoid the generation of high-risk decision-making strategies.

[0040] Feedback-driven real-time optimization capabilities, through closed-loop optimization of incremental parameter adjustments and emergency decision-making mode switching, enable the system to quickly perceive sudden changes in the environment and reconstruct decision-making logic, significantly improving the robustness of strategies in dynamic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the intelligent agent autonomous decision-making control method based on multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] The present invention provides Figure 1 The paper presents a method for autonomous decision-making and control of intelligent agents based on multimodal data fusion. This method addresses the heterogeneity and environmental adaptation issues in multimodal data fusion through dynamic weight allocation and spatiotemporal correlation modeling. Furthermore, it combines a risk-sensitive reinforcement learning framework with a closed-loop feedback mechanism to achieve full-link collaborative control, from data fusion and strategy generation to execution optimization. At the mechanistic level, this method overcomes the limitations of static fusion, single-objective optimization, and offline training, enabling adaptive dynamic environments and ensuring the agent's ability to operate effectively in complex scenarios such as noise interference, sudden changes in illumination, and urgent task switching, while also meeting the requirements for decision-making efficiency, safety, and environmental robustness.

[0044] The method comprises the following steps:

[0045] S1. Synchronous acquisition of multi-source heterogeneous data: The agent's sensor suite and communication modules collect real-time visual and audio data for environmental perception, as well as textual data for the target task context. The data is then timestamped and pre-processed for denoising.

[0046] In step S1, the sensor group includes a binocular vision sensor, a lidar, and a microphone array. The visual modality data is subjected to an image semantic segmentation network to extract pixel-level semantic labels. The speech modality data is converted into text after voiceprint feature extraction and background noise suppression. The contextual text modality data is parsed into structured task instructions through a natural language processing model. Preprocessing includes dynamic noise filtering based on Kalman filtering and sampling rate normalization of the multimodal data stream using a time sliding window algorithm, so that the data of each modality meets the synchronization constraints of joint modeling in the spatiotemporal dimensions.

[0047] S2. Dynamic Weighted Adaptive Fusion: Based on the confidence level of multimodal data and the decision-making requirements of the current scenario, the fusion weight coefficients of each modality are dynamically assigned to generate a joint semantic representation vector that includes spatiotemporal correlation features.

[0048] In step S2, the confidence indicators include the image clarity score of the visual modality, the signal-to-noise ratio of the speech modality, and the semantic integrity coefficient of the text modality. During the dynamic weight allocation process, a scene classifier is introduced to identify the light intensity, ambient sound pressure level, and task urgency label in the current environmental attributes. Based on this, a weight coefficient calculation matrix is constructed. The matrix nonlinearly combines the confidence indicators with the scene attributes to generate weight coefficients for each modality that are adaptive to the environment. The cross-modal association of multimodal features is modeled through a graph convolutional network combined with the semantic representation vector.

[0049] The weight coefficient calculation matrix in step S2 is defined as:

[0050] ,in, is the fusion weight of the mth mode, For the The confidence index of the modal, is the mth modality adaptation score output by the scene classifier, α is the learnable temperature coefficient, is the confidence index of the nth mode, n is used as the denominator to traverse all modes, The scene adaptation score of the nth modality is dynamically calculated by the scene classifier, reflecting the applicability of the modality to the current environmental conditions;

[0051] The joint semantic representation vector encodes cross-modal correlation information through the following formula:

[0052] ;in, is a unimodal eigenvector, is the adjacency matrix of the graph convolutional network, Represents tensor multiplication;

[0053] The above-mentioned exponential weight distribution enhances inter-modal complementarity, and graph convolution fusion preserves cross-modal topological relationships, significantly improving the accuracy of semantic representation in dynamic environments.

[0054] S3. Task-driven decision generation: Based on the joint semantic representation vector and the preset task goal, a multi-level reinforcement learning model is used to calculate the action-value function of candidate decisions and select the decision strategy with the highest overall expected benefit.

[0055] In step S3, the multi-level reinforcement learning model includes a strategy planning layer and a behavior execution layer. The strategy planning layer uses the Deep Q-Network (DQN) algorithm to calculate the long-term benefits of different decision sequences. The behavior execution layer generates executable action instructions based on the Proximal Policy Optimization (PPO) algorithm. The calculation of the action value function is synchronously integrated with the real-time environmental risk estimate. When the target trajectory is detected to deviate from the expected threshold, the strategy is re-planned and the action candidate pool is updated.

[0056] In step S3, the action-value function is defined as:

[0057] ;

[0058] Where, It represents the immediate reward value obtained by the agent after performing action a in state s, which is used to directly feedback the short-term benefits of the current decision; γ represents the discount factor, which is used to balance the contribution weight of the current reward and future long-term benefits; is the estimated value of environmental risk;

[0059] λ is the risk sensitivity coefficient, which is adaptively optimized through the reinforcement learning model to adjust the impact of environmental risks on long-term value assessment; The Q value output by the target network is calculated by the target network of the multi-level reinforcement learning model to avoid overestimation of the Q value;

[0060] stands for expected value, an operator used in probability theory to calculate the average value of a random variable. P( ) represents the calculation condition of expected value;

[0061] The trigger conditions for policy replanning are:

[0062] ,in, is the characteristic vector of the actual target trajectory, The theoretical expected trajectory set for the task, The deviation tolerance threshold is set. When the actual trajectory deviation exceeds this threshold, the strategy is triggered to re-plan to correct the action candidate pool.

[0063] Formula, introduction The environmental risk estimation is embedded in the long-term benefit calculation of Q value, so that the intelligent agent can actively avoid high-risk scenarios while pursuing high rewards. The trigger mechanism β constraint based on trajectory deviation is used to realize real-time update of the strategy space. By adjusting γ and λ, the intelligent agent can flexibly adapt to the needs of different scenarios.

[0064] S4. Autonomous Decision-Making Closed-Loop Optimization: After executing the selected strategy, the weight allocation rules in step S2 and the reward function parameters of the reinforcement learning model in step S3 are dynamically adjusted based on feedback on goal achievement and changes in the environment.

[0065] In step S4, goal achievement feedback includes task completion time deviation, resource consumption rate, and decision safety score. Environmental state changes are monitored by real-time monitoring of the frequency and intensity of external interference events. The weight allocation rule is adjusted by the following process: if the confidence level of the speech modality is continuously lower than the preset threshold and the ambient sound pressure level exceeds the threshold, its weight coefficient is reduced and the visual modality redundancy verification mechanism is activated. The reward function parameters are adjusted based on the incremental gradient descent algorithm to ensure that the model maintains policy stability in a dynamic environment.

[0066] The adjustment formula of the reward function parameters in step S4 is:

[0067] ;in, is the reward function parameter, is the learning rate, is the task completion loss function, scoring decision safety;

[0068] The adjustment constraints of the weight distribution rule are:

[0069] , where is the indicator function, τ is the confidence threshold, and ϵ is the weight failure tolerance coefficient;

[0070] The above effects are: the gradient update process couples task efficiency and safety constraints, while limiting the negative impact of low-confidence data through failure mode detection.

[0071] In addition, steps S1 to S4 also include the following enhanced functions: in the fusion process of step S2,

[0072] An adversarial training mechanism is introduced to simulate abnormal feature samples in scenarios such as sensor failure or data contamination through a generative adversarial network (GAN), thereby improving the robustness of the joint semantic representation vector to noise. A real-time causal reasoning module is embedded in the decision generation phase of step S3 to analyze the causal relationship chain between the decision sequence and the task goal, thereby eliminating candidate strategies with logical conflicts.

[0073] Adversarial training mechanism against data perturbations:

[0074] In the fusion process of step S2, abnormal samples containing sensor failure and data pollution are actively generated through the generative adversarial network (GAN) and injected into the multimodal training dataset;

[0075] During training, the discriminator network learns to distinguish between real data and adversarial examples, while the generator network continuously optimizes to simulate more invisible abnormal features, driving the joint semantic representation vector to develop generalized encoding capabilities against noise interference.

[0076] As a result, when the intelligent agent encounters extreme noise in real scenarios, it can still extract reliable cross-modal features and avoid decision failure caused by semantic representation collapse.

[0077] The causal reasoning module ensures logical consistency:

[0078] In the decision generation phase of step S3, the causal reasoning module constructs a directed acyclic graph (DAG) between decision actions and task goals, and analyzes whether the causal chain of candidate strategies meets the preset logical constraints;

[0079] For candidate strategies with logical conflicts, the module automatically identifies their causal breakpoints and performs confidence reduction processing to ensure that the final execution strategy is self-consistent at the causal level;

[0080] This mechanism is particularly suitable for complex tasks, avoiding the risk of scene violations caused by incorrect action timing;

[0081] As mentioned above, through the dual enhancement of adversarial training and causal reasoning, the two core vulnerability problems of multimodal systems in data pollution and logical conflict scenarios are solved, filling the technical gap of traditional methods in semantic representation degradation and strategy contradictions under complex interference.

[0082] The method further sets an emergency decision-making mode. When step S4 detects that the environmental mutation index exceeds a critical value, the following process is initiated: the dynamic weight allocation in step S2 is frozen and switched to the preset modal priority. At the same time, a Monte Carlo tree search algorithm is used to generate a rapid response strategy in step S3. The execution result of the strategy is fed back to step S4 to evaluate the effectiveness of the emergency mode, and the weight allocation rule is iteratively optimized based on historical emergency data.

[0083] Step S3 switches to the Monte Carlo tree search algorithm, which generates candidate strategies by simulating high-probability action branches in parallel, skipping the long-term benefit calculation step in the conventional reinforcement learning model and reducing response delay;

[0084] The results of the executed strategy are fed back to the evaluation module in step S4, which generates an emergency mode effectiveness score based on time efficiency and safety redundancy indicators, driving the incremental optimization of the weight allocation rule;

[0085] If the visual sensor frequently fails due to smoke interference during multiple emergency modes, the radar mode's weight in the priority table will be automatically increased, forming a self-evolving capability for emergency scenarios.

[0086] The above-mentioned mechanism proposes a hierarchical response mechanism for environmental mutations, which breaks through the response delay bottleneck caused by the dynamic calculation of the existing system in emergency scenarios. Through the cascade design of preset priorities and fast search algorithms, it realizes survivability guarantee and continuous optimization capabilities under extreme conditions.

[0087] In summary, compared with the prior art, the present invention has the following effects:

[0088] Dynamic modeling capabilities for cross-modal spatiotemporal associations. Through the fusion of timestamp alignment and graph convolutional networks, the joint semantic representation vector can preserve the spatiotemporal topological relationships of multimodal data, resolving semantic distortion issues caused by temporal misalignment or modal fragmentation in traditional methods.

[0089] Scenario-driven weight adaptability, combined with confidence metrics and a dynamic weight allocation mechanism for scene classifiers, enables the system to prioritize voice modality in low-light conditions and enhance text command parsing when tasks are urgent, achieving on-demand optimization of modality resources.

[0090] A risk-sensitive benefit assessment mechanism introduces an objective function for environmental risk estimation and a trajectory deviation trigger condition, embedding risk aversion into the long-term benefit prediction of reinforcement learning to avoid the generation of high-risk decision-making strategies.

[0091] Feedback-driven real-time optimization capabilities, through closed-loop optimization of incremental parameter adjustments and emergency decision-making mode switching, enable the system to quickly perceive sudden changes in the environment and reconstruct decision-making logic, significantly improving the robustness of strategies in dynamic scenarios.

[0092] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An agent autonomous decision-making control method based on multimodal data fusion, characterized in that: The steps include: S1. Synchronous acquisition of multi-source heterogeneous data. Through the sensor group and communication module configured by the intelligent agent, real-time collection of visual modal data for environmental perception, voice modal data, and contextual text modal data for the target task is carried out. The data of each modality is then timestamp aligned and denoised. S2, dynamic weighted adaptive fusion, dynamically assigns fusion weight coefficients to each modality based on the confidence index of multimodal data and the decision-making requirements of the current scenario, and generates a joint semantic representation vector containing spatiotemporal correlation features; S3, task-driven decision generation, calculates the action value function of candidate decisions based on the joint semantic representation vector and the preset task goal through a multi-level reinforcement learning model, and selects the decision strategy with the maximum comprehensive expected benefit; S4, autonomous decision-making closed-loop optimization, after executing the selected strategy, combined with the goal achievement feedback and environmental state changes, dynamically adjust the weight distribution rules in step S2 and the reward function parameters of the reinforcement learning model in step S3.

2. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: In step S2, the confidence indicators include the image clarity score of the visual modality, the signal-to-noise ratio of the speech modality, and the semantic integrity coefficient of the text modality; During the dynamic weight allocation process, a scene classifier is introduced to identify the light intensity, ambient sound pressure level, and task urgency labels in the current environmental attributes. Based on this, a weight coefficient calculation matrix is constructed. The matrix nonlinearly combines the confidence index with the scene attributes to generate weight coefficients for each modality that adapt to the environment. The semantic representation vector is combined with the graph convolutional network to model the cross-modal association of multimodal features.

3. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 2, characterized in that: The weight coefficient calculation matrix in step S2 is defined as: ,in, is the fusion weight of the mth mode, For the The confidence index of the modal, is the mth modality adaptation score output by the scene classifier, α is the learnable temperature coefficient, is the confidence index of the nth mode, n is used as the denominator to traverse all modes, The scene adaptation score of the nth modality is dynamically calculated by the scene classifier, reflecting the applicability of the modality to the current environmental conditions; The joint semantic representation vector encodes cross-modal correlation information through the following formula: ;in, is a unimodal eigenvector, is the adjacency matrix of the graph convolutional network, Represents tensor multiplication; The above-mentioned exponential weight distribution enhances inter-modal complementarity, and graph convolution fusion preserves cross-modal topological relationships, significantly improving the accuracy of semantic representation in dynamic environments.

4. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: In step S3, the action-value function is defined as: ； Where, It represents the immediate reward value obtained by the agent after executing action a in state s, which is used to directly feedback the short-term benefits of the current decision; γ represents the discount factor, which is used to balance the contribution weight of the current reward and future long-term benefits; is the estimated value of environmental risk; λ is the risk sensitivity coefficient, which is adaptively optimized through the reinforcement learning model to adjust the impact of environmental risks on long-term value assessment; The Q value output by the target network is calculated by the target network of the multi-level reinforcement learning model to avoid overestimation of the Q value; stands for expected value, an operator used in probability theory to calculate the average value of a random variable. P( ) represents the calculation condition of the expected value.

5. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 4, characterized in that: The triggering conditions for policy re-planning are: ,in, is the characteristic vector of the actual target trajectory, The theoretical expected trajectory set for the task, The deviation tolerance threshold is set. When the actual trajectory deviation exceeds this threshold, the strategy is triggered to re-plan to correct the action candidate pool. Formula, introduction The environmental risk estimation is embedded in the long-term benefit calculation of Q value, so that the intelligent agent can actively avoid high-risk scenarios while pursuing high rewards. The trigger mechanism β constraint based on trajectory deviation is used to realize real-time update of the strategy space. By adjusting γ and λ, the intelligent agent can flexibly adapt to the needs of different scenarios.

6. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: In step S3, the multi-level reinforcement learning model includes a strategy planning layer and a behavior execution layer; The strategy planning layer uses a deep Q-network algorithm to calculate the long-term benefit values of different decision sequences. The behavior execution layer generates executable action instructions based on a proximal policy optimization algorithm. The calculation of the action value function is synchronously integrated with the real-time environmental risk estimation value. When it is detected that the target trajectory deviates from the expected threshold, the strategy is triggered to re-plan and update the action candidate pool.

7. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: In step S4, the goal achievement feedback includes task completion time deviation, resource consumption rate and decision safety score, and the environmental state change is achieved by real-time monitoring of the frequency and intensity of external interference events; Adjusting the weight distribution rule includes the following process: if the confidence of the speech modality is continuously lower than the preset threshold and the ambient sound pressure level exceeds the threshold, its weight coefficient is reduced and the visual modality redundancy verification mechanism is activated; the adjustment of the reward function parameters is based on an incremental gradient descent algorithm to ensure that the model maintains policy stability in a dynamic environment.

8. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: The following enhancements are also included in steps S1 to S4: In the fusion process of step S2, an adversarial training mechanism is introduced to improve the robustness of the joint semantic representation vector to noise by generating an adversarial network to simulate abnormal feature samples in scenarios of sensor failure or data pollution. In the decision generation stage of step S3, a real-time causal reasoning module is embedded to analyze the causal relationship chain between the decision sequence and the task goal, thereby eliminating candidate strategies with logical conflicts.

9. The method for autonomous decision-making and control of an intelligent agent based on multimodal data fusion according to claim 1, characterized in that: The method further sets an emergency decision-making mode. When step S4 detects that the environmental mutation index exceeds a critical value, the following process is initiated: The dynamic weight allocation of step S2 is frozen and switched to the preset modal priority. At the same time, a Monte Carlo tree search algorithm is used to generate a rapid response strategy in step S3. The execution result of the strategy is fed back to step S4 to evaluate the effectiveness of the emergency mode, and the weight allocation rules are iteratively optimized based on historical emergency data.

Images