Unified decision-making and control method and system fused with multi-modal perception

Abstract

The embodiment of the invention provides a unified decision-making and control method and system fused with multi-modal perception, and relates to the technical field of intelligent agent multi-modal control technologies. The method comprises the following steps: acquiring multi-mode state information of an intelligent agent; constructing a generalized state space map based on the multi-modal state information; searching and generating a generalized optimal path in the generalized state space map based on a preset dynamic weighted unified cost function; and generating and executing a control instruction sequence according to the generalized optimal path. According to the invention, the problem of low risk decision-making precision of the intelligent agent is solved, and the effect of improving the risk decision-making precision is further achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of multi-modal control of agents, and in particular, to a unified decision and control method and system fusing multi-modal perception. BACKGROUND

[0002] With the rapid development of autonomous mobile robots, unmanned aerial vehicles and autonomous driving technologies, the ability of agents to perform tasks in increasingly complex dynamic environments has become critical. Its core capabilities usually rely on three interrelated but often independently processed technical pillars: navigation and positioning, communication, and safety control.

[0003] The prior art generally has the problems of "split decision" and "information island" when implementing these functions, that is, each algorithm module optimizes for its single goal independently, for example, the navigation module pursues the physically shortest path, the communication module pursues the strongest signal link, and the safety module only passively avoids physical obstacles. Such split decision mechanism often leads to suboptimal behavior of the agent, or even dangerous in some cases. For example, a physically shortest path may just pass through a communication signal blind area, causing the agent to lose contact at a critical moment, thereby causing unpredictable safety risks.

[0004] There is currently no good solution to the above problems. SUMMARY

[0005] Embodiments of the present application provide a unified decision and control method and system fusing multi-modal perception to at least solve the problem of low risk decision precision of agents in related technologies.

[0006] According to an embodiment of the present application, a unified decision and control method fusing multi-modal perception is provided, comprising: obtaining multi-modal state information of an agent, the multi-modal state information comprising physical state information, communication state information and safety state information; constructing a generalized state space map based on the multi-modal state information, wherein any position in the generalized state space map comprises a generalized cost, the generalized cost being used to indicate the comprehensive risk of performing a task at the position; searching and generating a generalized optimal path in the generalized state space map based on a preset dynamic weighted unified cost function, wherein the weight coefficients in the dynamic weighted unified cost function are dynamically adjusted according to the context of the current task; generating and executing a control instruction sequence according to the generalized optimal path.

[0007] In an exemplary embodiment, the obtaining of the multi-modal state information of the agent comprises: The physical state information, including the agent's position, attitude, and velocity, is obtained by fusing data from the Global Navigation Satellite System, Inertial Measurement Unit, and Instant Positioning and Mapping Algorithms. The communication status information is obtained by monitoring the link quality indicators of at least one communication link, and a real-time communication quality map covering the task area is generated. The safety status information is obtained by fusing environmental sensor data with the agent's own sensor data. The safety status information includes external environmental risks and the agent's internal health status.

[0008] In an exemplary embodiment, constructing a generalized state space map based on the multimodal state information includes: Build a basic environment map; The communication risk cost represented by the real-time communication quality map, the location confidence cost represented by the physical state information, and the environmental security cost represented by the security state information are attached as attributes to the nodes or edges of the basic environment map to obtain the generalized state space map.

[0009] In one exemplary embodiment, generating and executing a sequence of control instructions based on the generalized optimal path includes: The generalized optimal path is parsed into a sequence containing behavioral instructions, which include path point switching instructions, communication mode switching instructions, and speed adjustment instructions. The behavioral instructions are converted into control instructions for the underlying hardware and sent to the corresponding actuators. The execution status of the actuator is fed back in real time.

[0010] According to another embodiment of the present invention, a unified decision and control system integrating multimodal sensing is provided, comprising: A multimodal fusion perception module is used to acquire multimodal state information of an intelligent agent, including physical state information, communication state information, and security state information; The situational awareness and risk quantification module is used to construct a generalized state space map based on the multimodal state information, wherein any location in the generalized state space map includes a generalized cost, which is used to indicate the comprehensive risk of performing a task at that location; A unified decision-making and planning engine is used to search for and generate a generalized optimal path in the generalized state space map based on a preset dynamic weighted unified cost function, wherein the weight coefficients in the dynamic weighted unified cost function are dynamically adjusted according to the context of the current task. The closed-loop control and execution module is used to generate and execute a sequence of control instructions based on the generalized optimal path.

[0011] In one exemplary embodiment, obtaining the multimodal state information of the agent includes: The physical state information, including the agent's position, attitude, and velocity, is obtained by fusing data from the Global Navigation Satellite System, Inertial Measurement Unit, and Instant Positioning and Mapping Algorithms. The communication status information is obtained by monitoring the link quality indicators of at least one communication link, and a real-time communication quality map covering the task area is generated. The safety status information is obtained by fusing environmental sensor data with the agent's own sensor data. The safety status information includes external environmental risks and the agent's internal health status.

[0012] In an exemplary embodiment, constructing a generalized state space map based on the multimodal state information includes: Build a basic environment map; The communication risk cost represented by the real-time communication quality map, the location confidence cost represented by the physical state information, and the environmental security cost represented by the security state information are attached as attributes to the nodes or edges of the basic environment map to obtain the generalized state space map.

[0013] In one exemplary embodiment, generating and executing a sequence of control instructions based on the generalized optimal path includes: The generalized optimal path is parsed into a sequence containing behavioral instructions, which include path point switching instructions, communication mode switching instructions, and speed adjustment instructions. The behavioral instructions are converted into control instructions for the underlying hardware and sent to the corresponding actuators. The execution status of the actuator is fed back in real time.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0015] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] This invention expands the operational space of intelligent agents from a single physical space to a "generalized state space" encompassing multiple dimensions such as physics, communication, and security. Within this space, it performs globally optimal path planning and behavioral decisions based on a dynamically weighted unified cost function. Therefore, it can solve the problem of low risk decision-making accuracy caused by "decision fragmentation" and "information silos" in existing technologies, thereby improving the accuracy of risk decision-making, enhancing the accuracy and efficiency of path planning, and improving the survivability, task execution efficiency, and overall security of intelligent agents in complex, dynamic, and adversarial environments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a unified decision-making and control method that integrates multimodal perception according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a unified decision and control system that integrates multimodal perception according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of a specific embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0019] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0021] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0022] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0023] In one specific implementation of this application, a unified decision-making and control system integrating multimodal perception is provided. In this implementation, the system employs the collaborative work of a multimodal fusion perception module, a situational awareness and risk quantification module, a unified decision-making and planning engine, and a closed-loop control and execution module. This enables the system to plan globally optimal action sequences for the agent within a generalized state space encompassing multiple dimensions, including physical, communication, and security dimensions. This system solves the technical problem in existing technologies where the fragmented decision-making processes of navigation, communication, and security control modules lead to suboptimal or even dangerous behavior. It achieves the beneficial effect of significantly improving the agent's autonomous operation capability, task success rate, and safety in complex dynamic environments.

[0024] Example 1 Reference Figures 1-2 This diagram illustrates the structural block diagram of a unified decision-making and control system integrating multimodal perception, according to an embodiment of this application. The system can be implemented on intelligent agents such as autonomous mobile robots, drones, or autonomous vehicles.

[0025] The system includes: a multimodal fusion perception module 110, a situational awareness and risk quantification module 120, a unified decision-making and planning engine 130, and a closed-loop control and execution module 140, wherein: The multimodal fusion perception module 110 is responsible for collecting and fusing multi-dimensional information of the intelligent agent in the generalized state space in real time, providing a data foundation for upper-level decision-making; preferably, the module 110 can be further divided into a physical state perception unit, a communication state perception unit and a security state perception unit, but this is not limited here.

[0026] The Situation Awareness and Risk Quantification Module 120 is responsible for transforming heterogeneous data from the perception layer into unified quantitative risk indicators. Specifically, it constructs and maintains a dynamic generalized state space map and expresses various potential risks in the form of "costs" on the map.

[0027] The unified decision-making and planning engine 130 performs multi-objective intelligent decision-making and planning on a generalized state space map constructed by the situational awareness and risk quantification module 120. This engine does not pursue optimization in any single dimension, but rather seeks the optimal solution for global dynamic equilibrium.

[0028] The closed-loop control and execution module 140 is responsible for parsing the generated abstract behavior sequence into specific control instructions that can be executed by the underlying hardware (such as motors and communication modules), and providing real-time feedback on the execution status to form a complete "perception-decision-control" closed loop in order to cope with sudden changes in the environment.

[0029] The following will combine Figure 2 The flowchart shown illustrates in detail a unified decision-making and control method integrating multimodal perception provided in this application embodiment.

[0030] Step S100: Obtain the multimodal state information of the agent, which includes physical state information, communication state information and security state information.

[0031] This step involves capturing all relevant information about the agent itself and its environment. This information is multimodal and heterogeneous, covering everything from macroscopic geographical location to microscopic link quality and potential security threats. Specifically, it includes the following sub-steps: S110: Obtain physical state information.

[0032] In this embodiment, the physical state sensing unit fuses measurement data from different sensors in a tightly coupled manner to output a high-precision, high-frequency, and highly robust physical pose (position, attitude, velocity) estimate; wherein, the sensor set may include a Global Navigation Satellite System (GNSS) receiver, an inertial measurement unit (IMU), and a lidar (LiDAR) or visual camera.

[0033] To achieve high-precision fusion, the system employs an extended Kalman filter or a more advanced factor graph optimization algorithm, and the agent's state vector... It can be defined as ,in It is a three-dimensional position vector. It is a three-dimensional velocity vector. It is a quaternion representing attitude. and These are the zero bias values ​​of the accelerometer and gyroscope, respectively.

[0034] The IMU provides high-frequency acceleration and angular velocity measurements for state propagation (i.e., the prediction step) between two GNSS update cycles. When new GNSS position measurements or pose estimates from the SLAM algorithm are received, the system performs an update step, using the residuals between the measurements and predictions to correct the state estimate and reduce its covariance matrix. To mitigate the limitations of a single sensor; for example, an IMU can compensate for positioning interruptions in areas where GNSS signals are blocked, while GNSS can correct long-term drift of the IMU.

[0035] For example, suppose in At time t, the agent's state is estimated as Its covariance is ;from arrive At time t, the acceleration measured by the IMU is angular velocity is .

[0036] At this point, the prior state estimate is first calculated based on the kinematic equations. Its predicted location is: in It is composed of quaternions The resulting rotation matrix It is a time interval.

[0037] At the same time, the covariance matrix is ​​also propagated: in It is the Jacobian matrix of the state transition matrix. It is the process noise covariance.

[0038] exist At that moment, the GNSS receiver provided a position measurement value. Its measurement noise covariance is Then the Kalman gain is calculated: in It is the Jacobian matrix of the measurement matrix.

[0039] Then, update the state estimate: And update the covariance: Through this iterative process, the system can continuously output a physical state flow with ever-increasing confidence.

[0040] The final output physical state information includes not only the point estimate of the pose, but also a quantitative representation of its uncertainty (i.e., the covariance matrix). ); where the elements on the diagonal of the covariance matrix are... The square root of the value directly reflects the standard deviation of the current positioning along each axis.

[0041] S120: Obtain communication status information.

[0042] The communication state awareness unit is responsible for monitoring all available communication links in parallel, including but not limited to 5G / 4G cellular networks, Wi-Fi networks, and ad-hoc networks, and establishing a time-series database of key quality indicators for each link; these indicators constitute a multi-dimensional communication quality vector, for example: .

[0043] Among them, the Received Signal Strength Indicator (RSSI) reflects the absolute strength of the signal, usually measured in dBm; the Reference Signal Received Quality (RSRQ) considers both signal strength and interference level, and is a comprehensive indicator; the Signal to Interference plus Noise Ratio (SINR) is directly related to the reliability of data transmission; and latency and bandwidth determine the real-time performance and data throughput of communication.

[0044] Subsequently, based on these real-time acquired, discrete measurement points, a real-time communication quality map covering the entire task area is dynamically generated and maintained. To generate a continuous map from sparse measurement points, the system employs a spatial interpolation algorithm, utilizing spatial autocorrelation (i.e., neighboring locations are more likely to have similar communication quality) to estimate the communication quality of unmeasured locations. Simultaneously, the system combines historical data to form a priori communication quality map. When the agent enters a new area, it can first use the priori map for preliminary evaluation, and then continuously update it using real-time measurement data via Bayesian methods to obtain a more accurate real-time map.

[0045] For example, suppose the task area is divided into The grid. The agent is in position. A set of 5G signal quality data was measured. The system associates this data with the corresponding raster. For a raster with no measurement data Its communication quality vector It can be estimated using the inverse distance weighted method: in, It's weight. It is a grid and The Euclidean distance at the center It is an exponential parameter, usually taken as 2.

[0046] By performing calculations on all unmeasured grids, a continuous communication quality map is obtained. This map is not static, but dynamically updated as new measurement data arrives, reflecting the real-time impact of environmental changes (such as obstacle movement) and network load variations on communication quality.

[0047] S130: Obtain security status information.

[0048] The safety status perception unit is responsible for assessing various risks from the external environment and the agent's internal environment. External risk identification mainly relies on environmental sensors. For example, LiDAR is used to scan the surrounding environment, generating point cloud data, and clustering and segmentation algorithms are used to identify static obstacles (such as walls and trees) and dynamic obstacles (such as pedestrians and vehicles). Cameras are used to collect image data, and deep learning-based target detection algorithms (such as YOLOv-series) are used to identify specific categories of obstacles. Binocular vision or monocular depth estimation algorithms can be used to obtain their distance information. Radar has better robustness in adverse weather conditions (such as rain and fog). After fusing these sensor data, a dynamic list of obstacles is generated, in which each obstacle contains information such as position, speed, size, and predicted trajectory.

[0049] Internal risk monitoring is achieved by monitoring onboard sensors. For example, monitoring the voltage, current, and temperature reported by the battery management system (BMS) to assess the battery's state of health (SoH) and remaining charge (SoC), and monitoring the temperature and current of the motor driver to provide early warnings of overheating or overload risks. These internal state parameters constitute the agent's health vector. And so on.

[0050] For example, a lidar sensor returns 10,000 point clouds in a single scan frame. The system first removes ground points using a ground segmentation algorithm, then performs Euclidean distance clustering on the remaining point cloud to identify three independent point clusters. Subsequently, it calculates the bounding box for each point cluster to obtain its position and size in the agent's coordinate system. Assuming that one point cluster is continuously observed across several frames and its position undergoes systematic changes, the system tracks this point cluster using a Kalman filter, estimating its velocity. This information is then categorized as dynamic obstacles. Simultaneously, the BMS reports the current battery SoC at 35%. This information will be integrated for subsequent calculations of environmental safety and internal risk costs; for example, the predicted trajectory of a dynamic obstacle will generate a high-cost area, while a SoC below 40% may increase the cost of all actions, encouraging the agent to adopt more energy-efficient strategies or plan a return trip, and so on.

[0051] Step S200: Based on the multimodal state information, construct a generalized state space map, wherein any location in the generalized state space map is assigned a quantified generalized cost, and the generalized cost represents the comprehensive risk of performing a task at that location.

[0052] This step involves transforming the heterogeneous multimodal information obtained into a generalized state space map to determine a multidimensional decision space. Specifically, it includes the following sub-steps: S210: Generalized state-space modeling.

[0053] Generalized state-space maps can be represented using a rasterization method, where the environment is divided into uniform two-dimensional or three-dimensional grids. Each grid or node in the map... In addition to traditional physical attributes (such as whether it is occupied by obstacles), it is also assigned a series of cost attributes calculated by the perception layer to form a generalized cost vector: in: Communication risk costs This cost is used to indicate the cost in the grid. The risks and costs of communication within the network can be calculated based on the generated communication quality map; for example, assuming a grid... The 5G signal RSSI at the location is (Unit: dBm); Since RSSI is negative and the smaller the value, the worse the signal, the cost function of communication risk cost is... in It is the lowest acceptable signal strength threshold (e.g., -110 dBm). It is a positive scaling factor. When near hour, Approaching 1; when At very high levels (e.g., -70 dBm), If a region is a signal dead zone, the value approaches 0. Below Its cost can be set to an extremely large value or infinity, indicating that the area is unreachable by communication, and so on.

[0054] Locational reliability cost This cost is used to indicate the cost in the grid. The uncertainty of positioning within the environment. It can be determined based on the output physical state covariance matrix. To calculate.

[0055] For example, the location covariance matrix Determinant of the position-related submatrix or trace It can be used as a measure of uncertainty. For example, cost can be defined as follows: in, and It is the standard deviation of the position in the x and y directions. It is a scaling factor; in open areas with good GNSS signal and abundant SLAM features, the covariance is small and the cost is low. In tunnels, urban canyons, or corridors with sparse features, the covariance increases and the cost rises accordingly, but this is not limited here.

[0056] Environmental safety costs This cost takes into account both external environmental threats and internal health risks.

[0057] For example, for a static obstacle, the grid it occupies and the adjacent expansion region, its A very large value is set to indicate impassable; for dynamic obstacles, the system predicts their possible trajectory within the next T seconds and marks the area covered by that trajectory as having a higher cost that changes over time. The environmental safety cost can then be set as: in This represents the collision probability, where c is the number of individuals colliding. It is a scaling factor; internal risks can also be included, for example, when the battery power SoC is below a certain threshold, all grids... Each of these will be given an additional parameter that is proportional to the distance from the grid to the charging station, thereby guiding the agent to plan an energy-saving or return route. By following the steps above, a generalized state space map containing multi-dimensional risks can be obtained.

[0058] S220: Risk Prediction.

[0059] The risk prediction module runs before constructing the generalized state space map to predict the changing trends of each cost item on the map within a short future time window; preferably, the module uses a machine learning-based model, such as a spatiotemporal graph neural network (STGNN).

[0060] First, the rasterized generalized state space map is constructed into a graph. .in, It is a set of nodes, where each grid cell is a node. ; It is an edge set. If two grid cells are physically adjacent, then there is an edge between their corresponding nodes. It is an adjacency matrix, each node At any moment eigenvectors It consists of the multimodal state information at that moment, for example: The goal of the STGNN model is to learn the mapping function. Given history Frame of the image signal Able to predict the future Frame of the image signal STGNN models are typically composed of multiple stacked spatiotemporal convolutional blocks. Each block contains a temporal convolutional layer and a spatial convolutional layer (i.e., a graph convolutional layer). The temporal convolutional layer usually employs a gated convolutional network (Gated-CNN) or a long short-term memory network (LSTM) to capture the evolution of each node's features over time; for example, communication quality may fluctuate periodically (e.g., network load differs between day and night). The graph convolutional layer is used to capture spatial dependencies. Graph convolution updates the representation of the central node by aggregating information from neighboring nodes, thereby learning spatial correlations. The model is trained using supervised learning with historical data, and the loss function of the model is typically the mean squared error (MSE) between the predicted and actual values.

[0061] For example, the constructed STGNN model is used to predict communication risk costs. The model takes as input communication quality data for all rasters over the past 60 minutes (60 time steps in 1-minute intervals). Its architecture consists of two spatiotemporal convolutional blocks, each containing a 1D convolution (temporal dimension, kernel size 3) and a graph convolutional network (GCN) layer. The model outputs a predicted communication quality for each raster over the next 10 minutes. During training, several weeks of historical communication data are used. In actual operation, the model receives the latest real-time data and outputs predictions for the future. For example, the model might predict that a surge in pedestrian traffic during the midday rush hour will significantly reduce the 5G network bandwidth in a certain area over the next 10 minutes. This prediction is then used to update the generalized state space map for future time steps. This value enables the planning engine to anticipate this communication quality deterioration event in advance and plan a detour or switch to other backup communication methods in advance, thereby achieving proactive risk avoidance, and so on.

[0062] Step S300: Based on the preset dynamic weighted unified cost function, search for and generate a generalized optimal path in the generalized state space map, wherein the weight coefficients in the dynamic weighted unified cost function are dynamically adjusted according to the context of the current task.

[0063] Specifically: S310: Dynamically weighted unified cost function.

[0064] The cost function is set as follows: or, For edges (from node) arrive The cost of (movement), the function can be set as: in: or This is the cost of the physical path length, typically expressed as Euclidean distance or Manhattan distance. These are the corresponding weight coefficients, and they satisfy... .

[0065] It should be noted that these weighting coefficients are not fixed, but are dynamically adjusted according to the nature and context of the current task, thereby making intelligent trade-offs between different sub-objectives.

[0066] For example, the task management system dynamically sets the weight vector based on the task type. : Mission Type A: Emergency Rescue.

[0067] The core of this task is time efficiency; therefore, the system will assign the highest weight to path length. This means that the system will prioritize the shortest path while ensuring basic communication and security.

[0068] Task Type B: High-Resolution Surveying.

[0069] This task requires high-quality data acquisition and transmission; therefore, the requirements for positioning accuracy and communication bandwidth are extremely high. In this case, its weights may be set to... In this setup, the system may choose a physically longer, more circuitous path, as long as that path can provide a continuous, stable, high-precision positioning and a high-speed communication link.

[0070] Mission Type C: Covert Reconnaissance.

[0071] The core of this task is to avoid detection; security (in a broad sense, possibly including the risk of being detected) is paramount, and the weights may be set as follows: This means the system will prioritize routes that can avoid monitored areas and use terrain or communication silence for cover.

[0072] This dynamic weighting mechanism gives the method of the present invention a high degree of adaptability and intelligence, enabling it to generate action plans that best meet the task intent for diverse task requirements.

[0073] S320: Generalized path planning.

[0074] After determining the cost function and constructing the generalized state space map, the planning engine searches for the "generalized optimal path" from the starting point to the ending point. This path is the one with the minimum total cost under a unified cost function metric, rather than simply the physical shortest path. Its evaluation function is: From the starting point to the current node The actual cumulative cost. Within our framework, It is all the edges on the path. The summation of costs is calculated based on a dynamically weighted unified cost function.

[0075] From the current node The estimated cost to reach the destination, i.e., the heuristic function: In the formula, The distance is Euclidean; because all other cost terms are non-negative, the heuristic function will not overestimate the remaining cost.

[0076] The algorithm flow is as follows: 1. Maintain an "open set" and a "closed set", where the OPEN set contains nodes to be examined and the CLOSED set contains nodes that have already been examined.

[0077] 2. Add the starting point to the OPEN set.

[0078] 3. Execute in a loop: a. Select from OPEN set The node with the smallest value .

[0079] b. Move from OPEN set to CLOSED set.

[0080] c. If If it is the destination, then the path has been found, and the complete path can be obtained by tracing back the parent node pointer.

[0081] d. Traversal All neighboring nodes : i. If In the CLOSED set, it is ignored.

[0082] ii. Calculate the distance from the starting point to... arrive new value: iii. If Not in the OPEN set, or Smaller than its old If the value is not updated, then update. The parent node is Update it Value and The value is added to (or updated in) the OPEN set.

[0083] Reference Figure 3 The diagram visually compares the differences between the traditional path and the generalized optimal path.

[0084] Path 1 is the physically shortest path, but it crosses a communication dead zone (high). ) and a weak positioning signal area (high) Traditional navigation algorithms would choose path 1. However, the planning engine of this invention uses, for example, weights suitable for high-definition mapping tasks. At that time, the total cost of path 1 will be calculated to be very high; conversely, the physically longer path 2, although... It is more expensive, but it operates entirely within areas with good communication and positioning signals. and The extremely low cost results in an even lower total cost under a unified cost function. Therefore, the system intelligently selects path 2, ensuring the high-quality completion of the surveying task, and so on.

[0085] S330: Generation of behavior instruction sequences.

[0086] The planned generalized optimal path is a sequence of generalized state nodes. For example: The information contained in this sequence includes the geometric trajectory and the corresponding cost. This node sequence is then decoded into an executable instruction sequence. The decoding process checks the changes in the state attributes between adjacent nodes on the path and generates corresponding instructions accordingly.

[0087] For example, suppose from node Move to : Physical movement command: Generates a navigation command with the target point as... The physical coordinates.

[0088] Communication switching command: During system planning, the cost of all available communication methods may have been evaluated at each node, and the lowest-cost communication method selected for each node on the path. If a node... The best communication method for the connection is 5G, and the nodes The best communication method for the association is Wi-Fi (possibly because it is predicted that the 5G signal will weaken in the foreground), so in generating from arrive When a movement command is issued, a "on arrival" message will be inserted. Previously, the instruction was to "switch the main communication link to the Wi-Fi network".

[0089] Safety behavior instructions: If node of The cost is significantly higher than (For example, if entering an indoor area where GPS signal is blocked), the system will generate a command to "reduce the maximum driving speed to V_safe" to increase the system's reaction time and ensure safety. If the node of The system is more costly (e.g., predicting a trajectory close to a dynamic obstacle), and it will also generate instructions to slow down or maintain a safe distance.

[0090] Finally, the output behavior script is as follows: ["Proceed to coordinate A", "Switch to 5G communication", "Proceed to coordinate B", "Speed ​​down to 1m / s", "Switch to Wi-Fi communication", "Proceed to coordinate C"], and so on.

[0091] Step S400: Generate and execute a sequence of control instructions based on the generalized optimal path.

[0092] This step is the final implementation stage of the decision-making process. It is responsible for converting the sequence of behavioral instructions generated in the previous step into precise control signals for the underlying hardware and forming a closed-loop feedback. After receiving the sequence of behavioral instructions, the closed-loop control and execution module 140 will parse and execute them one by one.

[0093] For the command "Move to coordinate A", the motion controller (such as a PID controller or a model predictive controller MPC) in the module will calculate the voltage or PWM signal required to drive the motor, so as to drive the agent to reach the target point smoothly and accurately.

[0094] When the "Switch to 5G communication" command is received, the module will call the API interface of the communication manager to perform the network switching operation.

[0095] In response to the command "Speed ​​reduced to 1 m / s", the module will update the speed setting value of the motion controller.

[0096] During execution, the multimodal fusion perception module 110 continues to operate, reporting the latest state of the agent in real time. At this time, the closed-loop control and execution module 140 compares the actual execution state (such as current actual position, actual speed, and actual communication quality) with the command requirements. If a significant deviation occurs, or if the perception module 110 detects an unexpected major change in the environment (e.g., a new obstacle suddenly appears, or the signal quality in a certain area of ​​the communication map suddenly deteriorates), this new information is immediately updated in the generalized state space map. Subsequently, if this update significantly increases the total cost of the currently executed path, or even makes it infeasible, the system can trigger a replanning. Specifically, if D... Incremental search algorithms like Lite do not require starting from scratch; instead, they efficiently utilize existing information to quickly repair paths and generate new optimal solutions. This rapid response and replanning capability ensures the robustness of the agent in highly dynamic and uncertain environments.

[0097] Example 2 The difference from Embodiment 1 is that it includes the following steps: Step S100': Construct a multi-level probabilistic world model representing uncertainty.

[0098] In this embodiment, the agent no longer assumes that each region on the map has a fixed cost, but instead considers these costs / states to be random variables and strives to estimate their probability distribution.

[0099] First, the physical layer is a topology graph or grid graph describing physical connectivity. The probability distribution model is attached to the graph elements.

[0100] Communication state probability layer: for each location in the map or path The system no longer assigns it a single communication cost. Instead, it uses historical and real-time data to learn and maintain vectors of communication quality metrics. Conditional probability distribution (e.g., including bandwidth and latency) To capture the volatility and uncertainty of communication quality.

[0101] For example, a Gaussian mixture model (GMM) can be used to characterize this distribution; for a certain region Its bandwidth The distribution may be a mixture of two Gaussian components: in, =100Mbps, =20, =10Mbps, =5; This means that in this area, there is a 70% probability that the bandwidth will fluctuate at a high level of around 100Mbps, but there is also a 30% probability that it will drop to a low level of around 10Mbps. This representation is more accurate than a single average cost value (e.g., Mbps (Mbps) better reflects the true nature of the risk.

[0102] Location-based confidence probability layer: For each location The system directly utilizes the predicted state covariance matrix output by a fusion localization algorithm (such as EKF). As a measure of positioning uncertainty, this matrix is ​​used to describe the probability distribution of positioning errors in various directions, rather than just a scalar cost.

[0103] Environmental security probability layer: for each location The system calculates future time windows based on the tracking and trajectory prediction of dynamic obstacles (e.g., using Kalman filters or more complex interactive perception models). Inside, the intelligent agent is located The probability of a collision occurring This probability value directly reflects the dynamic safety risks brought about by the environment.

[0104] Step S200': Construct the state-dependent task utility function.

[0105] Task utility function In the middle, state This includes the agent's position, speed, internal health status, and actions. These are control instructions, easy to understand, and task utility functions. It consists of multiple components with non-linear relationships between them, designed to capture complex decision-making logic, specifically including: 1. Task progress utility : Used to reward the agent's behavior of moving towards the target; it can be set to the distance moved towards the target point, or the amount of reduction in the Euclidean distance from the target point: in It is a distance function. It is the target state. It is a positive number, for example (Unit: utility / meter).

[0106] 2. Survival utility (or risk penalty) This function is used to penalize behaviors that enter a high-risk state; it is defined based on the "probability of survival." First, a series of task failure events are defined. and their probability of occurrence. These probabilities are obtained from the aforementioned probabilistic world model: Communication failure events, such as bandwidth falling below the minimum threshold required by the task. Its probability By analyzing the probability distribution In the interval The result is obtained by integration.

[0107] Location failure events, such as locating the trace of the covariance matrix. Exceeding the maximum allowed value .

[0108] A collision event, with a probability of . .

[0109] The agent in state Survival probability It can be approximated as: .

[0110] At this point, the survival utility function is: in It is a normal weight, for example , It is a small normal number that prevents the logarithm from being negative infinity (e.g.) ).

[0111] When the probability of survival When the value is close to 1, its logarithm is close to 0, and the penalty is small; but when... As it approaches 0, its logarithm drops sharply to a huge negative value, creating a strong penalty that prevents the agent from entering these dangerous areas, and so on.

[0112] 3. Controlling cost-effectiveness This component penalizes overly strenuous controlled movements to ensure smoothness of motion and energy efficiency. in It is an acceleration or angular velocity control vector. It is a small positive number, for example .

[0113] The total task utility function is the sum of these components: in It is to perform an action The next state; this function is state-dependent because and Both depend on the state of the agent. .

[0114] S300': The generalized optimal trajectory is planned using a stochastic optimal control algorithm.

[0115] Since the world model is probabilistic and the utility function is nonlinear, the Model Predictive Path Integral (MPPI) control model is used here to plan the optimal control sequence online.

[0116] The MPPI workflow is as follows: 1. Batch forward simulation: at the current time... and current state The algorithm randomly samples in the control space. Candidate control sequences ,in , It is the length of the prediction time domain; for example, it can be derived from a Gaussian distribution. Mid-sampling, of which It is the optimal control sequence calculated at the previous time step. Assume... , (That is, predicting the next 5 seconds and controlling the frequency at 10Hz), the system will generate 2048 random control commands (such as angular velocity and linear velocity) sequences of length 50.

[0117] 2. Trajectory Cost Evaluation: For each sampled control sequence The system simulates a corresponding state trajectory forward. Then, calculate the total utility of this trajectory (or its opposite, i.e., the total cost). ): When calculating utility, for each simulated state The system will query the corresponding survival probability from the probabilistic world model. And calculate the nonlinear survival utility. .

[0118] 3. Control Update: After the utility of all sampled trajectories has been calculated, the algorithm uses a "soft maximization" operation to calculate the optimal control sequence with weighted averages. It is based on the total utility of each trajectory Calculated using an exponential function: in It is a "temperature" parameter that controls the balance between exploration and exploitation, in which the trajectory with higher utility has a greater weight.

[0119] Final update control sequence It is a weighted average of all sampled control sequences: 4. Execution & Replanning: The system will calculate the optimal control sequence. The first control command The message is sent to the underlying actuator. In the next control cycle (e.g., after 0.1 seconds), the agent reaches the new state. And received new sensor data. Then from this new state... To begin, repeat the above steps to achieve scrolling time-domain optimization.

[0120] For example, when faced with a shortcut but the communication quality distribution shows a high probability of disconnection (leading to...) When faced with a region with a sudden drop in traffic and a slightly longer but more stable communication path, MPPI assigns very low weights to the most sampled trajectories that pass through shortcuts because they have gained huge negative utility, and ultimately chooses the safer path, and so on.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0122] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0123] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0124] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0125] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0126] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A unified decision-making and control method integrating multimodal sensing, characterized in that, include: Acquire multimodal state information of the intelligent agent, including physical state information, communication state information, and security state information; Based on the multimodal state information, a generalized state space map is constructed, wherein any location in the generalized state space map includes a generalized cost, which is used to indicate the comprehensive risk of performing a task at that location; Based on a preset dynamic weighted unified cost function, a generalized optimal path is searched and generated in the generalized state space map, wherein the weight coefficients in the dynamic weighted unified cost function are dynamically adjusted according to the context of the current task. Based on the generalized optimal path, a sequence of control instructions is generated and executed.

2. The method according to claim 1, characterized in that, The acquisition of the agent's multimodal state information includes: The physical state information, including the agent's position, attitude, and velocity, is obtained by fusing data from the Global Navigation Satellite System, Inertial Measurement Unit, and Instant Positioning and Mapping Algorithms. The communication status information is obtained by monitoring the link quality indicators of at least one communication link, and a real-time communication quality map covering the task area is generated. The safety status information is obtained by fusing environmental sensor data with the agent's own sensor data. The safety status information includes external environmental risks and the agent's internal health status.

3. The method according to claim 2, characterized in that, The construction of the generalized state space map based on the multimodal state information includes: Build a basic environment map; The communication risk cost represented by the real-time communication quality map, the location confidence cost represented by the physical state information, and the environmental security cost represented by the security state information are attached as attributes to the nodes or edges of the basic environment map to obtain the generalized state space map.

4. The method according to claim 1, characterized in that, The step of generating and executing a sequence of control instructions based on the generalized optimal path includes: The generalized optimal path is parsed into a sequence containing behavioral instructions, which include path point switching instructions, communication mode switching instructions, and speed adjustment instructions. The behavioral instructions are converted into control instructions for the underlying hardware and sent to the corresponding actuators. The execution status of the actuator is fed back in real time.

5. A unified decision-making and control system integrating multimodal sensing, characterized in that, include: A multimodal fusion perception module is used to acquire multimodal state information of an intelligent agent, including physical state information, communication state information, and security state information; The situational awareness and risk quantification module is used to construct a generalized state space map based on the multimodal state information, wherein any location in the generalized state space map includes a generalized cost, which is used to indicate the comprehensive risk of performing a task at that location; A unified decision-making and planning engine is used to search for and generate a generalized optimal path in the generalized state space map based on a preset dynamic weighted unified cost function, wherein the weight coefficients in the dynamic weighted unified cost function are dynamically adjusted according to the context of the current task. The closed-loop control and execution module is used to generate and execute a sequence of control instructions based on the generalized optimal path.

6. The system according to claim 5, characterized in that, The acquisition of the agent's multimodal state information includes: The physical state information, including the agent's position, attitude, and velocity, is obtained by fusing data from the Global Navigation Satellite System, Inertial Measurement Unit, and Instant Positioning and Mapping Algorithms. The communication status information is obtained by monitoring the link quality indicators of at least one communication link, and a real-time communication quality map covering the task area is generated. The safety status information is obtained by fusing environmental sensor data with the agent's own sensor data. The safety status information includes external environmental risks and the agent's internal health status.

7. The system according to claim 6, characterized in that, The construction of the generalized state space map based on the multimodal state information includes: Build a basic environment map; The communication risk cost represented by the real-time communication quality map, the location confidence cost represented by the physical state information, and the environmental security cost represented by the security state information are attached as attributes to the nodes or edges of the basic environment map to obtain the generalized state space map.

8. The system according to claim 5, characterized in that, The step of generating and executing a sequence of control instructions based on the generalized optimal path includes: The generalized optimal path is parsed into a sequence containing behavioral instructions, which include path point switching instructions, communication mode switching instructions, and speed adjustment instructions. The behavioral instructions are converted into control instructions for the underlying hardware and sent to the corresponding actuators. The execution status of the actuator is fed back in real time.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 4 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method described in any one of claims 1 to 4.

Images