Intelligent recommendation control system and method fusing multi-modal perception and decision

Abstract

The embodiment of the invention provides an intelligent recommendation control system and method fusing multi-modal perception and decision, and relates to the technical field of intelligent agent control technologies. The method comprises the following steps: acquiring related multi-modal heterogeneous sensing data when autonomous equipment executes a task in an environment; based on the multi-modal heterogeneous sensing data, constructing a dynamic digital twinborn model comprising a plurality of state dimensions; according to a preset task target, generating at least two candidate action schemes; based on the dynamic digital twinborn model, evaluating the at least two candidate action schemes by using a multi-objective cost function to obtain a comprehensive cost value of each candidate action scheme; and generating a sorting recommendation list containing the at least two candidate action plans based on the comprehensive cost value. Through the intelligent agent control method and device, the problem that the intelligent agent control safety is insufficient is solved, and the effect of improving the intelligent agent control safety is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent agent control technology, and more specifically, to an intelligent recommendation control system and method that integrates multimodal perception and decision-making. Background Technology

[0002] Autonomous mobile robots (AMRs), drones, autonomous vehicles and other intelligent devices are increasingly being used in logistics warehousing, smart manufacturing, security inspection and other scenarios. These devices need to complete complex task planning, path navigation, status monitoring and safety management under a unified control system.

[0003] However, existing intelligent device control systems generally suffer from the technical problems of "information silos" and "fragmented decision-making." Specifically, navigation, communication, and security modules in the system are typically modular and operate independently. Navigation algorithms focus on finding the geometrically shortest path, ignoring the quality of wireless communication signals along the path; communication switching strategies tend to connect to the network nodes with the strongest signals, but may not meet the dependence of the work area the device is about to enter on high-precision positioning networks (such as 5G network slices). This decision-making approach, where each module pursues only local optima, prevents the system from achieving a globally optimal solution.

[0004] Therefore, how to deeply integrate multimodal perception information to achieve global situational awareness and intelligent decision-making, in order to solve the technical problem of insufficient security caused by fragmented decision-making in existing technologies, is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention provides an intelligent recommendation control system and method that integrates multimodal perception and decision-making, in order to at least address the problem of insufficient security performance in related technologies.

[0006] According to an embodiment of the present invention, an intelligent recommendation control method integrating multimodal perception and decision-making is provided, comprising: Acquire multimodal heterogeneous sensing data related to autonomous devices performing tasks in an environment, wherein the multimodal heterogeneous sensing data includes at least spatial positioning data, communication channel data, and operational safety data; Based on the multimodal heterogeneous sensing data, a dynamic digital twin model containing multiple state dimensions is constructed, wherein the dynamic digital twin model is associated with geographic location information; Based on the preset task objectives, generate at least two candidate action plans; Based on the dynamic digital twin model, at least two candidate action plans are evaluated using a multi-objective cost function to obtain the comprehensive cost value of each candidate action plan. Based on the comprehensive cost value, a ranked recommendation list containing the at least two candidate action plans is generated.

[0007] In an exemplary embodiment, constructing a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: The target geographic location is associated with the communication quality, environmental risk level, and device health status information corresponding to the target geographic location to obtain a multi-dimensional state vector. The multi-dimensional state vector includes a location vector, an attitude vector, a communication quality score, a security risk score, and an energy status score.

[0008] In one exemplary embodiment, the method further includes: Select the optimal action plan from the ranked recommendation list; The optimal action plan is parsed into an executable sequence of low-level control instructions and sent to the execution mechanism of the autonomous device; the actual execution status of the autonomous device is continuously acquired and fed back as new sensing data.

[0009] In an exemplary embodiment, constructing a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: Based on the historical records of the multimodal heterogeneous sensing data, a multimodal volatility vector is associated with each geographic location in the dynamic digital twin model. The multimodal volatility vector is used to quantify the historical volatility of at least one state dimension at the geographic location.

[0010] In one exemplary embodiment, evaluating at least two candidate action plans using a multi-objective cost function based on the dynamic digital twin model includes: Based on the dynamic digital twin model, the cumulative expected cost and cumulative variance of each candidate action plan on the preset risk dimension are calculated respectively; Based on the preset risk aversion coefficient, the cumulative expected cost, and the cumulative variance, the risk-adjusted total cost of each candidate action plan is determined, and the risk-adjusted total cost is used as the comprehensive cost value.

[0011] According to another embodiment of the present invention, an intelligent recommendation control system integrating multimodal perception and decision-making is provided, comprising: A multimodal heterogeneous sensing module is used to acquire multimodal heterogeneous sensing data related to autonomous devices performing tasks in an environment. The multimodal heterogeneous sensing data includes at least spatial positioning data, communication channel data, and operational safety data. The situational awareness and digital twin construction module is used to construct a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data, wherein the dynamic digital twin model is associated with geographic location information; The intelligent recommendation and decision engine module is used to generate at least two candidate action plans according to the preset task objectives, and to evaluate the at least two candidate action plans using a multi-objective cost function based on the dynamic digital twin model to obtain the comprehensive cost value of each candidate action plan; and to generate a ranked recommendation containing the at least two candidate action plans based on the comprehensive cost value.

[0012] In an exemplary embodiment, constructing a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: The target geographic location is associated with the communication quality, environmental risk level, and device health status information corresponding to the target geographic location to obtain a multi-dimensional state vector. The multi-dimensional state vector includes a location vector, an attitude vector, a communication quality score, a security risk score, and an energy status score.

[0013] In an exemplary embodiment, constructing a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: Based on the historical records of the multimodal heterogeneous sensing data, a multimodal volatility vector is associated with each geographic location in the dynamic digital twin model. The multimodal volatility vector is used to quantify the historical volatility of at least one state dimension at the geographic location.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0015] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] This invention integrates previously isolated decision-making dimensions such as positioning, communication, and security into a unified decision framework by constructing a unified digital twin model and multi-objective cost function. This framework generates and recommends truly globally optimal action plans, thus avoiding the overall performance degradation caused by locally optimal decisions. Furthermore, through comprehensive analysis of historical data and multi-dimensional information, the system can predict and proactively avoid potential, intangible risks such as communication blind spots and densely populated areas, elevating security control from reactive post-event response to proactive pre-event prevention. The dynamic weight adjustment mechanism and real-time closed-loop feedback enable the system to flexibly adjust decision preferences according to task nature and environmental changes, exhibiting strong adaptability and robustness. Therefore, it can address the insufficient security of agent control and improve the overall security of agent control. Attached Figure Description

[0017] Figure 1 This is a structural block diagram of an intelligent recommendation control system that integrates multimodal perception and decision-making according to an embodiment of the present invention; Figure 2 This is a flowchart of an intelligent recommendation control method that integrates multimodal perception and decision-making according to an embodiment of the present invention; Figure 3 This is a schematic diagram of path selection according to a specific embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0019] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0021] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0022] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0023] Example 1 Reference Figure 1 This embodiment provides an intelligent recommendation control system that integrates multimodal perception and decision-making. The system employs a multimodal heterogeneous perception module 110, a situational awareness and digital twin construction module 120, an intelligent recommendation and decision-making engine module 130, and a control command parsing and closed-loop execution module 140. This allows for deep integration of multimodal communication, positioning, and security perception information, enabling global situational awareness and intelligent decision-making recommendations. This system solves the technical problems of low operational efficiency and poor security of autonomous equipment caused by information silos and fragmented decision-making in existing technologies. It achieves the beneficial effects of improving the overall operational efficiency of autonomous equipment, realizing proactive safety avoidance, and optimizing human-machine collaboration efficiency.

[0024] The multimodal heterogeneous sensing module 110 is used to collect multi-dimensional information from the inside and outside of the autonomous device in real time and synchronously. The module 110 can be further divided into a spatial positioning unit, a communication channel unit, and an operational safety unit. Specifically, the module is physically integrated into the autonomous device and includes a GNSS receiver, a high-precision IMU, at least one 16-line LiDAR, multiple wide-angle cameras, a millimeter-wave radar, a UWB tag, a multi-mode communication module supporting 5G / Wi-Fi 6 / BT, and a CAN bus interface connected to the BMS and motor driver.

[0025] The situational awareness and digital twin construction module 120 is used to process, align, and fuse the raw heterogeneous data collected by the perception module 110 to construct a dynamic, multi-dimensional digital twin model of the environment and device status. This module runs on a server and is used to receive uploaded data from all online autonomous devices and maintain a global dynamic digital twin model of the entire scene. The training and inference of the aforementioned ST-GNN model are also completed in this module.

[0026] The intelligent recommendation and decision engine module 130 is used to generate multiple candidate action plans for the device based on the digital twin model and evaluate these candidate action plans. This module runs on the server, and when it receives a task request, it performs path planning, cost evaluation and recommendation list generation based on the digital twin model.

[0027] The control command parsing and closed-loop execution module 140 is used to parse the optimal solution selected by the decision engine module 130 into a specific and executable sequence of low-level control commands and send it to the device's actuators. At the same time, this module 140 continuously feeds back the actual execution status of the device to the perception module 110, forming a complete closed loop of "perception-decision-control-feedback" to ensure continuous dynamic optimization.

[0028] In addition, the system includes a mobile computing platform and actuators. The platform is a high-performance embedded computing unit (such as NVIDIA Jetson AGX Orin) responsible for running sensor data acquisition and preprocessing, SLAM localization algorithms, target detection algorithms, and underlying motion control and communication management. It uploads the processed perception data in real time via a wireless network. The actuators include wheel hub motor drivers, steering controllers, etc.

[0029] This collaborative architecture places computationally intensive tasks (such as digital twin maintenance, machine learning model inference, and multi-scenario evaluation) in the cloud or at the edge, reducing the burden on the vehicle-side computing platform while enabling centralized scheduling and optimization from a global perspective. The vehicle focuses on real-time perception, execution, and rapid response. Communication between the two occurs via low-latency, highly reliable wireless networks (such as 5G private networks).

[0030] Reference Figure 2 This embodiment also provides an intelligent recommendation control method that integrates multimodal perception and decision-making, corresponding to the above system. The specific process is as follows: Step S100: Acquire multimodal heterogeneous sensing data related to the autonomous device performing tasks in the environment. The multimodal heterogeneous sensing data includes at least spatial positioning data, communication channel data, and operational safety data.

[0031] This step can be further broken down into three parallel sub-steps: S110, S120, and S130. S110: Acquire spatial positioning data.

[0032] The spatial positioning unit provides high-precision real-time pose (position and attitude) information in the global coordinate system by tightly coupling and fusing multiple positioning technology sources. Since a single positioning technology is often insufficient in complex indoor / outdoor mixed scenarios, this unit adopts a multi-source fusion strategy. The fused positioning sources include, but are not limited to: 1) Global Navigation Satellite System (GNSS) receiver, used to obtain absolute outdoor position coordinates (longitude, latitude, altitude).

[0033] 2) Inertial Measurement Unit (IMU), which includes a three-axis accelerometer and a three-axis gyroscope, is used to measure the device's own three-dimensional linear acceleration and angular velocity; the IMU can provide high-frequency relative pose updates, but there is drift error that accumulates over time.

[0034] 3) LiDAR or visual cameras, combined with Simultaneous Localization and Mapping (SLAM) algorithms; LiDAR SLAM estimates motion and builds an environment map by matching point cloud data between consecutive frames; visual SLAM achieves this by analyzing image feature points. They perform well in environments with rich textures or structural features.

[0035] 4) Ultra-wideband (UWB) positioning system: By using UWB base stations pre-deployed in the environment, the system measures the position of UWB tags on the device using the time difference of arrival (TDOA) or time of flight (TOF) principle, achieving centimeter-level accuracy.

[0036] This multi-source localization information is fed into an extended Kalman filter (EKF) or a particle filter (PF) for fusion; the filter's state vector This typically includes the location of the equipment. ,speed , attitude (in quaternions) (represented) and the zero bias error of the IMU Meanwhile, the state equation (prediction model) is mainly driven by IMU measurements to predict the state at the next moment; the measurement equation (update model) uses the absolute position of GNSS, the relative pose increment of SLAM, or the absolute position of UWB to correct the predicted state, thereby suppressing the cumulative error of the IMU. Thus, the spatial positioning unit can output real-time pose information that maintains high accuracy (e.g., positioning accuracy better than 10 cm) and a high update rate (e.g., 100 Hz) in any environment.

[0037] For example, within a specific fusion cycle, the state prediction step of EKF is based on the state at the previous time step. and current IMU readings To predict the state at the current moment Assuming the IMU operates at time intervals... The internally measured acceleration is and angular velocity The prediction process can be represented as: in, It is composed of quaternions The resulting rotation matrix It is the zero-bias estimate of the accelerometer and gyroscope. This represents quaternion multiplication; subsequently, if a measurement position is received from the UWB at this moment... Then, the EKF update step begins, specifically calculating the Kalman gain. and using measurement residuals To update the state vector and covariance matrix: in It is a measurement matrix that maps the state vector to the measurement space. Through this process, the high-frequency dynamic information of the IMU is effectively combined with the absolute accuracy of UWB, maintaining centimeter-level positioning accuracy even in indoor environments where GNSS signals are lost. This ensures that all information in the digital twin model can be accurately anchored to the physical space. The computational complexity of this process has been optimized to run in real time on the embedded processor at the device end, guaranteeing the immediacy of the data.

[0038] S120: Acquire communication channel data to construct a communication quality map.

[0039] Modern industrial scenarios typically deploy a variety of wireless communication technologies, such as 5G networks for wide-area coverage and low-latency control, Wi-Fi 6 for high-speed local data transmission, Bluetooth for short-range communication between devices, and LoRa for low-power IoT sensing. Autonomous devices may have different communication requirements (such as high bandwidth, low latency, and high reliability) at different mission stages and in different areas. Therefore, the communication channel unit continuously monitors the performance indicators of all available communication links in parallel.

[0040] For 5G networks, the metrics that may be monitored include Reference Signal Received Power (RSRP), Reference Signal Received Quality (RSRQ), and Signal-to-Noise Ratio (SINR).

[0041] For Wi-Fi networks, the metrics monitored include Received Signal Strength Indicator (RSSI) and Signal-to-Noise Ratio (SNR).

[0042] For all networks, common performance metrics also include end-to-end latency, effective bandwidth, and packet loss rate.

[0043] After these raw physical and network layer metrics are collected, they need to be quantified and normalized to form a unified and comparable communication quality score. : in, These are the weights of each indicator, which can be configured according to the application scenario; It is a normalization function. For example, for RSSI (the larger the value, the better), a sigmoid function can be used to map it to the [0, 1] interval; for time delay (the smaller the value, the better), an inverse proportional or exponential decay function can be used.

[0044] The communication channel unit will record the data at each sampling time. This is correlated with the precise location information provided above, thereby dynamically updating and maintaining this communication quality map in the digital twin model.

[0045] For example, suppose the autonomous device is located at coordinate point At this time, the communication channel unit simultaneously monitored the following data: 5G network: RSRP = -85 dBm, RSRQ = -9 dB, SINR = 15 dB.

[0046] Wi-Fi 6 network: RSSI = -60 dBm, SNR = 30 dB.

[0047] By sending ping packets to both networks, the end-to-end latency was obtained as follows: 12 ms for 5G and 25 ms for Wi-Fi 6. To calculate the overall latency at this point... First, these raw values ​​need to be normalized; assuming we have defined a normalization function, for example, for RSRP, the normalization function is... Defined as: 1 when RSRP > -80 dBm, 0 when RSRP < -110 dBm, linearly mapped between [-110, -80]; then the normalized value for -85 dBm is... Similarly, other indicators are also normalized to obtain a set of values ​​in the range [0,1].

[0048] Assuming after normalization: 5G metrics: Wi-Fi 6 specifications: Then, a quality score is calculated for each network.

[0049] Assume the weights are set as follows: signal strength 40%, signal-to-noise ratio 30%, and latency 30%.

[0050] Ultimately, the overall communication quality score for this location... You can take the larger of the two, which is 0.827, or perform a more complex fusion based on application requirements (such as whether simultaneous connections are needed). This value is 0.827. To be tagged in the digital twin model At a given position, it becomes one dimension of the state vector at that point, and so on.

[0051] S130: Obtain runtime security data.

[0052] The operational safety unit is responsible for identifying potential internal health risks and external environmental risks. Internal health risk monitoring includes monitoring the operational status of critical components via onboard sensors, such as: 1. Battery Management System (BMS) provides information on remaining battery charge (SoC), voltage, current, and temperature.

[0053] 2. Motor temperature, speed, and load current fed back by the motor driver.

[0054] 3. The current load status detected by the load sensor (whether it is overloaded).

[0055] This data is used to assess the "health status" of the device and can be quantified as an energy status score. ) and health score ( ).For example, It can be directly mapped to a SoC, and The system can then use a rule engine or a simple machine learning model to determine whether the motor temperature and load exceed the limits.

[0056] External environmental risk monitoring involves understanding the dynamic environment surrounding the equipment through a series of environmental sensing sensors. 1. Camera: Uses computer vision algorithms (such as YOLO, SSD) to perform real-time target detection and identify pedestrians, other vehicles, obstacles, etc.

[0057] 2. LiDAR: Provides high-precision 3D point cloud data for obstacle detection, local map building, and assisted localization.

[0058] 3. Millimeter-wave radar: It can penetrate rain, fog and dust, and has a good detection capability for moving targets at long distances. It is especially suitable for adverse weather conditions.

[0059] 4. Microphone array: Through sound source localization and sound event recognition, it can detect abnormal sounds outside the line of sight, such as alarms and shouts, as a warning of potential risks.

[0060] These sensor data, after being fused, are used to generate a security risk score. ), The calculation is dynamic and location-dependent; for example, a region The risk level can spike instantly upon detecting a pedestrian entering the intersection; furthermore, the system combines historical data to assign a static baseline risk value to certain areas. For example, an intersection historically prone to congestion or human-machine interaction will have a baseline risk value even if no obstacle is currently detected. It will also be compared: in, It is a location Historical / static risks It is a moment In position Dynamic risks sensed in real time by sensors.

[0061] For example, an autonomous device is approaching an intersection. The intersection was marked as a "high-traffic area" in the historical database, therefore its static risk score was [high / low]. The value is set to 0.6 (assuming the score range is [0, 1]); at this time, the sensors of the running safety unit collect the following information: 1. The front-facing camera detected two pedestrians within a 3-meter range, with confidence levels of 0.95 and 0.92 respectively.

[0062] 2. The lidar detected a dense cluster of point clouds on the path ahead. After clustering and classification, it was identified as an unknown obstacle with a height of 0.5 meters and a width of 1 meter.

[0063] 3. No abnormal sound events were detected by the microphone array.

[0064] Dynamic risk score The calculation can be based on a set of rules; for example, the risk value increases by 0.2 for each nearby pedestrian detected, and by 0.3 for each obstacle detected on the path. Assuming weights for static and dynamic risks and If the scores are 0.3 and 0.7 respectively, then the overall safety risk score at this location is... for: This value is 0.67. This reflects a high security risk at the current location, and it will be updated in real time in the digital twin model, directly affecting subsequent path decisions. Simultaneously, the device's own energy status score... (For example, the current battery SoC is 0.78) will also be recorded as part of the state vector, without limitation here.

[0065] Step S200: Based on multimodal heterogeneous sensing data, construct a dynamic digital twin model that is associated with geographic location information and contains multiple state dimensions.

[0066] This step is executed by the situational awareness and digital twin construction module 120. Specifically, it involves the deep fusion of various types of collected and preliminarily processed data to construct a multi-dimensional digital model that reflects the state of the physical world. This includes the following sub-steps: S210: Data fusion and state vectorization.

[0067] A specific geographic location is strongly correlated with all relevant state information of that location to form a structured multidimensional state vector. For example, the entire working environment can be discretized into a grid map, with each grid cell (or more finely, each coordinate point) accompanied by a state vector.

[0068] For example, a specific location state vector It can be set to: in: Used to describe pose, including position. and posture .

[0069] : Communication quality score for this location.

[0070] : The overall security risk score for this location.

[0071] The current energy status score of the device is independent of its specific location but is related to the device's own status. However, it will be combined with the path length when making decisions (such as assessing energy consumption).

[0072] Other possible dimensions include, for example, terrain slope, ground friction coefficient, ambient temperature and humidity, etc., which are not limited here.

[0073] By continuously executing steps S100 and S210, the system maintains a dynamically updated digital twin model in memory. When the autonomous device moves in the environment, it experiences and reports the status information of each point along the way in real time, constantly enriching and refreshing the model. At the same time, other fixed environmental sensors (if present) can also inject data into this model.

[0074] For example, suppose that after a period of operation and data collection, the digital twin model stores state vectors for multiple locations: :Location Posture , , (Safe Zone) .

[0075] :Location Posture , (Strong 5G signal) (Currently high risk) .

[0076] :Location Posture , (Wi-Fi dead zone) , .

[0077] This structured dataset achieves information alignment and correlation, solving the "information silo" problem, and at the same time... and It will be updated in real time as the real world changes.

[0078] S220: Dynamic Scene Modeling and Prediction.

[0079] This embodiment uses a machine learning model to predict the time-varying state in digital twins, especially for the two dimensions of communication quality and security risk, which have obvious spatiotemporal correlation (the Wi-Fi signal quality in a certain area may periodically decrease due to the increase in the number of users during office hours, or the security risk of a certain channel may regularly increase due to the commuting peak hours).

[0080] To capture these complex spatiotemporal dependencies, this embodiment constructs a spatiotemporal graph neural network (ST-GNN) model. This ST-GNN model macroscopically consists of three main functional modules: a graph construction module, a spatiotemporal feature extraction module, and a prediction output module. The data flow first passes through the graph construction module, abstracting the physical environment into graph-structured data. Then, the spatiotemporal feature extraction module learns the spatiotemporal dependencies in the graph data. Finally, the prediction output module generates predictions of future states based on the learned features. Specifically: Graph building module: Used to convert the physical space represented by the digital twin model into a graph. .

[0081] node( ): Each node in the graph This corresponds to a discrete region or critical path point in the physical environment (e.g., the center of a grid cell).

[0082] side( Edges between nodes Indicates the physical connectivity between two areas (e.g., whether they are adjacent or directly accessible).

[0083] Adjacency matrix ( ): Edge weight It can be defined by the inverse of the geographical distance between two nodes or other correlation measures (such as historical traffic) to represent the strength of their influence.

[0084] Node characteristics: Each node At any moment The input features are a part of its state vector, that is: Spatiotemporal Feature Extraction Module: Composed of multiple stacked spatiotemporal convolutional blocks (ST-Conv Blocks). Each ST-Conv block is designed to capture dependencies in both spatial and temporal dimensions simultaneously. An ST-Conv block contains one temporal-gated convolutional layer and one spatial graph convolutional layer (Graph Conv). Its workflow is as follows: a. Input tensor definition: The shape of the tensor input to the first ST-Conv block is... ,in It is the number of nodes in the graph. It is the input time step (for example, using data from the past 60 minutes, with one sample point per minute). ), It is the feature dimension of the input (e.g., containing and ,but ).

[0085] b. Core processing and parameterization (taking an ST-Conv block as an example): Temporal Convolutional Layer: This layer uses a gated convolutional network (Gated CNN) to perform one-dimensional convolutions along the time axis to capture local patterns in the time series.

[0086] Input tensor: shape is .

[0087] Processing: Contains two parallel 1D convolutional layers with a kernel size of [size missing]. One convolutional layer is followed by a sigmoid activation function as a gate, and another is followed by a standard activation function (such as ReLU); the outputs of the two are then multiplied element-wise.

[0088] Example parameters: temporal kernel size It can be set to 3, which is the number of channels in the convolution (hidden layer dimension). It can be set to 64.

[0089] Output tensor: shape is .

[0090] Spatial Graph Convolutional Layer: This layer uses graph structures to propagate and aggregate information on the output feature map of temporal convolution, capturing spatial dependencies.

[0091] Input tensor: from the output of the temporal convolutional layer, with shape... .

[0092] Processing: Graph Convolutional Networks (GCNs) or variants thereof are employed. For each time step, the graph convolution operation can be represented as follows: ,in It is an adjacency matrix with self-loops added. It is its degree matrix. It is the feature matrix of the input nodes. It is a learnable weight matrix.

[0093] Example parameters: The output feature dimension of a graph convolutional layer can be set to... .

[0094] Output tensor: shape is .

[0095] c. Prediction output module: After multiple ST-Conv blocks are stacked, the final feature map is mapped to the prediction target through one or more fully connected layers.

[0096] Input tensor: The output of the last ST-Conv block, with the following shape. .

[0097] Processing: First, there might be a temporal convolutional layer to compress the time dimension to 1, followed by a fully connected layer to compress the feature dimension. Dimensions mapped to the predicted output .

[0098] Example parameters: To predict the next 15 minutes and ,but , .

[0099] Output tensor: shape is This refers to the predicted communication quality and security risk values ​​for each node in the graph over the next 15 time steps.

[0100] By training this ST-GNN model (using historical data as the training set), the system gains predictive capabilities. For example, it can predict that the Wi-Fi signal in an office area will deteriorate in 10 minutes due to the end of lunch break and employees returning to their workstations, or that a peak in pedestrian traffic will occur at the entrance of a warehouse in 30 minutes due to shift changes, thus adjusting route planning in advance, and so on.

[0101] Step S300: Generate at least two candidate action plans for the preset task objectives.

[0102] When the system receives a task objective (such as "move the autonomous device from point A to point B"), the intelligent recommendation and decision engine module 130 is activated to generate multiple candidate action plans with different characteristics.

[0103] To address this, the K-shortest path algorithm or its variants can be used to select paths. Furthermore, diverse paths can be generated by introducing a penalty term into the cost function of the path search algorithm. For example, in the heuristic function, in addition to considering distance, penalties can be imposed for behaviors such as frequent turns and passing through nodes on already selected paths, encouraging the algorithm to explore different corridors and passages. The resulting candidate action plans are not merely geometric paths but also include schemes with complete execution strategies. For instance, a scheme might specify a recommended speed along the path, the communication network that needs to be switched, etc.

[0104] For example, the task is to get from point A in the warehouse to point B on the assembly line. The decision engine module 130 calls the path planner, sets K=3, and generates three candidate solutions: Option A (Path 1): The shortest geometric distance is 120 meters, passing through the center of the production workshop.

[0105] Option B (Route 2): The second shortest distance is 150 meters, which goes around the equipment passage at the edge of the workshop.

[0106] Option C (Route 3): The longest distance, 180 meters, completely avoids the workshop and goes around through the external corridor.

[0107] These three options represent different strategies: Option A prioritizes efficiency, Option B strikes a balance between efficiency and risk, while Option C places security and communication stability as the highest priority.

[0108] Step S400: Based on the dynamic digital twin model, evaluate at least two candidate action plans using a multi-objective cost function to obtain the comprehensive cost value of each candidate action plan.

[0109] Reference Figure 3 For each generated candidate action, the decision engine module 130 uses a constructed digital twin model (including the prediction results of ST-GNN) to perform a comprehensive simulation and scoring, based on a dynamically weighted multi-objective cost function: The calculation methods for each item are as follows: Task time ( For a path consisting of n segments: The total time is the sum of the times for each segment: The recommended speed It can be based on the road section Make dynamic adjustments, such as slowing down in high-risk areas.

[0110] Communication interruption risk ( This risk is the cumulative effect of insufficient communication quality along the path; it can be caused by various factors along the path. Below a certain threshold (e.g.) The penalty points; Here, predictions are used. ,in The expected location of the equipment The time.

[0111] Security risks Similar to communication risk, it is the accumulation of security risk scores along the path: Similarly, predictions are used here. .

[0112] Energy consumption ( Energy consumption is related to path length, speed, acceleration, load, and terrain (such as slope). A simplified energy consumption model can be built, for example... ,in It is a coefficient related to distance and load. It is a coefficient related to the change in velocity.

[0113] Weighting coefficient The sum is usually normalized to 1. These weights are not fixed, but can be dynamically adjusted according to the type and priority of the current task.

[0114] For example, for "emergency material delivery" tasks, time efficiency is paramount, therefore It will be set to a higher value, such as 0.6.

[0115] For the "high-precision map data upload" task, communication stability and bandwidth are crucial, therefore It will be set to the highest value, for example, 0.7.

[0116] For "nighttime security patrol" tasks, avoiding potential risks and saving energy are key priorities, therefore and The weight will be relatively high.

[0117] Assuming the current task is "routine material delivery", the weights are set as follows: Subsequently, the decision engine module 130 began evaluating the three options: Evaluation Option A (Path 1, 120 meters, through the workshop): Estimated time: 4 minutes (240 seconds). Normalized (assuming time cost is proportional to the number of seconds). .

[0118] The path passes through a large area of ​​metal equipment; the digital twin model shows (and predicts) the Wi-Fi signal strength in that area. It will drop to 0.4 and remain there for 20 meters. After normalization, .

[0119] Passing through the center of the workshop, ST-GNN predicts that there will be frequent pedestrian activity in the area over the next 10 minutes, with an average of It is 0.5. After normalization, .

[0120] Shortest distance, lowest energy consumption. After normalization, .

[0121] Overall Cost .

[0122] Assessment Option B (Route 2, 150 meters, bypassing equipment passage): Estimated time: 5 minutes (300 seconds). .

[0123] The entire route has good 5G coverage. All are above 0.9 . .

[0124] The equipment passageway has sparse pedestrian traffic, on average It is 0.1. . .

[0125] Distance is moderate. .

[0126] Overall Cost .

[0127] Evaluation Option C (Path 3, 180 meters, external corridor): Estimated time: 6 minutes (360 seconds) .

[0128] External corridor communication quality is excellent. , .

[0129] External corridors separate pedestrians and vehicles, posing extremely low safety risks, on average. It is 0.05. . .

[0130] Longest distance, highest energy consumption. .

[0131] Overall Cost .

[0132] After evaluation, the combined costs of the three options are as follows: The lower the cost, the better.

[0133] Step S500: Based on the comprehensive cost value, generate a ranked recommendation list containing at least two candidate action plans.

[0134] The output of this step is a recommendation list with explanations for human-machine collaborative decision-making, which greatly reduces the cognitive load on back-end monitoring personnel. Operators do not need to analyze massive amounts of raw data; they can make quick and accurate judgments simply by looking at this intuitive list.

[0135] The list will be sorted from low to high based on the total value. In addition to path information and total value, each entry will also include key quantitative indicators and concise textual explanations.

[0136] For example, based on the aforementioned evaluation results, the system generates the following recommendation list: Option 1, Option B (Recommended): Path 2 (Device Channel) Total cost: 0.64 Estimated time: 5 minutes Communication reliability: 99.9% (5G throughout) Safety risk: Low Evaluation: Best overall performance, with moderate time cost while ensuring extremely high communication and security.

[0137] 2. Option C: Path 3 (External Corridor) Total cost: 0.73 Estimated time: 6 minutes Communication reliability: 99.9% (5G / Wi-Fi throughout) Safety risk: Extremely low Evaluation: The safest option, but it is time-consuming and energy-intensive, and is suitable for implementation in non-emergency situations.

[0138] 3. Option A: Path 1 (through the workshop) Total cost: 0.88 Estimated time: 4 minutes (fastest) Communication reliability: 85% (weak signal areas exist along the way). Safety risk: High (dense crowds) Evaluation: Fastest speed, but comes with a higher risk of communication interruption and security collisions, not recommended.

[0139] This list clearly illustrates the trade-offs between different decisions, providing operators with a clear overview. The system can be configured to automatically select the top-ranked option by default, but in certain critical tasks, operators can manually intervene to select alternatives.

[0140] Once an optimal course of action (e.g., option B) is selected, the control instruction parsing and closed-loop execution module 140 will parse it into a series of executable control instruction sequences, such as: { "action": "set_speed", "value": 1.0} { "action": "follow_path", "waypoints": [...]} { "action": "switch_network", "target": "5G"} These instructions are issued to the motion controller, communication management module, and other actuators of the autonomous equipment.

[0141] During execution, the multimodal heterogeneous sensing module 110 continues to work, constantly feeding back the actual execution status of the device (actual location, actual speed, real-time perceived communication quality and security risks) to the situational awareness and digital twin building module 120, so that the system can cope with unexpected dynamic changes.

[0142] For example, suppose the equipment is executing plan B. Halfway through its journey, a maintenance team suddenly sets up a temporary work area in the equipment aisle, causing disruption to that area. It surged from 0.1 to 0.9.

[0143] The closed-loop execution module 140 detected the actual sensed data. Compared with planning based on forecasts If a significant deviation occurs, the intelligent recommendation and decision-making engine module 130 will be immediately reactivated, starting from the current location and ending at point B, to re-execute processes S300, S400, and S500. At this point, option C (external corridor) becomes the least costly option because it completely avoids this sudden danger zone. The system will then issue new instructions to the device, guiding it to path 3, and push an updated recommendation list to the back-end operator, and so on.

[0144] Example 2 In the real world, risks not only have an expected value (mean) but also volatility (variance). A region with an average safety risk of 0.5 may have a risk that remains stable at 0.5, or it may fluctuate wildly between 0.1 and 0.9. For mission-critical tasks, the latter (high uncertainty) poses a far greater potential threat than the former.

[0145] To solve this technical problem, the method differs from Embodiment 1 in that a new step is introduced in the aforementioned steps S300 and S400: S230: Construct the multimodal volatility vector MVV.

[0146] In addition to each geographic location Associate a state vector In addition, this step also associates it with a multimodal volatility vector. This is to quantify the degree of uncertainty or historical fluctuation in each key dimension of the state vector: in: Communication quality score The variance at this location is calculated by analyzing historical datasets over similar time periods (e.g., 3 PM on a weekday). The fluctuations were calculated; high This value indicates that the communication quality in the area is extremely unstable and may be affected by occasional strong interference or network congestion, even if its average value is low. It's acceptable.

[0147] Safety risk score The variance at that location is also calculated based on historical data. For example, a fixed area of ​​physical obstacles... The mean is high, but the variance is very low (the risk is certain); while a temporary cargo loading and unloading area, its... It may be 0 (idle) at times and 1 (busy and chaotic) at other times, which may result in a low mean but a very large variance, representing a high degree of unpredictability.

[0148] : The variance of travel time at this location or road segment; by analyzing historical trajectory data, calculate the fluctuation in the time spent by different devices passing through this area under similar conditions, high The value indicates that there are random delay factors such as congestion in the area.

[0149] For example, the situational awareness and digital twin building module 120 supports two regions in the digital twin model. and An analysis was conducted: area (Main Road): Historical data shows that the average safety risk in this area is... The mean is 0.6, but the data points are very concentrated, and the calculated variance is... This indicates that the risks in the region are persistent and predictable, representing a "high risk."

[0150] area (Temporary parking area): Historical data shows that this area is safe 90% of the time. However, there are 10% of the times when the sudden stopping and operation of large vehicles can become extremely dangerous. ), at this point, its calculation is obtained The mean is approximately It has a low mean; however, its variance is low. However, it is as high as 0.0648, far exceeding... This reveals a dangerous region of "low average risk but high volatility," which traditional methods are likely to mistakenly favor due to their lower average risk. By constructing... The system gained the ability to perceive the "second moment" of risk, laying a data foundation for subsequent risk portfolio assessment.

[0151] S401 (alternative to S400): Evaluation of action plans based on risk-adjusted utility theory.

[0152] This step replaces the aforementioned linear weighted cost function with an evaluation framework that maximizes a utility function that characterizes decision-maker satisfaction, taking into account both the expected cost (mean) and the uncertainty (variance) of the cost.

[0153] S402: Construct the risk-reward profile of candidate solutions.

[0154] For each candidate action plan (path) generated, the system no longer calculates a single cost, but constructs its complete risk-reward profile, which includes calculating the cumulative expected cost and cumulative variance of the plan across various risk dimensions.

[0155] For a path Its total expected cost vector Sum of total variance vectors The calculation is as follows: Each component is calculated by integrating or summing the corresponding values ​​of all points / segments along the path: (This is a simplified assumption, assuming that the risks of each road segment are independent.) Here It is a location Expected cost value (e.g.) (mean), and It comes from The variance.

[0156] S403: Use the risk aversion utility function for evaluation.

[0157] Set a configurable risk aversion factor (dimensionless, This coefficient represents the degree to which the current task is averse to uncertainty. This indicates that they are completely unconcerned about risk fluctuations and only care about expected costs (degenerates into the case of Example 1). The larger the value, the more the system tends to choose the option that, although the expected cost may be slightly higher, has a more certain and predictable outcome.

[0158] For each scheme The system calculates its risk-adjusted total cost (RAC), which is a linear combination of the expected total cost and the variance total cost, determined by the risk coefficient. Regulation: in: Weight here The effect on variance implies different levels of aversion to different types of uncertainty.

[0159] The system will select The smallest solution is recommended as the optimal solution.

[0160] For the aforementioned schemes A and B, assuming the task is "transporting precision optical instruments," this is a highly risk-averse task. Therefore, the system will use a risk aversion coefficient. Set to a higher value, for example Weight Same as before.

[0161] Assessment Option A (Path 1, through the workshop, high volatility): Expected cost vector: Assuming analysis The variance vector (after normalization) is calculated as follows: (Time, communication, and security are all subject to high uncertainty) Expected total cost: Total cost of variance: Risk-adjusted total cost Evaluation Option B (Path 2, Equipment Channel, Low Fluctuation): Expected cost vector: Assuming the environment along this path is stable and controllable, its variance vector is extremely low: Expected total cost: Total cost of variance: Risk-adjusted total cost Comparison results Much larger Although the expected cost of option A (0.88) is only slightly higher than that of option B (0.64), its huge uncertainty (variance 0.73) is severely penalized under the risk aversion model. Therefore, the system will firmly recommend option B because it provides an action strategy that, although the expected cost is slightly higher, has a highly predictable outcome and controllable risk.

[0162] In addition, the decision engine does not only select a passive path, but also embeds proactive risk hedging actions. These actions are designed to proactively offset or mitigate the potential negative impact of key risk points (especially high variance areas) identified on the path by sacrificing a small amount of efficiency.

[0163] At this point, the candidate action plan is no longer just a combination of <path, velocity>, but has expanded to a complex strategy of <path, velocity profile, hedging action sequence>.

[0164] Exemplary hedging actions include: Communication risk hedging: If the assessment finds that a certain route must pass through a region with acceptable average communication quality but high variance (e.g., The system can add an instruction to the recommended scheme: "5 meters before entering coordinates (x1, y1), start the data pre-buffering mechanism and switch the critical status reporting frequency from 1Hz to the LoRa-based 0.1Hz heartbeat mode until leaving coordinates (x2, y2)". This instruction proactively "hedges" the potential communication interruption risk by preparing in advance and switching to a more robust backup channel.

[0165] Security risk hedging: If a certain plan must be located near an area with high security risk volatility (such as the aforementioned temporary docking area) The system can generate hedging instructions: "At a distance..." At a distance of 20 meters from the center point, the maximum speed limit was reduced from 1.5 m / s to 0.5 m / s, and the maximum power scanning mode of all long-range sensing sensors (such as millimeter-wave radar) was activated. This strategy of proactively reducing speed and enhancing perception, at the cost of time, gained a longer reaction window and stronger situational awareness, effectively mitigating the sudden dangers in the area.

[0166] Time risk hedging: If a route passes through a congested point with a large travel time variance, and the task has strict requirements for on-time arrival (high... (Given the time cost), the system can suggest a time-hedging path. Alternatively, if congestion is predicted to be temporary (e.g., ST-GNN predicts it will ease in 1 minute), the system can generate a timed wait instruction: "Pause the task at coordinates (x, y) for 60 seconds, and resume when the congestion risk decreases."

[0167] Finally, the system will present the new recommendation list to the operator or user, and each recommendation entry will include: 1. Solution name and path summary; 2. Expected cost / time; 3. Risk Exposure Index: This index comprehensively reflects... Risk measures (such as "low / medium / high"); 4. Result predictability: For example, the confidence interval for time is "estimated time is 5 minutes ± 15 seconds"; 5. Built-in hedging strategies: Clearly list the proactive risk hedging actions included in the plan, such as "communication pre-buffering has been enabled" and "automatic deceleration has been planned when approaching a danger zone".

[0168] Such a list allows operators to make decisions not based on a single "cost" metric, but within a multi-dimensional "risk-reward" framework. Users can clearly see the efficiency costs of each option in order to control risk, thus making more informed and safer choices based on the actual needs of the task, and so on.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0170] This embodiment also provides an intelligent recommendation control system that integrates multimodal perception and decision-making. This system is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0171] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0172] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0173] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0174] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0175] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent recommendation control method integrating multimodal perception and decision-making, characterized in that, include: Acquire multimodal heterogeneous sensing data related to autonomous devices performing tasks in an environment, wherein the multimodal heterogeneous sensing data includes at least spatial positioning data, communication channel data, and operational safety data; Based on the multimodal heterogeneous sensing data, a dynamic digital twin model containing multiple state dimensions is constructed, wherein the dynamic digital twin model is associated with geographic location information; Based on the preset task objectives, generate at least two candidate action plans; Based on the dynamic digital twin model, at least two candidate action plans are evaluated using a multi-objective cost function to obtain the comprehensive cost value of each candidate action plan. Based on the comprehensive cost value, a ranked recommendation list containing the at least two candidate action plans is generated.

2. The method according to claim 1, characterized in that, The construction of a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: The target geographic location is associated with the communication quality, environmental risk level and equipment health status information corresponding to the target geographic location to obtain a multi-dimensional state vector. The multi-dimensional state vector includes a location vector, an attitude vector, a communication quality score, a security risk score and an energy status score.

3. The method according to claim 1, characterized in that, The method further includes: Select the optimal action plan from the ranked recommendation list; The optimal action plan is parsed into an executable sequence of low-level control instructions and sent to the execution mechanism of the autonomous device; the actual execution status of the autonomous device is continuously acquired and fed back as new sensing data.

4. The method according to claim 1, characterized in that, The construction of a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: Based on the historical records of the multimodal heterogeneous sensing data, a multimodal volatility vector is associated with each geographic location in the dynamic digital twin model. The multimodal volatility vector is used to quantify the historical volatility of at least one state dimension at the geographic location.

5. The method according to claim 4, characterized in that, The evaluation of at least two candidate action plans based on the dynamic digital twin model using a multi-objective cost function includes: Based on the dynamic digital twin model, the cumulative expected cost and cumulative variance of each candidate action plan on the preset risk dimension are calculated respectively; Based on the preset risk aversion coefficient, the cumulative expected cost, and the cumulative variance, the risk-adjusted total cost of each candidate action plan is determined, and the risk-adjusted total cost is used as the comprehensive cost value.

6. An intelligent recommendation control system integrating multimodal perception and decision-making, characterized in that, include: A multimodal heterogeneous sensing module is used to acquire multimodal heterogeneous sensing data related to autonomous devices performing tasks in the environment. The multimodal heterogeneous sensing data includes at least spatial positioning data, communication channel data, and operational safety data. The situational awareness and digital twin construction module is used to construct a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data, wherein the dynamic digital twin model is associated with geographic location information; The intelligent recommendation and decision engine module is used to generate at least two candidate action plans based on preset task objectives, and to evaluate the at least two candidate action plans using a multi-objective cost function based on the dynamic digital twin model to obtain the comprehensive cost value of each candidate action plan. Based on the comprehensive cost value, a ranked recommendation list containing the at least two candidate action plans is generated.

7. The system according to claim 6, characterized in that, The construction of a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: The target geographic location is associated with the communication quality, environmental risk level and equipment health status information corresponding to the target geographic location to obtain a multi-dimensional state vector. The multi-dimensional state vector includes a location vector, an attitude vector, a communication quality score, a security risk score and an energy status score.

8. The system according to claim 6, characterized in that, The construction of a dynamic digital twin model containing multiple state dimensions based on the multimodal heterogeneous sensing data includes: Based on the historical records of the multimodal heterogeneous sensing data, a multimodal volatility vector is associated with each geographic location in the dynamic digital twin model. The multimodal volatility vector is used to quantify the historical volatility of at least one state dimension at the geographic location.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 5.

Images