A method for edge-cloud collaborative multimodal conflict resolution based on deep evidence uncertainty

Abstract

The present invention discloses a method for edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty, the method comprising step 1, multi-sensor data preprocessing; step 2, building a multimodal fusion network; step 3, uncertainty measurement and allocation; step 4, detecting evidence conflicts; step 5, incremental multimodal learning; step 6, synchronous edge-cloud feedback and optimization. The present invention improves robustness and accuracy, and at the same time, through the collaborative work of the edge and the cloud, realizes real-time processing and multi-perspective fusion of multimodal data in a dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal perception and data fusion, and in particular relates to an edge-cloud collaborative multimodal conflict processing method based on deep evidence uncertainty. Background Art

[0002] With the rapid development of intelligent driving technology, the application of edge-cloud collaboration and multimodal perception technology in unmanned driving systems has become more and more common. By integrating data from multiple sensors, such as RGB cameras, lidar, radar, etc., multimodal perception systems can more comprehensively understand the environmental information around the vehicle, thereby improving the safety and reliability of autonomous driving systems. In the multimodal perception process, deep learning and neural networks play an important role in feature extraction, environmental modeling, and scene understanding. However, although deep learning has made significant progress in image classification, object detection, and environmental perception, there are still some challenges when facing multi-source data fusion and conflict processing.

[0003] In the actual application scenarios of autonomous driving, the multimodal data collected by edge devices, such as sensors on vehicles, have inherent uncertainties and conflicts. For example, the detection results of RGB cameras and radar sensors may be inconsistent in data accuracy, time synchronization, and environmental adaptability, which will affect the decision-making effect of the autonomous driving system. To solve this problem, traditional methods usually adopt fusion strategies based on probabilistic models or prior knowledge. These methods often rely on a large amount of computing resources and complex model structures when dealing with conflicts and uncertainties in different modal data. However, this computing demand is difficult to meet when the resources of edge devices are limited.

[0004] In recent years, with the application of deep evidence theory in the fields of uncertainty quantification and multimodal fusion, edge-cloud collaborative perception systems have gained new development opportunities. Deep evidence theory can effectively evaluate the uncertainty of data in the multimodal perception process and optimize the fusion effect of different modalities through adaptive weight adjustment. The uncertainty quantification method based on deep evidence can not only process multimodal data in real time at the edge, but also achieve global modeling and conflict resolution of the environment through collaborative reconstruction with the cloud. The advantage of this method is that it can handle conflicts in multimodal data with less computing resources and improve the real-time perception capability of edge devices.

[0005] However, there are still some urgent problems to be solved in the current multimodal perception and edge-cloud collaborative systems. On the one hand, in the multimodal fusion process at the edge, the uncertainty and conflict of data in different modalities are difficult to be effectively handled by a single strategy. On the other hand, in the multi-perspective reconstruction of the cloud, how to efficiently globally fuse the data uploaded by the edge to improve the accuracy of environmental modeling is still a technical difficulty. In addition, how to achieve dynamic optimization and feedback of perception strategies in the edge-cloud collaborative process so that the edge and cloud can achieve true collaborative perception is also the focus of current research. Summary of the invention

[0006] The purpose of the present invention is to provide an edge-cloud collaborative multimodal conflict processing method based on deep evidence uncertainty, so as to realize real-time processing and multi-perspective fusion of multimodal data in a dynamic environment and improve accuracy and robustness.

[0007] In order to achieve the purpose of the present invention, the present invention provides an edge-cloud collaborative multimodal conflict processing method based on deep evidence uncertainty, comprising the following steps:

[0008] Step 1: Multi-sensor data preprocessing: denoising, segmentation, and normalization of sensor data;

[0009] Step 2: Construct a multimodal fusion network: Design a multimodal fusion network based on deep evidence theory to generate multimodal evidence weights and uncertainty metrics;

[0010] Step 3: Uncertainty measurement and allocation: The uncertainty of multimodal data is measured and allocated based on the adaptive weight allocation mechanism of the Gaussian process model;

[0011] Step 4: Detect evidence conflicts: quantify and adjust the conflicts using bidirectional entropy analysis and minimize the global conflict measure;

[0012] Step 5, incremental multimodal learning: dynamically updating the Gaussian process model using an incremental learning method;

[0013] Step 6: Synchronize edge-cloud feedback and optimization: Use edge-cloud collaboration to perform global feedback and optimization on multimodal data to complete environmental reconstruction.

[0014] Compared with the prior art, the significant progress of the present invention lies in: (1) Based on the deep evidence theory, the present invention accurately quantifies the uncertainty of multimodal data, combines the Gaussian process model to realize adaptive weight allocation, and can dynamically adjust the weight according to the real-time changes in the quality of sensor data, thereby effectively enhancing the robustness and accuracy of environmental reconstruction; (2) The present invention realizes the quantification and adjustment of modal conflicts through bidirectional entropy analysis, and optimizes the modal fusion process by minimizing the global conflict metric, which significantly reduces the conflict impact between multimodal data and ensures the reliability and consistency of the fusion results; (3) The present invention designs an edge-cloud collaboration mechanism to make full use of the real-time processing capabilities of edge devices and the global optimization capabilities of the cloud. The edge realizes preliminary fusion and preprocessing, and the cloud performs deep optimization based on a global perspective, and continuously improves the accuracy and real-time performance of environmental reconstruction through an iterative feedback mechanism; (4) The present invention uses an incremental learning method to dynamically update the model weights, and combines the time decay factor to effectively balance historical data and real-time data.

[0015] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 is a flow chart of the steps of the present invention;

[0018] Figure 2 is the weight-time dynamic change trend of different modalities (RGB camera, LiDAR, Radar) of the present invention;

[0019] Figure 3 is the changing trend of the global conflict metric of the present invention;

[0020] Figure 4 is the change of the dynamic weight adjustment amplitude of the present invention over time;

[0021] Figure 5 This is the variation trend of the fusion accuracy of the present invention. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] The present invention provides a method for edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty, combined with Figure 1 , including the following steps:

[0024] Step 1: Multi-sensor data preprocessing: De-noising, segmentation, and normalization of different types of sensor data to provide high-quality input data for subsequent multimodal fusion. The different types of sensors are RGB cameras, lidars, millimeter-wave radars, etc.

[0025] 1-1. De-noise, edge enhancement and normalization of RGB camera images; images denoised by Gaussian filtering for:

[0026] ;

[0027] in, is the original image, is the standard Gaussian kernel, is the filter strength parameter;

[0028] 1-2. LiDAR point cloud data Perform 3D KD tree segmentation and downsampling to obtain denoised point cloud data :

[0029] ;

[0030] in, is the sampling radius;

[0031] 1-3. Millimeter wave radar point cloud data Perform voxel segmentation and normalization to obtain processed radar point cloud data :

[0032] ;

[0033] in, is the voxel size.

[0034] Step 2: Construct a multimodal fusion network: Design a multimodal fusion network based on deep evidence theory, generate multimodal evidence weights and uncertainty metrics, and realize the fusion of multimodal information;

[0035] Assume that modal data sources, and the generated multimodal evidence weights and uncertainty measures for:

[0036] ;

[0037] ;

[0038] in, Any mode The value of evidence, Any mode The value of evidence, is the number of modes;

[0039] The multimodal evidence network generates evidence weights through the fully connected layer and the Softmax layer, and performs nonlinear transformation on the uncertainty metric through the ReLU activation function. The specific process is as follows:

[0040] 2-1. Network input and feature extraction

[0041] Input: The network receives pre-processed feature data from multimodal sensors, such as RGB images, lidar point clouds, and millimeter-wave radar point clouds;

[0042] Feature extraction: Mapping input modality features into high-dimensional feature representations through convolutional layers (CNN) or other feature extraction modules , to characterize the characteristics of each modal data;

[0043] 2-2. Fully connected layer generates evidence value

[0044] Each modal feature Input the fully connected layer to generate the corresponding evidence value :

[0045] ;

[0046] in, and are the weights and bias parameters of the fully connected layer, and the evidence value is the strength of support provided by the modal data to the final result;

[0047] 2-3. Softmax layer generates evidence weights

[0048] Use the Softmax function to convert the evidence value Convert to weight , satisfying the normalization constraint, the weight sum is 1:

[0049] ;

[0050] Among them, the weight is the relative contribution of the 𝑖th mode to the fusion result;

[0051] 2-4. Uncertainty measurement and ReLU nonlinear transformation

[0052] Uncertainty measures The residual of the weight is calculated as:

[0053] ;

[0054] When the weights of all modal data are uniform, the uncertainty is high; when the weight of a certain modal is dominant, the uncertainty is low. Figure 2 ;

[0055] Use the ReLU activation function to Perform a nonlinear transformation to ensure that the metric value is non-negative:

[0056] ;

[0057] ReLU can effectively filter out outliers less than zero and enhance the robustness of the model.

[0058] Step 3: Uncertainty measurement and allocation: The uncertainty of multimodal data is measured and allocated through an adaptive weight allocation mechanism based on the Gaussian process model to achieve dynamic adjustment of the multimodal fusion results;

[0059] The adaptive weight distribution mechanism uses a Gaussian process model to distribute the weights of modal evidence. The confidence prediction of the Gaussian process model is:

[0060] ;

[0061] in, is the mean function, is the covariance function, is the input modal feature, is the output confidence mapping function;

[0062] The weight for:

[0063] ;

[0064] in, and It also represents any mode, is the number of modes.

[0065] Step 4: Detect evidence conflicts: quantify and adjust the conflicts using bidirectional entropy analysis, and minimize the global conflict measure to optimize the modal fusion effect;

[0066] Calculating the conflict metric :

[0067] ;

[0068] in, and Any two modes and The confidence distribution of Represents a variety of conflicting events, such as sensor failure or error, environmental changes, occlusion or interference, etc.

[0069] Using the minimized global conflict metric , optimize modal fusion:

[0070] ;

[0071] Combination Figure 3 It can be seen that the global conflict metric in the present invention gradually decreases with time.

[0072] Step 5: Incremental multimodal learning: Use the incremental learning method to dynamically update the Gaussian process model in step 3 to adapt to the newly arrived data and improve the system's adaptability to the multimodal environment;

[0073] The incremental learning method is to use the time decay factor Dynamically adjust the modal weights of historical data:

[0074] ;

[0075] in, Representing modality exist The weight of the moment, Representing modality exist The weight of the moment, Representing modality exist The confidence level of the moment, combined with Figure 4 , we can see how the weight adjustment range of the present invention changes over time;

[0076] Based on the above dynamic weights, the present invention can flexibly adjust and optimize the parameters of the Gaussian process model by calculating weighted error values ​​when receiving new data samples, thereby improving the response speed of the Gaussian process model to new situations and enhancing its generalization ability.

[0077] Step 6: Synchronous edge-cloud feedback and optimization: Use edge-cloud collaboration to provide global feedback and optimization for multimodal data, thereby achieving environmental reconstruction;

[0078] 6-1. Edge data processing and preliminary fusion: The edge device collects and pre-processes multimodal data (RGB cameras, LiDAR, and millimeter-wave radar are used in this patent) in real time, including denoising, downsampling, and normalization, and uses a lightweight multimodal fusion model to generate preliminary fusion results;

[0079] 6-2. Edge-cloud collaborative data synchronization: Through the edge-cloud communication protocol, the edge uploads the pre-processed data and the preliminary fusion results to the cloud;

[0080] 6-3. Global feedback optimization: The cloud uses the integrated multimodal fusion network to perform secondary fusion and conflict detection based on the data uploaded by the edge and the global historical data, and optimizes the distribution of modal weights. and conflict measure , feedback the parameters and optimization strategy of the current global model;

[0081] 6-4. Dynamic environment reconstruction: Based on the modal weights and conflict metrics optimized in the cloud, new global features are integrated to use multimodal SLAM algorithms (such as DeepFusion SLAM, GraphSLAM with Multi-Modal Data, Edge-Cloud SLAM, etc.) to reconstruct the environment in three dimensions and generate a real-time updated global map. At the same time, the optimized model parameters are sent to the edge for the next stage of data fusion and processing.

[0082] 6-5. Iterative updates and feedback: Combination Figure 5 ,In a dynamic environment, the edge-cloud collaboration mechanism continuously optimizes the ,global model based on the newly collected multimodal data to ensure the ,accuracy and real-time nature of environmental reconstruction.

[0083] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0084] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty, characterized in that: The following steps are involved: Step 1: Multi-sensor data preprocessing: denoising, segmentation, and normalization of sensor data; Step 2: Construct a multimodal fusion network: Design a multimodal fusion network based on deep evidence theory to generate multimodal evidence weights and uncertainty metrics; Step 3: Uncertainty measurement and allocation: The uncertainty of multimodal data is measured and allocated based on the adaptive weight allocation mechanism of the Gaussian process model; Step 4: Detect evidence conflicts: quantify and adjust the conflicts using bidirectional entropy analysis and minimize the global conflict measure; Calculating the conflict metric : ； in, and Any mode and The confidence distribution of It is a variety of different conflict events; Using the minimized global conflict metric , optimize modal fusion: ； Step 5, incremental multimodal learning: dynamically updating the Gaussian process model using an incremental learning method; The incremental learning update method utilizes a time decay factor Dynamically adjust the modal weights of historical data: ； in, is modal exist The weight of the moment, is modal exist The weight of the moment, is modal exist Confidence at the moment; Step 6: Synchronize edge-cloud feedback and optimization: Use edge-cloud collaboration to perform global feedback and optimization on multimodal data to complete environmental reconstruction.

2. According to the method of edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty according to claim 1, it is characterized in that: The step 1 specifically comprises the following steps: 1-1. De-noise, edge enhancement and normalization of RGB camera images; images denoised by Gaussian filtering for: ； in, is the original image, is the standard Gaussian kernel, is the filter strength parameter; 1-2. LiDAR point cloud data Perform 3D KD tree segmentation and downsampling to obtain denoised point cloud data : ； in, is the sampling radius; 1-3. Millimeter wave radar point cloud data Perform voxel segmentation and normalization to obtain processed radar point cloud data : ； in, is the voxel size.

3. According to the method of edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty according to claim 2, it is characterized in that: The step 2 is specifically as follows: Assume that modal data sources, the confidence level generated and uncertainty measures for: ； ； in, Any mode The value of evidence, Any mode The value of evidence, is the number of modes; The multimodal fusion network generates evidence weights through a fully connected layer and a Softmax layer, and performs nonlinear transformation on uncertainty metrics through a ReLU activation function.

4. According to the method of edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty according to claim 3, it is characterized in that: The step 3 is specifically as follows: The adaptive weight distribution mechanism uses a Gaussian process model to distribute the weights of modal evidence. The confidence prediction of the Gaussian process model is: ； in, is the mean function, is the covariance function, is the input modal feature, is the output confidence mapping function; The weight for: ； in, and is any mode.

5. According to the method of edge-cloud collaborative multimodal conflict processing based on deep evidence uncertainty according to claim 1, it is characterized in that: The step 6 specifically comprises the following steps: 6-1. Edge data processing and preliminary fusion: The edge device collects and pre-processes multimodal data in real time, including denoising, downsampling and normalization, and uses a lightweight multimodal fusion model to generate preliminary fusion results; 6-2. Edge-cloud collaborative data synchronization: Through the edge-cloud communication protocol, the edge uploads the pre-processed data and the preliminary fusion results to the cloud; 6-3. Global feedback optimization: The cloud uses the integrated multimodal fusion network to perform secondary fusion and conflict detection based on the data uploaded by the edge and the global historical data, optimize the modal weight distribution and conflict measurement, and feedback the parameters and optimization strategy of the current global model; 6-4. Dynamic environment reconstruction: Based on the modal weights and conflict metrics optimized in the cloud, new global features are integrated, and the multimodal SLAM algorithm is used to reconstruct the environment in three dimensions to generate a real-time updated global map. At the same time, the optimized model parameters are sent to the edge for the next stage of data fusion and processing; 6-5. Iterative update and feedback: In a dynamic environment, the edge-cloud collaboration mechanism continuously optimizes the global model based on newly collected multimodal data to ensure the accuracy and real-time nature of environmental reconstruction.

Images