Sensor Weight Optimization Method and System Integrating Multi-Source Data and User Feedback

Abstract

The present invention belongs to the technical field of autonomous driving sensor data fusion, and particularly relates to a method for optimizing sensor weights by fusing multi-source data and user feedback. This method fuses multi-source sensor data (such as cameras, lidar, millimeter-wave radars, etc.), environmental data (such as weather, lighting, road types, etc.), and user feedback data (such as fatigue level, attention distribution, voice commands, etc.), and combines Bayesian networks, improved DS evidence theory, and reinforcement learning mechanisms to dynamically optimize sensor weights, improve the perception accuracy, environmental adaptability, and user satisfaction of the autonomous driving system. The present invention is particularly suitable for autonomous driving decision-making support in complex scenarios, can effectively handle multi-source data conflicts, environmental changes, and user requirements, and provides technical guarantees for the safety and reliability of the autonomous driving system.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving sensor data fusion, and particularly relates to a method and system for optimizing sensor weights by fusing multi-source data and user feedback. Background Art

[0002] In the field of autonomous driving technology, multi-source sensor data fusion is one of the key technologies for achieving environmental perception and decision support. Existing research mainly focuses on how to effectively fuse data from different sensors (such as cameras, lidars, millimeter-wave radars, etc.) to improve the perception accuracy and robustness of the system.

[0003] Multi-source sensor data fusion is one of the core research directions in the field of autonomous driving. Existing research usually adopts probability reasoning methods (such as Bayesian networks) and evidence theory methods (such as DS evidence theory, also known as Dempster-Shafer evidence theory) to handle the uncertainty and conflicts of sensor data. For example, Reference [1] proposed a multi-sensor fusion method based on Bayesian networks, which calculates the reliability weights of sensors by constructing the probability dependence relationships of sensors and the environment. Reference [2] uses an improved DS evidence theory to handle the conflicting evidence between sensors and generates the confidence after fusion. However, these methods usually consider less about user feedback and it is difficult to dynamically adjust the system weights according to user needs.

[0004] In recent years, with the rapid development of artificial intelligence and machine learning technologies, researchers have gradually incorporated user feedback into the optimization process of autonomous driving systems. For example, by using in-vehicle sensors (such as cameras, steering wheel torque sensors, microphones, etc.) to collect user behavior data (such as fatigue level, attention distribution, voice commands, etc.), and analyzing them in combination with algorithms such as neural networks, so that the behavior of the system better conforms to the actual scenario and user needs. For example, Reference [3] proposed a driver fatigue state recognition algorithm based on deep learning, and Reference [4] proposed using neural networks to recognize the user's voice and analyze information such as their emotions, providing technical support for optimizing autonomous driving decisions. These studies show that user feedback can significantly improve the adaptability and user satisfaction of the system. However, existing research usually uses the user feedback mechanism alone and does not attempt to combine it with probability reasoning and evidence theory, resulting in insufficient adaptability and robustness of the system in complex environments.

[0005] Deep learning and reinforcement learning techniques have also been widely applied to autonomous driving decision-making. For example, reference [5] proposed an anthropomorphic decision-making method for autonomous driving based on deep reinforcement learning, which analyzed and classified drivers' driving styles through reinforcement learning and had a high accuracy rate. Reference [6] studied and discussed an autonomous driving decision-making model based on end-to-end deep learning, which directly extracted features from sensor data through deep learning and generated decision instructions. These studies show that deep learning and reinforcement learning can effectively improve the decision-making ability and adaptability of autonomous driving systems. However, existing studies usually use these technologies separately and do not attempt to combine them with multi-source sensor data fusion and user feedback mechanisms, resulting in the need to improve the comprehensive performance of the system in complex scenarios.

[0006] Although existing studies have made certain progress in multi-source sensor data fusion, user feedback optimization, and the application of deep learning and reinforcement learning, there are still the following problems and drawbacks:

[0007] 1. Insufficient consideration of user feedback: Existing methods (such as Bayesian networks and DS evidence theory) usually consider user feedback less and it is difficult to dynamically adjust the system weights according to user needs. For example, when users show fatigue or inattention, existing methods cannot quickly adjust the sensor weights to enhance the safety of the system.

[0008] 2. Lack of a comprehensive optimization mechanism: Existing methods usually use a certain technology alone (such as Bayesian networks, DS evidence theory, or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance, and user feedback. For example, in complex scenarios, existing methods cannot consider sensor performance, environmental conditions, and user needs simultaneously, resulting in a decline in system performance.

[0009] [1] Chen Jiena, Zhang Mingzhuo, Du Dehui, et al. Autonomous Driving Behavior Decision-Making Based on Building a RoboSim Model with Bayesian Networks [J]. Journal of Software, 2023, 34(8): 3836-3852. DOI: 10.13328 / j.cnki.jos.006594.

[0010] [2] Hefei Zhongke Automatic Control System Co., Ltd. An Asynchronous Multimodal Target-Level Information Fusion Method Based on Temporal DS Theory: CN202311034586.9 [P]. 2023-11-14.

[0011] [3] Zhou Hui, Zhou Liang, Ding Qiulin. Fatigue State Recognition Algorithm Based on Deep Learning [J]. Computer Science, 2015, 42(3): 191-194, 200. DOI: 10.11896 / j.issn.1002-137X.2015.3.039.

[0012] [4] Dai Hang. Research on Speech-based Driver Emotion Recognition [D]. Heilongjiang: Harbin University of Science and Technology, 2023.

[0013] [5] Yang Chonghui. Research on Anthropomorphic Decision-making Method for Autonomous Driving Based on Deep Reinforcement Learning [D]. Chongqing: Chongqing University, 2023.

[0014] [6] Liu Wei. Research on Autonomous Driving Decision-making Model Based on End-to-End Deep Learning [D]. Chongqing University of Technology, 2022. Summary of the Invention

[0015] Aiming at the situation that the multi-source sensor data fusion method in the existing autonomous driving system lacks sufficient consideration of user feedback and is difficult to dynamically adjust the system weights according to user needs. At the same time, the existing methods usually use a certain technology alone (such as Bayesian network, DS evidence theory or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance and user feedback, resulting in insufficient perception accuracy, environmental adaptability and user satisfaction of the system in complex environments. The present invention provides an autonomous driving sensor weight optimization method and system based on multi-source data fusion and user feedback, aiming to dynamically optimize the sensor weights and improve the comprehensive performance of the system by combining Bayesian network, improved DS evidence theory and reinforcement learning mechanism.

[0016] According to one aspect of the specification of the present invention, a sensor weight optimization method for fusing multi-source data and user feedback is provided, including:

[0017] Obtain multi-source data, including external vehicle sensor data, internal vehicle sensor data and environmental sensor data;

[0018] Construct a Bayesian network system for defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node;

[0019] Construct a system based on improved DS evidence theory for defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidences of multiple sensors, reallocating the conflicting evidences, calculating the weights of each evidence and substituting them into the improved DS synthesis formula to calculate the confidence, and outputting the weight values of each sensor;

[0020] Construct a user feedback system for defining states and actions through a reinforcement learning mechanism, calculating the immediate reward and updating the Q value, and calculating the weight values of each sensor according to the updated Q value;

[0021] Input the weight values corresponding to the three systems into the weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function.

[0022] According to the adjusted weights of the three systems and in combination with the weight distribution of each system corresponding to each sensor, the optimized sensor weight values are obtained.

[0023] As a further technical solution, obtaining multi-source data further includes:

[0024] Collecting vehicle exterior sensor data, vehicle interior sensor data, and environmental sensor data;

[0025] Performing time synchronization and spatial alignment on the collected sensor data of each type;

[0026] Performing noise filtering on the sensor data of each type after time synchronization and spatial alignment.

[0027] As a further technical solution, defining each node includes: defining a sensor node, an environmental node, a user feedback node, and a risk node.

[0028] As a further technical solution, inputting the weight values corresponding to the three systems into a weight fusion network includes:

[0029] Setting the initial weights of the weight fusion network, where the initial weights include a Bayesian network system weight, an improved DS evidence theory system weight, and a user feedback system weight;

[0030] The input layer of the weight fusion network receives the initial weight vector and passes it to the hidden layer for feature extraction and non-linear transformation, and outputs the optimized system weights.

[0031] As a further technical solution, constructing a loss function based on user feedback and perception error, and adjusting the weight distribution of the three systems by optimizing the loss function includes:

[0032] Constructing the loss function as: , where α is a weight coefficient used to balance the importance of perception error and user satisfaction, L perception is the perception error, and L user is the user feedback;

[0033] Performing optimization through forward propagation, loss calculation, backpropagation, and parameter update to dynamically adjust the weights of each system.

[0034] As a further technical solution, the optimized sensor weight values are as follows:

[0035] ,

[0036] where, represents the fused sensor data, denotes the output data of the i-th sensor, which are respectively generated by the Bayesian network system, the improved DS evidence theory-based system, and the user feedback system, representing the weight allocation of each system to the sensors, respectively representing the importance of the Bayesian network system, the improved DS evidence theory-based system, and the user feedback system in the final fusion.

[0037] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization system that fuses multi-source data and user feedback, including:

[0038] A first main module for acquiring multi-source data, including external vehicle sensor data, internal vehicle sensor data, and environmental sensor data;

[0039] A second main module for constructing a Bayesian network system, by defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node;

[0040] A third main module for constructing an improved DS evidence theory-based system, by defining the output of each sensor as evidence, calculating a conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocating the conflicting evidence, calculating the weight of each evidence and substituting it into the improved DS synthesis formula to calculate the confidence level, and outputting the weight value of each sensor;

[0041] A fourth main module for constructing a user feedback system, by means of a reinforcement learning mechanism, defining states and actions, calculating immediate rewards and updating Q values, and calculating the weight values of each sensor according to the updated Q values;

[0042] A fifth main module for inputting the weight values corresponding to the three systems into a weight fusion network, constructing a loss function based on user feedback and perception error, and adjusting the weight allocation of the three systems by optimizing the loss function;

[0043] A sixth main module for obtaining the optimized sensor weight values according to the adjusted weights of the three systems and combining the weight allocation of each system corresponding to each sensor.

[0044] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization system that fuses multi-source data and user feedback, including an internal vehicle sensor, an external vehicle sensor, an environmental sensor, and a processor. The internal vehicle sensor, the external vehicle sensor, and the environmental sensor respectively collect external vehicle data, internal vehicle data, and environmental data and transmit them to the processor, and the processor is used to implement the optimization of the autonomous driving sensor weight by using the method described above.

[0045] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization device that fuses multi-source data and user feedback, including a memory and a processor. The memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to execute the steps of the method for optimizing sensor weights by fusing multi-source data and user feedback.

[0046] According to one aspect of the specification of the present invention, there is provided a non-transitory computer-readable storage medium that stores computer instructions, and the computer instructions cause the computer to execute the steps of the method for optimizing sensor weights by fusing multi-source data and user feedback.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] By combining a Bayesian network, an improved DS evidence theory, and a reinforcement learning mechanism, the present invention significantly improves the perception accuracy, environmental adaptability, and user satisfaction of the autonomous driving system. The Bayesian network calculates the reliability weights of sensors by constructing the probabilistic dependence relationships among sensors, the environment, and user feedback, reducing the uncertainty of sensor data; the improved DS evidence theory generates the fused confidence by processing the conflicting evidence among sensors, further improving the accuracy of the perception results; the user feedback system enhances the safety and comfort of the system through the reinforcement learning mechanism. These technical features work together to significantly improve the perception accuracy of the system.

[0049] Meanwhile, by combining the external vehicle sensor data, user feedback data, and environmental data, the present invention dynamically adjusts the sensor weights, enabling the system to adapt to different environmental conditions. For example, under rainy conditions, the system automatically reduces the weight of the camera and increases the weight of the lidar to ensure that the perception accuracy is not affected by the environment. In addition, through the reinforcement learning mechanism, the system can dynamically adjust the sensor weights according to user feedback data (such as fatigue level, attention distribution, voice commands, etc.), enhancing the safety and comfort of the system. When the user shows fatigue, the system automatically adjusts the sensor weights to enhance the safety of the system, thereby significantly improving user satisfaction.

[0050] Finally, by designing a weight fusion network, the present invention synthesizes the outputs of the Bayesian network, DS evidence theory, and user feedback mechanism, dynamically adjusts the system weights, reduces unnecessary calculations and energy consumption, and lowers the energy consumption and computational complexity of the system. These technical features enable the system to improve the operational simplicity and stability while ensuring performance. Brief Description of the Drawings

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 It is a schematic flowchart of a sensor weight optimization method that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0053] Figure 2 It is a data processing flowchart of a sensor weight optimization method that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0054] Figure 3 It is a schematic structural diagram of a sensor weight optimization system that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0055] Figure 4 It is a schematic structural diagram of a sensor weight optimization system that fuses multi-source data and user feedback provided by another embodiment of the present invention. Detailed implementation manners

[0056] The method of the present invention fuses multi-source sensor data (such as cameras, lidars, millimeter-wave radars, etc.), environmental data (such as weather, lighting, road types, etc.), and user feedback data (such as fatigue, attention distribution, voice commands, etc.), and combines Bayesian networks, improved DS evidence theory, and reinforcement learning mechanisms to dynamically optimize sensor weights, improve the perception accuracy, environmental adaptability, and user satisfaction of the autonomous driving system. The present invention is particularly suitable for autonomous driving decision support in complex scenarios, can effectively handle multi-source data conflicts, environmental changes, and user requirements, and provides technical guarantees for the safety and reliability of the autonomous driving system.

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Additionally, the technical features in each embodiment or single embodiment provided by the present invention can be arbitrarily combined with each other to form a new technical solution. Such combination is not restricted by the order of steps and / or the structural composition mode, but must be based on the fact that those of ordinary skill in the art can implement it. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0058] The present invention provides a method for optimizing sensor weights by integrating multi-source data and user feedback. Please refer to Figure 1 and Figure 2 , which specifically includes the following steps:

[0059] Step S1: Data collection and preprocessing, including multi-source data acquisition, time synchronization, spatial alignment, and noise filtering.

[0060] In the multi-source data acquisition step, the vehicle is equipped with external sensors (such as cameras, lidar, millimeter-wave radars), internal sensors (such as cameras, microphones), and environmental sensors (such as cameras, photosensors, temperature sensors) for multi-source data acquisition.

[0061] The external sensor data includes: camera data , representing the image data collected at time t; lidar data , representing the point cloud at time t; millimeter-wave radar data , representing the distance measurement value at time t.

[0062] The internal sensor data includes: user behavior data , representing the user behavior characteristics (such as fatigue degree, attention distribution) at time t; voice commands , representing the voice input at time t. The in-vehicle microphone captures the driver's voice commands for voice interaction and control functions; user satisfaction , representing the driver's subjective evaluation of the autonomous driving system, obtained through a real-time feedback mechanism for optimizing system performance.

[0063] The environmental data includes: weather , representing the weather condition at time t; light , representing the light intensity at time t.

[0064] Step S1.2: Time synchronization. Since the sampling frequencies of different sensors are different, they need to be aligned to a unified timestamp , and an interpolation function is used to align the data.

[0065] For the data of sensor i at time t , the data at the unified time is obtained through interpolation:

[0066]

[0067] Step S1.3: Spatial alignment. The data of different sensors are in different coordinate systems and need to be mapped to the vehicle coordinate system. An extrinsic parameter matrix (rotation matrix and translation vector ) is used to align the data.

[0068] The lidar point cloud is aligned to the camera coordinate system :

[0069]

[0070] Step S1.4: Noise filtering.

[0071] There is noise in the sensor data and filtering is required. Kalman filter (KF) or Extended Kalman filter (EKF) is used to filter the noise.

[0072] The prediction-update formula of the Kalman filter:

[0073] Prediction:

[0074] ,

[0075] Update:

[0076] ,

[0077] represents the state prediction value at time k, based on the information at time k-1; F represents the state transition matrix, represents the state estimation value at time k-1 (the optimal estimate after the update step); B represents the control input matrix; represents the control input at time k (if any). represents the prediction error covariance matrix at time k, indicating the uncertainty of the state prediction; represents the error covariance matrix at time k-1 (the optimal estimate after the update step); Q represents the process noise covariance matrix. represents the Kalman gain; H represents the observation matrix; R represents the observation noise covariance matrix. represents the state update value at time k; represents the actual observation value at time k; represents the observation prediction based on the predicted state; represents the observation residual. represents the updated error covariance matrix at time k; represents the identity matrix.

[0078] It should be noted that in step S1, more advanced sensors (such as high-resolution cameras, solid-state lidars, 4D millimeter-wave radars) can be used or the number of sensors can be increased (such as multi-camera systems, multi-lidar systems) to provide richer and more accurate data. In actual applications, although the aforementioned alternative solutions can improve data quality, they will increase system costs and computational complexity. The solution of the present invention can achieve high perception accuracy under the existing sensor configuration by optimizing sensor weights.

[0079] In data preprocessing, more complex filtering algorithms (such as particle filtering, unscented Kalman filtering) or deep learning-based preprocessing methods (such as neural network-based denoising algorithms) can also be used to better handle noise and nonlinear problems. Although the aforementioned alternative solutions can improve data quality, they will increase computational complexity and real-time requirements. The solution of the present invention uses Kalman filtering or extended Kalman filtering and can effectively handle noise while ensuring real-time performance.

[0080] Step S2: Construct a Bayesian network system for defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node. Specifically, it includes:

[0081] Step S2.1: Node definition.

[0082] Sensor nodes (S): cameras, lidars, millimeter-wave radars, etc.

[0083] Environmental nodes (E): weather, lighting, road type, etc.

[0084] User feedback nodes (U): driver fatigue, attention distribution, operation habits, etc.

[0085] Risk nodes (R): system risk levels (low, medium, high).

[0086] Step S2.2: Construction of the conditional probability table (CPT).

[0087] The conditional probability table (CPT) is used to describe the dependency relationships between nodes in a Bayesian network. The CPT for each node defines the conditional probability distribution of that node under different states of its parent nodes. represents the reliability of the sensor under a specific environment There are several methods for initializing the CPT. One method is to use a large amount of historical data and calculate the conditional probabilities between nodes through statistical methods or machine learning algorithms (such as maximum likelihood estimation, Bayesian estimation). Alternatively, in the absence of sufficient historical data, the CPT can be manually defined based on the expert knowledge of the domain.

[0088] Step S2.3: Inference process.

[0089] The goal of the inference process is to calculate the posterior probability of the risk node R through the Bayesian network and derive the weight values of each sensor. The risk node R represents the overall risk level of the system (low, medium, high), and its state is determined by the joint probability distribution of the sensor node S, the environment node E, and the user feedback node U.

[0090]

[0091] where, is the joint probability of the sensor, the environment, and the user feedback given the risk level, is the prior probability of the risk node, is the normalization constant, and its calculation formula is:

[0092]

[0093] Here, represents all possible states of the risk node. To simplify the calculation, the joint probability can be factorized as the product of the conditional probabilities of each node:

[0094]

[0095] where, , and represent the sets of parent nodes of the sensor node , the environment node and the user feedback node respectively.

[0096] Step S2.4: Sensor weight value calculation.

[0097] Through the inference process of the Bayesian network, the weight value of each sensor can be calculated. Using Bayesian network inference, calculate the reliability probability of each sensor , the calculated sensor reliability probability is normalized to obtain the weight value of each sensor , where represents the weight value of the i-th sensor under Bayesian network derivation, and the output is used for subsequent weighted fusion.

[0098] It should be noted that Bayesian network inference can also use other probability inference methods (such as Markov random field, conditional random field) or deep learning-based probability models (such as variational autoencoder, generative adversarial network). These alternative solutions can handle more complex dependencies. Although the aforementioned alternative solutions can improve the inference accuracy, they will increase the model complexity and computational cost. The solution of the present invention uses a Bayesian network, which can reduce the computational complexity while ensuring the inference accuracy.

[0099] Step S3: Construct a theory based on the improved DS evidence weight calculation to define the output of each sensor as evidence, calculate the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocate the conflicting evidence, calculate the weight of each evidence and substitute it into the improved DS synthesis formula to calculate the confidence, and output the weight value of each sensor.

[0100] Specifically, it includes:

[0101] Step S3.1: Evidence definition.

[0102] The output of each sensor is used as evidence to support or deny a certain hypothesis. For example, the detection result of the camera can be used as evidence for the existence of the target, and the obstacle distance of the lidar can be used as evidence for the position of the obstacle. Each piece of evidence i is represented by the basic probability assignment (BPA) to represent the degree of support for hypothesis A. The BPA satisfies the following conditions: . Where represents the set of all possible hypotheses.

[0103] Step S3.2: Conflict handling.

[0104] When there is a conflict among the evidence of multiple sensors, the conflict coefficient is used to quantify the degree of this conflict. The calculation formula of the conflict coefficient is:

[0105]

[0106] Define as the average support degree of the evidence for hypothesis A, and its calculation formula is: . In order to effectively handle the conflicting evidence, an improved conflict allocation function is introduced, and its calculation formula is: , and this function reduces the impact of the conflict on the fusion result by reallocating the conflicting evidence.

[0107] Step S3.3: Weight assignment.

[0108] The weight assignment is based on sensor reliability and environmental importance, and the comprehensive weight represents the importance of the i-th evidence in the fusion process. The probability calculated in the Bayesian network is used as the sensor reliability weight, as follows: . Since the performance of different sensors varies in different environments, the environmental sensor weight is defined , where represents the performance score of the i-th sensor in environment E. An example of the scoring rule is as follows:

[0109] For each sensor, its performance score is defined according to the environmental data. For example:

[0110] Camera: On sunny days, the performance score is 0.9; on rainy days, the performance score is 0.5 (since rain may cause image blurring); at night, the performance score is 0.4 (insufficient lighting).

[0111] LiDAR: On sunny days, the performance score is 0.8; on rainy days, the performance score is 0.7 (less affected by rain); at night, the performance score is 0.8 (not affected by lighting).

[0112] Millimeter-wave radar: On sunny days, the performance score is 0.7; on rainy days, the performance score is 0.6 (less affected by rain); at night, the performance score is 0.7 (not affected by lighting).

[0113] It should be noted that the above definitions can be made through historical data or expert experience (the same as the Bayesian network part).

[0114] Step S3.4: Weight fusion.

[0115] The comprehensive weight is the weighted sum of the sensor reliability weight and the environmental importance weight, and is used to represent the importance of the i-th evidence in the fusion process. Its formula is:

[0116]

[0117] where is the weight coefficient, which is used to balance sensor reliability and environmental importance.

[0118] Step S3.5: Update the confidence.

[0119] Substitute the comprehensive weight into the improved DS synthesis formula to obtain the fused confidence :

[0120]

[0121] wherein is the comprehensive weight of the i-th evidence, is the interaction coefficient between evidences i and j.

[0122] Step S3.6: Sensor weight calculation.

[0123] Normalize the fused confidence to obtain the final weight value of each sensor. The formula is:

[0124]

[0125] wherein, represents the weight value of the i-th sensor under the Dempster-Shafer theory derivation. Output the weight value of each sensor for subsequent weighted fusion.

[0126] Step S4: Construct a user feedback system to define states and actions through a reinforcement learning mechanism, calculate the immediate reward and update the Q value, and calculate the weight value of each sensor according to the updated Q value.

[0127] Specifically including:

[0128] Step S4.1: Feedback collection.

[0129] The user feedback mechanism obtains the behavior and status data of the driver or passengers in real time through in-vehicle sensors (such as cameras, steering wheel torque sensors, voice interaction systems, etc.). These data include the driver's fatigue level, attention distribution, voice commands, etc., and are used to evaluate the driver's status and satisfaction with the autonomous driving system. Define as the user feedback vector, where F is the fatigue level, A is the attention, and V is the voice command.

[0130] Step S4.2: Feedback processing.

[0131] The user feedback data is processed through a reinforcement learning mechanism to dynamically adjust the sensor weights. The core of reinforcement learning is Q value update, and its formula is:

[0132]

[0133] where: represents the value of action in state , is the immediate reward, is the learning rate, is the discount factor, represents in the next state

[0134] Under the condition, select the action that maximizes the Q value . The state corresponds to the current state of the system, including environmental

[0135] data (such as weather, light), user feedback data (such as fatigue, attention), and sensor data (such as the output of cameras, lidar). The action corresponds to the adjustment of the sensor weights. For example, increasing the weight of the camera or reducing the weight of the lidar. The immediate reward is calculated by combining user satisfaction and system performance. The formula is: , where represents the user satisfaction score, which is calculated from user feedback data (such as fatigue, attention, voice commands). represents the system performance score, which is calculated from indicators such as perception error and decision accuracy. represents the weight coefficient, which is used to balance the importance of user satisfaction and system performance and is determined by the user.

[0136] Step S4.3: Calculation of sensor weight values.

[0137] According to the updated Q value, calculate the weight value of each sensor. The formula is:

[0138]

[0139] where represents the weight value of the i-th sensor under the user feedback mechanism. Finally, output the weight value of each sensor for subsequent fusion.

[0140] Step S5: Input the weight values corresponding to the three systems into the weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function.

[0141] Specifically include:

[0142] Step S5.1: Initial weight definition.

[0143] The initial weights of the weight fusion network include the Bayesian network system weight ( ), the DS evidence theory system weight ( ), and the user feedback system weight ( ). The initial weights can be set to equal weights (such as ) or adjusted according to the specific environment. For example, increasing the weights of the Bayesian network and DS evidence theory and reducing the weight of user feedback under bad weather conditions.

[0144] Step S5.2: Fusion network structure

[0145] The input layer of the weight fusion network receives the initial weight vector , and passes it to the hidden layer for feature extraction and non-linear transformation. The hidden layer consists of multiple neural networks. The calculation formula of the first layer is , where is the weight matrix, is the bias vector, is the activation function (such as ReLU). The calculation formula of the second layer is , where is the weight matrix, is the bias vector. The output layer generates the optimized system weight , where is the weight matrix, is the bias vector. The weight of the output layer is used for subsequent sensor weight fusion.

[0146] Step S5.3: Optimization method

[0147] The optimization process of the weight fusion network includes loss function design and parameter update. The loss function consists of two parts: perception error and user satisfaction. The perception error measures the difference between the sensor output and the true value, and the formula is , where is the output of the i-th sensor, is the true value, and N is the number of sensors. The user satisfaction measures the subjective evaluation of the user on the system performance, and the formula is , where represents the satisfaction score of the j-th user, and M is the number of users. The total loss function combines the perception error and user satisfaction, and the formula is , where α is the weight coefficient, which is used to balance the importance of the perception error and user satisfaction. The optimization process includes forward propagation, loss calculation, backpropagation and parameter update. Forward propagation calculates the output of the fusion network . Calculating the loss calculates the total loss according to the perception error and user satisfaction . Backpropagation calculates the gradient of the loss function with respect to the network parameters . Parameter update uses the gradient descent method to update the network parameters, and the formula is , where η is the learning rate, which controls the step size of parameter update. Through the above optimization process, the weight fusion network can dynamically adjust the system weight to ensure that the system can operate in an optimal state under different environmental and user feedback conditions.

[0148] It should be noted that for the optimization of the weight fusion network, other network structures (such as convolutional neural networks, recurrent neural networks) or rule-based fusion methods (such as weighted average, voting method) can be used to handle different types of fusion problems. Although the aforementioned alternative solutions can improve the fusion effect, they will increase the model complexity and computational cost. The solution of the present invention uses a multi-layer neural network for weight fusion, which can reduce the computational complexity while ensuring the fusion effect.

[0149] Step S6: Weighted fusion and output. According to the adjusted weights of the three systems, combined with the weight distribution of each system for each sensor, the optimized sensor weight values are obtained.

[0150] Step S6.1: Sensor weight output.

[0151] The sensor weight output module generates the final sensor weights based on the optimized system weights and the weight distribution of each system for the sensors ( ), and is used for the weighted fusion of the sensed data, where are calculated through the weight fusion network, and respectively represent the importance of the Bayesian network system, the DS evidence theory system, and the user feedback system in the final fusion. The weight distribution of each system for the sensors ( ) are respectively generated by the Bayesian network, the DS evidence theory, and the user feedback mechanism, reflecting the reliability, conflict handling ability, and user preferences of the sensors under different conditions.

[0152] Step S6.2: Calculation of the final sensor weights.

[0153] The final sensor weights are the weighted sum of the optimized system weights and the weight distribution of each system for the sensors. The calculation formula is:

[0154]

[0155] Step S6.3: Weighted fusion.

[0156] The final sensor weights are used for the weighted fusion of the sensed data. The formula is:

[0157]

[0158] is the fused sensor data, is the output data of the i-th sensor. The final output is the weighted fused sensor data, which is used for subsequent decision-making or analysis.

[0159] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a sensor weight optimization system that integrates multi-source data and user feedback, and this system is used to execute the method for optimizing the sensor weight by integrating multi-source data and user feedback in the above method embodiments.

[0160] See Figure 3 , this system includes: a first main module for acquiring multi-source data, including vehicle exterior sensor data, vehicle interior sensor data, and environmental sensor data; a second main module for constructing a Bayesian network system by defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; a third main module for constructing an improved DS evidence theory system by defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocating the conflicting evidence, calculating the weight of each evidence and substituting it into the improved DS synthesis formula to calculate the confidence, and outputting the weight value of each sensor; a fourth main module for constructing a user feedback system by a reinforcement learning mechanism, defining states and actions, calculating immediate rewards and updating Q values, and calculating the weight value of each sensor according to the updated Q values; a fifth main module for inputting the weight values corresponding to the three systems into a weight fusion network, constructing a loss function based on user feedback and perception error, and adjusting the weight distribution of the three systems by optimizing the loss function; a sixth main module for obtaining the optimized sensor weight values according to the adjusted weights of the three systems and combining the weight distribution of each sensor corresponding to each system.

[0161] The sensor weight optimization system that integrates multi-source data and user feedback provided by the embodiment of the present invention aims at the situation that the existing multi-source sensor data fusion methods in autonomous driving systems lack sufficient consideration of user feedback and are difficult to dynamically adjust the system weights according to user needs; at the same time, the existing methods usually use a certain technology alone (such as Bayesian network, DS evidence theory, or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance, and user feedback, resulting in insufficient perception accuracy, environmental adaptability, and user satisfaction of the system in complex environments. By adopting Figure 3 several modules in , through combining Bayesian network, improved DS evidence theory, and reinforcement learning mechanism, the sensor weights are dynamically optimized to improve the comprehensive performance of the system.

[0162] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in setting corresponding functional modules, and the principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art, on the basis of the above system embodiments, refer to the specific technical solutions in other method embodiments, obtain corresponding technical means by combining technical features, and the technical solutions constituted by these technical means, and on the premise of ensuring the practicability of the technical solutions, improve the modules in the above system embodiments to obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0163] Based on the same inventive concept as the above embodiments, an embodiment of the present invention further provides a sensor weight optimization system that integrates multi-source data and user feedback, as Figure 4 shown, including in-vehicle sensors, out-of-vehicle sensors, environmental sensors, and a processor. The in-vehicle sensors, out-of-vehicle sensors, and environmental sensors respectively collect out-of-vehicle data, in-vehicle data, and environmental data and transmit them to the processor. The processor is used to implement the optimization of the weights of the autonomous driving sensors by using the method for optimizing the weights of the autonomous driving sensors based on multi-source data fusion and user feedback.

[0164] Based on the same inventive concept as the above embodiments, an embodiment of the present invention further provides a sensor weight optimization device that integrates multi-source data and user feedback, including a memory and a processor. The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for optimizing the weights of the sensors that integrates multi-source data and user feedback.

[0165] Based on the same inventive concept as the above embodiments, an embodiment of the present invention further provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the steps of the method for optimizing the weights of the sensors that integrates multi-source data and user feedback.

[0166] Embodiment 1

[0167] Suppose an autonomous vehicle is driving on an urban road. The environmental condition is rainy, the lighting condition is daytime, and the road type is the main urban road. The driver in the vehicle shows mild fatigue, and the attention is distributed on the road ahead without issuing voice commands. The system needs to integrate data from cameras, lidar, and millimeter-wave radars to generate a comprehensive perception result and dynamically adjust the sensor weights according to user feedback.

[0168] The implementation steps are as follows:

[0169] Step 1, data acquisition and preprocessing.

[0170] · Vehicle exterior sensor data: The camera captures images of the road ahead, the lidar generates point cloud data of obstacles ahead, and the millimeter-wave radar measures the distance to the vehicle ahead.

[0171] · Environmental data: The weather is rainy, the lighting is during the day, and the road type is the urban arterial road.

[0172] · User feedback data: The in-vehicle camera monitors that the driver is slightly fatigued, with attention distributed on the road ahead and no voice commands issued.

[0173] · Synchronize the data of the camera, lidar, and millimeter-wave radar in time and align them spatially.

[0174] · Use Kalman filtering to filter the noise of the sensor data.

[0175] Step 2, Bayesian network inference.

[0176] · Construct a Bayesian network, defining sensor nodes (camera, lidar, millimeter-wave radar), environmental nodes (rainy day, daytime, urban arterial road), user feedback nodes (slightly fatigued, attention ahead), and risk nodes (low, medium, high risk).

[0177] · Describe the dependence relationship between nodes through the conditional probability table and calculate the reliability weights of the sensors. For example, under rainy conditions, the reliability weight of the camera decreases, and the reliability weight of the lidar increases.

[0178] Step 3, improved DS evidence theory fusion.

[0179] · Take the outputs of the camera, lidar, and millimeter-wave radar as evidence and define the basic probability assignment.

[0180] · Calculate the conflict coefficient and reassign the conflicting evidence through the improved conflict assignment function.

[0181] · Combine the sensor reliability weights and environmental importance weights to calculate the comprehensive weight.

[0182] · Substitute the comprehensive weight into the improved DS synthesis formula to generate the fused confidence.

[0183] Step 4, user feedback optimization.

[0184] · Through the reinforcement learning mechanism, dynamically adjust the sensor weights in combination with user feedback data (slightly fatigued, attention ahead).

[0185] · Calculate the immediate reward and update the Q value, and calculate the sensor weights under the user feedback mechanism according to the updated Q value.

[0186] Step 5, Optimization of the Weight Fusion Network.

[0187] · Design a weight fusion network to comprehensively integrate the outputs of the Bayesian network, DS evidence theory, and the user feedback mechanism to generate optimized system weights.

[0188] · Use the final sensor weights for weighted fusion of the sensed data to generate a comprehensive sensing result.

[0189] Step 6, Weighted Fusion and Output.

[0190] · Use the final sensor weights for weighted fusion of the sensed data to generate a comprehensive sensing result.

[0191] · Output the data after weighted fusion for subsequent decision-making or analysis.

[0192] Comparative Example 1

[0193] Compared with Example 1, the user feedback mechanism is not used in this comparative example.

[0194] The implementation steps are as follows:

[0195] Step 1, Data Acquisition and Preprocessing: The same as in Example 1.

[0196] Step 2, Construction of the Bayesian Network: The same as in Example 1.

[0197] Step 3, Inference Calculation: The same as in Example 1.

[0198] Step 4, Without Using the User Feedback Mechanism: Directly use the sensor weights generated by the Bayesian network for data fusion.

[0199] Result Analysis:

[0200] · When the user shows fatigue, the system cannot dynamically adjust the sensor weights, resulting in a decrease in safety.

[0201] · The user satisfaction is low, and the system performance is poor in complex environments.

[0202] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the processes Figure 1 one process or multiple processes and / or blocks Figure 1means for the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one Figure 1 process or processes and / or boxes Figure 1 means for the functions specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one Figure 1 process or processes and / or boxes Figure 1 means for the functions specified in one or more boxes.

[0205] In summary, in the above embodiments, the present invention uses a method of making decisions by combining a neural network with multiple decision-making models such as DS, Bayesian, and reinforcement learning, designs a weight fusion network, and combines the DS evidence theory, Bayesian network, and reinforcement learning mechanism to dynamically optimize the sensor weights. The present invention combines models of various complex factors such as external vehicle sensors, user feedback, and environmental factors, constructs a comprehensive model, and combines external vehicle sensor data (such as cameras, lidar, millimeter-wave radars, etc.), user feedback data (such as fatigue degree, attention distribution, voice commands, etc.), and environmental data (such as weather, light, road type, etc.) to dynamically optimize the sensor weights. The present invention uses user feedback and sensor errors together as a loss to guide the optimization of parameters such as sensors, designs a loss function based on user feedback and sensor errors, and combines perception errors and user satisfaction to guide the optimization of sensor weights.

[0206] The terms "comprising" and "having" and any variations thereof in the description and claims of the present invention and the above drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or apparatuses.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing sensor weights by integrating multi-source data and user feedback, characterized in that Including: Obtain multi-source data, including vehicle external sensor data, vehicle internal sensor data, and environmental sensor data; Construct a Bayesian network system for defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; Construct an improved DS evidence theory system for defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidences of multiple sensors, reallocating the conflicting evidences, calculating the weight of each evidence and substituting it into the improved DS combination formula to calculate the confidence level, and outputting the weight value of each sensor; the improved DS combination formula is: ， Among them is the combined weight of the th evidence, and is the interaction coefficient between the evidences, is the th evidence's degree of support for hypothesis A, is the evidence conflict distribution function, , where is the average degree of support of the evidence for hypothesis A, , is the conflict coefficient, ; Construct a user feedback system for defining states and actions through a reinforcement learning mechanism, calculating the immediate reward and updating the Q value, and calculating the weight value of each sensor according to the updated Q value; Input the weight values corresponding to the three systems into a weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function; According to the adjusted weights of the three systems, combined with the weight allocation of each sensor corresponding to each system, obtain the optimized sensor weight values.

2. The sensor weight optimization method for integrating multi-source data and user feedback according to claim 1, wherein Obtaining multi-source data further includes: Collect vehicle external sensor data, vehicle internal sensor data, and environmental sensor data; Perform time synchronization and spatial alignment on the collected sensor data of each sensor; Perform noise filtering on the sensor data of each sensor after time synchronization and spatial alignment.

3. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 1, characterized in that Define each node, including: defining sensor nodes, environmental nodes, user feedback nodes, and risk nodes.

4. The method for optimizing sensor weights by fusing multi-source data and user feedback according to claim 1, wherein Input the weight values corresponding to the three systems into a weight fusion network, including: Set the initial weights of the weight fusion network, and the initial weights include the weights of the Bayesian network system, the improved DS evidence theory system, and the user feedback system; The input layer of the weight fusion network receives the initial weight vector and passes it to the hidden layer for feature extraction and non-linear transformation, and outputs the optimized system weights.

5. The method for optimizing sensor weights by fusing multi-source data and user feedback according to claim 4, characterized in that, Construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function, including: The constructed loss function is as follows: , where α is the weight coefficient, which is used to balance the importance of the perception error and the user satisfaction. L perception is the perception error, and L user is the user feedback; Optimize through forward propagation, loss calculation, backpropagation, and parameter update, and dynamically adjust the weights of each system.

6. The sensor weight optimization method for integrating multi-source data and user feedback according to claim 1, wherein The optimized sensor weight values are as follows: ， Among them, represents the fused sensor data, represents the output data of the th sensor, which are generated by the Bayesian network system, the improved DS evidence theory system, and the user feedback system respectively, and represent the weight allocation of each system to the sensor. respectively represent the importance of the Bayesian network system, the improved DS evidence theory system, and the user feedback system in the final fusion.

7. A sensor weight optimization system that integrates multi-source data and user feedback, characterized in that Including: The first main module is used to obtain multi-source data, including vehicle external sensor data, vehicle internal sensor data, and environmental sensor data; The second main module is used to construct a Bayesian network system by defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; The third main module is used to construct an improved DS evidence theory system by defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidences of multiple sensors, reallocating the conflicting evidences, calculating the weight of each evidence and substituting it into the improved DS combination formula to calculate the confidence level, and outputting the weight value of each sensor; the improved DS combination formula is: ， where is the combined weight of the th piece of evidence, is the interaction coefficient between evidence and , is the degree of support of the th piece of evidence for hypothesis A, is the evidence conflict allocation function, , where is the average degree of support of the evidence for hypothesis A, , is the conflict coefficient, ; The fourth main module is used to construct a user feedback system by defining states and actions through a reinforcement learning mechanism, calculating the immediate reward and updating the Q value, and calculating the weight value of each sensor according to the updated Q value; The fifth main module is used to input the weight values corresponding to the three systems into the weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function; The sixth main module is used to obtain the optimized sensor weight values according to the adjusted weights of the three systems and in combination with the weight allocation of each sensor corresponding to each system.

8. Sensor weight optimization system integrating multi-source data and user feedback, characterized in that, It includes in-vehicle sensors, out-of-vehicle sensors, environmental sensors and a processor. The in-vehicle sensors, out-of-vehicle sensors and environmental sensors respectively collect out-of-vehicle data, in-vehicle data and environmental data and transmit them to the processor. The processor is used to optimize the weights of the autonomous driving sensors by using the method described in any one of claims 1 to 6.

9. A sensor weight optimization device that fuses multi-source data and user feedback, characterized in that, It includes a memory and a processor. The memory stores program instructions executed by the processor. The processor calls the program instructions to execute the steps of the method for optimizing the weights of sensors by fusing multi-source data and user feedback described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the steps of the method for optimizing the weights of sensors by fusing multi-source data and user feedback described in any one of claims 1 to 6.

Images