Sensor weight optimization method and system fusing multi-source data and user feedback

Abstract

The invention belongs to the technical field of automatic driving sensor data fusion, and particularly relates to a sensor weight optimization method fusing multi-source data and user feedback. According to the method, multi-source sensor data (such as a camera, a laser radar and a millimeter-wave radar), environment data (such as weather, illumination and road types) and user feedback data (such as fatigue, attention distribution and voice instructions) are fused, a Bayesian network, an improved DS evidence theory and a reinforcement learning mechanism are combined, the sensor weight is dynamically optimized, and the user experience is improved. And the sensing precision, the environmental adaptability and the user satisfaction of the automatic driving system are improved. The method is especially suitable for automatic driving decision support in a complex scene, can effectively process multi-source data conflicts, environment changes and user requirements, and provides technical guarantee for safety and reliability of an automatic driving system.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving sensor data fusion, and particularly relates to a method and system for optimizing sensor weights by fusing multi-source data and user feedback. Background Art

[0002] In the field of autonomous driving technology, multi-source sensor data fusion is one of the key technologies for realizing environmental perception and decision support. Existing research mainly focuses on how to effectively fuse data from different sensors (such as cameras, lidars, millimeter-wave radars, etc.) to improve the perception accuracy and robustness of the system.

[0003] Multi-source sensor data fusion is one of the core research directions in the field of autonomous driving. Existing research usually adopts probabilistic reasoning methods (such as Bayesian networks) and evidence theory methods (such as DS evidence theory, also known as Dempster-Shafer evidence theory) to handle the uncertainty and conflict of sensor data. For example, Reference [1] proposed a multi-sensor fusion method based on Bayesian networks, which calculates the reliability weights of sensors by constructing the probabilistic dependence relationship between sensors and the environment. Reference [2] uses an improved DS evidence theory to process the conflicting evidence between sensors and generates the fused confidence. However, these methods usually consider less user feedback and it is difficult to dynamically adjust the system weights according to user needs.

[0004] In recent years, with the rapid development of artificial intelligence and machine learning technologies, researchers have gradually incorporated user feedback into the optimization process of autonomous driving systems. For example, user behavior data (such as fatigue, attention distribution, voice commands, etc.) is collected through in-vehicle sensors (such as cameras, steering wheel torque sensors, microphones, etc.), and analyzed in combination with algorithms such as neural networks, so that the behavior of the system is more in line with the actual scenario and user needs. For example, Reference [3] proposed a driver fatigue state recognition algorithm based on deep learning, and Reference [4] proposed using neural networks to recognize the user's voice and analyze its emotions and other information, providing technical support for autonomous driving decision optimization. These studies show that user feedback can significantly improve the adaptability and user satisfaction of the system, but existing research usually uses the user feedback mechanism alone and does not attempt to combine it with probabilistic reasoning and evidence theory, resulting in insufficient adaptability and robustness of the system in complex environments.

[0005] Deep learning and reinforcement learning technologies have also been widely applied to autonomous driving decision-making. For example, reference [5] proposed an anthropomorphic decision-making method for autonomous driving based on deep reinforcement learning, which analyzed and classified drivers' driving styles through reinforcement learning with high accuracy. Reference [6] studied and explored an autonomous driving decision-making model based on end-to-end deep learning, which directly extracted features from sensor data and generated decision instructions through deep learning. These studies show that deep learning and reinforcement learning can effectively improve the decision-making ability and adaptability of autonomous driving systems. However, existing studies usually use these technologies separately and do not attempt to combine them with multi-source sensor data fusion and user feedback mechanisms, resulting in the need to improve the comprehensive performance of the system in complex scenarios.

[0006] Although existing studies have made certain progress in multi-source sensor data fusion, user feedback optimization, and the application of deep learning and reinforcement learning, there are still the following problems and disadvantages: 1. Insufficient consideration of user feedback: Existing methods (such as Bayesian networks and DS evidence theory) usually consider user feedback less and are difficult to dynamically adjust the system weights according to user needs. For example, when users show fatigue or inattention, existing methods cannot quickly adjust the sensor weights to enhance the safety of the system.

[0007] 2. Lack of comprehensive optimization mechanism: Existing methods usually use a certain technology alone (such as Bayesian networks, DS evidence theory, or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance, and user feedback. For example, in complex scenarios, existing methods cannot consider sensor performance, environmental conditions, and user needs simultaneously, resulting in a decline in system performance.

[0008] [1] Chen Jiena, Zhang Mingzhuo, Du Dehui, et al. Autonomous Driving Behavior Decision-Making Based on Constructing RoboSim Model with Bayesian Network [J]. Journal of Software, 2023, 34(8): 3836-3852. DOI: 10.13328 / j.cnki.jos.006594.

[0009] [2] Hefei Zhongke Automatic Control System Co., Ltd. An Asynchronous Multimodal Target-Level Information Fusion Method Based on Temporal DS Theory: CN202311034586.9 [P]. 2023-11-14.

[0010] [3] Zhou Hui, Zhou Liang, Ding Qiulin. Fatigue State Recognition Algorithm Based on Deep Learning [J]. Computer Science, 2015, 42(3): 191-194, 200. DOI: 10.11896 / j.issn.1002-137X.2015.3.039.

[0011] [4] Dai Hang. Research on Voice-based Driver Emotion Recognition [D]. Heilongjiang: Harbin University of Science and Technology, 2023.

[0012] [5] Yang Chonghui. Research on Anthropomorphic Decision-making Method for Autonomous Driving Based on Deep Reinforcement Learning [D]. Chongqing: Chongqing University, 2023.

[0013] [6] Liu Wei. Research on Decision-making Model for Autonomous Driving Based on End-to-End Deep Learning [D]. Chongqing University of Technology, 2022. Summary of the Invention

[0014] In view of the fact that the multi-source sensor data fusion method in the existing autonomous driving system insufficiently considers user feedback and it is difficult to dynamically adjust the system weights according to user needs. At the same time, the existing methods usually use a certain technology alone (such as Bayesian network, DS evidence theory, or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance, and user feedback, resulting in insufficient perception accuracy, environmental adaptability, and user satisfaction of the system in complex environments. The present invention provides an autonomous driving sensor weight optimization method and system based on multi-source data fusion and user feedback, aiming to dynamically optimize the sensor weights and improve the comprehensive performance of the system by combining Bayesian network, improved DS evidence theory, and reinforcement learning mechanism.

[0015] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization method that fuses multi-source data and user feedback, including: Obtain multi-source data, including external vehicle sensor data, internal vehicle sensor data, and environmental sensor data; Construct a Bayesian network system for defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; Construct a system based on improved DS evidence theory for defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocating the conflicting evidence, calculating the weight of each evidence and substituting it into the improved DS synthesis formula to calculate the confidence, and outputting the weight value of each sensor; Construct a user feedback system for defining states and actions through a reinforcement learning mechanism, calculating the immediate reward and updating the Q value, and calculating the weight value of each sensor according to the updated Q value; Input the weight values corresponding to the three systems into a weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function; According to the adjusted weights of the three systems, combined with the weight allocation of each sensor corresponding to each system, obtain the optimized sensor weight values.

[0016] As a further technical solution, obtaining multi-source data further includes: Collecting vehicle exterior sensor data, vehicle interior sensor data, and environmental sensor data; Performing time synchronization and spatial alignment on the collected sensor data of each type; Performing noise filtering on the sensor data of each type after time synchronization and spatial alignment.

[0017] As a further technical solution, defining each node includes: defining a sensor node, an environmental node, a user feedback node, and a risk node.

[0018] As a further technical solution, inputting the weight values corresponding to the three systems into a weight fusion network, including: Setting the initial weights of the weight fusion network, where the initial weights include the Bayesian network system weight, the weight of the system based on the improved DS evidence theory, and the user feedback system weight; The input layer of the weight fusion network receives the initial weight vector and passes it to the hidden layer for feature extraction and non-linear transformation, and outputs the optimized system weights.

[0019] As a further technical solution, constructing a loss function based on user feedback and perception error, and adjusting the weight distribution of the three systems by optimizing the loss function, including: Constructing the loss function as: , where α is the weight coefficient, used to balance the importance of perception error and user satisfaction, L perception is the perception error, L user is the user feedback; Performing optimization through forward propagation, calculating the loss, backpropagation, and parameter update, and dynamically adjusting the weights of each system.

[0020] As a further technical solution, the optimized sensor weight values are as follows: , where, represents the fused sensor data, represents the output data of the i-th sensor, are respectively generated by the Bayesian network system, the system based on the improved DS evidence theory, and the user feedback system, representing the weight allocation of each system to the sensor, respectively represent the importance of the Bayesian network system, the system based on the improved DS evidence theory, and the user feedback system in the final fusion.

[0021] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization system that fuses multi-source data and user feedback, including: The first main module is used to obtain multi-source data, including vehicle external sensor data, vehicle internal sensor data, and environmental sensor data; The second main module is used to construct a Bayesian network system by defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; The third main module is used to construct an improved DS evidence theory system. By defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocating the conflicting evidence, calculating the weight of each evidence and substituting it into the improved DS synthesis formula to calculate the confidence level, and outputting the weight value of each sensor; The fourth main module is used to construct a user feedback system. Through a reinforcement learning mechanism, defining states and actions, calculating immediate rewards and updating Q values, and calculating the weight values of each sensor based on the updated Q values; The fifth main module is used to input the weight values corresponding to the three systems into a weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function; The sixth main module is used to obtain the optimized sensor weight values according to the adjusted weights of the three systems and in combination with the weight allocation of each sensor corresponding to each system.

[0022] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization system that fuses multi-source data and user feedback, including vehicle internal sensors, vehicle external sensors, environmental sensors, and a processor. The vehicle internal sensors, vehicle external sensors, and environmental sensors respectively collect vehicle external data, vehicle internal data, and environmental data and transmit them to the processor, and the processor is used to implement the optimization of the weights of autonomous driving sensors by using the method described above.

[0023] According to one aspect of the specification of the present invention, there is provided a sensor weight optimization device that fuses multi-source data and user feedback, including a memory and a processor. The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for optimizing the weights of sensors that fuses multi-source data and user feedback.

[0024] According to one aspect of the specification of the present invention, there is provided a non-transitory computer-readable storage medium that stores computer instructions, and the computer instructions cause the computer to execute the steps of the method for optimizing the weights of sensors that fuses multi-source data and user feedback.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention significantly improves the perception accuracy, environmental adaptability, and user satisfaction of the autonomous driving system by combining Bayesian networks, improved DS evidence theory, and reinforcement learning mechanisms. The Bayesian network calculates the reliability weights of sensors by constructing the probabilistic dependence relationships among sensors, the environment, and user feedback, reducing the uncertainty of sensor data; the improved DS evidence theory further improves the accuracy of perception results by processing the conflicting evidence among sensors and generating the fused confidence; the user feedback system enhances the safety and comfort of the system through the reinforcement learning mechanism. These technical features work together to significantly improve the perception accuracy of the system.

[0026] Meanwhile, the present invention dynamically adjusts the sensor weights by combining the data of external sensors, user feedback data, and environmental data, enabling the system to adapt to different environmental conditions. For example, under rainy conditions, the system automatically reduces the weight of the camera and increases the weight of the lidar to ensure that the perception accuracy is not affected by the environment. In addition, through the reinforcement learning mechanism, the system can dynamically adjust the sensor weights according to user feedback data (such as fatigue level, attention distribution, voice commands, etc.), enhancing the safety and comfort of the system. When the user shows fatigue, the system automatically adjusts the sensor weights to enhance the safety of the system, thus significantly improving user satisfaction.

[0027] Finally, the present invention designs a weight fusion network to comprehensively integrate the outputs of Bayesian networks, DS evidence theory, and user feedback mechanisms, dynamically adjusting the system weights, reducing unnecessary calculations and energy consumption, and lowering the energy consumption and computational complexity of the system. These technical features enable the system to improve the operational simplicity and stability while ensuring performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0029] Figure 1 It is a schematic flowchart of the sensor weight optimization method that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0030] Figure 2 It is a data processing flowchart of the sensor weight optimization method that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0031] Figure 3 It is a schematic structural diagram of the sensor weight optimization system that fuses multi-source data and user feedback provided by an embodiment of the present invention.

[0032] Figure 4 This is a schematic structural diagram of a sensor weight optimization system that integrates multi-source data and user feedback provided by another embodiment of the present invention. Detailed implementation manners

[0033] The method of the present invention dynamically optimizes sensor weights by integrating multi-source sensor data (such as cameras, lidars, millimeter-wave radars, etc.), environmental data (such as weather, lighting, road types, etc.), and user feedback data (such as fatigue, attention distribution, voice commands, etc.), combines Bayesian networks, improved DS evidence theory, and reinforcement learning mechanisms, and improves the perception accuracy, environmental adaptability, and user satisfaction of the autonomous driving system. The present invention is particularly applicable to autonomous driving decision-making support in complex scenarios, can effectively handle multi-source data conflicts, environmental changes, and user requirements, and provides technical guarantees for the safety and reliability of the autonomous driving system.

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. In addition, the technical features in various embodiments or individual embodiments provided by the present invention can be combined with each other arbitrarily to form a new technical solution, and this combination is not restricted by the order of steps and / or the mode of structural composition, but must be based on what can be achieved by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0035] The present invention provides a method for optimizing sensor weights by integrating multi-source data and user feedback. Please refer to Figure 1 and Figure 2 , and specifically includes the following steps: Step S1: Data collection and preprocessing, including multi-source data acquisition, time synchronization, spatial alignment, and noise filtering.

[0036] In the multi-source data acquisition step, the vehicle is equipped with external sensors (cameras, lidars, millimeter-wave radars, etc.), internal sensors (cameras, microphones, etc.), and environmental sensors (cameras, photosensors, temperature sensors, etc.) to perform multi-source data acquisition.

[0037] The external sensor data includes: camera data , representing the image data collected at time t; lidar data , representing the point cloud at time t; millimeter-wave radar data , representing the distance measurement value at time t.

[0038] In-vehicle sensor data includes: user behavior data , representing the user behavior characteristics at time t (such as fatigue, attention distribution); voice commands , representing the voice input at time t. The in-vehicle microphone captures the driver's voice commands for voice interaction and control functions; user satisfaction , representing the driver's subjective evaluation of the autonomous driving system, obtained through a real-time feedback mechanism for optimizing system performance.

[0039] Environmental data includes: weather , representing the weather condition at time t; lighting , representing the lighting intensity at time t.

[0040] Step S1.2: Time synchronization. Since the sampling frequencies of different sensors are different, they need to be aligned to a unified timestamp , using an interpolation function to align the data.

[0041] For the data of sensor i at time t , the data at the unified time is obtained through interpolation:

[0042] Step S1.3: Spatial alignment. The data of different sensors are in different coordinate systems and need to be mapped to the vehicle coordinate system. Use the extrinsic parameter matrix (rotation matrix and translation vector ) to align the data.

[0043] Lidar point cloud aligned to the camera coordinate system :

[0044] Step S1.4: Noise filtering.

[0045] There is noise in the sensor data and it needs to be filtered. Use Kalman filter (KF) or Extended Kalman filter (EKF) to filter the noise.

[0046] Prediction-update formula of Kalman filter: Prediction: , Update: , Represents the predicted state value at time k, based on the information at time k-1; F represents the state transition matrix, Represents the state estimate value at time k-1 (the optimal estimate after the update step); B represents the control input matrix; Represents the control input at time k (if any). Represents the predicted error covariance matrix at time k, indicating the uncertainty of state prediction; Represents the error covariance matrix at time k-1 (the optimal estimate after the update step); Q represents the process noise covariance matrix. Represents the Kalman gain; H represents the observation matrix; R represents the observation noise covariance matrix. Represents the state update value at time k; Represents the actual observation value at time k; Represents the observation prediction based on the predicted state; Represents the observation residual. Represents the updated error covariance matrix at time k; Represents the identity matrix.

[0047] It should be noted that in step S1, more advanced sensors (such as high-resolution cameras, solid-state lidars, 4D millimeter-wave radars) can be used or the number of sensors can be increased (such as multi-camera systems, multi-lidar systems) to provide richer and more accurate data. In actual applications, although the foregoing alternative solutions can improve data quality, they will increase system costs and computational complexity. The solution of the present invention can achieve high sensing accuracy under the existing sensor configuration by optimizing sensor weights.

[0048] In data preprocessing, more complex filtering algorithms (such as particle filtering, unscented Kalman filtering) or deep learning-based preprocessing methods (such as neural network-based denoising algorithms) can also be used to better handle noise and nonlinear problems. Although the foregoing alternative solutions can improve data quality, they will increase computational complexity and real-time requirements. The solution of the present invention uses Kalman filtering or extended Kalman filtering, which can effectively handle noise while ensuring real-time performance.

[0049] Step S2: Construct a Bayesian network system for defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node. Specifically, it includes: Step S2.1: Node definition.

[0050] Sensor nodes (S): cameras, lidars, millimeter-wave radars, etc.

[0051] Environmental nodes (E): weather, lighting, road type, etc.

[0052] User feedback node (U): Driver fatigue level, attention distribution, operating habits, etc.

[0053] Risk node (R): System risk level (low, medium, high).

[0054] Step S2.2: Construction of the conditional probability table (CPT).

[0055] The conditional probability table (CPT) is used to describe the dependency relationships between nodes in a Bayesian network. The CPT for each node defines the conditional probability distribution of that node under different states of its parent nodes. Represents the reliability of the sensor under a specific environment . Initialization methods for the CPT include using a large amount of historical data to calculate the conditional probabilities between nodes through statistical methods or machine learning algorithms (such as maximum likelihood estimation, Bayesian estimation), or manually defining the CPT relying on the expert knowledge of the domain in the absence of sufficient historical data.

[0056] Step S2.3: Inference process.

[0057] The goal of the inference process is to calculate the posterior probability of the risk node R through the Bayesian network and derive the weight values of each sensor. The risk node R represents the overall risk level of the system (low, medium, high), and its state is determined by the joint probability distribution of the sensor node S, the environment node E, and the user feedback node U.

[0058]

[0059] Among them, is the joint probability of the sensor, the environment, and the user feedback given the risk level, is the prior probability of the risk node, is the normalization constant, and the calculation formula is:

[0060] Here, represents all possible states of the risk node. To simplify the calculation, the joint probability can be factorized as the product of the conditional probabilities of each node:

[0061] Among them, , and respectively represent the sets of parent nodes of the sensor node , the environment node and the user feedback node .

[0062] Step S2.4: Calculation of sensor weight values.

[0063] Through the inference process of the Bayesian network, the weight value of each sensor can be calculated. Using Bayesian network inference, calculate the reliability probability of each sensor under the given environment E and user feedback U , and normalize the calculated sensor reliability probability to obtain the weight value of each sensor , where represents the weight value of the i-th sensor under the derivation of the Bayesian network, and the output is used for subsequent weighted fusion.

[0064] It should be noted that Bayesian network inference can also use other probability inference methods (such as Markov random field, conditional random field) or deep learning-based probability models (such as variational autoencoder, generative adversarial network). These alternative solutions can handle more complex dependencies. Although the aforementioned alternative solutions can improve the inference accuracy, they will increase the model complexity and computational cost. The solution of the present invention uses a Bayesian network, which can reduce the computational complexity while ensuring the inference accuracy.

[0065] Step S3: Construct an improved DS evidence weight calculation theory, which is used to define the output of each sensor as evidence, calculate the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocate the conflicting evidence, calculate the weight of each evidence and substitute it into the improved DS synthesis formula to calculate the confidence, and output the weight value of each sensor.

[0066] Specifically, it includes: Step S3.1: Evidence definition.

[0067] The output of each sensor is used as evidence to support or deny a certain hypothesis. For example, the detection result of the camera can be used as evidence for the existence of the target, and the obstacle distance of the lidar can be used as evidence for the position of the obstacle. Each piece of evidence i is represented by the basic probability assignment (BPA) indicating the degree of support for hypothesis A. The BPA satisfies the following conditions: . Where represents the set of all possible hypotheses.

[0068] Step S3.2: Conflict handling.

[0069] When there is a conflict among the evidence of multiple sensors, the conflict coefficient is used to quantify the degree of this conflict. The calculation formula of the conflict coefficient is:

[0070] Define is the average degree of support of the evidence for Hypothesis A, and its calculation formula is: To effectively handle conflicting evidence, an improved conflict distribution function is introduced Its calculation formula is: This function reduces the impact of conflicts on the fusion result by redistributing the conflicting evidence.

[0071] Step S3.3: Weight assignment.

[0072] The weight assignment is based on sensor reliability and environmental importance, and the comprehensive weight represents the importance of the i-th evidence in the fusion process. The probability calculated in the Bayesian network is used as the sensor reliability weight, as follows: . Since the performance of different sensors varies in different environments, the environmental sensor weight is defined , where represents the performance score of the i-th sensor in environment E. An example of the scoring rule is as follows: For each sensor, its performance score is defined according to the environmental data. For example: Camera: On sunny days, the performance score is 0.9; on rainy days, the performance score is 0.5 (since rain may cause image blurring); at night, the performance score is 0.4 (insufficient lighting).

[0073] LiDAR: On sunny days, the performance score is 0.8; on rainy days, the performance score is 0.7 (less affected by rain); at night, the performance score is 0.8 (not affected by lighting).

[0074] Millimeter-wave radar: On sunny days, the performance score is 0.7; on rainy days, the performance score is 0.6 (less affected by rain); at night, the performance score is 0.7 (not affected by lighting).

[0075] It should be noted that the above definitions can be determined through historical data or expert experience (the same as the Bayesian network part).

[0076] Step S3.4: Weight fusion.

[0077] The comprehensive weight is the weighted sum of the sensor reliability weight and the environmental importance weight, and is used to represent the importance of the i-th evidence in the fusion process. Its formula is:

[0078] where is the weight coefficient, which is used to balance sensor reliability and environmental importance.

[0079] Step S3.5: Update confidence.

[0080] The comprehensive weight Substitute into the improved DS synthesis formula to obtain the fused confidence :

[0081] where is the comprehensive weight of the i-th evidence, is the interaction coefficient between evidence i and j.

[0082] Step S3.6: Sensor weight calculation.

[0083] Normalize the fused confidence to obtain the final weight value of each sensor. The formula is:

[0084] where, represents the weight value of the i-th sensor under the DS theory derivation. Output the weight value of each sensor for subsequent weighted fusion.

[0085] Step S4: Construct a user feedback system to define states and actions through a reinforcement learning mechanism, calculate immediate rewards and update Q values, and calculate the weight values of each sensor according to the updated Q values.

[0086] Specifically including: Step S4.1: Feedback collection.

[0087] The user feedback mechanism obtains real-time behavior and status data of the driver or passengers through in-vehicle sensors (such as cameras, steering wheel torque sensors, voice interaction systems, etc.). These data include the driver's fatigue level, attention distribution, voice commands, etc., and are used to evaluate the driver's status and satisfaction with the autonomous driving system. Define as the user feedback vector, where F is the fatigue level, A is the attention, and V is the voice command.

[0088] Step S4.2: Feedback processing.

[0089] The user feedback data is processed through a reinforcement learning mechanism to dynamically adjust the sensor weights. The core of reinforcement learning is Q value update, and its formula is:

[0090] where: represents the value of action in state , is the immediate reward, is the learning rate, is the discount factor, represents in the next state

[0091] Select the action that maximizes the Q value . The state corresponds to the current state of the system, including environmental data (such as weather, light), user feedback data (such as fatigue, attention), and sensor data (such as the outputs of cameras, lidars), and the action corresponds to the adjustment of the sensor weights, for example, increasing the weight of the camera or decreasing the weight of the lidar, and the immediate reward is calculated by combining user satisfaction and system performance, and its formula is: , where represents the user satisfaction score, which is calculated from user feedback data (such as fatigue, attention, voice commands). represents the system performance score, which is calculated from metrics such as perception error and decision accuracy. represents the weight coefficient, which is used to balance the importance of user satisfaction and system performance and is determined by the user.

[0092] Step S4.3: Calculate the sensor weight values.

[0093] According to the updated Q value, calculate the weight value of each sensor, and the formula is:

[0094] where represents the weight value of the i-th sensor under the user feedback mechanism. Finally, output the weight value of each sensor for subsequent fusion.

[0095] Step S5: Input the weight values corresponding to the three systems into the weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight allocation of the three systems by optimizing the loss function.

[0096] Specifically, it includes: Step S5.1: Define the initial weights.

[0097] The initial weights of the weight fusion network include the weights of the Bayesian network system ( ), the weights of the DS evidence theory system ( ), and the weights of the user feedback system ( ). The initial weights can be set to equal weights (such as ) or adjusted according to a specific environment. For example, increase the weights of the Bayesian network and DS evidence theory and decrease the weight of user feedback under bad weather conditions.

[0098] Step S5.2: Fusion network structure The input layer of the weight fusion network receives the initial weight vector , and pass it to the hidden layer for feature extraction and non-linear transformation. The hidden layer consists of multiple neural networks. The calculation formula for the first layer is , where is the weight matrix, is the bias vector, is the activation function (such as ReLU). The calculation formula for the second layer is , where is the weight matrix, is the bias vector. The output layer generates the optimized system weights , where is the weight matrix, is the bias vector. The weights of the output layer are used for subsequent sensor weight fusion.

[0099] Step S5.3: Optimization method The optimization process of the weight fusion network includes loss function design and parameter update. The loss function consists of two parts: perception error and user satisfaction. The perception error measures the difference between the sensor output and the true value, and the formula is , where is the output of the i-th sensor, is the true value, and N is the number of sensors. The user satisfaction measures the subjective evaluation of the user on the system performance, and the formula is , where represents the satisfaction score of the j-th user, and M is the number of users. The total loss function combines the perception error and user satisfaction, and the formula is , where α is the weight coefficient, which is used to balance the importance of the perception error and user satisfaction. The optimization process includes forward propagation, calculating the loss, backpropagation, and parameter update. Forward propagation calculates the output of the fusion network. Calculating the loss calculates the total loss according to the perception error and user satisfaction. Backpropagation calculates the gradient of the loss function with respect to the network parameters. Parameter update uses the gradient descent method to update the network parameters, and the formula is , where η is the learning rate, which controls the step size of parameter update. Through the above optimization process, the weight fusion network can dynamically adjust the system weights to ensure that the system can operate in an optimal state under different environmental and user feedback conditions.

[0100] It should be noted that for the optimization of the weight fusion network, other network structures (such as convolutional neural networks, recurrent neural networks) or rule-based fusion methods (such as weighted average, voting method) can be used to handle different types of fusion problems. Although the aforementioned alternative solutions can improve the fusion effect, they will increase the model complexity and computational cost. The solution of the present invention uses a multi-layer neural network for weight fusion, which can reduce the computational complexity while ensuring the fusion effect.

[0101] Step S6: Weighted fusion and output. According to the adjusted weights of the three systems, combined with the weight distribution of each system corresponding to each sensor, the optimized sensor weight values are obtained.

[0102] Step S6.1: Sensor weight output.

[0103] The sensor weight output module generates the final sensor weights based on the optimized system weights and the weight distribution of each system for the sensors ( ), and is used for the weighted fusion of the sensed data, where are calculated through the weight fusion network, respectively representing the importance of the Bayesian network system, the DS evidence theory system, and the user feedback system in the final fusion. The weight distribution of each system for the sensors ( ) are respectively generated by the Bayesian network, the DS evidence theory, and the user feedback mechanism, reflecting the reliability, conflict handling ability, and user preferences of the sensors under different conditions.

[0104] Step S6.2: Calculation of the final sensor weights.

[0105] The final sensor weights are the weighted sum of the optimized system weights and the weight distribution of each system for the sensors. The calculation formula is:

[0106] Step S6.3: Weighted fusion.

[0107] The final sensor weights are used for the weighted fusion of the sensed data. The formula is:

[0108] is the fused sensor data, is the output data of the i-th sensor. The final output is the weighted fused sensor data, which is used for subsequent decision-making or analysis.

[0109] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this actual situation, on the basis of the above embodiments, an embodiment of the present invention provides a sensor weight optimization system that fuses multi-source data and user feedback, and this system is used to execute the method for optimizing the sensor weight by fusing multi-source data and user feedback in the above method embodiments.

[0110] See Figure 3 , the system includes: a first main module for obtaining multi-source data, including vehicle exterior sensor data, vehicle interior sensor data, and environmental sensor data; a second main module for constructing a Bayesian network system by defining each node and constructing a conditional probability table, and deriving the weight values of each sensor based on the posterior probability of the risk node; a third main module for constructing an improved DS evidence theory system by defining the output of each sensor as evidence, calculating the conflict coefficient when there is a conflict among the evidence of multiple sensors, reallocating the conflicting evidence, calculating the weight of each evidence and substituting it into the improved DS synthesis formula to calculate the confidence, and outputting the weight value of each sensor; a fourth main module for constructing a user feedback system by a reinforcement learning mechanism, defining states and actions, calculating the immediate reward and updating the Q value, and calculating the weight value of each sensor according to the updated Q value; a fifth main module for inputting the weight values corresponding to the three systems into a weight fusion network, constructing a loss function based on user feedback and perception error, and adjusting the weight allocation of the three systems by optimizing the loss function; a sixth main module for obtaining the optimized sensor weight values according to the adjusted weights of the three systems and combining the weight allocation of each sensor corresponding to each system.

[0111] The sensor weight optimization system that fuses multi-source data and user feedback provided by the embodiment of the present invention aims at the situation that the existing multi-source sensor data fusion methods in autonomous driving systems lack sufficient consideration of user feedback and are difficult to dynamically adjust the system weights according to user needs; at the same time, the existing methods usually use a certain technology alone (such as Bayesian network, DS evidence theory, or reinforcement learning), lacking a comprehensive optimization mechanism to balance sensor reliability, environmental importance, and user feedback, resulting in insufficient perception accuracy, environmental adaptability, and user satisfaction of the system in complex environments. By adopting Figure 3 several modules therein, the sensor weights are dynamically optimized by combining a Bayesian network, an improved DS evidence theory, and a reinforcement learning mechanism, thereby improving the comprehensive performance of the system.

[0112] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in setting corresponding functional modules, and the principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art, on the basis of the above system embodiments, refer to the specific technical solutions in other method embodiments, obtain corresponding technical means by combining technical features, and the technical solutions constituted by these technical means, and on the premise of ensuring the practicability of the technical solutions, improve the modules in the above system embodiments to obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0113] Based on the same inventive concept as the above embodiments, the embodiments of the present invention further provide a sensor weight optimization system that fuses multi-source data and user feedback, as Figure 4 shown, including in-vehicle sensors, out-of-vehicle sensors, environmental sensors, and a processor. The in-vehicle sensors, out-of-vehicle sensors, and environmental sensors respectively collect out-of-vehicle data, in-vehicle data, and environmental data and transmit them to the processor. The processor is used to implement the optimization of the weights of the autonomous driving sensors by using the method for optimizing the weights of the autonomous driving sensors based on multi-source data fusion and user feedback.

[0114] Based on the same inventive concept as the above embodiments, the embodiments of the present invention further provide a sensor weight optimization device that fuses multi-source data and user feedback, including a memory and a processor. The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for optimizing the weights of the sensors that fuses multi-source data and user feedback.

[0115] Based on the same inventive concept as the above embodiments, the embodiments of the present invention further provide a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the steps of the method for optimizing the weights of the sensors that fuses multi-source data and user feedback.

[0116] Embodiment 1 Suppose an autonomous driving vehicle is driving on an urban road. The environmental condition is rainy, the lighting condition is daytime, and the road type is the main urban road. The driver in the vehicle shows mild fatigue, the attention is distributed on the road ahead, and no voice commands are issued. The system needs to fuse the data from cameras, lidars, and millimeter-wave radars to generate a comprehensive perception result and dynamically adjust the sensor weights according to user feedback.

[0117] The implementation steps are as follows: Step 1, data collection and preprocessing.

[0118] · External sensor data: The camera captures the images of the road ahead, the lidar generates the point cloud data of the obstacles ahead, and the millimeter-wave radar measures the distance to the vehicle ahead.

[0119] · Environmental data: The weather is rainy, the lighting is during the day, and the road type is the urban arterial road.

[0120] · User feedback data: The in-vehicle camera monitors that the driver is slightly fatigued, with attention distributed on the road ahead and no voice commands issued.

[0121] · Perform time synchronization and spatial alignment on the data of the camera, lidar, and millimeter-wave radar.

[0122] · Use Kalman filtering to filter the noise of the sensor data.

[0123] Step 2, Bayesian network inference.

[0124] · Construct a Bayesian network, defining sensor nodes (camera, lidar, millimeter-wave radar), environmental nodes (rainy day, day, urban arterial road), user feedback nodes (slightly fatigued, attention ahead), and risk nodes (low, medium, high risk).

[0125] · Describe the dependence relationship between nodes through the conditional probability table and calculate the reliability weights of the sensors. For example, under rainy conditions, the reliability weight of the camera decreases and the reliability weight of the lidar increases. Step 3, improved DS evidence theory fusion.

[0126] · Take the outputs of the camera, lidar, and millimeter-wave radar as evidence and define the basic probability assignment.

[0127] · Calculate the conflict coefficient and reassign the conflicting evidence through the improved conflict assignment function.

[0128] · Combine the sensor reliability weights and environmental importance weights to calculate the comprehensive weight.

[0129] · Substitute the comprehensive weight into the improved DS synthesis formula to generate the fused confidence.

[0130] Step 4, user feedback optimization.

[0131] · Through the reinforcement learning mechanism, dynamically adjust the sensor weights in combination with the user feedback data (slightly fatigued, attention ahead).

[0132] · Calculate the immediate reward and update the Q value, and calculate the sensor weights under the user feedback mechanism according to the updated Q value.

[0133] Step 5, weight fusion network optimization.

[0134] · Design a weight fusion network to comprehensively integrate the outputs of Bayesian networks, DS evidence theory, and user feedback mechanisms to generate optimized system weights.

[0135] · Use the final sensor weights for weighted fusion of sensing data to generate a comprehensive sensing result.

[0136] Step 6, Weighted Fusion and Output.

[0137] · Use the final sensor weights for weighted fusion of sensing data to generate a comprehensive sensing result.

[0138] · Output the data after weighted fusion for subsequent decision-making or analysis.

[0139] Comparative Example 1 Compared with Example 1, this comparative example does not use the user feedback mechanism.

[0140] The implementation steps are as follows: Step 1, Data Acquisition and Preprocessing: The same as in Example 1.

[0141] Step 2, Construct a Bayesian Network: The same as in Example 1.

[0142] Step 3, Inference Calculation: The same as in Example 1.

[0143] Step 4, Without Using the User Feedback Mechanism: Directly use the sensor weights generated by the Bayesian network for data fusion.

[0144] Result Analysis: · The system cannot dynamically adjust the sensor weights when the user shows fatigue, resulting in a decrease in safety.

[0145] · The user satisfaction is low, and the system performance is poor in complex environments.

[0146] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0147] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 specified in one block or multiple blocks.

[0149] In summary of the above embodiments, the present invention designs a weight fusion network by combining a neural network with multiple decision-making models such as DS, Bayesian, and reinforcement learning to make decisions, and combines the DS evidence theory, Bayesian network, and reinforcement learning mechanism to dynamically optimize the sensor weights. The present invention constructs a comprehensive model by combining models of various complex factors such as external vehicle sensors, user feedback, and environmental factors, and combines external vehicle sensor data (such as cameras, lidar, millimeter-wave radars, etc.), user feedback data (such as fatigue, attention distribution, voice commands, etc.), and environmental data (such as weather, lighting, road type, etc.) to dynamically optimize the sensor weights. The present invention uses user feedback and sensor errors together as losses to guide the optimization of parameters such as sensors, designs a loss function based on user feedback and sensor errors, and combines perception errors and user satisfaction to guide the optimization of sensor weights.

[0150] The terms "including" and "having" and any variations thereof in the specification, claims, and drawings of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A sensor weight optimization method integrating multi-source data and user feedback, characterized in that: include: Acquire multi-source data, including external sensor data, internal sensor data, and environmental sensor data; Construct a Bayesian network system to define each node and construct a conditional probability table, and derive the weight value of each sensor based on the posterior probability of the risk node; A system based on the improved DS evidence theory is constructed to define the output of each sensor as evidence, calculate the conflict coefficient when there is a conflict in the evidence of multiple sensors, redistribute the conflicting evidence, calculate the weight of each evidence and substitute it into the improved DS synthesis formula to calculate the confidence, and output the weight value of each sensor; Build a user feedback system to define states and actions, calculate instant rewards and update Q values ​​through reinforcement learning mechanisms, and calculate the weight value of each sensor based on the updated Q values; The weight values ​​corresponding to the three systems are input into the weight fusion network, a loss function based on user feedback and perception error is constructed, and the weight distribution of the three systems is adjusted by optimizing the loss function; According to the adjusted weights of the three systems and the weight allocation of each system to each sensor, the optimized sensor weight value is obtained.

2. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 1, characterized in that: Acquiring multi-source data also includes: Collect external sensor data, internal sensor data and environmental sensor data; Perform time synchronization and spatial alignment on the collected sensor data; Noise filtering is performed on the sensor data after time synchronization and spatial alignment.

3. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 1, characterized in that: Define each node, including: define sensor nodes, environment nodes, user feedback nodes, and risk nodes.

4. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 1, characterized in that: The weight values ​​corresponding to the three systems are input into the weight fusion network, including: Setting the initial weights of the weight fusion network, wherein the initial weights include the Bayesian network system weights, the system weights based on the improved DS evidence theory, and the user feedback system weights; The input layer of the weight fusion network receives the initial weight vector and passes it to the hidden layer for feature extraction and nonlinear transformation, and outputs the optimized system weight.

5. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 4 is characterized in that: Construct a loss function based on user feedback and perception error, and adjust the weight distribution of the three systems by optimizing the loss function, including: The loss function is constructed as: ,in α is the weight coefficient used to balance the importance of perceived error and user satisfaction, L perception is the perception error, L user Provide feedback to users; The weights of each system are dynamically adjusted through forward propagation, loss calculation, backpropagation and parameter update for optimization.

6. The sensor weight optimization method for fusing multi-source data and user feedback according to claim 1, characterized in that: The optimized sensor weight values ​​are as follows: ， in, represents the fused sensor data, Represents the output data of the th sensor, They are generated by the Bayesian network system, the improved DS evidence theory system and the user feedback system, respectively, and represent the weight distribution of each system to the sensor. They respectively represent the importance of the Bayesian network system, the system based on the improved DS evidence theory and the user feedback system in the final fusion.

7. A sensor weight optimization system integrating multi-source data and user feedback, characterized in that: include: The first main module is used to obtain multi-source data, including external sensor data, internal sensor data and environmental sensor data; The second main module is used to build a Bayesian network system by defining each node and building a conditional probability table, and deriving the weight value of each sensor based on the posterior probability of the risk node; The third main module is used to build a system based on the improved DS evidence theory. By defining the output of each sensor as evidence, the conflict coefficient is calculated when there is a conflict in the evidence of multiple sensors, the conflicting evidence is redistributed, the weight of each evidence is calculated and substituted into the improved DS synthesis formula to calculate the confidence, and the weight value of each sensor is output; The fourth main module is used to build a user feedback system. Through the reinforcement learning mechanism, it defines the state and action, calculates the immediate reward and updates the Q value, and calculates the weight value of each sensor according to the updated Q value; The fifth main module is used to input the weight values ​​corresponding to the three systems into the weight fusion network, construct a loss function based on user feedback and perception error, and adjust the weight distribution of the three systems by optimizing the loss function; The sixth main module is used to obtain an optimized sensor weight value according to the adjusted weights of the three systems and the weight distribution of each sensor corresponding to each system.

8. A sensor weight optimization system integrating multi-source data and user feedback, characterized in that: The method comprises an in-vehicle sensor, an out-vehicle sensor, an environmental sensor and a processor, wherein the in-vehicle sensor, the out-vehicle sensor and the environmental sensor respectively collect out-vehicle data, in-vehicle data and environmental data and transmit them to the processor, and the processor is used to realize automatic driving sensor weight optimization by adopting the method described in any one of claims 1 to 6.

9. A sensor weight optimization device integrating multi-source data and user feedback, characterized in that: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the sensor weight optimization method for fusing multi-source data and user feedback as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the sensor weight optimization method for fusing multi-source data and user feedback as described in any one of claims 1 to 6.

Images