Backward early warning method and device based on multi-sensor fusion
Through the fault detection and isolation model and dynamic weighted data fusion model, combined with the environment perception model, the problem of performance degradation of multi-sensor fusion solution during single point failure is solved, and the backward warning function that can maintain high performance when sensor failure is achieved is realized .
Patent Information
- Application Number
- CN202411069857.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-19
AI Technical Summary
The existing multi-sensor fusion scheme has significantly reduced performance during single point failure or redundant switching, resulting in poor performance of backward early warning functions.
The sensor status is monitored by fault detection and isolation models, the dynamic weighted data fusion model adjusts the sensor weight, and the early warning function is triggered through the environment perception model to realize intelligent fault detection and isolation, adaptive weighted data fusion and multi-level redundant design.
Maintaining high-performance backward warning function when a single or multiple sensors fail, improving system stability and reliability.
Smart Images

Figure CN118977726A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a backward warning method and device based on multi-sensor fusion. Background Art
[0002] As the number of cars increases, the frequency and severity of traffic accidents are also rising. Among them, rear-end collision is one of the most common types of accidents, especially in crowded urban environments. Rear-end collision accidents are prone to occur when the driver fails to notice the rear vehicle or obstacles in time. Therefore, developing an effective rear-end warning system is crucial to improving road safety.
[0003] The rear warning system is an advanced driver assistance system (ADAS) designed to improve the vehicle's perception of the rear environment during reversing or driving, and provide necessary warning and assistance functions, including: blind spot detection, lane change assistance, rear collision warning and door opening warning.
[0004] The current environmental perception implementation methods of the backward warning function mainly include the following:
[0005] 1) Rear warning method based on pure millimeter wave radar. This method uses electromagnetic waves to detect objects behind the vehicle and measure their distance, speed and direction. However, this method has low detection accuracy for small objects, slow objects or stationary objects, and high-performance radar systems are expensive.
[0006] 2) A rear warning method based on a pure visual camera. This method uses a camera to capture images and combines image processing algorithms to identify and analyze objects behind. However, the performance of this method degrades in low-light or strong-light conditions, and the image processing algorithm may require more computing time and have delays.
[0007] 3) The backward warning method based on laser radar uses laser pulses to measure the distance and shape of objects and generate three-dimensional information of environmental objects. However, the laser radar system used in this method is expensive and requires complex algorithms and high-performance computing platform support.
[0008] 4) Backward warning method based on multi-sensor fusion. This method combines data from multiple sensors such as radar, camera, ultrasonic sensor and lidar, and improves the reliability and accuracy of detection through data fusion algorithm. However, the multi-sensor system used in this method is expensive and requires complex integration and calibration. The data fusion algorithm is complex and requires support from a high-performance computing platform.
[0009] Among the above environmental perception implementation methods of the backward warning function, the implementation method of the backward warning system based on multi-sensor fusion has been paid attention to by many researchers in practical applications, because in principle it can integrate the advantages of different sensors to achieve complementary advantages, and many similar products have also appeared.
[0010] However, the main problem with the current various multi-sensor fusion solutions is that the performance is greatly reduced when a single point fails or redundant switching occurs. That is, when one or some of the multiple sensors fail, the performance of the backward warning function is often greatly reduced or even directly fails, which is inconsistent with the original intention of using multi-sensor fusion.
[0011] Therefore, the prior art needs to be improved. Summary of the invention
[0012] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a backward warning method and device based on multi-sensor fusion to solve the problem of single point failure or performance degradation during redundant switching in the multi-sensor fusion solution.
[0013] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0014] In a first aspect, the present invention provides a backward warning method based on multi-sensor fusion, comprising:
[0015] Obtain data collected by each sensor in real time;
[0016] Based on the acquired data, the fault detection and isolation model is used to monitor the working status of each sensor, and the data of the faulty sensor is isolated according to the working status of each sensor to obtain normal data and fault information;
[0017] According to the normal data and the fault information, a dynamic weighted data fusion model is used to adjust the weight corresponding to each sensor to obtain weight information and normal data corresponding to each sensor;
[0018] According to the weight information and normal data corresponding to each sensor obtained by dynamic weighting, the environmental perception model is used for perception to obtain the environmental perception result, and the corresponding backward warning function is triggered according to the environmental perception result.
[0019] In one implementation, the method of monitoring the working state of each sensor using a fault detection and isolation model based on the acquired data, isolating the data of the faulty sensor based on the working state of each sensor, and obtaining normal data and fault information includes:
[0020] Based on the fault detection and isolation model, the acquired data is preprocessed, feature extracted, and data evaluated to obtain the detection results corresponding to each sensor data;
[0021] The working state corresponding to each sensor is determined according to the obtained detection result, the data of the faulty sensor is screened, and the data of the faulty sensor is isolated to obtain the normal data and the fault information.
[0022] In one implementation, the method further includes: monitoring the working state of each sensor using a fault detection and isolation model based on the acquired data, isolating the data of the faulty sensor based on the working state of each sensor, and obtaining normal data and fault information;
[0023] Use historical data and environmental variables to perform long-term trend analysis on the working status of each sensor, predict potential sensor failures and perform preventive maintenance in advance.
[0024] In one implementation, the weights corresponding to the sensors are adjusted using a dynamic weighted data fusion model according to the normal data and the fault information to obtain the weight information and normal data corresponding to the sensors, including:
[0025] Evaluate the credibility of each sensor data according to the normal data and the fault information, and adjust the data fusion strategy in real time;
[0026] According to the adjusted data fusion strategy, the Bayesian network or fuzzy logic algorithm is used to adjust the weight corresponding to each sensor according to the consistency of real-time monitoring data and historical performance, and obtain the weight information and normal data corresponding to each sensor.
[0027] In one implementation, the weights corresponding to the sensors are adjusted according to the adjusted data fusion strategy by using a Bayesian network or a fuzzy logic algorithm according to the consistency of the real-time monitored data and the historical performance to obtain the weight information and normal data corresponding to the sensors, and then the following is further included:
[0028] The data from different types of sensors are used for cross-validation. When abnormal data is detected, data compensation is performed by enabling data from backup sensors or other sensors that are working normally.
[0029] In one implementation, the weight information and normal data corresponding to each sensor obtained according to the dynamic weighting are perceived using an environment perception model to obtain an environment perception result, and a corresponding backward warning function is triggered according to the environment perception result, including:
[0030] Perform multi-sensor fusion according to weight information corresponding to each sensor and normal data, and perform environmental perception based on the multi-sensor fusion data to obtain the environmental perception result;
[0031] Determining whether to trigger a preset backward warning function according to the environmental perception result;
[0032] If yes, the control triggers the corresponding backward warning function.
[0033] In one implementation, the method further includes:
[0034] The original fault detection and isolation model is trained based on the labeled original data to obtain a trained fault detection and isolation model; wherein the labeled original data includes: original data labeled normally and original data labeled abnormally.
[0035] In a second aspect, the present invention provides a backward warning device based on multi-sensor fusion, comprising:
[0036] Data acquisition module, used to obtain data collected by each sensor in real time;
[0037] A fault detection and isolation module is used to monitor the working status of each sensor using a fault detection and isolation model based on the acquired data, and isolate the data of the faulty sensor according to the working status of each sensor to obtain normal data and fault information;
[0038] A dynamic weighting module, used to adjust the weight corresponding to each sensor by using a dynamic weighted data fusion model according to the normal data and the fault information, so as to obtain the weight information and normal data corresponding to each sensor;
[0039] The environment perception module is used to perceive the environment using the environment perception model according to the weight information and normal data corresponding to each sensor obtained by dynamic weighting to obtain the environment perception result;
[0040] The backward warning trigger module is used to trigger the corresponding backward warning function according to the environmental perception result.
[0041] In a third aspect, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a backward warning program based on multi-sensor fusion, and when the backward warning program based on multi-sensor fusion is executed by the processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion as described in the first aspect.
[0042] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, and which stores a backward warning program based on multi-sensor fusion. When the backward warning program based on multi-sensor fusion is executed by a processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion as described in the first aspect.
[0043] The present invention adopts the above technical solution to achieve the following effects:
[0044] The present invention can monitor the working status of each sensor using a fault detection and isolation model by acquiring data collected by each sensor in real time, and isolate the data of the faulty sensor according to the working status of each sensor; and adjust the weights corresponding to each sensor through a dynamic weighted data fusion model, and can perceive using an environmental perception model according to the weight information and normal data corresponding to each sensor, thereby obtaining environmental perception results, and triggering corresponding backward warning functions according to the environmental perception results. The present invention proposes a backward warning scheme based on multi-sensor fusion, which realizes a backward warning function that can maintain high performance when a single or multiple sensors fail by introducing an intelligent fault detection and isolation mechanism, an adaptive weighted data fusion algorithm, and a multi-level redundant design. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0046] Figure 1 It is a flow chart of a backward warning method based on multi-sensor fusion in one implementation of the present invention.
[0047] Figure 2 It is a schematic diagram of a vehicle alarm system in one implementation of the present invention.
[0048] Figure 3 It is a schematic diagram of training a fault detection and isolation model in one implementation of the present invention.
[0049] Figure 4 It is a schematic diagram of a self-driving domain controller module in one implementation of the present invention.
[0050] Figure 5 It is a functional principle diagram of a terminal in one implementation of the present invention.
[0051] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0053] Exemplary Methods
[0054] Among the current environmental perception implementation methods of the backward warning function, the implementation method of the backward warning system based on multi-sensor fusion has attracted the attention of many researchers in practical applications because it can integrate the advantages of different sensors to achieve complementary advantages in principle. Therefore, many similar products have emerged. However, the main problem with the current various multi-sensor fusion solutions is that the performance is greatly reduced when a single point fails or redundant switching occurs. That is, when one or some of the multiple sensors fail, the performance of the backward warning function is often greatly reduced, or even directly fails, which is inconsistent with the original intention of using multi-sensor fusion.
[0055] In response to the above technical problems, an embodiment of the present invention provides a backward warning method based on multi-sensor fusion. By acquiring the data collected by each sensor in real time, the method can monitor the working status of each sensor using a fault detection and isolation model, and isolate the data of the faulty sensor according to the working status of each sensor; and adjust the weights corresponding to each sensor through a dynamic weighted data fusion model, and can use the environment perception model to perceive according to the weight information and normal data corresponding to each sensor, thereby obtaining the environment perception result, and triggering the corresponding backward warning function according to the environment perception result. Therefore, an embodiment of the present invention proposes a backward warning scheme based on multi-sensor fusion, which realizes the backward warning function that can maintain high performance when a single or multiple sensors fail by introducing an intelligent fault detection and isolation mechanism, an adaptive weighted data fusion algorithm, and a multi-level redundant design.
[0056] like Figure 1 As shown, an embodiment of the present invention provides a backward warning method based on multi-sensor fusion, comprising the following steps:
[0057] Step S100, obtaining data collected by each sensor in real time.
[0058] In this embodiment, the backward warning method based on multi-sensor fusion is implemented by a passenger car backward warning system with multi-sensor fusion; Figure 2 As shown, the rear warning system for passenger vehicles includes:
[0059] Rear-facing camera: installed behind the vehicle body to provide high-resolution image data;
[0060] Rear millimeter-wave radar: installed at the rear of the vehicle body, used to detect objects at a long distance, provide more accurate detection speed, and provide obstacle information in the blocked area when the camera is blocked;
[0061] Rear ultrasonic radar: installed at the rear of the vehicle body for close-range detection;
[0062] As well as the self-driving domain controller and vehicle alarm system;
[0063] like Figure 4 As shown, the self-driving domain controller is deployed with: a fault detection and isolation model, a dynamic weighted data fusion model, and an environment perception model.
[0064] In this embodiment, the three detection sensors, namely the rear camera, the rear millimeter wave radar and the rear ultrasonic radar, are installed behind the vehicle or at other locations of the vehicle, but are responsible for collecting environmental information behind the vehicle. Under normal circumstances, the three sensors all have normal data output, and the data is fused and processed in the self-driving controller through a high-performance computing platform to obtain environmental information behind the vehicle, and then determine whether to trigger a certain warning function in the rear warning, and finally send the trigger signal to the vehicle alarm system for the final alarm output.
[0065] In this embodiment, during the operation of sensors used for the rear warning function, such as the rear camera, the rear millimeter-wave radar, and the rear ultrasonic radar, in order to monitor in real time whether these sensors of the rear warning function fail, it is necessary to obtain the data collected by these sensors in real time, and then monitor the working status of the corresponding sensors based on the sensor data obtained in real time, so as to determine whether the corresponding sensors are in a normal working state.
[0066] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the backward warning method based on multi-sensor fusion also includes the following steps:
[0067] Step S200, based on the acquired data, the working state of each sensor is monitored using a fault detection and isolation model, and the data of the faulty sensor is isolated according to the working state of each sensor to obtain normal data and fault information.
[0068] In this embodiment, for the real-time collected data of sensors such as the rearward camera, the rearward millimeter-wave radar, and the rearward ultrasonic radar, an intelligent fault detection and isolation mechanism is introduced to analyze these data, that is, the real-time collected data of the sensors is analyzed through the fault detection and isolation model. If the real-time collected data of these sensors obviously deviates from the normal range, the corresponding sensor is determined to be a faulty sensor; for example, if the real-time collected data of these sensors does not match the empirical value obtained from the historical data (for example, the sensor data exceeds the empirical value by a lot or is less than the empirical value, etc.), or the real-time collected data of the sensor for a long time is 0, or the real-time collected data of the sensor obviously does not match the current scene data (for example, under the same target, the data collected by the rearward camera and the rearward millimeter-wave radar are normal, while the data collected by the rearward ultrasonic radar obviously has a large deviation from the data collected by the first two sensors), in these cases, after analyzing the real-time collected data of the sensors through the fault detection and isolation model, the faulty sensor can be determined, and the data of the faulty sensor can be isolated to prevent the erroneous data from affecting the overall performance of the system.
[0069] In this embodiment, before using the fault detection and isolation model to monitor the working status of each sensor and isolating the data of the faulty sensor according to the working status of each sensor, it is also necessary to train the fault detection and isolation model with some normal labeled original data and abnormal labeled original data, so as to realize the fault detection and isolation function by deep learning method.
[0070] In an implementation of this embodiment, the following steps are included before step S200:
[0071] Step S200a, training the original fault detection and isolation model based on the labeled original data to obtain a trained fault detection and isolation model; wherein the labeled original data includes: original data labeled normally and original data labeled abnormally.
[0072] like Figure 3 As shown, in this embodiment, a sensor combination including a rearward-facing camera, a rearward-facing millimeter-wave radar, and a rearward-facing ultrasonic radar is used to collect raw data; then, the manually screened data is labeled, and a label is given to indicate whether the data is normal data or abnormal data, for subsequent model training.
[0073] Then, the labeled raw data is processed through data preprocessing, feature extraction, data evaluation and other steps to obtain the analysis results of normal or abnormal data inferred by the fault detection and isolation model. Finally, the output results of the fault detection and isolation model are supervised by the results of manual annotation to train the fault detection and isolation model.
[0074] In this embodiment, the trained fault detection and isolation model is deployed in the self-driving domain controller to determine whether the input data of the current sensor is normal data. If it is normal data, the data will be given to the downstream environmental perception module for environmental perception. If the data is abnormal, the data will be discarded and the abnormal information will be sent to the downstream module.
[0075] Specifically, in an implementation of this embodiment, step S200 includes the following steps:
[0076] Step S201, preprocessing, feature extraction, and data evaluation are performed on the acquired data based on the fault detection and isolation model to obtain detection results corresponding to each sensor data;
[0077] Step S202, determining the working state corresponding to each sensor according to the obtained detection result, screening the data of the faulty sensor, and isolating the data of the faulty sensor to obtain the normal data and the fault information.
[0078] In this embodiment, for the real-time data collected by sensors such as a rearward camera, a rearward millimeter-wave radar, and a rearward ultrasonic radar, the acquired data is preprocessed, feature extracted, and data evaluated based on the fault detection and isolation model to obtain the detection results corresponding to the data of each sensor; the fault detection and isolation model is mainly divided into the following two processes:
[0079] 1) Intelligent fault detection: Use machine learning and deep learning algorithms to monitor the status of each sensor in real time and quickly identify abnormal sensor behavior. This includes but is not limited to: using neural network models to learn the characteristics of sensor data under normal working conditions, and immediately triggering fault isolation when sensor data deviates from the normal range.
[0080] 2) Fault isolation: After a sensor failure is detected, the data of the faulty sensor is isolated to prevent the erroneous data from affecting the overall performance of the system. The isolated data can be compensated by the data of redundant sensors or other healthy sensors. If there is no redundant sensor to compensate, an algorithm is used to compensate.
[0081] In this embodiment, the fault detection and isolation model is responsible for determining whether the data is normal data, and then sending the normal data and fault information to the dynamic weighted data fusion model, so as to assign weights to each sensor through the dynamic weighted data fusion model. The dynamic weighted data fusion model is obtained by data fusion training based on the data of multiple sensors; in the process of training the dynamic weighted data fusion model, the weight of each sensor in the data fusion process is dynamically adjusted according to the real-time status of each sensor by using machine learning and deep learning methods (for example, Bayesian network or fuzzy logic algorithm). Including but not limited to when the reliability of a certain sensor decreases, its weight can be reduced and the weight of other sensors can be increased. In this way, even if a certain sensor fails, the system can rely on the data of other sensors to maintain high performance. The weights are adaptively adjusted by using machine learning and deep learning algorithms; therefore, based on the trained dynamic weighted data fusion model, after obtaining the normal data and fault information sent by the fault detection and isolation model, in the data fusion process, when the reliability of a certain sensor decreases, its weight can be reduced and the weight of other sensors can be increased, thereby improving the stability of the system.
[0082] Specifically, in an implementation of this embodiment, the following steps are included before step S201:
[0083] Step S201a, using historical data and environmental variables to perform long-term trend analysis on the working status of each sensor, predict potential sensor failures and perform preventive maintenance in advance.
[0084] In this embodiment, in addition to analyzing the real-time collected data of sensors such as the rear-facing camera, the rear-facing millimeter-wave radar, and the rear-facing ultrasonic radar, the fault detection and isolation model can also use historical data and environmental variables to perform long-term trend analysis on the sensor status, predict potential sensor failures and perform preventive maintenance in advance; in the process of performing long-term trend analysis, first, the historical data of each sensor and the corresponding environmental variable data are obtained; then, based on the fault detection and isolation model, the performance (or status) change trend of the sensor is extracted from the historical data using time series analysis, machine learning and other methods, and based on the change trend extracted from the historical data and the environmental variable data, the fault detection and isolation model is used to predict the performance (or status) change trend of the sensor in the future period of time, thereby realizing the prediction function of potential sensor failures, thereby minimizing the impact of sensor failures on the overall stability of the system.
[0085] like Figure 1 As shown, an embodiment of the present invention provides a backward warning method based on multi-sensor fusion, comprising the following steps:
[0086] Step S300: According to the normal data and the fault information, a dynamic weighted data fusion model is used to adjust the weight corresponding to each sensor to obtain the weight information and normal data corresponding to each sensor.
[0087] In this embodiment, on the basis of introducing the intelligent fault detection and isolation mechanism, an adaptive weighted data fusion algorithm is also introduced to adjust the weights corresponding to each sensor, and a multi-level redundancy design is introduced to achieve a high-performance backward warning function when a single or multiple sensors fail; specifically, the intelligent fault detection and isolation mechanism refers to the use of machine learning and deep learning algorithms based on the fault detection and isolation model to monitor the status of each sensor in real time, quickly identify the abnormal behavior of the sensor, and when the sensor data deviates from the normal range, the data of the faulty sensor is isolated to prevent the erroneous data from affecting the overall performance of the system. The isolated data can be compensated by the data of redundant sensors or other healthy sensors. If there is no redundant sensor to compensate, the algorithm is used for compensation; and the purpose of introducing the adaptive weighted data fusion algorithm is to dynamically adjust the weight of each sensor according to its real-time status during the data fusion process. When the reliability of a sensor decreases, its weight can be reduced and the weight of other sensors can be increased; and the purpose of the multi-level redundancy design is to achieve redundancy compensation by performing redundant design at multiple levels such as the sensor level, data processing level, data fusion level, and communication level to improve the stability of the backward warning performance. Therefore, in this embodiment, through such a design method, a high-performance backward warning function is achieved when a single or multiple sensors fail.
[0088] Specifically, in an implementation of this embodiment, step S300 includes the following steps:
[0089] Step S301, evaluating the credibility of each sensor data according to the normal data and the fault information, and adjusting the data fusion strategy in real time;
[0090] Step S302, according to the adjusted data fusion strategy, using Bayesian network or fuzzy logic algorithm, adjust the weight corresponding to each sensor according to the consistency of real-time monitored data and historical performance, and obtain the weight information and normal data corresponding to each sensor.
[0091] In this embodiment, firstly, the fault detection and isolation model is responsible for judging whether the data is normal data. The specific judgment method is as follows: the data collected by the sensors in real time are analyzed by the fault detection and isolation model. If the data collected by the sensors in real time obviously deviate from the normal range (for example, the normal range of the sensor data is an empirical value set based on historical data or prior knowledge), the corresponding sensors are determined to be faulty sensors, and the fault information of the faulty sensors (for example, sensor type, abnormal time, and abnormal data, etc.) is obtained; if the data collected by the sensors in real time do not deviate from the normal range, the corresponding sensors are determined to be sensors in normal state, and the data collected by the sensors in real time are normal data;
[0092] Then, the normal data and fault information are sent to the dynamic weighted data fusion model. The dynamic weighted data fusion model judges the reliability of the sensor based on the obtained data and assigns weights to each sensor. The specific method of allocating weights is as follows: based on the received normal data and fault information, the sensors with reduced reliability are determined, and the weights of these sensors with reduced reliability in the data fusion process are reduced, while the weights of other sensors in the data fusion process are increased. The specific sensor weights and the weights of these sensors after the reliability is reduced can be set according to the importance of the sensors in the entire data fusion process and the corresponding empirical values.
[0093] Finally, the dynamically adjusted weight information and the original data are given to the environmental perception model for perception, so as to obtain the environmental perception result, and the corresponding backward warning function is triggered according to the environmental perception result.
[0094] In this embodiment, the principle of the dynamic weighted data fusion mechanism is:
[0095] Separate data training for a single sensor: Use the data of a single sensor to train the environment perception model separately, and other sensors participate in the training as supervisory data, so that each sensor can provide output with relatively small performance degradation when used alone.
[0096] Dynamic weighted data fusion model processing: During the data fusion process, the weight of each sensor is dynamically adjusted according to its real-time status. This includes but is not limited to when the reliability of a sensor decreases, its weight can be reduced and the weight of other sensors can be increased. In this way, even if a sensor fails, the system can rely on the data of other sensors to maintain high performance. Use machine learning and deep learning algorithms to perform adaptive weight adjustments.
[0097] As an example, the dynamic weighted data fusion model in this embodiment can dynamically adjust the weight of each sensor according to the real-time status of each sensor; and adjust the data fusion strategy in real time by evaluating the credibility of sensor data, specifically: using Bayesian network or fuzzy logic algorithm, weight adjustment is performed according to the consistency of real-time monitored data and historical performance, so as to obtain the weight information corresponding to each sensor; wherein, Bayesian network can use historical data to evaluate the sensor performance, in weight adjustment, Bayesian network can learn the relationship between weight and performance, according to the consistency of sensor real-time data and historical data, and the historical performance of the sensor corresponding to the historical data, the probability model is used to guide the adjustment of weight, so as to achieve the global optimum. Fuzzy logic algorithm allows a subtle real measure between "completely true" and "completely false", which can handle uncertainty and ambiguity. In weight adjustment, fuzzy logic can be used to process fuzzy input information, and fuzzy rules are used to infer the appropriate weight adjustment strategy, that is, to determine whether the input sensor data is consistent with the historical data, and to adjust the weight according to the corresponding sensor performance in the historical data.
[0098] For example, in the process of realizing the rear warning function of multi-sensor fusion, before the reliability of each sensor decreases, the weights of the rear camera, rear millimeter-wave radar and rear ultrasonic radar are different. The weight of the rear camera is greater than that of the rear ultrasonic radar, and the weight of the rear ultrasonic radar is greater than that of the rear millimeter-wave radar. The weights of these sensors can be set according to empirical values; if the reliability of the rear camera decreases, at this time, the weight of the rear camera is reduced by increasing the weights of the rear millimeter-wave radar and the rear ultrasonic radar, and these dynamically adjusted weight values can be adjusted according to the results of the Bayesian network or fuzzy logic algorithm.
[0099] Specifically, in an implementation of this embodiment, the following steps are further included after step S302:
[0100] Step S303, cross-validation is performed using data from sensors of different types, and when abnormal data is detected, data compensation is performed by enabling data from standby sensors or other sensors that are working normally.
[0101] In this embodiment, on the basis of the dynamic weighted data fusion mechanism, a multi-level redundant design is also introduced: redundant compensation is achieved by performing redundant design at multiple levels such as the sensor level, data processing level, data fusion level, and communication level to improve the stability of the backward warning performance.
[0102] In addition to timely fault detection, it is also very important to have enough valid sensor data for downstream use when a fault occurs. To this end, a multi-level redundancy design is performed in this embodiment:
[0103] 1) Redundancy design at the sensor level:
[0104] Multi-sensor configuration: Multiple sensors are installed on the rear and sides of the vehicle, with at least two or more of each type, to ensure that if one sensor fails, the system still has other sensors to provide reliable data.
[0105] Real-time monitoring: Using fault detection algorithms, the working status of each sensor is monitored in real time to ensure its normal operation.
[0106] Automatic switchover: After a sensor failure is detected, the system automatically switches to the backup sensor.
[0107] 2) Redundancy design at the data processing level:
[0108] Redundant processing unit: Configure dual redundant or multi-redundant processors to ensure that when one processor fails, the backup processor can immediately take over the task.
[0109] Data synchronization and verification: Multiple processing units synchronize data in real time to ensure the consistency of processing results, and perform cross-verification to ensure data accuracy and consistency.
[0110] 3) Redundancy design at the data fusion level:
[0111] Multi-algorithm fusion: At the data fusion level, configure multiple data fusion algorithms (such as Kalman filtering, particle filtering, deep learning models, etc.) to process sensor data in parallel.
[0112] Dynamic weight adjustment: Based on real-time monitoring results, the weight of each sensor in data fusion is dynamically adjusted to ensure that even if the data of a certain sensor is unreliable, the system can still rely on other sensor data to maintain high performance.
[0113] Fault-tolerant data fusion: Develop data fusion algorithms with fault-tolerant capabilities (such as robust Kalman filtering and fault-tolerant particle filtering) to maintain high fusion accuracy even when sensor data is noisy or partially lost.
[0114] 4) Redundancy design at the communication level:
[0115] Redundant communication links: Configure multiple communication links to ensure that when one communication link fails, other links can continue to work.
[0116] Communication protocol redundancy: multiple communication protocols (such as CAN bus, Ethernet, etc.) are adopted to ensure the reliability of communication.
[0117] Data backup and recovery: Real-time backup of critical data to ensure that data is not lost when the communication link fails, and data can be quickly restored from the backup.
[0118] In this embodiment, the multi-level redundancy design includes: designing a multi-level sensor redundancy structure to achieve redundancy at the sensor level and the data processing level; using data between different types of sensors for cross-validation. Specifically, the cross-validation step includes:
[0119] Compare and analyze the data of different types of sensors. When an abnormality is detected, compensation is made by enabling spare sensors or using data from other sensors that are working normally.
[0120] like Figure 1 As shown, an embodiment of the present invention provides a backward warning method based on multi-sensor fusion, comprising the following steps:
[0121] Step S400, based on the weight information and normal data corresponding to each sensor obtained by dynamic weighting, the environment perception model is used for perception to obtain the environment perception result, and the corresponding backward warning function is triggered according to the environment perception result.
[0122] Specifically, in an implementation of this embodiment, step S400 includes the following steps:
[0123] Step S401, performing multi-sensor fusion according to weight information corresponding to each sensor and normal data, and performing environmental perception based on the multi-sensor fusion data to obtain the environmental perception result;
[0124] Step S402, judging whether to trigger a preset backward warning function according to the environmental perception result;
[0125] Step S403, if yes, control triggering the corresponding backward warning function.
[0126] In this embodiment, the dynamic weighted data fusion model makes judgments based on the obtained data, assigns weights to each sensor, and finally gives the dynamically weighted weight information and the original data to the environmental perception model. The environmental perception model processes the data to obtain the final environmental perception result. Based on the environmental perception result, it is determined whether to trigger the preset backward warning function. If a certain backward warning function needs to be triggered, the corresponding backward warning function is controlled to be triggered.
[0127] For the environmental perception model, each individual sensor will be used as the main data source, and the remaining sensor data will be used as auxiliary supervision information to train the model, and then the fusion model will be trained. This ensures that the perception performance when only a single sensor is effective remains relatively stable without causing a significant performance degradation.
[0128] In this embodiment, the rear warning system based on multi-sensor fusion includes, in addition to the above sensor combination, self-driving domain controller and vehicle alarm system, a data processing unit; for the data processing unit, a distributed computing architecture can be adopted to realize real-time data processing and decision-making through edge computing devices, thereby improving the response speed and processing efficiency of the system.
[0129] This embodiment achieves the following technical effects through the above technical solution:
[0130] This embodiment obtains the data collected by each sensor in real time, and can use the fault detection and isolation model to monitor the working status of each sensor, and isolate the data of the faulty sensor according to the working status of each sensor; and adjusts the weights corresponding to each sensor through the dynamic weighted data fusion model, and can use the environment perception model to perceive according to the weight information and normal data corresponding to each sensor, so as to obtain the environment perception result, and trigger the corresponding backward warning function according to the environment perception result. This embodiment proposes a backward warning scheme based on multi-sensor fusion, which realizes the backward warning function that can maintain high performance when a single or multiple sensors fail by introducing an intelligent fault detection and isolation mechanism, an adaptive weighted data fusion algorithm and a multi-level redundant design.
[0131] Exemplary Devices
[0132] Based on the above embodiments, the present invention further provides a backward warning device based on multi-sensor fusion, comprising:
[0133] Data acquisition module, used to obtain data collected by each sensor in real time;
[0134] A fault detection and isolation module is used to monitor the working status of each sensor using a fault detection and isolation model based on the acquired data, and isolate the data of the faulty sensor according to the working status of each sensor to obtain normal data and fault information;
[0135] A dynamic weighting module, used to adjust the weight corresponding to each sensor by using a dynamic weighted data fusion model according to the normal data and the fault information, so as to obtain the weight information and normal data corresponding to each sensor;
[0136] The environment perception module is used to perceive the environment using the environment perception model according to the weight information and normal data corresponding to each sensor obtained by dynamic weighting to obtain the environment perception result;
[0137] The backward warning trigger module is used to trigger the corresponding backward warning function according to the environmental perception result.
[0138] This embodiment achieves the following technical effects through the above technical solution:
[0139] This embodiment obtains the data collected by each sensor in real time, and can use the fault detection and isolation model to monitor the working status of each sensor, and isolate the data of the faulty sensor according to the working status of each sensor; and adjusts the weights corresponding to each sensor through the dynamic weighted data fusion model, and can use the environment perception model to perceive according to the weight information and normal data corresponding to each sensor, so as to obtain the environment perception result, and trigger the corresponding backward warning function according to the environment perception result. This embodiment proposes a backward warning scheme based on multi-sensor fusion, which realizes the backward warning function that can maintain high performance when a single or multiple sensors fail by introducing an intelligent fault detection and isolation mechanism, an adaptive weighted data fusion algorithm and a multi-level redundant design.
[0140] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Figure 5 shown.
[0141] The terminal includes: a processor, a memory, an interface, a display screen and a communication module connected via a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.
[0142] When the computer program is executed by a processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion.
[0143] It can be understood by those skilled in the art that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0144] In one embodiment, a terminal is provided, which includes: a processor and a memory, the memory storing a backward warning program based on multi-sensor fusion, and the backward warning program based on multi-sensor fusion is used to implement the operation of the above-mentioned backward warning method based on multi-sensor fusion when executed by the processor.
[0145] In one embodiment, a storage medium is provided, wherein the storage medium stores a backward warning program based on multi-sensor fusion, and when the backward warning program based on multi-sensor fusion is executed by a processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion as described above.
[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and volatile memory.
[0147] In summary, the present invention provides a backward warning method and device based on multi-sensor fusion, the method comprising: acquiring data collected by each sensor in real time; based on the acquired data, using a fault detection and isolation model to monitor the working state of each sensor, and isolating the data of the faulty sensor according to the working state of each sensor, to obtain normal data and fault information; based on the normal data and the fault information, using a dynamic weighted data fusion model to adjust the weights corresponding to each sensor, to obtain weight information and normal data corresponding to each sensor; based on the weight information and normal data corresponding to each sensor obtained by dynamic weighting, using an environmental perception model to perceive, to obtain environmental perception results, and triggering the corresponding backward warning function according to the environmental perception results. The present invention proposes a backward warning function that can maintain high performance when a single or multiple sensors fail.
[0148] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A backward warning method based on multi-sensor fusion, It is characterized in that include: Obtain data collected by each sensor in real time; Based on the acquired data, the fault detection and isolation model is used to monitor the working status of each sensor, and the data of the faulty sensor is isolated according to the working status of each sensor to obtain normal data and fault information; According to the normal data and the fault information, a dynamic weighted data fusion model is used to adjust the weight corresponding to each sensor to obtain weight information and normal data corresponding to each sensor; According to the weight information and normal data corresponding to each sensor obtained by dynamic weighting, the environmental perception model is used for perception to obtain the environmental perception result, and the corresponding backward warning function is triggered according to the environmental perception result.
2. According to the backward warning method based on multi-sensor fusion according to claim 1, It is characterized in that The method of monitoring the working state of each sensor using a fault detection and isolation model based on the acquired data and isolating the data of the faulty sensor based on the working state of each sensor to obtain normal data and fault information includes: Based on the fault detection and isolation model, the acquired data is preprocessed, feature extracted, and data evaluated to obtain the detection results corresponding to each sensor data; The working state corresponding to each sensor is determined according to the obtained detection result, the data of the faulty sensor is screened, and the data of the faulty sensor is isolated to obtain the normal data and the fault information.
3. According to the backward warning method based on multi-sensor fusion according to claim 1, It is characterized in that The method further includes: monitoring the working state of each sensor using a fault detection and isolation model based on the acquired data, isolating the data of the faulty sensor based on the working state of each sensor, and obtaining normal data and fault information. Use historical data and environmental variables to perform long-term trend analysis on the working status of each sensor, predict potential sensor failures and perform preventive maintenance in advance.
4. According to claim 1, the backward warning method based on multi-sensor fusion, It is characterized in that The method of adjusting the weights corresponding to the sensors by using a dynamic weighted data fusion model according to the normal data and the fault information to obtain the weight information and normal data corresponding to the sensors includes: Evaluate the credibility of each sensor data according to the normal data and the fault information, and adjust the data fusion strategy in real time; According to the adjusted data fusion strategy, the Bayesian network or fuzzy logic algorithm is used to adjust the weight corresponding to each sensor according to the consistency of real-time monitoring data and historical performance, and obtain the weight information and normal data corresponding to each sensor.
5. According to claim 4, the backward warning method based on multi-sensor fusion, It is characterized in that According to the adjusted data fusion strategy, the Bayesian network or fuzzy logic algorithm is used to adjust the weight corresponding to each sensor according to the consistency of real-time monitored data and historical performance, and the weight information and normal data corresponding to each sensor are obtained, and then it also includes: Data from different types of sensors are used for cross-validation. When abnormal data is detected, data compensation is performed by enabling data from backup sensors or other sensors that are working normally.
6. The backward warning method based on multi-sensor fusion according to claim 1, It is characterized in that The weight information and normal data corresponding to each sensor obtained according to the dynamic weighting are perceived using the environment perception model to obtain an environment perception result, and the corresponding backward warning function is triggered according to the environment perception result, including: Perform multi-sensor fusion according to weight information corresponding to each sensor and normal data, and perform environmental perception based on the multi-sensor fusion data to obtain the environmental perception result; Determining whether to trigger a preset backward warning function according to the environmental perception result; If yes, the control triggers the corresponding backward warning function.
7. The backward warning method based on multi-sensor fusion according to claim 1, It is characterized in that The method further comprises: The original fault detection and isolation model is trained based on the labeled original data to obtain a trained fault detection and isolation model; wherein the labeled original data includes: original data labeled normally and original data labeled abnormally.
8. A backward warning device based on multi-sensor fusion, It is characterized in that include: Data acquisition module, used to obtain data collected by each sensor in real time; A fault detection and isolation module is used to monitor the working status of each sensor using a fault detection and isolation model based on the acquired data, and isolate the data of the faulty sensor according to the working status of each sensor to obtain normal data and fault information; A dynamic weighting module, used to adjust the weight corresponding to each sensor by using a dynamic weighted data fusion model according to the normal data and the fault information, so as to obtain the weight information and normal data corresponding to each sensor; The environment perception module is used to perceive the environment using the environment perception model according to the weight information and normal data corresponding to each sensor obtained by dynamic weighting to obtain the environment perception result; The backward warning trigger module is used to trigger the corresponding backward warning function according to the environmental perception result.
9. A terminal, It is characterized in that include: A processor and a memory, wherein the memory stores a backward warning program based on multi-sensor fusion, and when the backward warning program based on multi-sensor fusion is executed by the processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion as described in any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a backward warning program based on multi-sensor fusion, and when the backward warning program based on multi-sensor fusion is executed by the processor, it is used to implement the operation of the backward warning method based on multi-sensor fusion as described in any one of claims 1-7.
Citation Information
Cited By
Controller fault prediction method based on data acquisition
CN119690050A
Multi-sensor self-adaptive fault compensation system and method thereof
JP2026098171A
Multi-sensor self-adaptive fault compensation system and method
JP7744495B1
