Intelligent driving redundancy perception enhancement method and system based on virtual sensor

Abstract

The invention discloses an intelligent driving redundancy perception enhancement method and system based on a virtual sensor, and the system comprises a data collection module which is used for obtaining the detection data of a real sensor; the virtual sensor construction module is used for processing the collected multi-source data based on a deep learning algorithm so as to construct a virtual sensor; the virtual sensor is used for outputting identification data based on input real-time multi-source data; the multi-source data are original data collected by various real sensors; the sensor state monitoring module is used for detecting the working state of a real sensor; the data fusion module is used for fusing the identification data with detection data of a real sensor to obtain sensing data; and the decision execution module is used for generating a decision instruction according to the identification data or the sensing data, and the decision instruction responds to an execution mechanism of the intelligent driving system.

Description

Technical Field

[0001] This application relates to the field of intelligent perception technology. Specifically, it relates to an intelligent driving redundancy perception enhancement method and system based on virtual sensors. Background Art

[0002] At present, with the continuous development of intelligent driving technology, the perception system of vehicles plays a crucial role in safe and accurate driving. Currently, intelligent driving vehicles mainly rely on a variety of sensors such as cameras, radars, and infrared thermal imagers to obtain surrounding environment information. However, these existing sensors have many limitations in practical applications.

[0003] First of all, as an important part of the intelligent driving perception system, although the camera can provide rich visual information, it has certain defects when facing complex environments and special scenarios. For example, in bad weather conditions such as heavy rain, thick fog, and sandstorms, the imaging quality of the camera will be severely affected, resulting in blurred images and lost details, thus greatly reducing the accuracy of functions such as object detection and material recognition based on image recognition. In addition, for some materials with special optical properties, such as transparent glass, the camera relying only on visible light imaging is difficult to accurately identify its material properties and states, and is prone to misjudgment or missed judgment.

[0004] Secondly, radar sensors mainly detect information such as the distance, speed, and angle of target objects by emitting and receiving electromagnetic waves. Although radar has high accuracy in ranging and speed measurement, its ability to identify object materials is relatively weak. Radar echo signals mainly reflect the geometric shape and motion state of objects, and the discrimination of different materials is limited. For example, for objects of different materials such as metal and plastic, radar may only be able to detect their presence and position information, but it is difficult to accurately determine their specific materials.

[0005] Furthermore, infrared thermal imaging sensors use the infrared radiation emitted by objects themselves to form images and can effectively detect target objects at night or in low-light environments. However, infrared thermal imaging sensors also have limitations. On the one hand, it is sensitive to temperature changes, but it is difficult to accurately distinguish some objects of different materials with similar temperatures. On the other hand, the resolution of infrared thermal imaging images is relatively low, and the ability to capture object details is not as good as that of cameras, which also limits its application in material recognition and other aspects.

[0006] In addition, when any of the above real sensors fails or is affected by external interference, the performance of the entire perception system will be severely affected, and it may even cause the intelligent driving vehicle to lose the accurate perception of the surrounding environment, thus triggering safety accidents. For example, when the radar is affected by electromagnetic interference, the signals it outputs may be abnormal, resulting in incorrect judgments of the distance and speed of the obstacles in front of the vehicle, thereby affecting driving decisions.

[0007] In summary, the existing intelligent driving perception systems have deficiencies in material recognition, and the reliability and redundancy of sensors need to be improved. There is an urgent need for a technical solution that can make up for these deficiencies to enhance the safety and reliability of intelligent driving. Summary of the Invention

[0008] To solve the above problems, the embodiments of the present application provide an intelligent driving redundancy perception enhancement method and system based on virtual sensors. Through the construction and application of virtual sensors, redundant perception enhancement is realized, and the performance of the intelligent driving perception system is comprehensively improved.

[0009] In a first aspect, the embodiments of the present application provide an intelligent driving redundancy perception enhancement system based on virtual sensors, and the system includes: A data acquisition module, configured to acquire the detection data of real sensors; A virtual sensor construction module, configured to process the acquired multi-source data based on a deep learning algorithm to construct a virtual sensor; the virtual sensor is used to output recognition data based on the input real-time multi-source data; the multi-source data is the original data collected by various real sensors; A sensor status monitoring module, configured to detect the working status of real sensors; A data fusion module, configured to fuse the recognition data with the detection data of the real sensors to obtain perception data; A decision execution module, configured to generate a decision instruction according to the recognition data or the perception data, and the decision instruction is responsive to the actuator of the intelligent driving system.

[0010] Preferably, the working status of the real sensors includes: normal status, fault status; When the working status is in a fault status, the decision execution module generates the decision instruction according to the recognition data; When the working statuses are all in a normal status, the decision execution module generates the decision instruction according to the perception data.

[0011] Preferably, the real sensors at least include cameras, ultrasonic radars, lidars, and infrared thermal imagers; The camera is used to collect image data around the vehicle, and the image data includes appearance, color, and texture features; The ultrasonic radar is used to detect the distance, speed, and angle information between the target object and the vehicle, and form dynamic data; The lidar is used to collect three-dimensional spatial information of the target object and form point cloud data; Radar data is generated based on the point cloud data and the dynamic data; The infrared imager is used to collect infrared thermal radiation images of objects around the vehicle and generate infrared thermal imaging data.

[0012] Preferably, the virtual sensor construction module constructs multiple virtual sensors based on the types of multi-source data.

[0013] Preferably, the virtual sensor construction module at least includes a first construction unit and a second construction unit; Among them, the first construction unit has a first deep learning model built in, and trains the first deep learning model based on the first source data collected by the real sensor to obtain the first virtual sensor; the first source data is image data, and the first virtual sensor is used to process the first detection data of the camera, and the first detection data includes the image data of the camera; The second construction unit has a second deep learning model built in, and trains the second deep learning model based on the second source data collected by the real sensor to obtain the second virtual sensor; the second source data at least includes the dynamic data and the point cloud data, and the second virtual sensor is used to process the second detection data, and the second detection data includes the dynamic data of the ultrasonic radar and the point cloud data of the lidar.

[0014] Preferably, the second source data further includes the infrared thermal imaging data of the infrared imager; When training the second deep learning model based on the second source data, it further includes: Training the second deep learning model based on the dynamic data and the point cloud data, and at the same time, optimizing the second deep learning model based on the infrared thermal imaging data to obtain the second virtual sensor, and the second detection data further includes the infrared thermal imaging data of the infrared imager.

[0015] Preferably, the data fusion module includes a feature-level fusion unit and a decision-level fusion unit; among them, the feature-level fusion unit is used to perform feature fusion on the recognition data output by the virtual sensor and the detection data of the real sensor; The decision-level fusion unit is used to perform comprehensive decision-making on the feature fusion to obtain the perception data.

[0016] In a second aspect, an embodiment of the present application provides an intelligent driving redundancy perception enhancement method based on a virtual sensor, which is applicable to an intelligent driving redundancy perception enhancement system based on a virtual sensor in the first aspect. The method includes: Obtain the original data collected by each real sensor mounted on the vehicle, and generate detection data based on the original data; Input the original data into a pre-constructed virtual sensor, and output recognition data. Fuse the recognition data with the detection data to obtain perception data; Generate a decision instruction according to the recognition data or the perception data, and the decision instruction is responsive to an actuator of the intelligent driving system.

[0017] Preferably, pre-constructing the virtual sensor specifically includes: Obtain first source data and a first deep learning model suitable for processing the first source data; the first source data is image data; Generate a first training set based on the first source data of different objects of different materials in different scenarios; Train the first deep learning model based on the first training set to obtain a first virtual sensor, and the first virtual sensor is used to process the image data in the detection data; Obtain second source data and a second deep learning model suitable for processing the second source data; the second source data is radar data; Generate a second training set based on the second source data of different objects of different materials in different scenarios; Train the second deep learning model based on the second training set to obtain a second virtual sensor, and the second virtual sensor is used to process the radar data in the detection data.

[0018] Preferably, the second source data further includes infrared thermal imaging data, and an optimized training set is generated based on the infrared thermal imaging data of different objects of different materials in different scenarios; While training the second deep learning model based on the second training set, optimize the second deep learning model based on the optimized training set to obtain an optimized second virtual sensor, so that the optimized second virtual sensor is also used to process the infrared thermal imaging data in the detection data.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method provided in the first aspect or any possible implementation manner of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect or any possible implementation manner of the first aspect is implemented.

[0021] The beneficial effects of the present invention are as follows: The present invention relates to an intelligent driving redundancy perception enhancement method and system based on a virtual sensor. Based on existing sensor data, namely camera image data, radar data, and infrared thermal imaging data, a virtual sensor is constructed using a deep learning algorithm, enabling the algorithm model optimized for specific materials of the virtual sensor to have a more accurate material recognition and state judgment ability, providing more detailed environmental information for the intelligent driving system, and helping the system make more reasonable decisions.

[0022] Under normal circumstances, a method of fusing decisions between the virtual sensor and the real sensor is adopted. Through a data fusion algorithm, the high material sensitivity information provided by the virtual sensor is combined with other accurate information of the real sensor to further improve the overall perception accuracy; when the real sensor fails or is interfered with, the virtual sensor seamlessly takes over the work and continues to provide reliable information for intelligent driving, ensuring the continuity of the perception system and avoiding decision-making errors caused by sensor failures.

[0023] By constructing a virtual sensor and implementing redundancy perception enhancement, the present invention solves the deficiencies in material recognition of existing intelligent driving perception systems and the reliability problems of sensors, providing a more powerful guarantee for the safe and stable operation of intelligent driving.

[0024] The present invention relies on a deep learning algorithm to deeply mine and analyze multi-source sensor data such as cameras, radars, and infrared thermal imagings, generating a virtual sensor with high sensitivity to specific materials and breaking through the limitations of traditional sensor material recognition.

[0025] Under normal working conditions, the virtual sensor data and the real sensor data are fused at the feature level and the decision level to comprehensively improve the perception accuracy and enable the system to have a more comprehensive and accurate understanding of the environment.

[0026] Based on the virtual sensor, the present invention develops a redundancy perception mechanism to continuously monitor the state of the real sensor. Once a failure or interference occurs, the virtual sensor seamlessly switches to take over the work to ensure continuous and accurate perception and enhance the reliability of the system. Description of the Drawings

[0027] To more clearly illustrate the technical solutions in the embodiments of the present application, the accompanying drawings required for the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0028] Figure 1 It is a structural diagram of an intelligent driving redundant perception enhancement system based on a virtual sensor provided by an embodiment of the present application; Figure 2 It is a schematic diagram of a method for generating a decision instruction; Figure 3 It is a structural diagram of the fusion of identification data and detection data; Figure 4 It is a schematic diagram of a method for data fusion; Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0030] In the following description, the terms "first" and "second" are only for the purpose of description and cannot be construed as indicating or implying relative importance. The following description provides multiple embodiments of the present application. Different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing all other possible combinations of A, B, C, and D, even though such embodiments may not be explicitly described in the following content.

[0031] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes can be made to the functions and arrangements of the described elements without departing from the scope of the content of the present application. Each example can appropriately omit, substitute, or add various processes or components. For example, the described method can be executed in a different order than the described order, and various steps can be added, omitted, or combined. In addition, the features described in some examples can be combined into other examples.

[0032] See Figure 1 , Figure 1 It is a structural diagram of an intelligent driving redundant perception enhancement system based on a virtual sensor provided by an embodiment of the present application. In the embodiment of the present application, the system includes: The data acquisition module is used to obtain the detection data of real sensors; The virtual sensor construction module is used to process the multi-source data collected based on a deep learning algorithm to construct a virtual sensor; the virtual sensor is used to output recognition data based on the input real-time multi-source data; the multi-source data is the original data collected by various real sensors; The sensor status monitoring module is used to detect the working status of real sensors; The data fusion module is used to fuse the recognition data with the detection data of real sensors to obtain perception data; The decision execution module is used to generate a decision instruction according to the recognition data or the perception data, and the decision instruction is responsive to the actuator of the intelligent driving system.

[0033] In this application, the real sensors are the sensors installed on the vehicle, which may include sensors for perceiving the external environment such as cameras, ultrasonic radars, lidars, and infrared thermal imagers, and may also include sensors for perceiving the vehicle interior environment, such as in-vehicle cameras, temperature sensors, etc.

[0034] Taking several types of sensors such as cameras, ultrasonic radars, lidars, and infrared thermal imagers as examples, the perceivable content that can be obtained includes: The camera is used to collect the image data around the vehicle, and the image data includes appearance, color, and texture features; The ultrasonic radar is used to detect the distance, speed, and angle information between the target object and the vehicle to form dynamic data; The lidar is used to collect the three-dimensional space information of the target object to form point cloud data; The detection data of the two radars can be combined, which is called radar data, that is, radar data is generated based on the point cloud data and the dynamic data; The infrared imager is used to collect the infrared thermal radiation images of the objects around the vehicle to generate infrared thermal imaging data.

[0035] The data acquisition module can be responsible for collecting the detection data from cameras, radars, and infrared thermal imagers.

[0036] Based on the actual situation, various real sensors can be arranged according to actual needs. Exemplarily: Camera data acquisition: Multiple different types of cameras, including front-view cameras, rear-view cameras, and surround-view cameras, can be installed at different positions of the vehicle, such as the front of the vehicle, the rear of the vehicle, both sides of the vehicle body, and the interior rearview mirror. These cameras collect the image data of the vehicle surrounding environment at a certain frame rate (such as 30 frames per second), covering the visible light band, and can obtain rich visual information such as the appearance, color, and texture of objects.

[0037] Radar data acquisition: Millimeter-wave radar and lidar can be arranged. The millimeter-wave radar can detect the distance, speed, and angle information of target objects in real time by transmitting and receiving millimeter-wave signals and analyzing the frequency change of the echo signal. The lidar can accurately obtain the three-dimensional spatial information of objects by emitting laser beams and measuring the time of the reflected light, generating point cloud data.

[0038] Infrared thermal imaging data acquisition: An infrared thermal imager can be installed at a suitable position on the vehicle to continuously collect the infrared thermal radiation images of the objects around the vehicle, which can detect the thermal characteristics of objects at night or in low-light environments, especially suitable for detecting objects with temperature differences from the surrounding environment.

[0039] In this application, the purpose of constructing virtual sensors is to process the real-time data collected by real sensors to obtain more accurate perception results. Based on the type of sensors, it can be divided into two types, one is image data, and the other is detection data. Among them, image data is the image data collected by cameras, and detection data is the radar data detected by the two radars and the infrared thermal imaging data detected by the infrared thermal imager; different deep learning models can be selected to construct virtual sensors for the processing of different data.

[0040] In the embodiments of this application, multiple virtual sensors can be constructed based on the types of multi-source data, which are respectively used to process the corresponding data types, making the virtual sensors more targeted with the data types and enhancing the accuracy of recognition and perception.

[0041] Taking the above two data types as an example, the virtual sensor construction module at least includes a first construction unit and a second construction unit, and the corresponding virtual sensors are constructed based on the two construction units respectively.

[0042] Exemplarily, a first deep learning model can be built into the first construction unit. The first deep learning model is suitable for processing image data. The first deep learning model is trained based on the first source data collected by real sensors to obtain a first virtual sensor; the first source data is image data, and the first virtual sensor trained based on image data is suitable for processing the first detection data of the camera, and the first detection data includes the image data of the camera. Exemplarily, a second deep learning model can be built into the second construction unit. The second deep learning model is suitable for processing detection data. The second deep learning model is trained based on the second source data collected by real sensors to obtain a second virtual sensor; the second source data at least includes dynamic data and point cloud data, and the second virtual sensor trained based on detection data is suitable for processing the second detection data, and the second detection data includes the dynamic data of the ultrasonic radar and the point cloud data of the lidar.

[0043] Further, the second source data may further include infrared thermal imaging data of an infrared imager. When training the second deep learning model based on the second source data, it further includes: training the second deep learning model based on dynamic data and point cloud data, and at the same time, optimizing the second deep learning model based on the infrared thermal imaging data to obtain a second virtual sensor, so that the second virtual sensor can be applied to process infrared thermal imaging data. Correspondingly, the second detection data also includes infrared thermal imaging data of the infrared imager.

[0044] It can be understood that when the data type increases, the construction unit can be correspondingly increased. Based on the characteristics of the new data type, a suitable built-in deep learning model is selected, and in the same construction method as above, a virtual sensor applicable to the new data type is constructed.

[0045] In a specific implementation, to construct a virtual sensor based on a deep learning model, a large amount of multi-source sensor data containing objects of different materials (such as glass, metal, plastic, etc.) can be collected as a training data set, including images of objects of various materials in different scenarios captured by a camera, detection data of these objects by a radar, and thermal images obtained by an infrared imager.

[0046] Then, a suitable deep learning model is selected. For example, a convolutional neural network (CNN) is used to process the camera image data, a recurrent neural network (RNN) or its variant is used to process the sequential radar data, and at the same time, the model is optimized according to the characteristics of the infrared thermal imaging data. The training data set is used to train the model so that the model learns the feature representations of objects of different materials in various sensor data. For example, through the CNN, features such as the texture and reflectivity of objects of different materials in the camera image are learned, enabling the model to accurately identify material-related features such as the transparent characteristics of glass and the luster of metal.

[0047] The trained deep learning model constitutes a virtual sensor. For the input real-time sensor data (real-time detection data), the virtual sensor can output sensitivity information about specific material objects, that is, identification data. For example, based on the current camera image data, the virtual sensor can output the probability that the object in the image is glass or metal, and at the same time, combine the radar and infrared thermal imaging data to further correct and refine the material identification result.

[0048] It can be understood that the identification data can be probability-based information, such as the probability information of a single type of material or multiple types of materials, or a deterministic identification result, or a combination of both. Exemplarily, the identification of color, shape, etc. is deterministic, while the identification of material is probabilistic. When outputting the identification data, information such as the color, shape, and material of the object can be included at the same time. Therefore, the identification data can include multiple types of information.

[0049] In the embodiments of the present application, please refer to Figure 3 , the data fusion module includes a feature-level fusion unit and a decision-level fusion unit. Among them, the feature-level fusion unit is used to perform feature fusion on the recognition data output by the virtual sensor and the detection data of the real sensor to obtain a multi-dimensional feature map; the decision-level fusion unit is used to perform comprehensive decision-making on the multi-dimensional feature map to obtain perception data.

[0050] In the embodiments of the present application, when the real sensor is in a normal working state, the recognition data and the detection data can be subjected to feature fusion to obtain a multi-dimensional feature map, and based on the multi-dimensional feature map, comprehensive decision-making is performed through a decision-level fusion algorithm to obtain a more accurate perception result of the target object.

[0051] Specifically, based on the multi-dimensional feature map obtained by feature fusion, decision-level fusion algorithms such as the voting method and Bayesian inference can be used to perform comprehensive decision-making on the features after the fusion of multiple sensors to obtain a more accurate perception result of the target object.

[0052] For example, for an object in front, the camera identifies it as a possible metal object, the radar provides its accurate distance information, and the virtual sensor further confirms that the probability of it being metal is relatively high. Through the decision-level fusion algorithm, these information are integrated to determine that the object is metal and more accurate information such as its position, material, and state is given.

[0053] Based on the actual situation, vehicle-mounted sensors are at risk of being interfered with or malfunctioning, which can be collectively referred to as sensor failures. Therefore, when a vehicle-mounted sensor fails, the obtained detection data may not be real. If decision-making instructions are made based on untrue data, there are potential safety hazards, and the virtual sensor of the present application can well solve this technical defect.

[0054] It should be clear that when there are sensor failures, the same type of sensors often have redundant configurations. Even if the redundant configurations also fail, other types of sensors can also make alternative judgments. For example, to judge whether there is a failure ahead. When the camera fails, the radar can also detect and make a judgment on whether there is a failure.

[0055] However, in actual applications, if the camera judges that there is a failure ahead and the radar judges that there is none, it often intervenes in the way of there being a failure. Although the risk of misjudgment can be reduced, there are still potential safety hazards, and the virtual sensor of the present application can solve this problem by detecting the working state of the sensor to judge the validity of its data. When there is a failure, the recognition data is used as the decision-making basis instead of the faulty data.

[0056] Therefore, the intelligent perception through virtual sensors in this application can effectively solve the misjudgment risk when the sensors malfunction, ensuring the effectiveness and accuracy of intelligent perception.

[0057] In an embodiment of this application, the sensor status monitoring module may include a fault detection unit, an interference identification unit, and a fault output unit. Among them, the fault detection unit is used to judge the working status of the sensor based on preset monitoring indicators; the interference identification unit uses signal processing and data analysis to identify whether the sensor is affected by external interference; the fault output unit is used to receive the judgment result of the fault detection unit and the identification result of the interference identification unit, and output the fault status of the corresponding sensor.

[0058] Exemplarily, by setting a series of monitoring indicators, such as the clarity and noise level of camera images, the intensity and stability of radar signals, the uniformity of infrared thermal imaging images, etc., to judge whether the sensor is working properly. For example, if the camera image appears severely blurred or has excessive noise and lasts for more than a certain threshold, it is determined that the camera may malfunction. At this time, the fault output unit can output the camera as a fault status and send the corresponding status information to the data fusion module; if the radar signal shows abnormal fluctuations or loss, it is considered that the radar is interfered or malfunctioned. At this time, the fault output unit can output the radar as a fault status and send the corresponding status information to the data fusion module.

[0059] Exemplarily, signal processing and data analysis techniques are used to identify whether the sensor is affected by external interference. For example, spectrum analysis is used to judge whether the radar is affected by electromagnetic interference, and image feature analysis is used to judge whether the camera is affected by strong light, smoke, etc. Once it is detected that the sensor malfunctions or is interfered, the fault output unit can output the sensor as a fault status and send the corresponding status information to the data fusion module.

[0060] After receiving the corresponding status information, the data fusion module can exclude the detection data collected by the sensors in the fault state from feature fusion, and only perform data fusion on the detection data of the sensors in the normal state, thereby obtaining semi-perceived state data with a semi-fusion nature where some identification data is not fused. In the semi-perceived state data, there are some perceived data that have been fused and some identification data that lack fusion objects and have not been fused. That is, the identification data dominates in the semi-perceived state data. Therefore, the semi-perceived state data can be defined as identification data, which is different from the fully fused perceived data.

[0061] When generating a decision instruction based on the recognition data, the recognition data is dominant. When generating a decision instruction based on the semi-perceptual data, the recognition data is also dominant. When generating a decision instruction based on the perceptual data, the perceptual data is dominant. Therefore, from the perspective of dominance, when generating a decision instruction in this application, it is generated based on the recognition data or the perceptual data.

[0062] In the embodiments of this application, the working states of the real sensors include: normal state and fault state. Thus, the working state of the vehicle-mounted sensors can be judged first, and more effective data can be selected according to the working state to generate a decision instruction, specifically including: When the working state is in the fault state, the decision execution module generates a decision instruction according to the recognition data; When the working states are all in the normal state, the decision execution module generates a decision instruction according to the perceptual data.

[0063] In a possible implementation manner, if all the sensors are in the fault state and detection data cannot be obtained at this time, an alarm message can be directly sent, including the fault type and the fault code.

[0064] In another possible implementation manner, if all the sensors of the same type are in the fault state, although detection data and recognition data can be obtained based on other sensors, there are still relatively large hidden dangers. At this time, a warning message can be sent, including the type of the faulty sensor and the fault code.

[0065] In a preferred embodiment, a critical threshold for the number of faults of various types of sensors can be preset. When the number of faults reaches the critical threshold, an alarm message is sent, and at the same time, it is informed that the perception system is limited, reducing the driver's dependence on the related functions of the intelligent driving system and reducing the safety risk.

[0066] In the embodiments of this application, the decision execution module generates a corresponding decision instruction according to the perception result output by the data fusion module, and the perception result is obtained based on the recognition data or the perceptual data. The decision instruction can be sent to the execution structure of the intelligent driving system of the vehicle, such as the engine, the brake, the steering system, etc., to achieve the safe and stable operation of the intelligent driving.

[0067] For example, when an obstacle made of glass material is detected ahead, the decision execution module decides whether to decelerate, avoid, etc. according to the relative distance, relative speed, relative angle, etc. between the obstacle and the vehicle, and generates a corresponding instruction to send to the relevant execution structure to achieve the purpose of intelligent driving.

[0068] The present invention also proposes an intelligent driving redundancy perception enhancement method based on a virtual sensor, which is applicable to the above-mentioned intelligent driving redundancy perception enhancement system based on a virtual sensor. The method includes: Obtain the original data collected by each real sensor carried on the vehicle, and generate detection data based on the original data; Input the original data into a pre-constructed virtual sensor, and output recognition data. Integrate the recognition data with the detection data to obtain perception data; Generate a decision instruction according to the recognition data or the perception data, and the decision instruction responds to the actuator of the intelligent driving system.

[0069] Among them, there are various types of real sensors carried on the vehicle, such as cameras, radars (ultrasonic, laser), infrared imagers, etc. When constructing a virtual sensor, an adapted virtual sensor can be constructed according to the differences in the data collected by the real sensors, and various types of detection data can be processed specifically to achieve the purpose of enhancing perception.

[0070] Exemplarily, different deep learning architectures can be selected to adapt to different types of detection data, and the corresponding models are trained using a training data set, so that the models learn the feature representations of objects of different materials in various sensor data.

[0071] For example, a convolutional neural network (CNN) is used to construct a virtual sensor for processing camera image data, and a recurrent neural network (RNN) or other variants are used to construct a virtual sensor for processing sequential radar data. At the same time, the model is optimized according to the characteristics of infrared thermal imaging data.

[0072] Through CNN, learn the texture, reflectivity and other features of objects of different materials in the images collected by the camera, so that the model can accurately identify material-related features such as the transparent characteristics of glass and the luster of metal. The radar data is processed by RNN to obtain three-dimensional point clouds, and the infrared thermal imaging data is processed to obtain temperature distribution characteristics. The combination of the two can further correct and refine the material recognition results.

[0073] For the input real-time sensor data, the virtual sensor can output the sensitivity information of specific material objects. For example, based on the current camera image data, the virtual sensor can output the probability that the object in the image is glass or metal, and at the same time, combine the radar and infrared thermal imaging data to further correct and refine the material recognition results.

[0074] When pre-constructing a virtual sensor, the original data collected by the data acquisition module can be used as a training data set, or the simulated data obtained by other means can be used as a training data set. The simulated data can be artificially designed data, or the original data and the simulated data can be combined as a training data set.

[0075] In the embodiments of the present application, taking the original data as an example for illustration, when other data is added, it can be supplemented in the corresponding training data set, which does not affect the specific training process.

[0076] In a specific embodiment, the pre-built virtual sensor includes: Obtain the first source data and the first deep learning model applicable to process the first source data; the first source data is image data; Generate a first training set based on the first source data of different objects of different materials in different scenarios; Train the first deep learning model based on the first training set to obtain a first virtual sensor, and the first virtual sensor is used to process the image data in the detection data; Obtain the second source data and the second deep learning model applicable to process the second source data; the second source data is radar data; Generate a second training set based on the second source data of different objects of different materials in different scenarios; Train the second deep learning model based on the second training set to obtain a second virtual sensor, and the second virtual sensor is used to process the radar data in the detection data.

[0077] Feasibly, the second source data further includes infrared thermal imaging data, and an optimized training set is generated based on the infrared thermal imaging data of different objects of different materials in different scenarios; While training the second deep learning model based on the second training set, optimize the second deep learning model based on the optimized training set to obtain an optimized second virtual sensor, so that the optimized second virtual sensor is further used to process the infrared thermal imaging data in the detection data.

[0078] Based on this, the second virtual sensor can process the radar data and the infrared thermal imaging data separately or synchronously.

[0079] In the embodiments of the present application, the first virtual sensor is for processing image data, such as the image data collected by a camera; the second virtual sensor is for processing detection data, such as radar data and infrared thermal imaging data.

[0080] In the embodiments of the present application, it is possible to determine whether to generate a decision instruction based on the recognition data or the perception data based on the working state of the real sensor, and the judgment basis can be the detection data of the real sensor; Please refer to Figure 2 , when the working state of the real sensor is normal, a decision instruction can be generated based on the fused perception data; when the real sensor fails, a decision instruction can be generated based on the recognition data, but the recognition data can still be combined with the detection data of the real sensor with normal working state to enhance perception and improve the perception accuracy.

[0081] Furthermore, both the recognition data and the perception data contain multi-source data, that is, data obtained from different types of sensors. Through the fusion processing of multi-source data, mutual verification can be achieved, and the recognition result of the target can be judged more accurately. Therefore, whether generating a decision instruction based on recognition data or perception data, data fusion processing can be implemented, that is, decision-level fusion based on multi-source data. Through the decision-level fusion algorithm, comprehensive decision-making is carried out on the target to obtain a more accurate perception result of the target object. For example, through camera recognition, it may be a metal object, but through radar detection, it is a wooden material. Through the decision-level fusion algorithm, combining this information can determine that the object is a wooden material; while the virtual sensor provides the probability that it is a metal and the probability that it is a wooden material. If the probability of being a wooden material is higher, through the decision-level fusion algorithm, combining the above two types of information can give the perception data that the target object is a wooden material.

[0082] Specifically, decision-level fusion algorithms such as the voting method and Bayesian inference can be used to comprehensively make decisions on multi-source data to obtain a more accurate perception result of the target object.

[0083] When the real sensor fails or is interfered with, the recognition data of the virtual sensor can be directly used as the perception input to ensure the continuity and stability of the perception system.

[0084] Please refer to Figure 4 , Figure 4 FIG. is a schematic diagram of a method for fusing recognition data and detection data to obtain perception data. The recognition data and the detection data are used as feature-level data inputs, and feature extraction and annotation are respectively performed on the recognition data and the detection data. Thus, two types of feature maps can be obtained. The first type is the material feature map based on the recognition data, and the second type is the target feature map based on the detection data. Since both types of maps are based on the same target object, they can be fused, and feature representation of the fusion area can be realized on the fused map, that is, by comparing the features of the same area on the fused map to judge the accuracy of the fusion.

[0085] For the features existing on the fused map, they have multi-dimensional attributes, that is, detection features from different types of sensors. Thus, a multi-dimensional feature map can be obtained, and the multi-dimensional feature map contains multi-dimensional feature attributes; based on the multi-dimensional feature map, the decision-level fusion algorithm can be used to comprehensively judge the multi-dimensional feature attributes to make a decision on the perception result represented by the multi-dimensional feature map, realize the decision-level fusion of the multi-dimensional feature attributes, and finally obtain the perception result, that is, the perception data, based on the multi-dimensional feature attributes contained in the multi-dimensional feature map.

[0086] Exemplarily, for camera image data, the material features output by the virtual sensor are fused with features such as object edges, shapes, and textures extracted based on traditional image processing algorithms; for radar data, the material recognition information of the virtual sensor is combined with the distance and speed information of the radar itself; for infrared thermal imaging data, the material judgment of the virtual sensor based on thermal features is fused with the temperature distribution features of the infrared image. Through this feature-level fusion, the information dimension of the data is enriched.

[0087] Based on the feature-level fusion, decision-level fusion algorithms such as voting method and Bayesian inference are adopted to comprehensively make decisions on the features after the fusion of multiple sensors, and a more accurate perception result of the target object is obtained.

[0088] For example, for an object ahead, the camera identifies it as a possible metal object, the radar provides its accurate distance information, and the virtual sensor further confirms that the probability of it being metal is relatively high. Through the decision-level fusion algorithm, these information are integrated to determine that the object is metal and more accurate information such as its position, material, and state are given.

[0089] This application realizes the enhancement of redundant perception for intelligent driving based on the virtual sensor, bringing significant effects in many aspects: From the aspect of improving the material recognition accuracy, the virtual sensor constructed by the deep learning algorithm can provide a higher sensitivity recognition ability for specific material objects, such as glass, metal, etc. This greatly makes up for the deficiencies of traditional cameras, radars, and infrared thermal imaging sensors in material recognition.

[0090] In the actual driving scenario, accurate material recognition is crucial. For example, when the vehicle is driving on an urban street and facing a building with a glass curtain wall, the virtual sensor can accurately identify its material, and the intelligent driving system can accordingly adjust the vision algorithm in advance to avoid visual interference caused by glass reflection, so as to more accurately perceive the surrounding environment and improve driving safety. Compared with traditional sensors, the virtual sensor significantly improves the recognition accuracy of the intelligent driving system for objects of different materials in complex scenarios. Through actual tests, the recognition accuracy for glass material objects has been significantly improved, and the recognition accuracy for metal material objects has also been correspondingly significantly improved. In terms of enhanced redundancy perception, when real sensors encounter failures or are interfered with, virtual sensors can seamlessly take over the work to ensure the continuity and accuracy of the perception system. This provides a reliable backup mechanism for the intelligent driving system, greatly reducing the risk of accidents caused by the failure of a single sensor. Under extreme weather conditions, such as when heavy rain obstructs the camera's line of sight or radar signals are interfered with in areas with complex electromagnetic environments, virtual sensors can continuously provide key environmental perception information for the intelligent driving system based on the multi-source data features learned previously, ensuring that the vehicle can continue to drive safely. According to statistics, this redundancy mechanism can improve the safety driving guarantee rate of the intelligent driving system in the case of sensor failures or interference.

[0091] In addition, in the normal working state, the fusion of virtual sensor data and real sensor data significantly improves the overall perception accuracy. Through feature-level and decision-level fusion, the advantages of different sensors are complementary, enriching the dimension of environmental perception information. For example, the combination of the object appearance information provided by the camera, the distance and speed information of the radar, and the material recognition information of the virtual sensor enables the intelligent driving system to have a more comprehensive and accurate understanding of the target object. This fusion can not only detect target objects more precisely but also make more reliable predictions about the behavior and potential risks of the objects. For example, for a metal obstacle on the road, by combining the radar's distance information and the virtual sensor's recognition of the metal material, the system can more accurately judge the possible impact on the vehicle and plan a reasonable avoidance path in advance, further improving the safety and reliability of intelligent driving. After actual verification, the overall perception accuracy after data fusion is significantly improved compared with that of a single sensor, laying a solid foundation for the intelligent driving system to make scientific and accurate decisions.

[0092] In summary, the present invention improves the performance of intelligent driving perception from multiple key aspects, strongly promoting the development of intelligent driving technology towards a safer and more reliable direction.

[0093] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented by means of software and / or hardware. The "units" and "modules" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.

[0094] Each processing unit and / or module of the embodiments of the present application can be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or can be implemented by software that executes the functions described in the embodiments of the present application.

[0095] See Figure 5, which shows a schematic structural diagram of an electronic device involved in an embodiment of the present application. This electronic device can be used in the method in the above embodiment. As Figure 3 shown, the electronic device 300 may include: at least one central processing unit 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0096] Among them, the communication bus 302 is used to realize the connection and communication between these components.

[0097] Among them, the user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may further include a standard wired interface and a wireless interface.

[0098] Among them, the network interface 304 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0099] Among them, the central processing unit 301 may include one or more processing cores. The central processing unit 301 uses various interfaces and lines to connect various parts within the entire electronic device 300. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305, it executes various functions of the terminal and processes data. Optionally, the central processing unit 301 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The central processing unit 301 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the central processing unit 301 and may be implemented separately by a single chip.

[0100] Among them, the memory 305 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area can store the data involved in the above-mentioned method embodiments. Optionally, the memory 305 may also be at least one storage device located far from the aforementioned central processor 301. As Figure 3 shown, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0101] In Figure 3 the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user to obtain the data input by the user; and the central processor 301 can be used to call the intelligent driving redundancy perception enhancement application program stored in the memory 305 and specifically perform the following operations: Obtain the raw data collected by each real sensor carried on the vehicle, and generate detection data based on the raw data; Input the raw data into a pre-built virtual sensor, and output recognition data, and fuse the recognition data with the detection data to obtain perception data; Generate a decision instruction according to the recognition data or the perception data, and the decision instruction responds to the actuator of the intelligent driving system.

[0102] This application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. Among them, the computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nano-systems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0103] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0104] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0105] In several embodiments provided by this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some service interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0106] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, in each embodiment of this application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0108] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned memory includes: USB flash drives, read-only memories (ROM), random access memories (RAM), mobile hard disks, magnetic disks, or optical discs, etc., all of which can store program codes.

[0109] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: flash drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs, etc.

[0110] The above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. After considering the specification and practicing the present disclosure herein, those skilled in the art will readily think of other embodiments of the present disclosure. This application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An intelligent driving redundant perception enhancement system based on virtual sensors, characterized in that, The system includes: A data acquisition module for acquiring the detection data of real sensors; A virtual sensor construction module for processing the acquired multi-source data based on a deep learning algorithm to construct a virtual sensor; the virtual sensor is used to output recognition data based on the input real-time multi-source data; the multi-source data is the original data collected by various real sensors; A sensor status monitoring module for detecting the working status of real sensors; A data fusion module for fusing the recognition data and the detection data of the real sensors to obtain perception data; A decision execution module for generating a decision instruction according to the recognition data or the perception data, and the decision instruction is responsive to the actuator of the intelligent driving system.

2. The system according to claim 1, characterized in that, The working status of the real sensors includes: normal status, fault status; When the working status is in a fault status, the decision execution module generates the decision instruction according to the recognition data; When the working statuses are all in a normal status, the decision execution module generates the decision instruction according to the perception data.

3. The system according to claim 1, characterized in that, The real sensors at least include a camera, an ultrasonic radar, a lidar, and an infrared thermal imager; The camera is used to collect image data around the vehicle, and the image data includes appearance, color, and texture features; The ultrasonic radar is used to detect the distance, speed, and angle information between the target object and the vehicle to form dynamic data; The lidar is used to collect the three-dimensional space information of the target object to form point cloud data; Based on the point cloud data and the dynamic data, radar data is generated; The infrared imager is used to collect the infrared thermal radiation image of the object around the vehicle to generate infrared thermal imaging data.

4. The system according to claim 3, wherein The virtual sensor construction module constructs multiple virtual sensors based on the types of multi-source data.

5. The system according to claim 4, characterized in that, The virtual sensor construction module at least includes a first construction unit and a second construction unit; Among them, the first construction unit has a first deep learning model built in, and trains the first deep learning model based on the first source data collected by the real sensor to obtain the first virtual sensor; the first source data is image data, and the first virtual sensor is used to process the first detection data of the camera, and the first detection data includes the image data of the camera; The second construction unit has a second deep learning model built in, and trains the second deep learning model based on the second source data collected by the real sensor to obtain the second virtual sensor; the second source data at least includes the dynamic data and the point cloud data, and the second virtual sensor is used to process the second detection data, and the second detection data includes the dynamic data of the ultrasonic radar and the point cloud data of the lidar.

6. The system according to claim 4, wherein The second source data further includes the infrared thermal imaging data of the infrared imager; When training the second deep learning model based on the second source data, it further includes: Based on training the second deep learning model with the dynamic data and point cloud data, and at the same time, optimizing the second deep learning model based on the infrared thermal imaging data to obtain the second virtual sensor, the second detection data further includes the infrared thermal imaging data of the infrared imager.

7. The system according to claim 1, wherein The data fusion module includes a feature-level fusion unit and a decision-level fusion unit; wherein, the feature-level fusion unit is used to perform feature fusion on the recognition data output by the virtual sensor and the detection data of the real sensor to obtain a multi-dimensional feature map; The decision-level fusion unit is used to make a comprehensive decision on the multi-dimensional feature map to obtain the perception data.

8. An intelligent driving redundancy perception enhancement method based on virtual sensors, characterized in that, Applicable to an intelligent driving redundant perception enhancement system based on a virtual sensor according to any one of claims 1-7, the method includes: Obtain the original data collected by each real sensor mounted on the vehicle, and generate detection data based on the original data; Input the original data into a pre-constructed virtual sensor, and output recognition data. Fuse the recognition data with the detection data to obtain perception data; Generate a decision instruction according to the recognition data or the perception data, and the decision instruction responds to the actuator of the intelligent driving system.

9. The method according to claim 8, wherein Pre-construct the virtual sensor, specifically including: Obtain the first source data and the first deep learning model suitable for processing the first source data; the first source data is image data; Generate a first training set based on the first source data of different materials and objects in different scenarios; Train the first deep learning model based on the first training set to obtain a first virtual sensor, and the first virtual sensor is used to process the image data in the detection data; Obtain the second source data and the second deep learning model suitable for processing the second source data; the second source data is radar data; Generate a second training set based on the second source data of different materials and objects in different scenarios; Train the second deep learning model based on the second training set to obtain a second virtual sensor, and the second virtual sensor is used to process the radar data in the detection data.

10. A method according to claim 9, wherein The second source data further includes infrared thermal imaging data, and an optimized training set is generated based on the infrared thermal imaging data of different materials and objects in different scenarios; While training the second deep learning model based on the second training set, optimize the second deep learning model based on the optimized training set to obtain an optimized second virtual sensor, so that the optimized second virtual sensor is also used to process the infrared thermal imaging data in the detection data.

Images