Multi-sensor data fusion method, device, system, equipment and storage medium

Abstract

The present application relates to a multi-sensor data fusion method, apparatus, system, device, and storage medium. Applied to a multi-sensor system including multiple sensors, the method includes: obtaining current environmental data of the multi-sensor system, and obtaining a weight correction value for each of the sensors according to the environmental data; obtaining an initial weight corresponding to each of the sensors, and using the weight correction value of each of the sensors to correspondingly correct the initial weight of each of the sensors to obtain a target weight for each of the sensors; obtaining data to be fused for each of the sensors, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensor, and the sensed data of each of the sensors is sensed by each of the sensors at the same time and in the same scenario; performing fusion processing on the data to be fused for each of the sensors according to the target weight of each of the sensors to obtain a multi-sensor data fusion result. Using this method can fully combine the characteristics of different sensors and improve the reliability of sensor data perception.

Description

Technical Field

[0001] The present application relates to the technical field of sensors, and particularly to a multi-sensor data fusion method, device, system, equipment and storage medium. Background Art

[0002] A sensor is a detection device that can convert the perceived information into an electrical signal or other required form of information output according to certain rules. With the advent of the information age, sensors are the main means and ways to obtain information in the natural and production fields during the process of using information.

[0003] For example, taking the field of intelligent transportation as an example, more and more sensors are installed on roads to sense road data, and the road conditions, traffic safety, etc. can be monitored by analyzing the sensed data. Currently, lidar sensors, millimeter-wave radar sensors and cameras are the main sensors in the field of intelligent transportation, and are often installed on the crossbars, vertical poles or gantry frames at road intersections for data sensing of roads. Different sensors have different characteristics. For example, in rainy or snowy weather, there are more noise interferences in the data sensed by lidar sensors, while at night or in insufficient light conditions, the imaging effect of cameras will be seriously affected, resulting in a large difference between the data sensed by cameras and the reality.

[0004] Therefore, how to fully combine the characteristics of different sensors to improve the reliability of sensor-sensed data has become an urgent problem to be solved at present. Summary of the Invention

[0005] Based on this, it is necessary to provide a multi-sensor data fusion method, device, system, equipment and storage medium that can fully combine the characteristics of different sensors and improve the reliability of sensor-sensed data for the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a multi-sensor data fusion method, which is applied to a multi-sensor system including multiple sensors. The method includes:

[0007] Obtain the current environmental data of the multi-sensor system, and obtain the weight correction value of each sensor according to the environmental data;

[0008] Obtain the initial weight corresponding to each sensor, and use the weight correction value of each sensor to correct the initial weight of each sensor correspondingly to obtain the target weight of each sensor;

[0009] Obtain the data to be fused of each sensor, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensor, and the sensed data of each sensor is sensed by each sensor at the same moment and in the same scenario;

[0010] Fuse the to-be-fused data according to the target weights of the sensors to obtain a multi-sensor data fusion result.

[0011] In one embodiment, the step of fusing the to-be-fused data according to the target weights of the sensors to obtain a multi-sensor data fusion result includes:

[0012] Use the target weights of the sensors to perform weighted summation calculation on the to-be-fused data to obtain the multi-sensor data fusion result.

[0013] In one embodiment, the step of obtaining the initial weights corresponding to the sensors includes:

[0014] Obtain the historical perception data of the sensors, where the historical perception data of the sensors are the data sensed by the sensors at the same historical time and in the same scenario;

[0015] Perform spatio-temporal synchronization processing on the historical perception data to obtain the synchronized data corresponding to the historical perception data;

[0016] Calculate the variance of the synchronized data, and calculate the initial weight of the corresponding sensor according to the variance of the synchronized data.

[0017] In one embodiment, the step of calculating the variance of the synchronized data includes:

[0018] For each sensor, group the synchronized data of the sensor to obtain multiple groups of data;

[0019] Calculate the variance of each group of data in the multiple groups of data of the sensor, and calculate the variance of the synchronized data of the sensor according to the variance of each group of data.

[0020] In one embodiment, the step of obtaining the to-be-fused data of the sensors includes:

[0021] Obtain multiple historical to-be-fused data of the sensors within a preset historical time period, where the historical to-be-fused data are obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensors within the preset historical time period;

[0022] Calculate the average value of the multiple historical to-be-fused data corresponding to each sensor, and determine the average value corresponding to each sensor as the to-be-fused data of each sensor.

[0023] In one embodiment, the step of obtaining the to-be-fused data of the sensors includes:

[0024] Obtain the sensing data sensed by each of the sensors at the same time and in the same scenario;

[0025] Adopt a preset spatio-temporal synchronization method to convert the sensing data corresponding to each of the sensors to a target coordinate system, and obtain the data to be fused corresponding to each of the sensors.

[0026] In one embodiment, the sensors include lidar sensors, cameras, or millimeter-wave radar sensors.

[0027] In one embodiment, the obtaining the current environmental data of the multi-sensing system includes:

[0028] Obtain the current environmental data of the multi-sensing system through an auxiliary sensor, where the auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor.

[0029] In a second aspect, an embodiment of the present application provides a multi-sensing data fusion device, and the device includes:

[0030] A first acquisition module, configured to acquire the current environmental data of the multi-sensing system, and obtain a weight correction value for each of the sensors according to the environmental data;

[0031] A correction module, configured to obtain the initial weight corresponding to each of the sensors, and use the weight correction value of each of the sensors to correct the initial weight of each of the sensors correspondingly, and obtain the target weight of each of the sensors.

[0032] A second acquisition module, configured to acquire the data to be fused for each of the sensors, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensing data of the corresponding sensors, and the sensing data of each of the sensors is sensed by each of the sensors at the same time and in the same scenario;

[0033] A processing module, configured to perform fusion processing on the data to be fused for each of the sensors according to the target weight of each of the sensors, and obtain a multi-sensing data fusion result.

[0034] In a third aspect, an embodiment of the present application provides a multi-sensing system, and the system includes a processor, an auxiliary sensor, and sensors, the sensors and the auxiliary sensor are connected to the processor, the sensors include at least two of lidar sensors, cameras, and millimeter-wave radar sensors, and the auxiliary sensor includes a high-precision optoelectronic sensor and / or a temperature and humidity sensor;

[0035] The auxiliary sensor is configured to obtain the current environmental data of the multi-sensing system, and send the environmental data to the processor;

[0036] The processor is configured to receive the environmental data, obtain the weight correction values of the sensors according to the environmental data, and obtain the initial weights of the sensors, and use the weight correction values of the sensors to correspondingly correct the initial weights of the sensors to obtain the target weights of the sensors;

[0037] The processor is further configured to obtain the data to be fused of the sensors, and perform fusion processing on the data to be fused of the sensors according to the target weights of the sensors to obtain a multi-sensor data fusion result, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensors, and the sensed data of the sensors are sensed by the sensors at the same time and in the same scenario.

[0038] In a fourth aspect, an embodiment of the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the method in the first aspect as described above when executing the computer program.

[0039] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the method in the first aspect as described above when executed by a processor.

[0040] The beneficial effects brought by the technical solutions provided by the embodiments of the present application at least include:

[0041] The above multi-sensor data fusion method, device, system, equipment and storage medium obtain the current environmental data of the multi-sensor system and obtain the weight correction values of each sensor according to the environmental data. Since different sensors have different characteristics in the same environment. For example, in the night or when the light is insufficient, the data perception effect of the camera is poor, while the data perception effects of the lidar sensor and the millimeter-wave radar sensor are good. Therefore, according to the current environmental data, the weight correction values of each sensor are obtained. After obtaining the initial weights corresponding to each sensor, the initial weights of each sensor are corrected correspondingly by using the weight correction values of each sensor to obtain the target weights of each sensor. In this way, the initial weights of each sensor can be dynamically adjusted in combination with the environmental data, so that the characteristics of different sensors can be fully combined. For example, when the current environment is night, the initial weight of the camera can be reduced by the weight correction value of the camera, while the initial weight of the lidar sensor can be increased by the weight correction value of the lidar sensor, and the initial weight of the millimeter-wave radar sensor can be increased by the weight correction value of the millimeter-wave radar sensor, thereby increasing the weights of the sensors with performance advantages in the current environment. Then, after obtaining the data to be fused of each sensor, the data to be fused of each sensor are fused according to the corrected weights of each sensor, that is, the target weights of each sensor, to obtain the multi-sensor data fusion result. By fully combining the characteristics of different sensors, the reliability of the sensor perception data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic flowchart of the multi-sensor data fusion method in one embodiment;

[0043] Figure 2 It is a schematic diagram of some refined steps of step S200 in another embodiment;

[0044] Figure 3 It is a schematic diagram of the refined steps of step S300 in another embodiment;

[0045] Figure 4 It is a schematic diagram of the refined steps of step S300 in another embodiment;

[0046] Figure 5 It is a structural block diagram of the multi-sensor data fusion device in one embodiment;

[0047] Figure 6 It is a structural block diagram of the multi-sensor data fusion device in one embodiment;

[0048] Figure 7 It is a structural block diagram of the multi-sensor data fusion device in one embodiment;

[0049] Figure 8 It is an internal structure diagram of a computer device in one embodiment. Specific Embodiments

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that for the multi-sensor data fusion method provided in the embodiments of the present application, the execution subject can be a multi-sensor data fusion device, and the multi-sensor data fusion device can be implemented as part or all of a computer device through software, hardware or a combination of software and hardware. In the following method embodiments, the execution subject is taken as a computer device for illustration. The computer device can be a server; it can be understood that the multi-sensor data fusion method provided in the following method embodiments can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0052] In one embodiment, as Figure 1 shown, a multi-sensor data fusion method is provided, which is applied to a multi-sensor system including multiple sensors, and includes the following steps:

[0053] Step S100, obtaining the current environmental data of the multi-sensor system, and obtaining the weight correction value of each sensor according to the environmental data.

[0054] In the embodiments of the present application, the computer device can obtain the current environmental data of the multi-sensor system through an auxiliary sensor in the multi-sensor system. The auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor. The high-precision optoelectronic sensor is used to measure the optical signal in the environment where the multi-sensor system is currently located, and the temperature and humidity sensor is used to measure the temperature and humidity in the environment where the multi-sensor system is currently located.

[0055] In the embodiments of the present application, the types of the sensors in the multi-sensor system are different. As an implementation manner, the multi-sensor system may include at least two of a lidar sensor, a camera, and a millimeter-wave radar sensor. For example, taking the field of intelligent transportation as an example, lidar sensors, millimeter-wave radar sensors, and cameras are the main sensors in the field of intelligent transportation, and are often installed on the crossbars, vertical poles or gantries at road intersections for data collection on the road side.

[0056] After the computer device obtains the current environmental data of the multi-sensor system, it can determine the weight correction value corresponding to each sensor in the multi-sensor system and the environmental data by looking up a table or using a mapping model, etc., that is, under each environmental data, each sensor has a corresponding weight correction value.

[0057] Since different sensors have different characteristics in the same environment. For example, in the night or when the light is insufficient, the data acquisition effect of the camera is poor, while the data acquisition effects of lidar sensors and millimeter-wave radar sensors are good. The weight correction values corresponding to the sensors can correct the initial weights of the sensors in the current environment, so that the weights of the sensors with performance advantages in the current environment can be improved.

[0058] Step S200: Obtain the initial weights corresponding to the sensors, and use the weight correction values of the sensors to correct the initial weights of the sensors correspondingly to obtain the target weights of the sensors.

[0059] In the embodiments of the present application, each sensor is set with an initial weight, and the sum of the initial weights of the sensors is 1. The initial weight can be determined according to the historical perception data of each sensor. The computer device obtains the initial weights corresponding to the sensors. For each sensor, the computer device uses the weight correction value corresponding to the sensor to correct the initial weight of the sensor to obtain the target weight of the sensor. Thus, the target weights of the sensors are obtained.

[0060] In a possible implementation manner, when the computer device uses the weight correction value of a sensor to correct the initial weight of the sensor correspondingly, it can be to add the weight correction value of the sensor and the initial weight of the sensor, and the obtained sum is the target weight of the sensor.

[0061] For example, when the current environmental data of the multi-sensor system represents that the current environment is night, since the data acquisition effect of the camera is poor at night, while the data acquisition effects of lidar sensors and millimeter-wave radar sensors are good, therefore, the weight correction value of the camera obtained by the computer device is negative, and the weight correction values of the lidar sensors and millimeter-wave radar sensors obtained are positive; the computer device adds the weight correction value of each sensor and the corresponding initial weight, then the initial weight of the camera can be reduced, while the initial weights of the lidar sensor and the millimeter-wave radar sensor can be increased, so as to improve the weights of the sensors with performance advantages in the current environment, realizing the dynamic adjustment of the initial weights of the sensors in combination with environmental data, and thus the advantageous characteristics of different sensors can be fully combined.

[0062] Step S300: Obtain the data to be fused of the sensors.

[0063] In the embodiments of the present application, the data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensors, and the perception data of each sensor is perceived by each sensor at the same moment and in the same scenario.

[0064] For example, at the same moment and in the same scenario, the perception data collected by a camera is an image, the perception data collected by a lidar sensor is point cloud data, and the perception data collected by a millimeter-wave radar sensor is electromagnetic waves. A computer device can perform processing such as classification, coordinate extraction, and angle calculation on the image, point cloud data, and electromagnetic waves respectively to obtain the target perception data of each sensor. Among them, the target perception data of the camera can include the category of the target in the image and the position box coordinates of the target in the image; the target perception data of the lidar sensor can include the category of the target extracted from the point cloud data, the center point coordinates of the target (equivalent to the horizontal and vertical coordinates), and the heading angle of the target; the target perception data of the millimeter-wave radar sensor can include the category of the target extracted from the electromagnetic waves, the coordinates of the target (equivalent to the horizontal and vertical coordinates), and the speed of the target.

[0065] In a possible implementation manner, the computer device can set the standard target perception data to include six types of information: category, horizontal coordinate, vertical coordinate, horizontal speed, vertical speed, and angle. For the information that cannot be obtained by each sensor, the computer device marks this type as "0"; for example, the target perception data of the camera includes the category of the target in the image and the position box coordinates of the target in the image (equivalent to the horizontal and vertical coordinates), but the horizontal speed, vertical speed, and angle cannot be obtained through the camera, so the computer device marks these three types as 0 in the target perception data of the camera; the target perception data of the lidar sensor includes the category of the target extracted from the point cloud data, the center point coordinates of the target (equivalent to the horizontal and vertical coordinates), and the heading angle of the target (equivalent to the angle), but the horizontal speed and vertical speed cannot be obtained through the lidar sensor, so the computer device marks these three types as 0 in the target perception data of the lidar sensor; the target perception data of the millimeter-wave radar sensor includes the category of the target extracted from the electromagnetic waves, the coordinates of the target (equivalent to the horizontal and vertical coordinates), and the speed of the target (equivalent to the horizontal speed and vertical speed), but the angle cannot be obtained through the millimeter-wave radar sensor, so the computer device marks this type as 0 in the target perception data of the millimeter-wave radar sensor. After the computer device fills in the data of each sensor, the target perception data of each sensor is obtained.

[0066] Since different sensors perform data perception based on different original coordinate systems, such as the camera performing data perception based on the camera coordinate system and the millimeter-wave radar sensor performing data perception based on the millimeter-wave radar coordinate system, the computer device converts the target perception data of the camera, the target perception data of the lidar sensor, and the target perception data of the millimeter-wave radar sensor to the same target coordinate system using a spatio-temporal synchronization method, so that the data of different sensors can be fused based on the same coordinate system.

[0067] As an implementation manner, the spatio-temporal synchronization method may be a calibration method, that is, the computer device uses a calibration algorithm to calibrate the target perception data of the camera, the target perception data of the lidar sensor, and the target perception data of the millimeter-wave radar sensor to the same target coordinate system. In the embodiments of the present application, the target coordinate system may be a pixel coordinate system, and the computer device obtains the data to be fused corresponding to each sensor after spatio-temporal synchronization of the target perception data of each sensor.

[0068] In the embodiments of the present application, the multi-sensor system may be set in the same scenario, and the computer device may set the same sampling frequency for each sensor, such as it may be set to collect once every 0.1 seconds, thereby ensuring that each sensor perceives the perception data at the same time and in the same scenario.

[0069] Step S400: Perform a fusion process on each data to be fused according to the target weights of each sensor to obtain a multi-sensor data fusion result.

[0070] The computer device performs a fusion process on each data to be fused according to the target weights of each sensor to obtain a multi-sensor data fusion result. In a possible implementation manner, the computer may implement step S400 by executing the following step A:

[0071] Step A: Use the target weights of each sensor to perform a weighted summation calculation on each data to be fused to obtain a multi-sensor data fusion result.

[0072] That is, for each sensor, the computer device may multiply the data to be fused of the sensor by the target weight of the sensor, and then add the multiplication results of each sensor to obtain a multi-sensor data fusion result.

[0073] In this embodiment, the computer device obtains the current environmental data of the multi-sensor system and obtains the weight correction values of each sensor according to the environmental data. Since different sensors have different characteristics in the same environment. For example, in the case of night or insufficient light, the data acquisition effect of the camera is poor, while the data acquisition effects of the lidar sensor and the millimeter-wave radar sensor are good. Therefore, according to the current environmental data, the weight correction values of each sensor are obtained. After obtaining the initial weights corresponding to each sensor, the initial weights of each sensor are corrected correspondingly by using the weight correction values of each sensor to obtain the target weights of each sensor. In this way, the initial weights of each sensor can be dynamically adjusted in combination with the environmental data, so that the characteristics of different sensors can be fully combined. For example, when the current environment is night, the initial weight of the camera can be reduced by the weight correction value of the camera, while the initial weight of the lidar sensor can be increased by the weight correction value of the lidar sensor, and the initial weight of the millimeter-wave radar sensor can be increased by the weight correction value of the millimeter-wave radar sensor, so as to increase the weights of the sensors with performance advantages in the current environment. Then, after obtaining the data to be fused of each sensor, according to the corrected weights of each sensor, that is, the target weights of each sensor, the data to be fused are fused to obtain the multi-sensor data fusion result. By fully combining the characteristics of different sensors, the reliability of the sensor perception data is improved.

[0074] In one embodiment, based on the Figure 1 embodiment shown, refer to Figure 2 , this embodiment relates to the process of how the computer device obtains the initial weights corresponding to each sensor. As Figure 2 shown, the computer device can obtain the initial weights corresponding to each sensor by performing the following steps S201, step S202, and step S203:

[0075] Step S201, obtain the historical perception data of each sensor.

[0076] Among them, the historical perception data of each sensor are the perceptions of each sensor at the same historical moment and in the same scenario. For example, the computer device can obtain the historical perception data of each sensor within a preset historical time period before the current moment. For each sampling moment in the preset historical time period, each sensor perceives the historical perception data.

[0077] Step S202, perform spatio-temporal synchronization processing on each historical perception data to obtain the synchronized data corresponding to each historical perception data.

[0078] After the computer device performs processing such as classification, coordinate extraction, and angle calculation on each historical perception data, and then performs spatio-temporal synchronization processing to the same target coordinate system, the synchronized data of each sensor are obtained.

[0079] Step S203: Calculate the variance of each synchronized data, and calculate the initial weight of the corresponding sensor according to the variance of each synchronized data.

[0080] In the embodiments of the present application, it is assumed that there are n sensors, and W1, W2...W n respectively represent the initial weights of the n sensors, then:

[0081] Then the total variance after fusion of each sensor is where δ i 2 represents the variance of the synchronized data of sensor i.

[0082] The computer device finds the minimum value of the total variance after fusion of each sensor, and then obtains:

[0083]

[0084] Formula 1 is the calculation formula for the initial weights corresponding to each sensor. After the computer device calculates the variances of the synchronized data of each sensor, substituting the variances corresponding to each sensor into Formula 1, the initial weights corresponding to each sensor can be obtained.

[0085] In a possible implementation manner, the computer device calculates the variances of the synchronized data of each sensor, which can be implemented by performing the following steps: for each sensor, group the synchronized data of the sensor to obtain multiple groups of data; calculate the variance of each group of data in the multiple groups of data of the sensor, and calculate the variance of the synchronized data of the sensor according to the variances of each group of data.

[0086] In the embodiments of the present application, the synchronized data of the sensor can be obtained by the computer device performing data processing on the historical perception data sensed by the sensor at multiple sampling moments within a historical time period; for example, if the historical time period is 3 minutes and the sensor senses data every 0.1 second, then the synchronized data of the sensor within the 3 minutes is obtained according to the historical perception data sensed at 1800 sampling moments, and the synchronized data corresponding to the sensor and the sampling moment can be obtained after processing the historical perception data at each sampling moment.

[0087] In the embodiments of the present application, the synchronized data at each sampling moment includes six types of information: category, horizontal coordinate, vertical coordinate, horizontal speed, vertical speed, and angle. Among them, for the information that a certain sensor cannot obtain among these six types of information, the computer device marks this type of information as "0" in the synchronized data corresponding to each sampling moment of the sensor.

[0088] As an implementation, the computer device can divide the synchronized data at multiple sampling moments with odd sampling times into one group according to the parity of the sampling times, and divide the synchronized data at multiple sampling moments with even sampling times into another group. Thus, two groups of data corresponding to each sensor are obtained. For example, the first group of data corresponding to sensor i is represented by Z i (1), Z i (3), Z i (5)... Z i (p), and the corresponding second group of data is Z i (2), Z i (4), Z i (6)... Z i (q), where p and q are both positive integers, p is odd, q is even, and the value of p + q is the total number of sampling times of sensor a.

[0089] The computer device calculates the arithmetic mean of the first group of data corresponding to sensor i and the arithmetic mean of the second group of data Then, the computer device calculates the mean square deviation δ of the first group of data using the following formula 2 i1 :

[0090]

[0091] The computer device calculates the mean square deviation δ of the second group of data using the following formula 3 i2 :

[0092]

[0093] Thus, the computer device obtains the variance δ i1 of the first group of data corresponding to sensor i and the mean square deviation δ i2 of the second group of data. Then, the variance of the synchronized data corresponding to sensor i

[0094] After the computer device calculates the variances of the synchronized data of each sensor, substituting the variances corresponding to each sensor into formula 1, the initial weights corresponding to each sensor are obtained.

[0095] In the embodiments of the present application, the computer device obtains the historical perception data of each sensor, and the historical perception data of each sensor is perceived by each sensor at the same historical moment and in the same scenario; performs spatio-temporal synchronization processing on each piece of historical perception data to obtain the synchronized data corresponding to each piece of historical perception data; calculates the variance of each piece of synchronized data, and calculates the initial weight of the corresponding sensor according to the variance of each piece of synchronized data. Thus, by setting the initial weight for each sensor and then using the target weight of each sensor to correct the initial weight of each sensor, it is convenient to fuse the data to be fused of each sensor, avoiding the situation where the perception data of only a single sensor may have a large error, and ensuring the reliability of the multi-sensor data fusion result.

[0096] In one embodiment, based on Figure 1 the embodiment shown, refer to Figure 3 , this embodiment relates to the process of how the computer device obtains the data to be fused of each sensor. As Figure 3 shown, step S300 of this embodiment includes step S301 and step S302:

[0097] Step S301, obtain multiple historical data to be fused of each sensor within a preset historical time period.

[0098] The historical data to be fused is obtained by the computer device through spatio-temporal synchronization processing of the perception data of the corresponding sensor within a preset historical time period. After the computer device classifies, extracts coordinates, calculates angles, etc. for the perception data sensed by each sensor at multiple sampling moments before the current moment, and then calibrates it to the same target coordinate system, multiple historical data to be fused of each sensor are obtained.

[0099] Step S302, calculate the average value of the multiple historical data to be fused corresponding to each sensor, and determine the average value corresponding to each sensor as the data to be fused of each sensor.

[0100] The computer device calculates the average value of the multiple historical data to be fused corresponding to each sensor, and determines the average value corresponding to each sensor as the target traffic data of each sensor.

[0101] In the embodiments of the present application, the computer device determines the average value corresponding to each sensor as the target traffic data of each sensor. In this way, even if each sensor does not currently sense data, for example, the sensor fails, the computer device can still obtain the data to be fused of each sensor according to the multiple historical data to be fused, so as to fuse the data to be fused corresponding to each sensor, ensure that the multi-sensor data fusion result can be normally output, and improve the reliability of the output data.

[0102] In one embodiment, based on Figure 1Based on the illustrated embodiments, refer to Figure 4 , this embodiment relates to the process of how a computer device obtains the data to be fused corresponding to each sensor through spatio-temporal synchronization. As Figure 4 shown, step S300 of this embodiment includes step S303 and step S304:

[0103] Step S303, obtain the perception data sensed by each sensor at the same moment and in the same scenario.

[0104] In the embodiments of the present application, the computer device obtains the perception data sensed by each sensor at the same moment and in the same scenario. This same moment can be the real-time acquisition moment. For example, the perception data sensed by a camera is an image, the perception data sensed by a lidar sensor is point cloud data, and the perception data sensed by a millimeter-wave radar sensor is electromagnetic waves. The computer device can perform processing such as classification, coordinate extraction, and angle calculation on the image, point cloud data, and electromagnetic waves respectively to obtain the target perception data of each sensor.

[0105] Step S304, adopt a preset spatio-temporal synchronization method to convert the perception data corresponding to each sensor to the target coordinate system to obtain the data to be fused corresponding to each sensor.

[0106] Since different sensors sense data based on different coordinate systems. For example, a camera senses data based on the camera coordinate system, and a millimeter-wave radar sensor senses data based on the millimeter-wave radar coordinate system. Therefore, the computer device uses a preset spatio-temporal synchronization method for the target perception data of the camera, the target perception data of the lidar sensor, and the target perception data of the millimeter-wave radar sensor. For example, it can be calibrated to the same target coordinate system using a calibration algorithm, so that the data of different sensors can be fused. The target coordinate system is such as the pixel coordinate system.

[0107] In a possible implementation manner, step S304 may include step a1 and step a2:

[0108] Step a1, obtain the initial rotational degree-of-freedom parameter and the initial translational degree-of-freedom parameter corresponding to each sensor.

[0109] Step a2, for each sensor, use the initial rotational degree-of-freedom parameter of the sensor, the initial translational degree-of-freedom parameter of the sensor, and a preset calibration algorithm to convert the target perception data corresponding to the sensor to the target coordinate system to obtain the data to be fused corresponding to each sensor.

[0110] In the embodiments of the present application, the initial rotational degree-of-freedom parameter and the initial translational degree-of-freedom parameter corresponding to each sensor can be set manually, and the computer device obtains this parameter.

[0111] Taking the target coordinate system as the pixel coordinate system as an example, the computer device can first convert the coordinates of the target perception data of each sensor to the reference coordinate system, which can be the world coordinate system, and then convert the coordinates of each sensor in the reference coordinate system to the pixel coordinate system.

[0112] As an implementation, the computer device can use the following formula 4 to convert the coordinates of the target perception data of each sensor to the reference coordinate system:

[0113]

[0114] Where, is the coordinate of any point in the target perception data of a sensor in the original coordinate system of the sensor, is the coordinate of the any point in the world coordinate system, R is the rotation matrix corresponding to the initial rotation degree of freedom parameter of the sensor, and t is the initial translation degree of freedom parameter of the sensor.

[0115] After the computer device converts the coordinates of the target perception data of each sensor to the reference coordinate system through formula 4, it then uses the following formula 5 to convert the coordinates of each sensor in the reference coordinate system to the pixel coordinate system:

[0116]

[0117] Where, (u, v) is the coordinate in the world coordinate system The pixel coordinates after being converted to the pixel coordinate system, the matrix with a size of 3*3 is the internal parameter matrix of the camera, and r, t are the initial rotation degree of freedom parameter and the initial translation degree of freedom parameter corresponding to each sensor.

[0118] As an implementation, the computer device can also use the distortion coefficient of the camera to correct the calibration result, so as to obtain the data to be fused corresponding to each sensor.

[0119] In a possible implementation, step S304 may include step a3, step a4, and step a5:

[0120] Step a3, obtain multiple calibrated data corresponding to at least two sensors and the target coordinate system during the historical calibration process, and obtain multiple historical rotation degree of freedom parameters and multiple historical translation degree of freedom parameters of each sensor.

[0121] In the embodiments of the present application, the historical calibration process may be a process in which the computer device converts the perception data sensed by the camera, lidar sensor, and millimeter wave radar sensor at a historical moment to the target coordinate system. The computer device obtains multiple calibrated data after conversion, and obtains multiple historical rotation degree of freedom parameters and multiple historical translation degree of freedom parameters of each sensor used in the calibration process.

[0122] Step a4: According to the multiple calibrated data of each sensor, select the target rotational degree-of-freedom parameter and the target historical translational degree-of-freedom parameter corresponding to each sensor from the multiple historical rotational degree-of-freedom parameters and the multiple historical translational degree-of-freedom parameters of each sensor.

[0123] In the embodiment of the present application, the computer device obtains the position information of the fusion area of each sensor. The fusion area may be the overlapping part of the coverage areas of each sensor. For the multiple calibrated data of the lidar sensor within the fusion area, the computer device calculates the minimum bounding rectangle of the point coordinates of each target in the multiple calibrated data, and then calculates the overlapping area between the minimum bounding rectangle of each target and the target box of the image recognition result of the target, and calculates the ratio Rl of the overlapping area of each target to the target box of the image recognition result. Finally, calculate the average value Ml of the ratios Rl of all targets. For the millimeter-wave radar sensor, the computer device draws a rectangle with the target pixel of each target in the multiple calibrated data as the center and p as the radius. p can be set according to the category of the target during implementation. The computer device calculates the overlapping area between the rectangle of each target and the target box of the image recognition result, and then calculates the ratio Rr of the overlapping area to the rectangle. Finally, calculate the average value Mr of the ratios Rr of all targets.

[0124] The computer device determines the maximum values of Ml and Mr, and determines the historical rotational degree-of-freedom parameter and the historical translational degree-of-freedom parameter corresponding to the maximum values of Ml and Mr as the target rotational degree-of-freedom parameter and the target historical translational degree-of-freedom parameter.

[0125] Step a5: For each sensor, use the target rotational degree-of-freedom parameter of the sensor, the target historical translational degree-of-freedom parameter of the sensor, and a preset calibration algorithm to convert the target sensing data corresponding to the sensor to the target coordinate system, and obtain the data to be fused corresponding to each sensor.

[0126] The computer device uses the target rotational degree-of-freedom parameter, the target historical translational degree-of-freedom parameter of the sensor, and a preset calibration algorithm, such as Zhang's calibration algorithm, to convert the target sensing data corresponding to each sensor to the target coordinate system, and obtain the data to be fused corresponding to each sensor.

[0127] Thus, by selecting the target rotational degree-of-freedom parameter and the target historical translational degree-of-freedom parameter with the best calibration effect from the multiple historical rotational degree-of-freedom parameters and the multiple historical translational degree-of-freedom parameters of each sensor, and using the target rotational degree-of-freedom parameter and the target historical translational degree-of-freedom parameter to convert the target sensing data corresponding to each sensor to the target coordinate system, the data accuracy of the data to be fused by the sensor is improved, and the data reliability is improved.

[0128] It should be understood that although Figures 1-4 each step in the flowchart is shown in sequence according to the indication of the arrow, these steps are not necessarily executed in sequence according to the order indicated by the arrow. Unless there is a clear description in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 1-4 at least a part of the steps in

[0129] In one embodiment, as Figure 5 shown, a multi-sensor data fusion device is provided, including:

[0130] A first acquisition module 10, configured to acquire the current environmental data of the multi-sensor system, and acquire a weight correction value of each sensor according to the environmental data;

[0131] A correction module 20, configured to acquire the initial weight corresponding to each sensor, and use the weight correction value of each sensor to correct the initial weight of each sensor correspondingly to obtain the target weight of each sensor;

[0132] A second acquisition module 30, configured to acquire the data to be fused of each sensor, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensor, and the perception data of each sensor is perceived by each sensor at the same moment and in the same scenario;

[0133] A processing module 40, configured to perform fusion processing on the data to be fused of each sensor according to the target weight of each sensor to obtain a multi-sensor data fusion result.

[0134] In one embodiment, the processing module 40 is specifically configured to use the target weight of each sensor to perform weighted summation calculation on the data to be fused of each sensor to obtain the multi-sensor data fusion result.

[0135] In one embodiment, when the correction module 20 acquires the initial weight corresponding to each sensor, it is specifically configured to acquire the historical perception data of each sensor, where the historical perception data of each sensor is perceived by each sensor at the same historical moment and in the same scenario; perform spatio-temporal synchronization processing on each historical perception data to obtain the synchronized data corresponding to each historical perception data; calculate the variance of each synchronized data, and calculate the initial weight of the corresponding sensor according to the variance of each synchronized data.

[0136] In one embodiment, when calculating the variance of each piece of synchronized data, the correction module 30 is specifically configured to, for each sensor, group the synchronized data of the sensor to obtain multiple groups of data; calculate the variance of each group of data in the multiple groups of data of the sensor, and calculate the variance of the synchronized data of the sensor according to the variance of each group of data.

[0137] In one embodiment, based on the above Figure 5 illustrated embodiment, as Figure 6 illustrated, the second acquisition module 30 includes:

[0138] A first acquisition unit 301, configured to acquire multiple historical data to be fused of each sensor within a preset historical time period, where the historical data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensor within the preset historical time period;

[0139] A calculation unit 302, configured to calculate the average value of the multiple historical data to be fused corresponding to each sensor, and determine the average value corresponding to each sensor as the data to be fused of each sensor.

[0140] In one embodiment, based on the above Figure 5 illustrated embodiment, as Figure 7 illustrated, the second acquisition module 30 includes:

[0141] A second acquisition unit 303, configured to acquire the sensed data sensed by each sensor at the same time and in the same scenario;

[0142] A spatio-temporal synchronization unit 304, configured to use a preset spatio-temporal synchronization method to convert the sensed data corresponding to each sensor to a target coordinate system to obtain the data to be fused corresponding to each sensor.

[0143] In one embodiment, the sensor includes a lidar sensor, a camera, or a millimeter-wave radar sensor.

[0144] In one embodiment, the first acquisition module 10 is specifically configured to acquire the current environmental data of the multi-sensor system through an auxiliary sensor, where the auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor.

[0145] For the specific limitations of the multi-sensor data fusion device, reference can be made to the limitations of the multi-sensor data fusion method in the above text, which will not be elaborated here. Each module in the above multi-sensor data fusion device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0146] In one embodiment, a multi-sensor system is provided. The system includes a processor, an auxiliary sensor, and a sensor. The sensor and the auxiliary sensor are connected to the processor. The sensor includes at least two of a lidar sensor, a camera, and a millimeter-wave radar sensor. The auxiliary sensor includes a high-precision optoelectronic sensor and / or a temperature and humidity sensor.

[0147] The auxiliary sensor is configured to obtain the current environmental data of the multi-sensor system and send the environmental data to the processor.

[0148] The processor is configured to receive the environmental data, obtain the weight correction value of each sensor according to the environmental data, and obtain the initial weight of each sensor, and use the weight correction value of each sensor to correspondingly correct the initial weight of each sensor to obtain the target weight of each sensor.

[0149] The processor is further configured to obtain the data to be fused of each sensor, and perform fusion processing on the data to be fused of each sensor according to the target weight of each sensor to obtain a multi-sensor data fusion result. Wherein, the data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensor, and the perception data of each sensor is perceived by each sensor at the same time and in the same scenario.

[0150] In one embodiment, the processor is further configured to execute the steps of the method in any of the above embodiments.

[0151] For the specific limitations of the processor of the multi-sensor system, reference can be made to the limitations of the multi-sensor data fusion method in the above embodiments, which will not be elaborated here.

[0152] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 8As shown in the figure. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of the multi-sensor data fusion method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a multi-sensor data fusion method.

[0153] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0154] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0155] Obtain the current environmental data of the multi-sensor system, and obtain the weight correction value of each sensor according to the environmental data;

[0156] Obtain the initial weight corresponding to each sensor, and use the weight correction value of each sensor to correct the initial weight of each sensor correspondingly to obtain the target weight of each sensor;

[0157] Obtain the data to be fused of each sensor, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensor, and the sensed data of each sensor is sensed by each sensor at the same time and in the same scenario;

[0158] Fuse the data to be fused of each sensor according to the target weight of each sensor to obtain the multi-sensor data fusion result.

[0159] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0160] Use the target weight of each sensor to perform weighted summation calculation on the data to be fused of each sensor to obtain the multi-sensor data fusion result.

[0161] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0162] Obtain the historical perception data of each of the sensors, where the historical perception data of each of the sensors is the data sensed by each of the sensors at the same historical moment and in the same scenario;

[0163] Perform spatio-temporal synchronization processing on each of the historical perception data to obtain the synchronized data corresponding to each of the historical perception data;

[0164] Calculate the variance of each of the synchronized data, and calculate the initial weight of the corresponding sensor according to the variance of each of the synchronized data.

[0165] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0166] For each of the sensors, group the synchronized data of the sensor to obtain multiple groups of data;

[0167] Calculate the variance of each group of data in the multiple groups of data of the sensor, and calculate the variance of the synchronized data of the sensor according to the variance of each group of data.

[0168] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0169] Obtain multiple historical data to be fused of each of the sensors within a preset historical time period, where the historical data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensor within the preset historical time period;

[0170] Calculate the average value of the multiple historical data to be fused corresponding to each of the sensors, and determine the average value corresponding to each of the sensors as the data to be fused of each of the sensors.

[0171] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0172] Obtain the perception data sensed by each of the sensors at the same moment and in the same scenario;

[0173] Adopt a preset spatio-temporal synchronization method to convert the perception data corresponding to each of the sensors to the target coordinate system to obtain the data to be fused corresponding to each of the sensors.

[0174] In one embodiment, the sensor includes a lidar sensor, a camera, or a millimeter wave radar sensor.

[0175] In one embodiment, when the processor executes the computer program, the following steps are further implemented:

[0176] Obtain the current environmental data of the multi-sensor system through an auxiliary sensor, where the auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0178] Obtain the current environmental data of the multi-sensor system, and obtain the weight correction value of each sensor according to the environmental data;

[0179] Obtain the initial weight corresponding to each sensor, and use the weight correction value of each sensor to correspondingly correct the initial weight of each sensor to obtain the target weight of each sensor;

[0180] Obtain the data to be fused of each sensor, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensing data of the corresponding sensor, and the sensing data of each sensor is sensed by each sensor at the same time and in the same scenario;

[0181] Perform fusion processing on the data to be fused of each sensor according to the target weight of each sensor to obtain the multi-sensor data fusion result.

[0182] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0183] Use the target weight of each sensor to perform weighted summation calculation on the data to be fused of each sensor to obtain the multi-sensor data fusion result.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0185] Obtain the historical sensing data of each sensor, and the historical sensing data of each sensor is sensed by each sensor at the same historical time and in the same scenario;

[0186] Perform spatio-temporal synchronization processing on each historical sensing data to obtain the synchronized data corresponding to each historical sensing data;

[0187] Calculate the variance of each synchronized data, and calculate the initial weight of the corresponding sensor according to the variance of each synchronized data.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0189] For each sensor, group the synchronized data of the sensor to obtain multiple groups of data;

[0190] Calculate the variance of each group of data in the multiple groups of data of the sensor, and calculate the variance of the synchronized data of the sensor according to the variance of each group of data.

[0191] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0192] Obtain a plurality of historical data to be fused of each of the sensors within a preset historical time period, where the historical data to be fused is obtained by performing spatio-temporal synchronization processing on the sensed data of the corresponding sensor within the preset historical time period;

[0193] Calculate the average value of the plurality of historical data to be fused corresponding to each sensor, and determine the average value corresponding to each sensor as the data to be fused of each sensor.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0195] Obtain the sensed data sensed by each of the sensors at the same time and in the same scenario;

[0196] Adopt a preset spatio-temporal synchronization method to convert the sensed data corresponding to each sensor to a target coordinate system, and obtain the data to be fused corresponding to each sensor.

[0197] In one embodiment, the sensor includes a lidar sensor, a camera, or a millimeter wave radar sensor.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0199] Obtain the current environmental data of the multi-sensor system through an auxiliary sensor, where the auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor.

[0200] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0201] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0202] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-sensor data fusion method, which is applied to a multi-sensor system including multiple sensors, and is characterized in that, The method includes: Obtaining the current environmental data of the multi-sensor system, and obtaining the weight correction value of each sensor according to the environmental data; Obtaining the initial weight corresponding to each sensor, and using the weight correction value of each sensor to correspondingly correct the initial weight of each sensor to obtain the target weight of each sensor; The obtaining the initial weight corresponding to each sensor includes: Performing data processing on the historical sensing data sensed by each sensor at multiple sampling moments within a historical time period to obtain the synchronized data corresponding to each historical sensing data; Dividing the synchronized data at multiple sampling moments with an odd sampling number into a first group of data, dividing the synchronized data at multiple sampling moments with an even sampling number into a second group of data, calculating the mean square error of the first group of data, and calculating the mean square error of the second group of data; According to the formula the variance of each of the synchronized data is obtained, where is the mean square error of the first group of data, is the mean square error of the second group of data; According to the formula determine the initial weights corresponding to each of the sensors, where represents the initial weight corresponding to sensor i, represents the variance of the data after synchronization of sensor i, and n represents n sensors; Obtaining the data to be fused for each sensor, where the data to be fused is obtained by performing spatio-temporal synchronization processing on the sensing data of the corresponding sensor, and the sensing data of each sensor is sensed by each sensor at the same moment and in the same scenario; Performing fusion processing on the data to be fused for each sensor according to the target weight of each sensor to obtain a multi-sensor data fusion result.

2. The method according to claim 1, characterized in that, The performing fusion processing on the data to be fused for each sensor according to the target weight of each sensor to obtain a multi-sensor data fusion result includes: Using the target weight of each sensor to perform weighted summation calculation on the data to be fused for each sensor to obtain the multi-sensor data fusion result.

3. The method according to claim 1, characterized in that, The obtaining the data to be fused for each sensor includes: Obtaining multiple historical data to be fused for each sensor within a preset historical time period, where the historical data to be fused is obtained by performing spatio-temporal synchronization processing on the sensing data of the corresponding sensor within the preset historical time period; Calculating the average value of the multiple historical data to be fused corresponding to each sensor, and determining the average value corresponding to each sensor as the data to be fused for each sensor.

4. The method according to claim 1, wherein The obtaining the data to be fused for each sensor includes: Obtaining the sensing data sensed by each sensor at the same moment and in the same scenario; Adopting a preset spatio-temporal synchronization method to convert the sensing data corresponding to each sensor to a target coordinate system to obtain the data to be fused corresponding to each sensor.

5. The method according to any one of claims 1-4, characterized in that The sensor includes a lidar sensor, a camera, or a millimeter-wave radar sensor.

6. The method according to any one of claims 1 to 4, characterized in that The obtaining the current environmental data of the multi-sensor system includes: Obtaining the current environmental data of the multi-sensor system through an auxiliary sensor, where the auxiliary sensor includes at least one of a high-precision optoelectronic sensor and a temperature and humidity sensor.

7. A multi-sensor data fusion device, characterized in that Applied to a multi-sensor system including multiple sensors, the device includes: A first obtaining module, configured to obtain the current environmental data of the multi-sensor system, and obtain the weight correction value of each sensor according to the environmental data; A correction module, configured to obtain the initial weights corresponding to the sensors, and use the weight correction values of the sensors to correct the initial weights of the sensors respectively, so as to obtain the target weights of the sensors; the obtaining the initial weights corresponding to the sensors includes: Performing data processing on the historical perception data sensed by the sensors at multiple sampling moments within a historical time period to obtain the synchronized data corresponding to the historical perception data; Dividing the synchronized data at multiple sampling moments with an odd sampling number into a first group of data, dividing the synchronized data at multiple sampling moments with an even sampling number into a second group of data, calculating the mean square error of the first group of data, and calculating the mean square error of the second group of data; According to the formula the variances of the synchronized data are obtained, where is the mean square error of the first group of data, is the mean square error of the second group of data; According to the formula determine the initial weights corresponding to each of the said sensors, where represents the initial weight corresponding to sensor i, represents the variance of the data after synchronization of sensor i, and n represents n sensors; A second acquisition module, configured to acquire the data to be fused of the sensors, wherein the data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensors, and the perception data of the sensors are the data sensed by the sensors at the same moment and in the same scenario; A processing module, configured to perform fusion processing on the data to be fused of the sensors according to the target weights of the sensors respectively to obtain a multi-sensor data fusion result.

8. A multi-sensing system, characterized in that, The system includes a processor, auxiliary sensors and sensors, the sensors and the auxiliary sensors are connected to the processor, the sensors include at least two of lidar sensors, cameras and millimeter wave radar sensors, and the auxiliary sensors include high-precision optoelectronic sensors and / or temperature and humidity sensors; The auxiliary sensors are configured to acquire the current environmental data of the multi-sensor system and send the environmental data to the processor; The processor is configured to receive the environmental data, obtain the weight correction values of the sensors according to the environmental data, and obtain the initial weights of the sensors, and use the weight correction values of the sensors to correct the initial weights of the sensors respectively to obtain the target weights of the sensors; The obtaining the initial weights corresponding to the sensors includes: Performing data processing on the historical perception data sensed by the sensors at multiple sampling moments within a historical time period to obtain the synchronized data corresponding to the historical perception data; Dividing the synchronized data at multiple sampling moments with an odd sampling number into a first group of data, dividing the synchronized data at multiple sampling moments with an even sampling number into a second group of data, calculating the mean square error of the first group of data, and calculating the mean square error of the second group of data; According to the formula the variance of each of the synchronized data is obtained, where is the mean square error of the first group of data, is the mean square error of the second group of data; According to the formula determine the initial weights corresponding to each of the sensors, where represents the initial weight corresponding to sensor i, represents the variance of the data after synchronization of sensor i, and n represents n sensors; The processor is further configured to acquire the data to be fused of the sensors and perform fusion processing on the data to be fused of the sensors according to the target weights of the sensors respectively to obtain a multi-sensor data fusion result, wherein the data to be fused is obtained by performing spatio-temporal synchronization processing on the perception data of the corresponding sensors, and the perception data of the sensors are the data sensed by the sensors at the same moment and in the same scenario.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images