Multi-dimensional pixel fusion method for environmental perception and storage medium

Abstract

The invention relates to the technical field of driving assistance, and provides a multi-dimensional pixel fusion method for environmental perception and a storage medium, and the method comprises the steps: carrying out the coordinate unification and time sequence alignment of the detection data of all sensors based on the different types of obstacle data collected by different sensors, the image and the radar data are subjected to pixel-level real-time'time-space alignment and synchronization 'and are output in a'multi-dimensional pixel' format through splicing and fusion. And multi-mode accurate sensing information of the target and the environment is provided for an automatic driving system, including comprehensive sensing of visual data of the sensor for sensing the target and the environment, 4D millimeter-wave radar data and infrared radiation data, so that accurate recognition of target obstacles in various driving scenes is realized, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of driving assistance technology, and in particular to a multi-dimensional pixel fusion method and storage medium for environment perception. Background Art

[0002] The Advanced Driving Assistance System (ADAS) uses a variety of sensors installed on the car (millimeter wave radar, laser radar, single / binocular cameras and satellite navigation) to sense the surrounding environment at any time during the driving process of the car, collect data, identify, detect and track static and dynamic objects, and combine navigation map data to perform systematic calculations and analysis, so as to allow drivers to be aware of possible dangers in advance, effectively increasing the comfort and safety of car driving. In recent years, the ADAS market has grown rapidly. Originally, this type of system was limited to the high-end market, but now it is entering the mid-range market. At the same time, many low-tech applications are more common in the entry-level passenger car field. Improved new sensor technologies are also creating new opportunities and strategies for system deployment.

[0003] Therefore, the perception system of an autonomous vehicle needs to accurately identify and understand the environment around the vehicle. Currently, the perception system of a single sensor has limitations, for example: cameras have high resolution but are greatly affected by lighting, radars can penetrate bad weather but have low resolution, and infrared cameras are suitable for nighttime use but have limited ability to capture details. Summary of the invention

[0004] The present invention provides a multi-dimensional pixel fusion method and storage medium for environmental perception, which solves the technical problem that the existing single-sensor perception system cannot accurately identify and understand the environment around the vehicle.

[0005] In order to solve the above technical problems, the present invention provides a multi-dimensional pixel fusion method for environment perception, comprising:

[0006] Based on the frame synchronization signal and the hard-line trigger signal, the sensor is driven to collect environmental information to obtain time-aligned raw environmental data, wherein the sensor includes a visual sensor, an infrared sensor and a millimeter-wave radar;

[0007] Preprocessing the raw environmental data, and performing feature extraction based on the processed raw environmental data to obtain corresponding visual features, infrared features, and dynamic features;

[0008] Perform spatial coordinate alignment processing to obtain the FOV overlap area of ​​spatial registration;

[0009] A weight matrix is ​​constructed, and then the visual features, infrared features and dynamic features of the overlapping area of ​​the FOV are fused to obtain multi-dimensional perception data.

[0010] In a further embodiment, the collecting of environmental information based on the frame synchronization signal and the hard-line trigger signal to obtain timing-aligned raw environmental data includes:

[0011] At the same time, control signals are sent to the visual deserializer, infrared deserializer, and millimeter wave radar;

[0012] Performing exposure control on the visual sensor based on the frame synchronization signal, acquiring visual data from the visual camera serializer, and deserializing the visual data through the visual deserializer to obtain the visual data;

[0013] Exposure control is performed on the infrared sensor based on the frame synchronization signal, infrared data is obtained from the infrared camera serializer, and the infrared data is deserialized through the infrared deserializer to obtain infrared data;

[0014] Drive the millimeter-wave radar to collect data based on hard-wire trigger signals to obtain point cloud data;

[0015] The original environment data includes visual data, infrared data and point cloud data.

[0016] In a further embodiment, the feature extraction based on the processed raw environmental data comprises:

[0017] Based on the deep learning model, visual features are extracted from the visual data; based on the deep learning model, infrared features are extracted from the infrared data; based on the optical flow method or the key point detection algorithm, dynamic features are extracted from the point cloud data.

[0018] In a further embodiment, the pair is subjected to spatial coordinate alignment processing to obtain a spatially registered FOV field of view overlap region, comprising:

[0019] Take the center point of the FOV of the millimeter-wave radar as the origin of the coordinate system and establish a plane coordinate system;

[0020] Calculate the coordinate offset values ​​of the center point of the visual FOV, the center point of the infrared FOV and the origin of the coordinate system;

[0021] Taking the plane coordinate system as a reference, the coordinate offset value and the sensor resolution are obtained, and the first coordinate group, the second coordinate group, and the third coordinate group corresponding to the radar field of view FOV range, the visual FOV range, and the infrared FOV range are obtained respectively;

[0022] Based on the first coordinate group, the second coordinate group, and the third coordinate group, a fourth coordinate group corresponding to the FOV overlapping area is calculated so that the coordinate space of the pixel points in the FOV overlapping area is aligned; the FOV overlapping area is the overlapping area of ​​the radar FOV range, the visual FOV range, and the infrared FOV range.

[0023] In a further implementation scheme, assuming that the resolution of the millimeter wave radar is U*V, the resolution of the visual sensor is M*N, and the resolution of the infrared sensor is R*T, then:

[0024] The fourth coordinate group of the FOV overlapping area is as follows:

[0025]

[0026] Where Axy, Bxy, Cxy, and Dxy are the coordinates of the four vertices of the FOV overlapping area; (m, n) is the coordinate offset between the center point of the visual FOV and the origin of the coordinate system; (p, q) is the coordinate offset between the center point of the infrared FOV and the origin of the coordinate system.

[0027] In a further embodiment, constructing a weight matrix comprises:

[0028] Obtain obstacle data in the overlapping area of ​​the FOV field of view and establish a probability distribution model;

[0029] According to the probability distribution estimated by the probability distribution model, the information entropy of the visible light camera, 4D millimeter wave radar and infrared camera are calculated respectively;

[0030] Performing normalization processing to convert the information entropy into a confidence weight coefficient;

[0031] All the confidence weight coefficients corresponding to the visual sensor, millimeter wave radar, and infrared sensor are integrated to obtain a weight matrix.

[0032] In a further embodiment, the information entropy is calculated as follows:

[0033]

[0034] Where H(X) is the information entropy of random variable X, p(x i ) is the probability of X taking the i-th value, and n is the total number of possible values;

[0035] The calculation formula of the confidence weight coefficient is as follows:

[0036]

[0037] In the formula, w i is the weight coefficient of the i-th sensor, H(Xi ) is the information entropy of the ith sensor, ∈ is a positive number.

[0038] In a further embodiment, the feature fusion of the visual features, infrared features and dynamic features of the FOV overlapping area to obtain multi-dimensional perception data includes:

[0039] Based on the FOV overlapping area, respectively obtain the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix corresponding to the visual features, infrared features, and dynamic features;

[0040] Perform feature splicing processing on the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix pixels to obtain a multi-dimensional pixel matrix;

[0041] Performing multidimensional pixel fusion according to the multidimensional pixel matrix and the weight matrix to obtain multidimensional perception data;

[0042] The obstacle data includes: the color channel values ​​(R, G, B) of the object image collected by the visual sensor, the distance L, speed S, direction R and height signal h of the object collected by the millimeter wave radar, and the thermal radiation H of the object collected by the infrared sensor.

[0043] In a further embodiment, the multidimensional pixel matrix is ​​as follows:

[0044]

[0045] The fusion formula of the multi-dimensional perception data is as follows:

[0046]

[0047] in,

[0048]

[0049] Where W is the weight matrix, are the confidence weight coefficients of the visual sensor, millimeter wave radar, and infrared sensor respectively; P Matrix_A , P Matrix_B , P Matrix_C They respectively represent the obstacle attribute characteristics detected by the visual sensor, the obstacle attribute characteristics detected by the millimeter wave radar, and the obstacle attribute characteristics detected by the infrared sensor.

[0050] The present invention also provides a storage medium on which a computer program is stored, wherein the computer program is used to implement the multi-dimensional pixel fusion method for environmental perception as described above. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0051] The beneficial effects of the present invention are as follows:

[0052] (1) Control signals are sent to the visual deserializer, infrared deserializer, and millimeter-wave radar at the same time. Frame synchronization signals and hard-wired trigger signals are used to drive the visual sensor, infrared sensor, and millimeter-wave radar to collect data at the same time. Data is collected at the same time to achieve timing alignment between the sensors, providing a fusion basis for multi-dimensional data fusion.

[0053] (2) Based on the particularity of the FOV center, the coordinate offset values ​​of the visual FOV center point, the infrared FOV center point and the origin of the coordinate system (the FOV center point of the millimeter wave radar) are calculated to determine the offset range of the entire field of view. Then, based on the coordinate offset values, the positions of the visual FOV center point and the infrared FOV center point relative to the plane coordinate system are determined. By fusing the data collected by the three sensors in the same coordinate system, spatial alignment is achieved. This allows sensors of different types and different FOV fields of view to find a multi-dimensional pixel field of view overlap area through software algorithms, and the coordinate space within the area is aligned.

[0054] (3) Based on the different characteristics of obstacle data types collected by different sensors, a probability distribution model is established according to real-time data, and then the information entropy is calculated and converted into a confidence weight coefficient. The confidence weight coefficient of each sensor is updated in real time to ensure the actual adaptability of the obstacle recognition coefficient and improve the recognition accuracy.

[0055] (4) Based on the different types of obstacle data collected by different sensors, the detection data of each sensor is "coordinate-unified and time-series-aligned", and the image and radar data are "synchronized in time and space" at the pixel level in real time and output in a "multi-dimensional pixel" format through splicing and fusion. Provide the autonomous driving system with multi-modal accurate perception information of the target and the environment: including the comprehensive perception of the sensor's visual data (brightness, texture, color, etc.) of the target and the environment, 4D millimeter-wave radar data (target distance, speed, direction, height, etc.) and infrared radiation data (texture, temperature, etc.), so as to achieve accurate identification of target obstacles in various driving scenarios and improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1is a workflow diagram of a multi-dimensional pixel fusion method for environment perception provided by an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the spatial mapping relationship of the detection domain of the multi-sensor combination provided by an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of synchronous acquisition of FPGA provided by an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of the sensor installation layout provided by an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of sensor coordinate alignment provided by an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram of feature splicing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0063] Example 1

[0064] The embodiment of the present invention provides a multi-dimensional pixel fusion method for environment perception, such as Figure 1 to Figure 6 As shown, in this embodiment, it includes:

[0065] S1. Drive the sensor to collect environmental information based on the frame synchronization signal and hard-wire trigger signal to obtain the original environmental data with time alignment, including:

[0066] At the same time, control signals are sent to the visual deserializer, infrared deserializer, and millimeter wave radar;

[0067] Performing exposure control on the visual sensor based on the frame synchronization signal, acquiring visual data from the visual camera serializer, and deserializing the visual data through the visual deserializer to obtain the visual data;

[0068] Exposure control is performed on the infrared sensor based on the frame synchronization signal, infrared data is obtained from the infrared camera serializer, and the infrared data is deserialized through the infrared deserializer to obtain infrared data;

[0069] Drive the millimeter-wave radar to collect data based on hard-wire trigger signals to obtain point cloud data;

[0070] The sensors include visual sensors (such as a front-view 8M camera), infrared sensors (i.e., infrared cameras) and millimeter-wave radars; the raw environmental data include visual data, infrared data and point cloud data.

[0071] In this embodiment, 4D millimeter-wave radar is preferably used for point cloud data collection. The 4D millimeter-wave radar collects millimeter-wave radar data through a hard-line trigger signal sent by the SOC, and inputs the point cloud data to the SOC through Ethernet for further data processing.

[0072] In this embodiment, the time synchronization of each sensor uses hard wires and image frame exposure control to align the timing of each sensor. Figure 3 , the specific steps are:

[0073] When the system is started, the exposure control center of the FPGA (field programmable gate array) sends a control signal to the deserializer 1 (i.e., the visual deserializer) through Ctrl0. The deserializer 1 sends the control signal to the exposure control pin of the sensor 1 (i.e., the visual sensor) through the GMSL link reverse channel for exposure control.

[0074] Similarly, the exposure control center of the FPGA sends the control signal from Ctrl1 to the deserializer 2 (i.e., the infrared deserializer), and the deserializer 2 sends the control signal to the exposure control pin of the sensor 2 through the GMSL link reverse channel for exposure control.

[0075] The time synchronization of the 4D millimeter-wave radar adopts hard-wire control, and the FPGA control center controls the radar data acquisition through the Ctrl2 control signal.

[0076] S2, preprocessing the original environmental data, and performing feature extraction based on the processed original environmental data to obtain corresponding visual features, infrared features and dynamic features;

[0077] Specifically, based on the deep learning model, visual features are extracted from the visual data; based on the deep learning model, infrared features are extracted from the infrared data; based on the optical flow method or the key point detection algorithm, dynamic features are extracted from the point cloud data. In the specific implementation process, the feature extraction algorithm can be selected according to actual needs, and this embodiment does not limit it.

[0078] S3, performing spatial coordinate alignment processing to obtain the FOV overlap area of ​​spatial registration, including:

[0079] S31, taking the center point of the FOV of the millimeter wave radar as the origin of the coordinate system, establishing a plane coordinate system (i.e., O1-XY plane);

[0080] S32, calculating the coordinate offset values ​​of the center point of the visual FOV, the center point of the infrared FOV and the origin of the coordinate system;

[0081] S33, taking the plane coordinate system as a reference, obtaining the coordinate offset value and the sensor resolution, and respectively obtaining a first coordinate group, a second coordinate group, and a third coordinate group corresponding to the radar field of view FOV range, the visual FOV range, and the infrared FOV range;

[0082] Coordinate alignment in the prior art is usually based on a spatial coordinate system, see Figure 4 , O1 is the origin of FOV1 of 4D millimeter wave radar; O2 is the center point of FOV2 of visible light camera; O3 is the center point of FOV3 of infrared camera. The multi-dimensional pixel sensor pixel coordinate system O1-XYZ is established with O1 as the coordinate origin.

[0083] Among them, the coordinate offset of the center point O2 of the visible light camera's FOV2 from O1 is (m, n, 0); the coordinate offset of the center point O2 of the infrared camera's FOV3 from O1 is (p, q, 0).

[0084] In this regard, in this embodiment, assuming that the resolution of the millimeter wave radar is U*V, the resolution of the visual sensor is M*N, and the resolution of the infrared sensor is R*T, then:

[0085] See also Figure 5 , taking the FOV coordinates of the 4D millimeter wave radar as a reference, the first coordinate group of the FOV range of the 4D millimeter wave radar is P1 on the O1-XY plane. xy , the second coordinate group of the visual sensor FOV range is P2 xy , the third coordinate group of the infrared sensor FOV range is P3 xy , from which we can get:

[0086]

[0087]

[0088] Among them, Pi xy (x, y are integers) are the pixel coordinates of the i-th sensor in the O1-XY plane and satisfy the following relationship:

[0089] x≤max{U,M,R}

[0090] y≤max{V,N,T}

[0091] S34, based on the first coordinate group, the second coordinate group, and the third coordinate group, calculate a fourth coordinate group corresponding to the FOV overlap area, so that the coordinate space of the pixel points in the FOV overlap area is aligned; the FOV overlap area is the overlap area of ​​the radar FOV range, the visual FOV range, and the infrared FOV range (such as Figure 5The rectangular pixel area enclosed by ABCD in the figure).

[0092] In this embodiment, the fourth coordinate group of the FOV overlapping area is as follows:

[0093]

[0094] Where Axy, Bxy, Cxy, and Dxy are the coordinates of the four vertices of the FOV overlapping area; (m, n) is the coordinate offset between the center point of the visual FOV and the origin of the coordinate system; (p, q) is the coordinate offset between the center point of the infrared FOV and the origin of the coordinate system.

[0095] In this embodiment, steps S1 and S3 synchronize and align the original environmental data collected by the visual sensor, infrared sensor and millimeter wave radar in time and space to ensure the consistency of the data.

[0096] S4. Construct a weight matrix, and then perform feature fusion on the visual features, infrared features, and dynamic features of the overlapping area of ​​the FOV to obtain multi-dimensional perception data.

[0097] In this embodiment, constructing a weight matrix includes:

[0098] A1. Obtain obstacle data in the overlapping area of ​​the FOV field of view and establish a probability distribution model;

[0099] Define data model: Define a unified data model to represent data from different sensors. For example, Bayesian estimation can be used to estimate the probability density function of the data.

[0100] For example, if you are dealing with image data, each pixel can be represented as a multi-dimensional vector containing color information and confidence levels obtained from different sensors.

[0101] A2. Calculate the information entropy of the visible light camera, 4D millimeter wave radar and infrared camera respectively according to the probability distribution estimated by the probability distribution model;

[0102] The calculation formula of the information entropy is as follows:

[0103]

[0104] Where H(X) is the information entropy of random variable X, p(x i ) is the probability of X taking the i-th value, and n is the total number of possible values.

[0105] A3, performing normalization processing to convert the information entropy into a confidence weight coefficient;

[0106] The calculation formula of the confidence weight coefficient is as follows:

[0107]

[0108] In the formula, w i is the weight coefficient of the i-th sensor, H(X i ) is the information entropy of the i-th sensor, and ∈ is a positive number. Among them, ∈ is a very small positive number to ensure numerical stability.

[0109] Normalize the credibility of each sensor to the same scale for easy comparison and fusion. This can be achieved by selecting maximum normalization or other appropriate methods as needed.

[0110] A4. Integrate all the confidence weight coefficients corresponding to the visual sensor, millimeter wave radar, and infrared sensor to obtain a weight matrix.

[0111] Assume that the confidence weight coefficients of the visual sensor, millimeter wave radar, and infrared sensor are α, β, and γ respectively. Then the weight matrix is:

[0112]

[0113] in,

[0114]

[0115] In the formula, They are the confidence weights of visual sensors, millimeter-wave radars, and infrared sensors respectively.

[0116] In this embodiment, the obstacle data includes: the color channel values ​​(R, G, B) of the object image collected by the visual sensor, the distance L, speed S, direction R and height signal h of the object collected by the millimeter wave radar, and the thermal radiation H of the object collected by the infrared sensor. In other embodiments, it can also be determined that the visual sensor collects the image signal in YUV or RAW format according to actual needs.

[0117] In this embodiment, the feature fusion of the visual features, infrared features and dynamic features of the FOV overlapping area to obtain multi-dimensional perception data includes:

[0118] B1. Based on the FOV overlapping area, the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix corresponding to the visual features, infrared features, and dynamic features are obtained respectively, as follows;

[0119]

[0120]

[0121] Where P RGBRepresents the pixel matrix of obstacles detected by the visible light camera; P hLSR P is the point cloud pixel matrix of obstacles detected by 4D millimeter wave radar; H The thermal radiation pixel matrix of the obstacle detected by the infrared camera.

[0122] B2. Perform feature stitching processing on the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix pixels to obtain a multi-dimensional pixel matrix.

[0123] See also Figure 6 , for each pixel or point, the fusion is performed according to its attributes and corresponding weights in different sensor data. If the data is an image, the attributes such as color and depth of each pixel can be weighted averaged. The multidimensional pixel matrix corresponding to the overlapping field of view area ABCD is as follows:

[0124]

[0125] B3. Perform multidimensional pixel fusion according to the multidimensional pixel matrix and the weight matrix to obtain multidimensional perception data;

[0126] The fusion formula of the multi-dimensional perception data is as follows:

[0127]

[0128] in,

[0129]

[0130] Where P Matrix_A , P Matrix_B , P Matrix_C They respectively represent the obstacle attribute characteristics detected by the visual sensor, the obstacle attribute characteristics detected by the millimeter wave radar, and the obstacle attribute characteristics detected by the infrared sensor.

[0131] In this embodiment, processing boundaries and outliers is also included.

[0132] Specifically, during the data fusion process, boundary problems or outliers may be encountered, which require data processing, including but not limited to outlier processing algorithms such as interpolation and filtering.

[0133] In this embodiment, optimization and post-processing are also included.

[0134] Specifically, the fused data may require further optimization and post-processing to improve its quality and usability, including but not limited to denoising, enhancement, correction and other steps.

[0135] In this embodiment, the acquired multi-dimensional perception data can be used for target recognition and tracking.

[0136] Apply target detection and tracking algorithms on the fused image to identify and track objects around the vehicle. Using a multi-dimensional pixel matrix can solve the following typical corner cases that cannot be covered by a single sensor:

[0137] When driving at night and vehicles in the oncoming lane turn on their headlights, the cyclists around your vehicle cannot be accurately sensed, thus bringing driving risks.

[0138] Accurate perception of the vehicle ahead on the highway can provide early information about the intersection ahead, thereby reserving more response time for system decision-making and execution.

[0139] At the same time, different confidence levels can be set for the perception results of different sensors in the multi-dimensional pixel matrix according to different weather conditions and visual environments, so as to achieve accurate identification of target obstacles in various driving scenarios and improve driving safety.

[0140] Example 2

[0141] The embodiment of the present invention further provides a storage medium on which a computer program is stored, and the computer program is used to implement the multi-dimensional pixel fusion method for environment perception as described above. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0142] The beneficial effects of the embodiments of the present invention are as follows:

[0143] (1) Control signals are sent to the visual deserializer, infrared deserializer, and millimeter-wave radar at the same time. Frame synchronization signals and hard-wired trigger signals are used to drive the visual sensor, infrared sensor, and millimeter-wave radar to collect data at the same time. Data is collected at the same time to achieve timing alignment between the sensors, providing a fusion basis for multi-dimensional data fusion.

[0144] (2) Based on the particularity of the FOV center, the coordinate offset values ​​of the visual FOV center point, the infrared FOV center point and the origin of the coordinate system (the FOV center point of the millimeter wave radar) are calculated to determine the offset range of the entire field of view. Then, based on the coordinate offset values, the positions of the visual FOV center point and the infrared FOV center point relative to the plane coordinate system are determined. By fusing the data collected by the three sensors in the same coordinate system, spatial alignment is achieved. This allows sensors of different types and different FOV fields of view to find a multi-dimensional pixel field of view overlap area through software algorithms, and the coordinate space within the area is aligned.

[0145] (3) Based on the different characteristics of obstacle data types collected by different sensors, a probability distribution model is established according to real-time data, and then the information entropy is calculated and converted into a confidence weight coefficient. The confidence weight coefficient of each sensor is updated in real time to ensure the actual adaptability of the obstacle recognition coefficient and improve the recognition accuracy.

[0146] (4) Based on the different types of obstacle data collected by different sensors, the detection data of each sensor is "coordinate-unified and time-series-aligned", and the image and radar data are "synchronized in time and space" at the pixel level in real time and output in a "multi-dimensional pixel" format through splicing and fusion. Provide the autonomous driving system with multi-modal accurate perception information of the target and the environment: including the comprehensive perception of the sensor's visual data (brightness, texture, color, etc.) of the target and the environment, 4D millimeter-wave radar data (target distance, speed, direction, height, etc.) and infrared radiation data (texture, temperature, etc.), so as to achieve accurate identification of target obstacles in various driving scenarios and improve driving safety.

[0147] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-dimensional pixel fusion method for environmental perception, characterized in that: include: Based on the frame synchronization signal and the hard-line trigger signal, the sensor is driven to collect environmental information to obtain time-aligned raw environmental data, wherein the sensor includes a visual sensor, an infrared sensor and a millimeter-wave radar; Preprocessing the raw environmental data, and performing feature extraction based on the processed raw environmental data to obtain corresponding visual features, infrared features, and dynamic features; Perform spatial coordinate alignment processing to obtain the FOV overlap area of ​​spatial registration; A weight matrix is ​​constructed, and then the visual features, infrared features and dynamic features of the overlapping area of ​​the FOV are fused to obtain multi-dimensional perception data.

2. A multi-dimensional pixel fusion method for environment perception as claimed in claim 1, characterized in that: The method of collecting environmental information based on the frame synchronization signal and the hard-line trigger signal to obtain timing-aligned original environmental data includes: At the same time, control signals are sent to the visual deserializer, infrared deserializer, and millimeter wave radar; Performing exposure control on the visual sensor based on the frame synchronization signal, acquiring visual data from the visual camera serializer, and deserializing the visual data through the visual deserializer to obtain the visual data; Exposure control is performed on the infrared sensor based on the frame synchronization signal, infrared data is obtained from the infrared camera serializer, and the infrared data is deserialized through the infrared deserializer to obtain infrared data; Drive the millimeter-wave radar to collect data based on hard-wired trigger signals to obtain point cloud data; The original environment data includes visual data, infrared data and point cloud data.

3. A multi-dimensional pixel fusion method for environment perception as claimed in claim 2, characterized in that: The feature extraction based on the processed original environment data includes: Based on the deep learning model, visual features are extracted from the visual data; based on the deep learning model, infrared features are extracted from the infrared data; based on the optical flow method or the key point detection algorithm, dynamic features are extracted from the point cloud data.

4. The multi-dimensional pixel fusion method for environment perception according to claim 1, characterized in that: The spatial coordinate alignment processing is performed to obtain the FOV overlap area of ​​the spatial registration, including: Take the center point of the FOV of the millimeter-wave radar as the origin of the coordinate system and establish a plane coordinate system; Calculate the coordinate offset values ​​of the center point of the visual FOV, the center point of the infrared FOV and the origin of the coordinate system; Taking the plane coordinate system as a reference, the coordinate offset value and the sensor resolution are obtained, and the first coordinate group, the second coordinate group, and the third coordinate group corresponding to the radar field of view FOV range, the visual FOV range, and the infrared FOV range are obtained respectively; Based on the first coordinate group, the second coordinate group, and the third coordinate group, a fourth coordinate group corresponding to the FOV overlapping area is calculated so that the coordinate space of the pixel points in the FOV overlapping area is aligned; the FOV overlapping area is the overlapping area of ​​the radar FOV range, the visual FOV range, and the infrared FOV range.

5. A multi-dimensional pixel fusion method for environment perception as claimed in claim 4, characterized in that: Assume that the resolution of the millimeter wave radar is U*V, the resolution of the visual sensor is M*N, and the resolution of the infrared sensor is R*T, then: The fourth coordinate group of the FOV overlapping area is as follows: Where Axy, Bxy, Cxy, and Dxy are the coordinates of the four vertices of the FOV overlap area; (m, n) is the coordinate offset between the center point of the visual FOV and the origin of the coordinate system; (p,q) is the coordinate offset between the center point of the infrared FOV and the origin of the coordinate system.

6. The multi-dimensional pixel fusion method for environment perception according to claim 1, characterized in that: The constructing of the weight matrix comprises: Obtain obstacle data in the overlapping area of ​​the FOV field of view and establish a probability distribution model; According to the probability distribution estimated by the probability distribution model, the information entropy of the visible light camera, 4D millimeter wave radar and infrared camera are calculated respectively; Performing normalization processing to convert the information entropy into a confidence weight coefficient; All the confidence weight coefficients corresponding to the visual sensor, millimeter wave radar, and infrared sensor are integrated to obtain a weight matrix.

7. A multi-dimensional pixel fusion method for environment perception as claimed in claim 6, characterized in that: The calculation formula of the information entropy is as follows: Where H(X) is the information entropy of random variable X, p(x i ) is the probability of X taking the i-th value, and n is the total number of possible values; The calculation formula of the confidence weight coefficient is as follows: In the formula, w i is the weight coefficient of the i-th sensor, H(X i ) is the information entropy of the ith sensor, ∈ is a positive number.

8. The multi-dimensional pixel fusion method for environment perception according to claim 6, characterized in that: The feature fusion of the visual features, infrared features and dynamic features of the FOV overlapping area to obtain multi-dimensional perception data includes: Based on the FOV overlapping area, respectively obtain the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix corresponding to the visual features, infrared features, and dynamic features; Perform feature splicing processing on the obstacle pixel matrix, point cloud pixel matrix, and thermal radiation pixel matrix pixels to obtain a multi-dimensional pixel matrix; Performing multidimensional pixel fusion according to the multidimensional pixel matrix and the weight matrix to obtain multidimensional perception data; The obstacle data includes: the color channel values ​​(R, G, B) of the object image collected by the visual sensor, the distance L, speed S, direction R and height signal h of the object collected by the millimeter wave radar, and the thermal radiation H of the object collected by the infrared sensor.

9. A multi-dimensional pixel fusion method for environment perception as claimed in claim 8, characterized in that: The multi-dimensional pixel matrix is ​​as follows: The fusion formula of the multi-dimensional perception data is as follows: in, Where W is the weight matrix, are the confidence weight coefficients of the visual sensor, millimeter wave radar, and infrared sensor respectively; P Matrix_A , P Matrix_B , P Matrix_C They respectively represent the obstacle attribute characteristics detected by the visual sensor, the obstacle attribute characteristics detected by the millimeter wave radar, and the obstacle attribute characteristics detected by the infrared sensor.

10. A storage medium having a computer program stored thereon, characterized in that: The computer program is used to implement a multi-dimensional pixel fusion method for environmental perception as described in any one of claims 1-9.

Images