A method, device, system, and storage medium for road environment sensing

Abstract

The present invention discloses a road environment perception method and device, system, and storage medium, including the following steps: S100, obtaining a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, and a radar device; S200, preprocessing the road environment image; S300, performing multi-data registration and fusion on the preprocessed road environment image; S400, performing segmentation, recognition, and perception on the registered and fused road image to achieve road environment recognition in high-altitude and cold regions. The technical solution of the present invention is adopted to realize vehicle external environment monitoring by combining thermal imaging images, visible light images, and point cloud data fusion, which can monitor the environment around the vehicle in real time and issue early warnings in a timely manner, improve recognition accuracy in harsh road environments, and significantly improve driving safety.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a road environment perception method and device, a system, and a storage medium. Background Art

[0002] In high-altitude and cold regions, driving safety faces huge challenges due to harsh climatic conditions and special situations such as road intrusion by pedestrians, non-motor vehicles, livestock, and wild animals. In these complex environments, it is difficult to accurately identify road conditions relying solely on computer vision technology.

[0003] At present, smart vehicles are generally equipped with laser radar, millimeter wave radar, and industrial cameras to identify road environments. The fusion of radar and camera can indeed obtain image environment information intuitively, but when there is extreme weather such as sandstorms or fog, radar and camera equipment cannot clearly perceive all the road conditions of the current road, which is very dangerous under complex road conditions in high altitudes and cold weather. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a road environment perception method and device, system, and storage medium, which realize vehicle external environment monitoring by combining thermal imaging images, visible light images and point cloud data fusion, can monitor the environment around the vehicle in real time, and issue early warnings in time, which can improve the recognition accuracy in harsh road environments and significantly improve driving safety.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] The present invention provides a road environment perception method, which recognizes and perceives the road environment based on multi-sensor equipment, uses information obtained by sensors such as visible light cameras, thermal imaging cameras, millimeter wave radars, and lidars to perform multi-source fusion of data, and transmits the information to a vehicle-mounted edge computer through narrowband transmission and wired transmission. The edge computer combines image fusion technology and deep learning algorithms to recognize and perceive the road environment, performs path planning based on this information, and transmits the road conditions to the road cloud through 5G signals and vehicle networking technology, so that the road control center can understand the road conditions in a timely manner. At the same time, if dangerous road conditions occur, the vehicle will promptly notify the driver to take corresponding measures to ensure safe driving; specifically, the following steps are included:

[0007] S100, obtaining a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, or a radar device;

[0008] S200, preprocessing the road environment image;

[0009] S300, performing multi-data registration and fusion on the pre-processed road environment image, including:

[0010] The visible light image captured by the visible light camera and the infrared image captured by the thermal imaging camera are multi-source data registered through the radar point cloud information;

[0011] The structure and texture information of the multi-source image after data registration is calculated by the visible light and infrared countermeasure fusion model based on texture information, which is used as the base layer of the fused image. The fused image is then obtained by color space conversion, and the processed point cloud information is mapped to the fused image.

[0012] S400: Segment, recognize and perceive the registered and fused road image to realize road environment recognition in high altitude and cold regions.

[0013] Preferably, preprocessing the road environment image includes: constructing a relationship between the image grayscale value and the temperature based on the infrared image data collected by the thermal imaging camera.

[0014] Preferably, preprocessing the road environment image includes: denoising the point cloud information of the point cloud data collected by the radar device.

[0015] Preferably, preprocessing the road environment image includes: ultra-definition processing of the road environment image data collected by the visible light camera.

[0016] Preferably, S400 includes:

[0017] Based on the image data after multi-data registration and fusion, the reflectance coefficient, surface texture and temperature distribution of the road surface are extracted through the improved Swin Transformer network;

[0018] Based on the reflection coefficient, surface texture and temperature distribution of the road surface, the road environment in plateau and cold areas can be identified.

[0019] The present invention also provides a road environment perception device, comprising:

[0020] An acquisition module is used to acquire a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, or a radar device;

[0021] A preprocessing module, used for preprocessing the road environment image;

[0022] A fusion module is used to perform multi-data registration and fusion on the pre-processed road environment image;

[0023] The recognition module is used to segment, identify and perceive the registered and fused road images, so as to realize the road environment recognition in plateau and cold areas.

[0024] The present invention also provides a road environment perception system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a road environment perception method when executed by the processor.

[0025] The present invention also provides a storage medium, on which a computer program is stored, and the computer program executes the road environment perception method when running.

[0026] The present invention realizes the perception of road environment in plateau and cold areas based on the fusion of thermal imaging, images and radar point clouds to ensure driving safety; adopts an image fusion method based on texture features to ensure the same fusion effect under different lighting conditions and improve fusion accuracy; uses an improved Transform network to improve the efficiency and accuracy of road environment perception; and promptly warns the driver when dangerous conditions are found on the road surface; when the vehicle is driving in plateau and cold areas, the identified road condition information will be fed back to the road control center through the cloud to facilitate the improvement of road condition information. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0028] Figure 1 A flowchart of a road environment perception method according to an embodiment of the present invention;

[0029] Figure 2 It is a schematic diagram of the layout of the on-board unit equipment;

[0030] Figure 3 This is a schematic diagram of the transformation relationship between pixel-image-camera-world coordinate system;

[0031] Figure 4 This is a schematic diagram of the joint calibration of radar equipment and multi-source cameras;

[0032] Figure 5 It is a schematic diagram of multi-source image data fusion algorithm;

[0033] Figure 6 Schematic diagram of the fusion image road environment recognition algorithm. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1:

[0037] The embodiment of the present invention provides a road environment perception method, which calibrates, aligns, and fuses visible light cameras, infrared thermal imaging cameras, and radar equipment, and uses deep learning algorithms to realize road environment status recognition and monitoring based on multi-source data fusion, and can perceive the road environment status in real time. In terms of equipment selection and application: multi-source cameras (visible light cameras and infrared thermal imaging cameras), radar equipment (lidar and millimeter wave radar) and edge computers for data processing are selected; in terms of data transmission: according to different scene requirements, multiple communication methods such as wired transmission, narrowband transmission, 5G transmission are used, and stable data communication is carried out in combination with the Internet of Vehicles to ensure timely and barrier-free transmission of data; in terms of data processing: while using edge computers to process the next frame of multi-source data to complete the registration and fusion, a deep learning algorithm is used to classify and identify the fused road environment image of the previous frame.

[0038] The embodiment of the present invention first deploys data acquisition equipment to complete the acquisition of road image data, and performs data preprocessing on the acquired information respectively; secondly, it completes the registration and calibration of multi-source data based on the relationship between pixel-image-camera-radar coordinate systems, and uses texture information fusion algorithm to realize the fusion of visible light image and thermal imaging image, and integrates point cloud information; finally, it uses deep learning algorithm to process the fused image information to complete the road recognition task, and transmits the road environment recognition result to the car's central control screen via narrowband transmission to remind the driver to pay attention to the road environment, and transmits the result to the cloud via 5G transmission, and the cloud feeds back to the road control center to understand the environmental road conditions in high-altitude and cold areas. Figure 1 As shown, the specific steps are as follows:

[0039] S000, Collection equipment layout

[0040] The layout of the equipment is as follows Figure 2 As shown, millimeter-wave radars are installed around the vehicle, a visible light camera is placed at the front of the vehicle, a lidar, an infrared thermal imaging camera, and a GNSS antenna for transmitting information to the cloud are placed on the top of the vehicle.

[0041] S100, according to Figure 2 The equipment is deployed as shown and the original road environment image information collected by each sensor is obtained, including thermal imaging cameras, radar equipment, and road environment information collected by cameras.

[0042] S200: Preprocessing the image data information collected by each sensor, specifically including:

[0043] S201, preprocessing the infrared image data collected by the thermal imaging camera, the specific steps are as follows.

[0044] 1) Image grayscale conversion

[0045] Since grayscale images have less information and faster calculation, they are convenient for subsequent fusion and recognition processing, so they are processed in grayscale. The formula for converting to grayscale is:

[0046] GS=0.299*R+0.587*G+0.114*B

[0047] Among them, R, G, and B refer to the three color channels of red, green, and blue respectively.

[0048] 2) Establish the relationship between image gray value and temperature

[0049] Experiments show that there is a first-order functional relationship between grayscale value and temperature value. Therefore, the relationship between image grayscale value and temperature value can be written based on the grayscale value and temperature value of any two points. By using the maximum temperature and minimum temperature of the road environment that are most easily obtained in the thermal imaging image and their corresponding image grayscale values, the conversion relationship between image grayscale value and temperature is obtained as follows:

[0050]

[0051] Where T is the temperature value of any point in the image, T max , T min are the highest and lowest temperatures of the road environment captured by the thermal imaging camera, and GS is the grayscale value of the point.

[0052] S202. Preprocessing of point cloud data collected by radar equipment

[0053] Perform point cloud information denoising and use the median filter method to denoise the image to improve the clarity and quality of the point cloud data.

[0054] S203: Preprocessing of road environment image data acquired by the visible light camera

[0055] Considering the weather conditions in the high-altitude and cold regions, there are more foggy days in spring and winter, and dusty days in summer and autumn. The pictures taken by the camera may be blurry, so the Sobel function is used to process the image into ultra-clear images, which can be expressed as:

[0056]

[0057] Among them, S x1 (x,y) and S y1 (x, y) are the coordinates of the midpoint of the original image and the new coordinates formed by convolution of the surrounding 8 points with the corresponding Sobel operator. x1 , g y1 Convolution kernels for edge detection in horizontal and vertical directions.

[0058] S300, multi-source sensor calibration and registration

[0059] To fuse multi-source image information data, we should first determine the transformation relationship of each image coordinate system, perform joint calibration of multi-source sensors based on the transformation relationship, and then perform data registration based on point cloud information to eliminate image differences caused by imaging conditions, viewing angles, lighting and other factors, so that the same objects in different images can be accurately matched, providing a basis for subsequent image fusion. The specific operations are as follows:

[0060] S301. Conversion relationship between pixels, images, cameras and world coordinate systems during calibration of visible light cameras and thermal imaging cameras

[0061] like Figure 3 As shown in the figure, the pixel coordinate system is defined as uov, with point O as the origin of the pixel coordinate system, and the unit of the coordinate system is one pixel. p (u 0 ,v 0 ) is the origin of the coordinate system, the x-axis is established in a position parallel to the u-axis, and the y-axis is established in a position parallel to the v-axis. p The y coordinate system is the camera coordinate system. The mapping relationship between the two coordinate systems is expressed in the form of a matrix:

[0062]

[0063] like Figure 3 As shown, the optical center O C As the origin, take the specified plane as the camera coordinate system plane, and establish the three-dimensional camera coordinate system XYZO. Then there is a point P in the camera coordinate system with the coordinates (X C ,Y C ,Z C ) T, assuming that the coordinates of point P in the image coordinate system are p(x,y), then the transformation relationship between p and P is expressed as:

[0064]

[0065] Among them, f is the focal length of the camera. Through the transformation relationship between p and P, the transformation relationship between the image coordinate system and the camera coordinate system can be obtained as follows:

[0066]

[0067] In order to describe the position relationship between the camera and the object in a three-dimensional environment, a reference coordinate system can be selected as the world coordinate system, which is composed of X w ,Y w ,Z w The rotation matrix R and the translation vector t constitute the external parameter matrix of the camera and world global coordinate systems. The transformation relationship between the world coordinate system and the camera coordinate system is:

[0068]

[0069] Among them, x c ,y c ,z c is the coordinate in the camera coordinate system, x w ,y w ,z w The coordinates are in the world coordinate system.

[0070] Through the mapping relationship between the image coordinate system and the camera coordinate system and the transformation relationship between the camera coordinate system and the world coordinate system, the corresponding relationship between the world coordinate system and the image pixel coordinate system is obtained as follows:

[0071]

[0072] Among them, K is the intrinsic parameter matrix of the camera

[0073] S302, Joint calibration of radar equipment and multi-source cameras

[0074] like Figure 4 As shown in the figure, joint calibration is to solve the matching problem between multiple coordinate systems, and the matching problem from the point cloud coordinate system to the image coordinate system is actually to solve the coefficients of its rigid transformation. First, through the formula P c =RP L+t determines the relationship between the point cloud coordinate system and the multi-source camera coordinate system, where R represents the rotation matrix and t represents the translation vector. Through the relationship between the camera coordinate system and the image coordinate system, the point on the camera coordinate system is projected onto the image coordinate system, and its transformation relationship is P I =AP C , where P I =[u,v,1] T is the three-dimensional coordinate of the pixel in the image coordinate system, u and v represent the coordinates of the image coordinate system in pixels, and A is the parameter matrix of the camera. Through the transformation relationship between the point cloud coordinate system and the camera coordinate system, and the camera coordinate system and the image coordinate system, the transformation relationship between the point cloud coordinate system and the image coordinate system can be obtained as follows:

[0075] p I =A[R,t]P L

[0076] Through the transformation relationship, the radar information can be mapped to the image coordinate system collected by the multi-source camera.

[0077] S303. Multi-source data registration based on radar point cloud information

[0078] The corresponding relationship between the point cloud coordinate system and the image pixel coordinate system is obtained by step S301 and step S302. Then, the visible light image captured by the visible light camera and the infrared image captured by the thermal imaging camera are multi-source data registered through the radar point cloud information, and the conversion relationship in the same atlas under the same coordinate system is:

[0079]

[0080] Among them, P VIS , P NIR is the coordinate of the point cloud information on the camera and thermal imaging camera image, C p Point cloud information obtained by radar, IS VIS ,ES VIS IS is the internal and external parameters of the camera. NIR ,ES NIR are the internal and external parameters of the infrared thermal imaging camera.

[0081] The calculated P VIS , P NIR The conversion relationship between the world coordinate system and the image coordinate system is used for spatial transformation, and the MASC algorithm is used to further refine the transformed coordinates to complete the multi-source image data registration based on radar point cloud information.

[0082] S304, Multi-source image data fusion

[0083] The embodiment of the present invention proposes a visible light and infrared countermeasure fusion model based on texture information, such as Figure 5 As shown. The visible light and infrared countermeasure fusion model is intended to solve the problems of low visibility and blurred images caused by snow, fog, dust and other weather conditions when vehicles are traveling in high-altitude and cold areas, so that the computer can better understand and identify the road environment. Although traditional multi-source image fusion algorithms can improve the visual effect of images, most of them do not take into account the needs of computer recognition tasks. Therefore, the visible light and infrared countermeasure fusion model of the embodiment of the present invention retains the discriminator to realize computer recognition of the image, and improves the generator, using the texture filtering fusion algorithm instead of the pure artificial intelligence iterative algorithm to clarify the fusion target, so that the image is closer to the visible light image faster, thereby quickly realizing the "deception" of the computer and forming a visible light picture recognized by the computer. For vehicles traveling in high-altitude and cold areas, the efficiency and high stability of fusion are always the first priority, which is also the reason for the optimization algorithm and a major innovation of the present invention. The texture filtering fusion algorithm calculates the structure and texture information of multi-source images through texture filters, combines color space conversion to obtain a fused image, and maps the point cloud information to the fused image. Through continuous correction, fusion, and judgment, it ultimately generates multi-source image information that integrates visible light, infrared thermal imaging, and point cloud information, effectively improving the image blur problem and enhancing the computer's ability to perceive the road environment in the special environment of high altitude and cold weather.

[0084] The image fusion algorithm proposed in the present invention includes the following seven steps: structure separation and extraction, texture extraction, texture filtering, color space conversion, image fusion, point cloud fusion, and fusion judgment.

[0085] (1) Structural separation and extraction

[0086] Visible light images and thermal imaging images are composed of two parts: main structure and texture. a =T a +S a , a∈(vis,nir),I a Represents the original image under different channels, T a represents the texture image, S a Represents a structural image, uses a texture information extraction algorithm to extract the texture structure of the image, separates the main structure of the image from the texture part of the image, and facilitates subsequent separation operations on the main structure of the image and the texture part of the image.

[0087] (2) Texture extraction

[0088] A pixel point p is divided into K intervals according to color. A region is set with pixel p as the center and r as the radius. A straight line with an angle of θ with the horizontal line is set to divide the region into two sub-regions of equal size. The texture information of these two sub-regions is expressed by g θ,r (p,k),h θ,r (p,k).θ,r (p,k),h θ,r (p, k) is vector data, which contains color histogram information, from which the histogram difference in any direction can be obtained:

[0089]

[0090] Among them, k is the index of the color interval, from which the texture edge detector mPb can be obtained, and its algorithm is

[0091]

[0092] Where, Ω = {r 1 ,r 2 ,...,r m} represents a set of radii. Different radius sets will lead to different filtering effects. mPb is used to improve the efficiency of obtaining texture edges.

[0093] (3) Texture filtering

[0094] The separated image texture structure is processed by texture filtering to remove the noise in the texture and optimize the texture structure of the image. The filter G generated by the joint bilateral filtering algorithm (JBL) is used for texture filtering:

[0095]

[0096] Among them, Sp, G p ,I q represents the smoothing, guiding and input images under p pixels, k represents the weight, and its value is the result when the fusion adversarial network is stable, f(p,q), g(G p ,G q ) is a Gaussian function used to calculate the spatial distance and color distance of p and q pixels.

[0097] (4) Color space conversion

[0098] Color space conversion is a crucial step, which involves converting an image from one color mode to another. This conversion plays a core role in the process of image fusion. Through color space conversion, images can be processed and fused in different color expressions, so that the color information in the image can be more effectively extracted and utilized. In the present invention, the main structures of the visible light image and the thermal imaging image are converted into the same color system, thereby facilitating the subsequent image fusion. The basic algorithm is as follows:

[0099] F v =V rgb +JBL(T vis,s)+T nir

[0100] Among them, V rgb is the visible light image, T vis is the value obtained by converting the RGB channel to the HSV channel and selecting the V channel as the base color for fusion. nir For infrared texture structure, JBL (T vis,s ) is the data obtained by improving the joint bilateral filtering (JBL) wave.

[0101] (5) Image fusion based on texture information

[0102] The calculated F v With H vis , S vis Convert to RGB image as the result of image fusion. Hvis and Svis are the visible light values ​​of H and S channels in HSV channel respectively. This improves the quality and visual effect of the image. It also brings temperature data to the image, making up for the problem of low visibility and blurred image caused by dust and fog in high-altitude cold areas. At the same time, temperature data can also better reflect the surrounding environment information. The steps are as follows:

[0103] (6) Point cloud fusion

[0104] Integrating point cloud information into the fused image facilitates subsequent computer recognition and can determine the location of obstacles in the image and their distance from the vehicle based on the point cloud information.

[0105] (7) Fusion judgment

[0106] The fused image F is generated by the above algorithm 融 , at this time, the original image captured by the visible light camera and the fused image F 融 Synchronously input into the visible light discriminator D for fusion judgment. This step is called the test process. In this process, the computer judges the visible light photo as the true value and the fused image as the false value. If it is a true value, it is output. If it is a false value, the parameters of the filter are modified to re-fuse the image. Through the continuous correction of the parameters of the discriminator and the filter, a fused image with more optical texture and that can "cheat" the computer is gradually formed. At the same time, when the correction is completed, the parameters of the filter are stable in the same section and the same road conditions, that is, a "deceptive visible light image" can be generated by inputting a visible light image, a thermal imaging image, and a point cloud data, which is convenient for the subsequent semantic segmentation and recognition of the computer. The discriminator D follows the following algorithm:

[0107]

[0108] Among them, x represents the real visible light image, z represents the noise input to the filter G, G(z) represents the fused image generated by the fusion algorithm after filtering through the filter G, D(*) represents the probability of the discriminator D judging whether the image is real visible light, and E(*) represents the mathematical expectation. Obviously, for the discriminator D, it is hoped to maximize logD(x) and log(1-D(G(z))), and for the filter G, the purpose of continuously modifying the weight k is to minimize log(1-D(G(z))) so as to achieve the goal of minimizing M(D,G).

[0109] S400, Road Environment Perception

[0110] The road environment perception algorithm is the key and core step of the embodiment method of the present invention. It directly determines the vehicle's understanding of the road environment and the accuracy of driving decisions. In terms of algorithm recognition efficiency and accuracy, the Transformer network is superior to the convolutional neural network (CNN). However, due to its low-resolution feature mapping and quadratic growth of complexity with image size, its structure is not suitable as a backbone network for dense, large-scale visual tasks or high-resolution images. These shortcomings are fatal to road environment recognition. Therefore, the present invention has made targeted improvements to the Transformer network and selected the Swin Transformer network to recognize the fused image. It uses a moving window method to significantly reduce the model parameters and optimize its efficiency and accuracy in recognizing the fused image. The algorithm flow is as follows: Figure 6 shown.

[0111] (1) Identification process

[0112] The improved Swin Transformer network can quickly extract key feature values ​​from the fused image. This network is uniquely designed, adopts a block strategy and a multi-stage structure to effectively process large-size images. At the same time, it integrates a window attention module and a moving window self-attention module. These two modules work together to accurately capture various features of the image, including image features, point cloud features, and temperature features.

[0113] The fused image is input into the Swin Transformer network, and the image is segmented through the Patch Partition layer of the Swin Transformer fusion module. The segmented data is subjected to feature mapping through the Linear Embedding layer to extract key features.

[0114] The output key features of each stage are processed by the ConvNext module. The ConvNext module first uses N*N convolution operations to increase the expression ability of local information. And the number of feature channels is enlarged through the multi-layer perceptron to provide richer feature representation.

[0115] Finally, the object classification and recognition module is constructed using nonlinear activation units such as ReLU functions and position encoding. The regression prediction module is responsible for converting the feature maps learned in the previous layers into specific results of target detection, and then classifying and outputting the results, thereby achieving effective recognition of road obstacles.

[0116] (2) Loss Function and Model Training

[0117] Cascade RCNN is used as the regression module, including classifier C x and the regressor f x Two parts, each element in the graph will be classified, and the classifier outputs a probability distribution vector to show the probability of each element being classified into its class. The classification loss is:

[0118]

[0119] Among them, L cls is the cross entropy loss, N is the number of recognized elements in the graph, x i is the network image input, y i is the category to which the classification belongs, c(x i ) is the posterior probability of M+1 (1≤M≤N) categories.

[0120] When classifying elements in an image, different IOU thresholds are usually used to determine which elements in the image block need to be classified and identified and which elements only serve as the background of the image block. If the IOU is higher than the threshold u, it indicates that there are elements in the image block that need to be classified. The calculation function is:

[0121]

[0122] Among them, g represents the true category, g y is the location label of the ground-truth box.

[0123] Use the regressor f(x,b) to regress the selection box b at a certain position to the real target box g. Let the coordinates of each box be (b x ,b y ,b w ,b h ), the loss of the regressor is:

[0124]

[0125] Among them, L loc For L 2 The loss value is obtained by optimizing the IOU threshold at each stage and retraining is performed to achieve a prediction closer to the target. The IOU value is recorded and kept stable for a certain period of time. The optimized cascade loss is expressed as:

[0126] L(x t ,g)=L ds (h i (x t ),y t )+λ[y t ≥1]L loc (f i (x t ,b t ),g)

[0127] Among them, b t =f t (x t-1 ,b t-1 ), λ is the weight coefficient, and the present invention takes λ as 1, y t is the label of the tth iteration, g represents the real frame object of the tth iteration. The training of this algorithm can significantly improve the detection performance and training efficiency, which is suitable for complex and changeable, real-time interactive, and huge data vehicle-mounted recognition.

[0128] The present invention performs segmentation, recognition and perception on the fused road image based on a deep learning algorithm to complete the road environment recognition in high-altitude and cold areas. After the registration and fusion of multi-source image data, the image information contains the image, temperature and location information of the road. Based on this, the road environment information can be judged to perceive whether there are stones, potholes, road intrusions, ice and snow accumulation on the road surface. The camera captures the image information of the road surface, the radar obtains the distance and reflection information of the road surface, and the thermal imaging camera detects the temperature distribution of the road surface. The reflection coefficient, surface texture, temperature distribution and other characteristic information of the road surface are extracted from the fused atlas. Classifiers and machine learning algorithms are used to classify and identify the extracted characteristic information, and finally output to the driver whether there are safety hazards on the road.

[0129] S500, recognition result feedback

[0130] The road environment recognition results obtained in step S4 are fed back to the driver to help him understand the driving environment in real time, and also provide data information support for subsequent path planning. The results are transmitted to the cloud through the GNSS antenna, so that the road control center can understand the real-time road conditions in plateau and cold areas, and form a data set to provide data reference for subsequent vehicles.

[0131] Embodiment 2:

[0132] The present invention also provides a road environment perception device, comprising:

[0133] An acquisition module is used to acquire a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, or a radar device;

[0134] A preprocessing module, used for preprocessing the road environment image;

[0135] A fusion module is used to perform multi-data registration and fusion on the pre-processed road environment image;

[0136] The recognition module is used to segment, identify and perceive the registered and fused road images, so as to realize the road environment recognition in plateau and cold areas.

[0137] Embodiment 3:

[0138] An embodiment of the present invention further provides a road environment perception system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a road environment perception method when executed by the processor.

[0139] Embodiment 4:

[0140] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the road environment perception method when running.

[0141] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A road environment perception method, characterized in that: The following steps are involved: S100, obtaining a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, or a radar device; S200, preprocessing the road environment image; S300, performing multi-data registration and fusion on the pre-processed road environment image, including: The visible light image captured by the visible light camera and the infrared image captured by the thermal imaging camera are multi-source data registered through the radar point cloud information; The structure and texture information of the multi-source image after data registration is calculated by the visible light and infrared countermeasure fusion model based on texture information, which is used as the base layer of the fused image. The fused image is then obtained by color space conversion, and the processed point cloud information is mapped to the fused image. S400: Segment, recognize and perceive the registered and fused road image to realize road environment recognition in high altitude and cold regions, specifically including: Based on the image data after multi-data registration and fusion, the reflectance coefficient, surface texture and temperature distribution of the road surface are extracted through the Swin Transformer network; Based on the reflection coefficient, surface texture and temperature distribution of the road surface, the road environment in plateau and cold areas can be identified.

2. The road environment perception method according to claim 1, characterized in that: Preprocessing the road environment image includes: constructing the relationship between the image grayscale value and the temperature based on the infrared image data collected by the thermal imaging camera.

3. The road environment perception method according to claim 2, characterized in that: Preprocessing the road environment image includes: denoising the point cloud information of the point cloud data collected by the radar equipment.

4. The road environment perception method according to claim 2, characterized in that: Preprocessing the road environment image includes: ultra-definition processing of the road environment image data collected by the visible light camera.

5. A road environment perception device for implementing the road environment perception method according to any one of claims 1 to 4, characterized in that: include: An acquisition module is used to acquire a road environment image; wherein the road environment image is road environment information collected by a visible light camera, a thermal imaging camera, or a radar device; A preprocessing module, used for preprocessing the road environment image; A fusion module is used to perform multi-data registration and fusion on the pre-processed road environment image; The recognition module is used to segment, identify and perceive the registered and fused road images, so as to realize the road environment recognition in plateau and cold areas.

6. A road environment perception system, comprising: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the road environment perception method as described in any one of claims 1 to 4 is executed.

7. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, executes the road environment perception method as described in any one of claims 1 to 4.

Images