Road Environment Perception Method, System and Medium Based on Air-Ground Multi-Sensor Fusion

Abstract

The present invention discloses a road environment perception method, system and medium based on ground-air multi-sensor fusion. The method includes: using a satellite sensor to obtain aerial information of the ground environment and extracting road information from the aerial information; using a vehicle-end sensor to obtain vehicle-end information of the environment around the vehicle body and extracting obstacle information from the vehicle-end information; and fusing the road information and the obstacle information to obtain a road environment perception result. The present invention can more conveniently and comprehensively obtain ground environment information for vehicle autonomous driving or assisted driving.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly to a road environment perception method, system and medium based on ground-air multi-sensor fusion. Background Art

[0002] In the prior art, autonomous driving perception solutions are mostly limited to vehicle-mounted sensors or vehicle-road collaborative sensor perception. Vehicle-mounted sensor perception mainly includes single visible light camera perception, lidar perception, millimeter wave radar perception, and their multi-modal fusion solutions. These solutions can mostly detect moving and static obstacles on the road in real time, but have less perception of the environmental information on the road or have certain detection difficulties. Vehicle-road collaborative sensor perception realizes accurate perception of traffic conditions and road conditions through information provided by infrastructure sensors such as traffic signals and roadside cameras and vehicle-mounted communication devices, which requires a large amount of infrastructure construction and vehicle communication device support. Summary of the Invention

[0003] In view of the above technical problems, the present invention provides a road environment perception method, system and medium based on ground-air multi-sensor fusion, which can obtain ground environment information more conveniently and comprehensively.

[0004] In the first aspect of the present invention, there is provided a road environment perception method based on ground-air multi-sensor fusion, including:

[0005] Using a satellite sensor to obtain air-end information of the ground environment, and extracting road information from the air-end information;

[0006] Using a vehicle-mounted sensor to obtain vehicle-end information of the environment around the vehicle body, and extracting obstacle information from the vehicle-end information;

[0007] Fusing the road information and the obstacle information to obtain a road environment perception result.

[0008] In a possible implementation manner, the method further includes:

[0009] Extracting lane information from the vehicle-end information, and improving the road information based on the lane information.

[0010] In a possible implementation manner, the satellite sensor includes at least one of a satellite positioning sensor, a high-resolution multi-spectral satellite sensor, and an interferometric synthetic aperture radar

[0011] In a possible implementation manner, the step of using a satellite sensor to obtain air-end information of the ground environment, and extracting road information from the air-end information, includes:

[0012] Segment the road area using InSAR (Interferometric Synthetic Aperture Radar), and obtain the road boundary using a boundary extraction algorithm.

[0013] In one possible implementation, the method of using a satellite sensor to obtain aerial information of the ground environment and extracting road information from the aerial information includes:

[0014] Extract the road area using high-precision multi-spectral images, and detect lane lines and road surface markings through feature differences.

[0015] In one possible implementation, the method of using a satellite sensor to obtain aerial information of the ground environment and extracting road information from the aerial information further includes:

[0016] Based on the periodic law and three-dimensional spatial information of remote sensing satellite images, splice and merge multiple satellite images, unify the resolution and perform spatial position matching;

[0017] Crop the spliced and merged satellite images into map tiles of a fixed size;

[0018] And, using the map tiles as the basic unit, store the vector data corresponding to the road information in a database.

[0019] In one possible implementation, the method of using a vehicle sensor to obtain vehicle-end information of the environment around the vehicle body and extracting obstacle information from the vehicle-end information includes:

[0020] Perform pre-fusion on radar point cloud data and visible light camera data to obtain a three-dimensional spatial feature information map;

[0021] Detect the obstacle information from the spatial feature information map.

[0022] In one possible implementation, the method of fusing the road information and the obstacle information to obtain a road environment perception result includes:

[0023] Use a satellite sensor to obtain the position information of the vehicle;

[0024] Based on the position information, obtain the surrounding road information at the corresponding position in the aerial information;

[0025] Integrate the surrounding road information and the obstacle information into the vehicle's ego-coordinate system to obtain the road environment perception result.

[0026] In one possible implementation, the method of extracting lane information from the vehicle-end information and improving the road information based on the lane information includes:

[0027] Judge the incomplete areas with missing information in the road information through a connectivity algorithm;

[0028] Search for road boundaries and / or lane lines with coordinates close to the position based on the position information of the incomplete area from the lane information;

[0029] Perform topological matching and merging on the incomplete area based on the road boundary and / or lane line to improve the road information.

[0030] In a second aspect of the present invention, there is provided a road environment perception system based on ground-air multi-sensor fusion, including:

[0031] An air-end data processing module, at least used to obtain air-end information of the ground environment by using a satellite sensor, and extract road information from the air-end information;

[0032] A vehicle-end data processing module, at least used to obtain vehicle-end information of the environment around the vehicle body by using vehicle-end sensors, and extract obstacle information from the vehicle-end information;

[0033] Wherein, the vehicle-end data processing module is further used to fuse the road information and the obstacle information to obtain a road environment perception result.

[0034] In a possible implementation manner, the air-end data processing module is further used to extract lane information from the vehicle-end information, and improve the road information based on the lane information.

[0035] In a possible implementation manner, the system further includes:

[0036] An air-end data management module, at least used to store the vector data corresponding to the road information in a database.

[0037] In a third aspect of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a computer, it executes the method described in the first aspect of the embodiments of the present invention.

[0038] The present invention uses the air-end information obtained by a satellite sensor and the vehicle-end information obtained by vehicle-end sensors, and obtains a road environment perception result by fusing road information and obstacle information, thereby realizing a comprehensive perception and understanding of the road environment. This method is independent of a high-precision map (HDMap), directly uses the images generated by a satellite sensor, directly synthesizes the ground road information at the air-end, eliminates the complex high-precision map production process, improves the accuracy and robustness of the autonomous driving system for road conditions, enhances the vehicle's perception ability of the surrounding environment, and thus improves driving safety and driving comfort. Description of the Drawings

[0039] Figure 1It is a flowchart of a road environment perception method based on ground-air multi-sensor fusion in an embodiment of the present invention.

[0040] Figure 2 It is a schematic diagram of the principle of interferometric synthetic aperture radar imaging in an embodiment of the present invention.

[0041] Figure 3 It is a schematic diagram of the principle of a map tile cutting process in an embodiment of the present invention.

[0042] Figure 4 It is a schematic diagram of vector data for storing air-end information in an embodiment of the present invention.

[0043] Figure 5 It is a schematic diagram of the output result of vehicle-end information in an embodiment of the present invention.

[0044] Figure 6 It is a logical architecture diagram of ground-air data transmission in an embodiment of the present invention.

[0045] Figure 7 It is a logic diagram of a ground-air data processing module in an embodiment of the present invention. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present invention.

[0047] It should be understood that the terms "first", "second", "third", etc. in the claims, the description and the drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "comprising" used in the description and claims of this disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations. It should also be understood that the terms used in the description of this disclosure are only for the purpose of describing specific embodiments, and are not intended to limit this disclosure.

[0048] The autonomous driving road environment perception provided by the present invention can be classified into the following categories according to actual applications:

[0049] First, road environment perception can be divided into two categories: environmental information and obstacle information;

[0050] Secondly, the environmental information mainly includes non-road information and road information. The non-road information refers to objects outside the vehicle's drivable area, mostly such as buildings, vegetation, etc. The road information refers to some symbol or area information within the spatial range of the vehicle's drivable area, including roads, lane lines, zebra crossings, traffic lights, traffic signs, etc.;

[0051] Thirdly, the obstacle information mainly refers to the object information in the area that the vehicle can reach. It includes moving obstacles and static obstacles. Among them, the moving obstacles include moving vehicles, pedestrians, cyclists, etc., and the static obstacles include cone barrels, vehicles parked by the roadside, etc.

[0052] Please refer to Figure 1 As shown, the present invention provides a road environment perception method based on ground-air multi-sensor fusion, including the following steps:

[0053] S1, Use satellite sensors to obtain the aerial information of the ground environment, and extract road information from the aerial information; when using satellite sensors to obtain aerial information, satellite remote sensing data, such as high-resolution remote sensing images, can be used to photograph a specific area or obtain corresponding remote sensing data; then, use image processing technology and computer vision algorithms to process the remote sensing data and extract road information, such as road contours, lane lines, etc.

[0054] S2, Use vehicle sensors to obtain the vehicle-end information of the environment around the vehicle body, and extract obstacle information from the vehicle-end information; when using vehicle sensors to obtain vehicle-end information, vehicle-mounted sensors, such as lidar, cameras, millimeter-wave radars, etc., can be used to obtain the environmental information around the vehicle in real time; then, use perception algorithms to extract obstacle information from the data obtained by the vehicle-mounted sensors, such as other vehicles, pedestrians, roadblocks, etc.

[0055] S3, Fuse the road information and the obstacle information to obtain the road environment perception result. When fusing the road information and the obstacle information, data fusion technology can be used to integrate the information obtained in the S1 and S2 stages; for example, data fusion algorithms (such as Kalman filtering, Bayesian fusion, etc.) can be used to merge the road information and the obstacle information to generate a road environment perception result, such as a road map, an obstacle position map, etc.

[0056] Exemplarily, first, a convolutional neural network is used to perform image segmentation and feature extraction on satellite images to extract pixel information of the road area. Secondly, data obtained from lidar and cameras is processed through a ground segmentation algorithm and an object detection algorithm to identify obstacle information such as vehicles and pedestrians. Finally, data fusion algorithms such as Bayesian filtering are used to fuse the road information in the satellite images with the obstacle information around the vehicle to obtain a comprehensive road environment perception result. This method does not rely on high-precision maps, directly uses the images generated by satellite sensors, and eliminates the complex high-precision map production process. At the same time, it improves the accuracy and robustness of the autonomous driving system to road conditions, enhances the vehicle's perception ability of the surrounding environment, and thus improves driving safety and driving comfort.

[0057] Further, as a specific implementation manner of the present invention, in step S1, satellite sensors are used to obtain aerial information of the ground environment, and road information is extracted from the aerial information, including:

[0058] An interferometric synthetic aperture radar is used to segment the road area, and a boundary extraction algorithm is used to obtain the road boundary.

[0059] Specifically, referring to Figure 2 , the special diffuse reflection reaction of an interferometric synthetic aperture radar (InSAR) on flat roads (asphalt roads, gravel roads, cement roads), which is different from the interferometric imaging effects of other objects, is used. The semantic segmentation method of a deep learning model (such as Deeplabv3) is used to segment the road area of the InSAR feature image to generate a road mask, and an image edge detection algorithm (such as the sobel algorithm) is used to extract the boundary of the road mask. Finally, the obtained road area and road boundary are vectorized through vectorization processing. The obtained road boundary is used to distinguish road information from other non-road information.

[0060] Further, as a specific implementation manner of the present invention, in step S1, satellite sensors are used to obtain aerial information of the ground environment, and road information is extracted from the aerial information, including:

[0061] High-precision multi-spectral images are used to extract the road area, and lane lines and road surface symbols are detected through feature differences.

[0062] Specifically, different ground objects in high-precision multispectral images have different spectral characteristics, and the value ranges of the characteristics of many building facilities are similar in the visible light range. To prevent interference, the hyperspectral image of the road area can be extracted first according to the road mask generated in the previous embodiment. Then, within the road area, using the obvious differences between asphalt roads and lane lines, road surface symbols or texts, the center line of the lane lines can be detected using a lane line detection algorithm (such as the Ultra-Fast-Lane-Detection algorithm), and road surface symbols can be detected using an object detection algorithm (such as YOLOv7). Finally, the vector lane lines and the central point coordinates of the road surface symbols and their attributes such as the included names are returned.

[0063] Further, as a specific embodiment of the present invention, in step S1, using a satellite sensor to obtain the aerial information of the ground environment and extracting road information from the aerial information further includes:

[0064] Based on the periodic law and three-dimensional spatial information of remote sensing satellite images, splicing and merging multiple satellite images, unifying the resolution and performing spatial position matching;

[0065] Cutting the spliced and merged satellite images into map tiles of a fixed size;

[0066] And storing the vector data corresponding to the road information into a database with the map tiles as the basic unit.

[0067] Exemplarily, satellites A and B periodically acquire image data of the Earth's surface at different time points. First, the image data acquired by satellites A and B are spliced and merged, which involves correcting and registering the images to ensure their spatial alignment. Then, the resolution of the spliced image data is unified, and the resolution of the image can be adjusted to a unified level through methods such as interpolation. Finally, the spatial position matching of the satellite image data at a relatively recent time is performed, and the image data acquired at different time points are registered in space to ensure their positions on the Earth's surface are consistent. Further, referring to Figure 3 , cutting the matched image data into fixed-size map tiles. These map tiles can be square or rectangular to meet the needs of subsequent map processing and analysis.

[0068] Based on the above map tiles as the basic unit, the point, line, and surface vector data such as road areas, lane lines, road boundaries, and road surface symbols extracted from the remote sensing satellite information of this area are stored in the database. The functions of this database include data directly obtained from remote sensing satellites and data in some areas that are difficult to obtain by remote sensing images collected by the vehicle end (crowdsourcing). Most of these data are vector data, symbols, or texts, and the required storage capacity is small. The schematic diagram of vector data is asFigure 4 as shown

[0069] Further, as a specific embodiment of the present invention, in step S2, vehicle-end sensors are used to obtain vehicle-end information of the surrounding environment of the vehicle body, and obstacle information is extracted from the vehicle-end information, including: performing pre-fusion on radar point cloud data and visible light camera data to obtain a three-dimensional spatial feature information map; detecting the spatial feature information map to obtain obstacle information.

[0070] Specifically, a visible light camera can obtain the characteristics of the surface texture of an object, and the use of the visible light camera data can be divided into two methods. One is to only use the front view image to obtain traffic signs (TSR), traffic lights (TLR), etc. in front of the vehicle itself; the other is to generate a fused three-dimensional image information in the TPV three directions by using data conversion technology for the images of the front, rear, left front, left rear, right front, and right rear cameras of the vehicle end and the lidar point cloud data, and identify freespace, lane lines, moving obstacles, and static obstacles, etc. on the road within the visible range of the vehicle itself through segmentation or detection methods.

[0071] Although radar point clouds cannot obtain the texture of the target object like a visible light camera, they have more accurate three-dimensional information about obstacles on the road surface. Performing pre-fusion on the radar point cloud data and the visible light image to output a detection or segmentation result, and the output result is as Figure 5 shown. Among them, a block can be regarded as a set of multiple vector points. For moving or static obstacles, the stored points are the vector points of the 8 corner points of their minimum circumscribed cube.

[0072] Further, as a specific embodiment of the present invention, in step S3, fusing the road information and the obstacle information to obtain a road environment perception result, including:

[0073] One is to use satellite sensors to obtain the position information of the vehicle; specifically, a satellite positioning system (such as Beidou, GPS) can be used to obtain the global positioning information (latitude and longitude coordinates) of the vehicle, and convert the vehicle position information into coordinates in the vehicle's own coordinate system for subsequent integration with the road information;

[0074] The other is to obtain the surrounding road information at the corresponding position in the air-end information based on the position information; specifically, the vehicle position information can be used to retrieve the map tile data at the corresponding position in the air-end database. Extract relevant surrounding road information such as road areas, lane lines, and road boundaries from the map tile data;

[0075] Thirdly, integrate the surrounding road information and the obstacle information into the vehicle's ego-coordinate system to obtain the road environment perception result; convert the vehicle position information obtained from the satellite sensor into the ego-coordinate system, and perform coordinate transformation on the road information obtained from the air end and the obstacle information obtained from the vehicle end according to the vehicle position, and convert them into the ego-coordinate system. Integrate the converted road information and obstacle information together to form the road environment perception result around the vehicle.

[0076] By implementing the above steps, the road information obtained from the air end and the obstacle information obtained from the vehicle end can be integrated together, and the perception result of the road environment can be obtained based on the ego-coordinate system of the vehicle, so as to provide real-time and accurate environment perception data for the autonomous driving system.

[0077] Furthermore, as a specific implementation manner of the present invention, the road environment perception method based on ground-air multi-sensor fusion further includes:

[0078] Extract lane information from the vehicle end information, and improve the road information based on the lane information.

[0079] Specifically, the present invention can further use sensors (such as cameras, lidars, etc.) carried by the vehicle to obtain lane information around the vehicle, including lane lines, lane widths, lane markings, etc. Process and analyze the lane information around the vehicle, and extract key lane features. Integrate the lane information extracted from the vehicle end information with the road information previously obtained from the air end information. Correct and supplement the road information according to the lane information, such as correcting the road width, marking missing lane lines, updating road signs, etc. Integrate the improved result of the lane information with other road environment information together to form a more accurate and comprehensive road environment perception result.

[0080] Further, as an implementation manner of the present invention, step S4, extract lane information from the vehicle end information, and improve the road information based on the lane information, includes: judging the incomplete area with missing information in the road information through a connectivity algorithm; searching for road boundaries and / or lane lines with coordinates close to the position from the lane information based on the position information of the incomplete area; performing topological matching and merging on the incomplete area based on the road boundaries and / or lane lines to improve the road information.

[0081] Specifically, first, to ensure the timely update and processing of road information, a fixed period is set, for example, once a week or once a month for the improvement process of incomplete areas. Next, a connectivity algorithm is used to detect the connectivity of the road areas on the current map tile and its surrounding 8 map tiles. Depth-First Search (DFS) or Breadth-First Search (BFS) can be used to find the connected road areas. Map tiles with poor connectivity are defined as incomplete areas, and based on the location information of the incomplete areas, road boundaries and / or lane lines with similar coordinates are searched from the lane information. After searching for road information at similar coordinate positions, a topological matching algorithm is used to determine the best matching result. The topological matching algorithm can find the most similar road information according to the topological structure and spatial relationship of the road network. According to the result of topological matching, the searched road information is merged with the incomplete part on the current tile in terms of area or line segment. This can be completed through geometric calculations and spatial analysis to ensure that the merged road information matches the surrounding environment and meets the requirements of road continuity and integrity.

[0082] Further, as a specific implementation manner of the present invention, step S4, extracting lane information from the vehicle-end information and improving the road information based on the lane information, further includes: outputting the merge result and obtaining the verification information manually input for the result output, and based on the verification information, storing the map tiles with correct merge into the database.

[0083] Specifically, the merged road information is output for manual verification. This can be presented to the operator through a visualization tool or other interfaces. The operator can view the merged road information and provide verification feedback on the result. According to the feedback information of manual verification, the map tiles with correct merge in the merge result are stored in the database. These correct tiles contain the improved and completed road information and can be used as updated map data for subsequent use.

[0084] By adding the step of manual verification and storing the correct map tiles in the database, repeated operations on the same area can be avoided in the future. Once the map tiles are verified and stored in the database after manual verification, the system can record this information to avoid reprocessing the tiles that have been processed in the next processing cycle. This can save system resources and time, improve processing efficiency, and ensure the consistency and accuracy of road information.

[0085] The present disclosure discloses a road environment perception method based on ground-air multi-sensor fusion. This method is independent of high-definition maps (HDMaps) and directly uses the images generated by satellite sensors to directly synthesize the road environment information on the ground at the air end. Utilizing the characteristics of periodic satellite acquisitions, it updates the extraction of road and surface information in a short cycle and manages this information. When the vehicle is in motion, it outputs in real time the environmental information within a certain range of the vehicle's position. At the same time, it fuses with the real-time obstacle and free space results detected by the vehicle-end sensors and outputs the perceived results to the downstream modules of autonomous driving. The satellite sensors described in the present invention mainly include air sensors such as the Beidou positioning system (COMPASS), high-resolution multi-spectral satellite sensors, and interferometric synthetic aperture radar sensors. The vehicle-end sensors mainly include in-vehicle sensors such as visible light cameras (Cameras) and lidars (Lidars).

[0086] As Figure 6 shown, the present invention also provides a road environment perception system based on ground-air multi-sensor fusion, including:

[0087] An air-end data processing module, at least for obtaining the air-end information of the ground environment using satellite sensors and extracting road information from the air-end information;

[0088] A vehicle-end data processing module, at least for obtaining the vehicle-end information of the environment around the vehicle body using vehicle-end sensors and extracting obstacle information from the vehicle-end information;

[0089] Wherein, the vehicle-end data processing module is further used for fusing the road information and the obstacle information to obtain a road environment perception result.

[0090] Further, as a specific implementation manner of the present invention, the air-end data processing module is further used for extracting lane information from the vehicle-end information and improving the road information based on the lane information.

[0091] Further, as a specific implementation manner of the present invention, the system further includes: an air-end data management module, at least for storing the vector data corresponding to the road information in a database.

[0092] Specifically, in the airborne data processing module, the high-resolution multispectral image and the interferometric synthetic aperture radar data are first subjected to spatio-temporal synchronization processing to match the two data sources according to the same spatial position and close acquisition time. Then, the two synchronized data are processed separately, and the resulting vector data is stored in the airborne database and managed by the airborne data management module. At the same time, the airborne data management module receives the supplementary data information transmitted from the vehicle end through the airborne data transmission module, in the format of vectors and text symbols. The data corresponding to the same area at the airborne and vehicle ends is placed in the spatial data processing module for improvement, and the processed results are transmitted back to the airborne data management module for storage, facilitating data transmission to the vehicle end next time.

[0093] In the vehicle-end data processing module, the visible light camera and the lidar point cloud data are synchronized spatio-temporally according to the sensor characteristics. Then, a part of the synchronized data is subjected to pre-fusion processing of the lidar point cloud and the visible light image to generate a pre-fusion feature map for processing. And the forward-looking camera is used to detect traffic lights and traffic signs. At the same time, the real-time position information of the vehicle itself is obtained through the vehicle-end real-time positioning system RTK and transmitted to the airborne end through the vehicle-end data transmission module to obtain the map tile vector information of the airborne end. The vehicle end transmits the vehicle-end perception result to the downstream of autonomous driving through the vehicle-end data processing module based on the transmitted vector information and the results obtained at the vehicle end. And some detection information at the vehicle end, such as traffic signs, is transmitted back to the airborne end to facilitate the improvement of the airborne end information.

[0094] Furthermore, referring to Figure S7, in the airborne data processing module, the high-resolution multispectral image is subjected to image segmentation processing to extract the road surface signs (TGS) of the traffic road, such as lane lines (Lane), zebra crossings, numbers or symbols, etc. The interferometric synthetic aperture radar separates the road (Road) and the road surrounding environment (Envirenment) by using the special diffuse reflection of the road / pavement, and finally the results are sent to the airborne data management module. In the vehicle-end data processing module, the lidar point cloud and the visible light camera are pre-fused to obtain a three-dimensional spatial feature information map, and then the drivable area (freespace) and obstacles (Obstacle) of the road are obtained by performing segmentation and detection respectively using these feature information maps. RTK obtains the real-time position information (Location) of the vehicle itself through the satellite positioning system. The forward-looking visible light image detects traffic lights (TLR) and traffic signs (TSR), and finally the results are sent to the airborne data management module.

[0095] Compared with single vehicle-end perception, the system described in the present invention provides more environmental information. Although the update frequency of the environmental information is lower than that of existing high-precision map products, it is sufficient to meet the vehicle-end's perception requirements for road conditions. Compared with a single satellite positioning solution, it obtains more road surface information from a top-down perspective and three-dimensional space information around, and is more likely to extract road environment information by using the fusion features of multi-spectral sensors and interferometric synthetic aperture radar.

[0096] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned road environment perception method based on ground-air multi-sensor fusion is implemented.

[0097] It can be understood that the computer-readable storage medium may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), and a software distribution medium, etc. The computer program includes computer program code. The computer program code can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), and a software distribution medium, etc.

[0098] In some embodiments of the present invention, the electronic device may include a controller or a processor. The controller is a single-chip microcomputer chip that integrates a processor, a memory, a communication module, etc. The processor may refer to the processor included in the controller. The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0099] Any process or method description depicted in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code that includes one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0100] Those of ordinary skill in the art will recognize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans can use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the present invention.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road environment perception method based on ground-air multi-sensor fusion, characterized in that: include: Using satellite sensors to obtain air-side information of the ground environment, and extracting road information from the air-side information. When using satellite sensors to obtain air-side information, using satellite remote sensing data, using image processing technology and computer vision algorithms, the remote sensing data is processed to extract road information; Using vehicle-side sensors to obtain vehicle-side information about the environment around the vehicle body, and extracting obstacle information from the vehicle-side information; fusing the road information and the obstacle information to obtain a road environment perception result; and, Extracting lane information from the vehicle-side information, and completing the road information based on the lane information; The extracting lane information from the vehicle-side information and improving the road information based on the lane information includes: Determine the incomplete area where the information is missing in the road information by a connectivity algorithm; Based on the position information of the defective area, searching the lane information for a road boundary and / or lane line with similar coordinates; The incomplete area is topologically matched and merged based on the road boundary and / or lane line to improve the road information.

2. The road environment perception method based on ground-air multi-sensor fusion according to claim 1 is characterized in that: The satellite sensor includes at least one of a satellite positioning sensor, a high-resolution multi-spectral satellite sensor, and an interferometric synthetic aperture radar.

3. The road environment perception method based on ground-air multi-sensor fusion according to claim 2 is characterized in that: The method of acquiring air-side information of a ground environment by using a satellite sensor and extracting road information from the air-side information includes: The interferometric synthetic aperture radar is used to segment the road area, and the road boundary is obtained using the boundary extraction algorithm.

4. The road environment perception method based on ground-air multi-sensor fusion according to claim 2 is characterized in that: The method of acquiring air-side information of a ground environment by using a satellite sensor and extracting road information from the air-side information includes: High-precision multispectral images are used to extract road areas, and lane lines and road surface symbols are detected through feature differences.

5. The road environment perception method based on ground-air multi-sensor fusion according to claim 3 or 4 is characterized in that: The method of acquiring air-side information of the ground environment by using a satellite sensor and extracting road information from the air-side information further includes: Based on the periodicity and three-dimensional spatial information of remote sensing satellite images, multiple satellite images are spliced ​​and merged to unify the resolution and perform spatial position matching; Crop the stitched and merged satellite images into map tiles of fixed size; And, taking the map tile as a basic unit, the vector data corresponding to the road information is stored in a database.

6. The road environment perception method based on ground-air multi-sensor fusion according to claim 1 is characterized in that: The method of acquiring vehicle-side information of the surrounding environment of the vehicle body by using the vehicle-side sensor and extracting obstacle information from the vehicle-side information includes: The radar point cloud data and the visible light camera data are forward-fused to obtain a three-dimensional spatial feature information map; Obstacle information is obtained by detecting the spatial feature information map.

7. The road environment perception method based on ground-air multi-sensor fusion according to claim 1 is characterized in that: The fusing the road information and the obstacle information to obtain a road environment perception result includes: Use satellite sensors to obtain vehicle location information; Based on the location information, obtain surrounding road information at a corresponding position in the air terminal information; The surrounding road information and the obstacle information are integrated into the vehicle's own vehicle coordinate system to obtain the road environment perception result.

8. A road environment perception system based on ground-air multi-sensor fusion, applied to the method according to any one of claims 1 to 7, characterized in that: include: The air-side data processing module is at least used to obtain air-side information of the ground environment using a satellite sensor and extract road information from the air-side information. When obtaining the air-side information using the satellite sensor, the remote sensing data is processed using satellite remote sensing data and image processing technology and computer vision algorithms to extract road information. A vehicle-side data processing module, at least used to obtain vehicle-side information of the vehicle body surrounding environment using a vehicle-side sensor, and extract obstacle information from the vehicle-side information; Among them, the vehicle-side data processing module is also used to fuse the road information and the obstacle information to obtain a road environment perception result.

9. The road environment perception system based on ground-air multi-sensor fusion according to claim 8 is characterized in that: The air-side data processing module is also used to extract lane information from the vehicle-side information and improve the road information based on the lane information.

10. The road environment perception system based on ground-air multi-sensor fusion according to claim 8 is characterized in that: The system further comprises: The air-side data management module is at least used to store the vector data corresponding to the road information into a database.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the road environment perception method based on ground-to-air multi-sensor fusion as described in any one of claims 1 to 7 is executed.

Images