Automatic driving vehicle positioning method and device based on vision assistance

Abstract

The present application discloses a method and device for positioning an autonomous driving vehicle based on vision assistance, wherein the method includes establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include feature elements on the roadside and / or road surface during the driving process of the autonomous driving vehicle; performing visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information; based on a fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by the visual odometer on the autonomous driving vehicle. This application improves the stability of vehicle positioning in special scenarios, and is not affected by whether a point cloud map exists.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method and device for positioning an autonomous driving vehicle based on vision assistance. Background Art

[0002] With the development of computer vision technology, visual positioning is increasingly being used in the positioning of autonomous vehicles, the most important of which is visual SLAM technology.

[0003] As an auxiliary positioning method, visual SLAM technology uses feature points extracted from images to pre-establish a feature point cloud map as a reference. When an autonomous driving vehicle subsequently passes through an area with a preset point cloud map, it can use feature point matching technology to calculate the vehicle's current position in the map and convert this position into absolute longitude and latitude as observation information for multi-sensor fusion positioning.

[0004] However, due to the influence of lighting, dynamic objects, and other factors, related technologies can easily cause feature point tracking and association loss during actual mapping and positioning. Secondly, the storage and import of maps require a large amount of resources. If the autonomous vehicle's driving area is large, large-area point cloud maps often have large errors, and feature point matching calculations are subject to unpredictable delays. In addition, due to the influence of vehicle speed, current visual SLAM technology has not been well applied to a wide range of scenarios and is limited to scenarios such as automatic parking in parking lots and closed campus distribution. Summary of the Invention

[0005] The embodiments of the present application provide a method and device for positioning an autonomous driving vehicle based on vision assistance to improve the stability of vehicle positioning in special scenarios.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In the first aspect, an embodiment of the present application provides a method for positioning an autonomous driving vehicle based on vision assistance, wherein the method includes: establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include feature elements on the roadside and / or road surface during the driving of the autonomous driving vehicle; performing visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information; based on a fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by the visual odometry on the autonomous driving vehicle.

[0008] In some embodiments, based on the fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by using the visual odometry on the autonomous driving vehicle includes:

[0009] Based on the Kalman filter fusion positioning framework, the GNSS information of the autonomous driving vehicle is used as the main observation value, and the image positioning information of the autonomous driving vehicle is used as the auxiliary observation value.

[0010] In some embodiments, the vehicle positioning based on the fusion positioning framework by matching the target map information, or the vehicle positioning by the visual odometry on the autonomous driving vehicle, further includes:

[0011] If the target map information is available in the current area and the current position data of the autonomous driving vehicle is available, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected;

[0012] If there is no target map information in the current area and the relative displacement can be calculated by the visual odometry assisted by the vehicle wheel speed, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected. The target map information includes point cloud map data.

[0013] In some embodiments, establishing a first feature descriptor point pair set based on preset feature points includes:

[0014] capturing, by a wide-angle fisheye camera mounted on the autonomous vehicle, characteristic elements on the roadside and / or road surface during the autonomous vehicle's driving process as preset characteristic points;

[0015] Performing semantic segmentation processing on the preset feature points, and classifying the semantic segmentation processing results to extract road surface elements and / or static semantic elements on the road;

[0016] The corner point information of the road surface elements and / or static semantic elements on the road is obtained to obtain feature descriptors, and a first feature descriptor point pair set is established.

[0017] In some embodiments, obtaining corner point information of the road surface elements and / or static semantic elements on the road to obtain feature descriptors and establishing a first feature descriptor point pair set includes:

[0018] According to the vehicle speed and vehicle heading angle change rate information, the displacement relationship between the collected feature elements and any two frames is calculated;

[0019] The position interval of the feature descriptor in the image is determined according to the displacement relationship, and data association is performed in combination with the corner point information of the road surface elements and / or static semantic elements on the road to obtain a first feature descriptor point pair set.

[0020] In some embodiments, based on a fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by using a visual odometry on the autonomous driving vehicle, includes:

[0021] If the target map information and / or the matched feature descriptor point pair set are pre-loaded, the vehicle is positioned by matching the target map information;

[0022] If there is no pre-loaded target map information and / or no matching feature descriptor point pair set, and the vehicle pose information cannot be obtained through relocalization, the visual odometry is used to perform vehicle localization.

[0023] In some embodiments, performing visual SLAM mapping to obtain target map information based on the first feature descriptor point pair set and the second feature descriptor point pair set includes:

[0024] In the process of combining the first feature descriptor point pair set with the second feature descriptor point pair set, a first weight is pre-assigned to the feature descriptor points in the first feature descriptor point pair set, and a second weight is pre-assigned to the feature descriptor points in the first feature descriptor point pair set, wherein the first weight is greater than the second weight;

[0025] The first feature descriptor point pair set and the second feature descriptor point pair set are combined and input into the original visual SLAM point cloud map building framework, and visual SLAM mapping is performed to obtain target map information.

[0026] In the second aspect, an embodiment of the present application also provides an autonomous driving vehicle positioning device based on vision assistance, wherein the device includes: a feature descriptor establishment module, used to establish a first feature descriptor point pair set based on preset feature points and a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving of the autonomous driving vehicle; a mapping module, used to perform visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information; a positioning module, used to perform vehicle positioning by matching the target map information based on a fusion positioning framework, or to perform vehicle positioning by the visual odometer on the autonomous driving vehicle.

[0027] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, enable the processor to perform the above method.

[0028] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0029] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0030] By establishing a first feature descriptor point pair set based on preset feature points and a second feature descriptor point pair set based on visual SLAM, visual SLAM mapping is then performed based on the first and second feature descriptor point pairs to obtain target map information. During positioning, vehicle positioning is performed based on a fusion positioning framework by matching the target map information, or using the visual odometry on the autonomous vehicle, thereby improving positioning stability and eliminating reliance on feature point cloud maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 This is a flow chart of a method for positioning an autonomous driving vehicle based on vision assistance in an embodiment of the present application;

[0033] Figure 2 This is a schematic diagram of the structure of a vision-assisted autonomous driving vehicle positioning device in an embodiment of the present application;

[0034] Figure 3 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] The inventors have discovered that the point cloud map pre-established through SLAM often requires more resources and is often affected by lighting, dynamic objects, etc., which may cause feature points to be lost in the point cloud map.

[0037] In response to the above-mentioned shortcomings, an embodiment of the present application provides a method for positioning an autonomous driving vehicle based on vision assistance, which uses feature points on highly discriminative, (relatively) static objects to make up for the feature points randomly selected by the SLAM framework in the related art for data association. Furthermore, it can be ensured that there are a stable number of feature point pairs between the two frames of images to calculate the posture, thereby preventing tracking failures caused by too few feature points. Preferably, the posture calculated by combining the wheel speed with the visual odometer is also input into the multi-sensor fusion filtering framework as additional observation information, which is not affected by the presence or absence of a map or reduces the impact of the presence or absence of a map.

[0038] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0039] The present application provides a method for positioning an autonomous driving vehicle based on vision assistance. Figure 1 As shown, a flow chart of a method for positioning an autonomous driving vehicle based on vision assistance in an embodiment of the present application is provided, wherein the method comprises at least the following steps S110 to S130:

[0040] Step S110: establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include feature elements on the roadside and / or road surface during the driving process of the autonomous driving vehicle.

[0041] Based on the preset feature point selection rules, as the autonomous vehicle drives, as many feature elements as possible are collected from both sides of the road (roadside) and the road surface. Multiple feature descriptors are derived from these designated / specific feature elements, and a first feature descriptor point pair set is established based on the multiple feature descriptors.

[0042] It can be understood that the plurality of feature descriptors are obtained based on designated or specific feature elements, which are more targeted than randomly generated feature elements.

[0043] In some embodiments, the characteristic elements include, but are not limited to, lane lines, road arrows, signboards, buildings, and other relatively static and distinct elements.

[0044] At the same time, according to the random feature point selection method used in related technologies, a second feature descriptor point pair set is established based on visual SLAM. Feature points are then selected through manually specified rules combined with the feature point random selection mechanism in the existing algorithm framework to ensure the quantity and quality of feature points.

[0045] Step S120: Perform visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information.

[0046] The obtained first feature descriptor point pair set and the second feature descriptor point pair set are input into a visual SLAM framework in related technology for mapping, thereby obtaining target map information.

[0047] It should be noted that the target map information here refers to the establishment within a specified area and does not limit the scene (it is not limited to scenes such as automatic parking in parking lots, closed park delivery, etc.).

[0048] Step S130: Based on the fusion positioning framework, the vehicle is positioned by matching the target map information, or the vehicle is positioned by the visual odometry on the autonomous driving vehicle.

[0049] The autonomous vehicle is positioned based on the target map information. The positioning process is based on a fusion positioning framework, whereby the positioning state is estimated through prediction and observation updates.

[0050] Furthermore, if there is a matching target map, a direct match can be performed to obtain the vehicle's current positioning information. If no matching target map information exists, the autonomous vehicle's visual odometry is used to locate the vehicle, while the positioning information is input as auxiliary observations. The combination of visual odometry and visual SLAM eliminates the need for pre-built point cloud maps.

[0051] The method in the embodiment of the present application provides additional observation information on the basis of matching only through road surface information in related technologies, which can improve the stability of the algorithm in special scenarios and reduce manual takeover when the positioning accuracy decreases due to insufficient information.

[0052] In one embodiment of the present application, based on a fusion positioning framework, vehicle positioning is performed by matching the target map information, or vehicle positioning is performed by the visual odometry on the autonomous driving vehicle, including: based on a Kalman filter fusion positioning framework, the GNSS information of the autonomous driving vehicle is used as the main observation value, and the image positioning information of the autonomous driving vehicle is used as the auxiliary observation value.

[0053] In specific implementation, when using target map information and feature descriptors for positioning, Kalman filtering is used as a fusion framework. GNSS is used as the main observation value and image positioning is used as an auxiliary observation value to prevent positioning deviation caused by GNSS signal instability / loss.

[0054] It should be noted that the positioning process requires the use of positioning data from the IMU+RTK / GNSS positioning module, vehicle body data (including vehicle speed and angular velocity change rate), point cloud map information, i.e. target map information, and image information (feature descriptors, feature points randomly selected by visual SLAM in related technologies).

[0055] In one embodiment of the present application, the vehicle positioning is performed by matching the target map information based on the fusion positioning framework, or the vehicle positioning is performed by the visual odometry on the autonomous driving vehicle, and further includes: when the target map information is available in the current area and the current posture data of the autonomous driving vehicle can be obtained, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the bias of the GNSS is corrected; when there is no target map information in the current area and the relative displacement can be calculated by the visual odometry assisted by the vehicle wheel speed, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the bias of the GNSS is corrected, and the target map information includes point cloud map data.

[0056] In specific implementation, when the target map information is available in the current area and the current posture data of the autonomous driving vehicle can be obtained (the current area belongs to the area where the posture with high confidence is calculated through repositioning, that is, the posture data is available), the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the bias of the GNSS is corrected.

[0057] In another case, if there is no target map information in the current area and the relative displacement can be calculated by the visual odometry (VO) assisted by the vehicle wheel speed, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected. The target map information includes point cloud map data.

[0058] It can be understood that the image positioning information here includes but is not limited to feature points randomly obtained based on an open source visual SLAM algorithm and image positioning information obtained by combining preset feature points and calculation.

[0059] In one embodiment of the present application, establishing a first feature descriptor point pair set based on preset feature points includes: collecting feature elements on the roadside and / or road surface during the driving of the autonomous driving vehicle through a camera installed on the autonomous driving vehicle as preset feature points; performing semantic segmentation processing on the preset feature points, and classifying the semantic segmentation processing results to extract road surface elements and / or static semantic elements on the road; obtaining corner point information in the road surface elements and / or static semantic elements on the road to obtain feature descriptors, and establishing a first feature descriptor point pair set.

[0060] In specific implementation, at least one wide-angle fisheye camera is used as an image acquisition device to ensure that the vehicle has a sufficiently large field of view to capture as many outlines of buildings on both sides of the road as possible during driving. Then, semantic segmentation is performed on the acquired image, and the results of semantic segmentation are classified to extract road elements such as arrows, lane lines, etc., as well as roadside static semantic elements such as signs and buildings. Finally, the corner point information of the above semantic elements is extracted (such as the four corner point information of a rectangular road sign). Since the roadside static semantic elements are on both sides of the image, have regular shapes and are highly distinguishable from the background (usually the sky is used as the image background), the feature descriptor of the corner information is calculated, such as the orb operator, and then the feature descriptor is generated to establish a feature descriptor point pair set.

[0061] In one embodiment of the present application, the acquiring of corner point information in the road surface elements and / or static semantic elements on the road to obtain feature descriptors and establishing a first feature descriptor point pair set includes: calculating the displacement relationship between the collected feature elements between any two frames based on the vehicle speed and vehicle body heading angle change rate information; determining the position interval of the feature descriptor in the image based on the displacement relationship, and performing data association in combination with the corner point information in the road surface elements and / or static semantic elements on the road to obtain the first feature descriptor point pair set.

[0062] In specific implementations, semantic element templates can be generated simultaneously with feature descriptors as optimization criteria for pose calculation. Specifically, when calculating the optimized pose, pixels from the same or multiple element templates in two frames should overlap as much as possible. It is understood that semantic templates can include traffic lights, signboards, and other such examples, and this is not specifically limited in this application.

[0063] Furthermore, based on the current vehicle speed and the rate of change of the vehicle angular velocity, the displacement relationship between the two frames of images is calculated. Based on this displacement relationship, the position interval of the extracted feature descriptors in the image is calculated, and combined with the corner point positions calculated by semantic segmentation, data association is performed to obtain the first feature descriptor point pair set S1.

[0064] When calculating the displacement relationship between two frames of images, since the driving area of the autonomous vehicle is relatively smooth and there are relatively few bumps, it can be approximated that the change in the height direction is very small.

[0065] In one embodiment of the present application, based on a fusion positioning framework, vehicle positioning is performed by matching the target map information, or by using a visual odometry on the autonomous driving vehicle, including: performing vehicle positioning according to the feature descriptor point pair set and / or the target map information; if the target map information is preloaded and / or there is a matching feature descriptor point pair set, vehicle positioning is performed by matching the target map information; if the target map information is not preloaded and / or there is no matching feature descriptor point pair set, and the vehicle posture information cannot be obtained through repositioning, the visual odometry is used to perform vehicle positioning.

[0066] In specific implementations, if target map information is available, map matching is performed directly. If pose information cannot be obtained through relocalization due to reasons such as lack of target map information or map matching failure, the results of semantic visual odometry (VO) are directly used. The actual calculation also uses both the first feature descriptor point pair set S1 and the second feature descriptor point pair set S2.

[0067] It is understandable that the visual odometry (VO) is always running during the above process and will not be run again after the target map information matching fails.

[0068] In one embodiment of the present application, visual SLAM mapping is performed based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information, including: in the process of combining the first feature descriptor point pair set with the second feature descriptor point pair set, the feature descriptor points in the first feature descriptor point pair set are pre-assigned a first weight and the feature descriptor points in the first feature descriptor point pair set are pre-assigned a second weight, and the first weight is greater than the second weight; the first feature descriptor point pair set and the second feature descriptor point pair set are combined and input into the original visual SLAM point cloud map establishment framework to perform visual SLAM mapping to obtain target map information.

[0069] First, the existing SLAM framework can be used to calculate the second feature descriptor point pair set S2.

[0070] After that, the map is constructed by combining the first feature descriptor point pair set S1 and the second feature descriptor point pair set S2 and inputting them into the existing SLAM framework for mapping. The existing SLAM framework is not an improvement point of this application and will not be described in detail here.

[0071] Furthermore, when combining the first feature descriptor point pair set S1 with the second feature descriptor point pair set S2, the feature descriptor point pairs in S1 are assigned a higher weight because the corner point information in S1 is stable and highly discriminative on static objects. The first and second feature descriptor point pairs are then combined and input into the original visual SLAM framework to create a point cloud map, obtaining target map information. Assigning different weights to different feature points ensures that stable and easily identifiable feature point pairs in the image are more likely to be selected for pose calculation.

[0072] The embodiment of the present application also provides a vision-assisted autonomous driving vehicle positioning device 200, such as Figure 2 As shown, a schematic diagram of the structure of a vision-assisted autonomous vehicle positioning device in an embodiment of the present application is provided. The vision-assisted autonomous vehicle positioning device 200 includes at least: a feature descriptor establishment module 210, a mapping module 220, and a positioning module 230, wherein:

[0073] In one embodiment of the present application, the feature descriptor establishment module 210 is specifically used to: establish a first feature descriptor point pair set based on preset feature points and establish a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving process of the autonomous driving vehicle.

[0074] Based on the preset feature point selection rules, as the autonomous vehicle drives, it collects as many characteristic elements of the road and the road surface as possible. Multiple feature descriptors are derived from these specified / specific feature elements, and a feature descriptor point pair set is established.

[0075] In some embodiments, the characteristic elements include, but are not limited to, lane lines, road arrows, signboards, buildings, and other relatively static and distinct elements.

[0076] At the same time, according to the method of randomly selecting feature points in the related art, a second feature descriptor point pair set is established based on visual SLAM.

[0077] In one embodiment of the present application, the mapping module 220 is specifically configured to perform visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information.

[0078] The obtained first feature descriptor point pair set and the second feature descriptor point pair set are input into a visual SLAM framework in related technology for mapping, thereby obtaining target map information.

[0079] It should be noted that the target map information here refers to the establishment within a specified area and does not limit the scene (it is not limited to scenes such as automatic parking in parking lots, closed park delivery, etc.).

[0080] In one embodiment of the present application, the positioning module 230 is specifically used to: perform vehicle positioning by matching the target map information based on a fusion positioning framework, or perform vehicle positioning through the visual odometry on the autonomous driving vehicle.

[0081] Based on the target map information, the autonomous vehicle is positioned. During the positioning process, it is based on a fusion positioning framework, which takes the relevant observation values as input and obtains the predicted value.

[0082] Furthermore, if there is target map information that can be matched, a direct match can be performed to obtain the vehicle's current positioning information. If there is no target map information that can be matched, the autonomous vehicle's visual odometry is used to locate the vehicle, and the relevant feature descriptors are input as auxiliary observation values.

[0083] It can be understood that the above-mentioned vision-assisted autonomous driving vehicle positioning device can implement the various steps of the vision-assisted autonomous driving vehicle positioning method provided in the aforementioned embodiment. The relevant explanations about the vision-assisted autonomous driving vehicle positioning method are applicable to the vision-assisted autonomous driving vehicle positioning device and will not be repeated here.

[0084] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0085] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0086] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0087] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a vision-assisted autonomous vehicle positioning device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0088] Establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving of the autonomous driving vehicle;

[0089] Performing visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information;

[0090] Based on the fusion positioning framework, vehicle positioning is performed by matching the target map information, or by using the visual odometry on the autonomous driving vehicle.

[0091] The above application Figure 1The method performed by the vision-assisted autonomous vehicle positioning device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0092] The electronic device may also perform Figure 1 The method of executing the positioning device of the autonomous driving vehicle based on vision assistance in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0093] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 The method performed by the vision-assisted autonomous driving vehicle positioning device in the illustrated embodiment is specifically used to perform:

[0094] Establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving of the autonomous driving vehicle;

[0095] Performing visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information;

[0096] Based on the fusion positioning framework, vehicle positioning is performed by matching the target map information, or by using the visual odometry on the autonomous driving vehicle.

[0097] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0102] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0103] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0105] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A vision-assisted autonomous vehicle positioning method, wherein: The method comprises: Establishing a first feature descriptor point pair set based on preset feature points and establishing a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving of the autonomous driving vehicle; Performing visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information; Based on a fusion positioning framework, the vehicle is positioned by matching the target map information, or the vehicle is positioned by a visual odometry on the autonomous driving vehicle; Based on the fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by using the visual odometry on the autonomous driving vehicle includes: Based on the Kalman filter fusion positioning framework, the GNSS information of the autonomous driving vehicle is used as the main observation value, and the image positioning information of the autonomous driving vehicle is used as the auxiliary observation value; If the target map information is available in the current area and the current position data of the autonomous driving vehicle is available, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected; If there is no target map information in the current area and the relative displacement can be calculated by the visual odometry assisted by the vehicle wheel speed, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected. The target map information includes point cloud map data.

2. The method according to claim 1, wherein: The step of establishing a first feature descriptor point pair set based on preset feature points includes: Capturing, by a camera installed on the autonomous vehicle, characteristic elements on the roadside and / or road surface during the driving of the autonomous vehicle as preset characteristic points; Performing semantic segmentation processing on the preset feature points, and classifying the semantic segmentation processing results to extract road surface elements and / or static semantic elements on the road; The corner point information of the road surface elements and / or static semantic elements on the road is obtained to obtain feature descriptors, and a first feature descriptor point pair set is established.

3. The method according to claim 2, wherein: The step of obtaining corner point information of the road surface elements and / or static semantic elements on the road to obtain feature descriptors and establishing a first feature descriptor point pair set includes: According to the vehicle speed and vehicle heading angle change rate information, the displacement relationship between the collected feature elements and any two frames is calculated; The position interval of the feature descriptor in the image is determined according to the displacement relationship, and data association is performed in combination with the corner point information of the road surface elements and / or static semantic elements on the road to obtain a first feature descriptor point pair set.

4. The method according to claim 1, wherein: Based on the fusion positioning framework, the vehicle is positioned by matching the target map information, or the vehicle is positioned by using the visual odometry on the autonomous driving vehicle, including: If the target map information and / or the matched feature descriptor point pair set are pre-loaded, the vehicle is positioned by matching the target map information; If there is no pre-loaded target map information and / or no matching feature descriptor point pair set, and the vehicle pose information cannot be obtained through relocalization, the visual odometry is used to perform vehicle localization.

5. The method of claim 1, wherein: Performing visual SLAM mapping to obtain target map information based on the first feature descriptor point pair set and the second feature descriptor point pair set includes: In the process of combining the first feature descriptor point pair set with the second feature descriptor point pair set, a first weight is pre-assigned to the feature descriptor points in the first feature descriptor point pair set, and a second weight is pre-assigned to the feature descriptor points in the first feature descriptor point pair set, wherein the first weight is greater than the second weight; The first feature descriptor point pair set and the second feature descriptor point pair set are combined and input into the original visual SLAM point cloud map building framework, and visual SLAM mapping is performed to obtain target map information.

6. A vision-assisted autonomous vehicle positioning device, wherein: The device comprises: A feature descriptor establishment module is configured to establish a first feature descriptor point pair set based on preset feature points and a second feature descriptor point pair set based on visual SLAM, wherein the preset feature points include characteristic elements on the roadside and / or road surface during the driving of the autonomous vehicle; A mapping module, configured to perform visual SLAM mapping based on the first feature descriptor point pair set and the second feature descriptor point pair set to obtain target map information; a positioning module, configured to perform vehicle positioning by matching the target map information based on a fusion positioning framework, or by using a visual odometry on the autonomous driving vehicle; Based on the fusion positioning framework, performing vehicle positioning by matching the target map information, or performing vehicle positioning by using the visual odometry on the autonomous driving vehicle includes: Based on the Kalman filter fusion positioning framework, the GNSS information of the autonomous driving vehicle is used as the main observation value, and the image positioning information of the autonomous driving vehicle is used as the auxiliary observation value; If the target map information is available in the current area and the current position data of the autonomous driving vehicle is available, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected; If there is no target map information in the current area and the relative displacement can be calculated by the visual odometry assisted by the vehicle wheel speed, the image positioning information is input into the Kalman filter fusion positioning framework as an auxiliary observation value, and the GNSS bias is corrected. The target map information includes point cloud map data.

7. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing one or more programs, wherein when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device executes the method according to any one of claims 1 to 5.

Images