System and method for feature matching to associate vehicle perception data

Abstract

A method and system for feature matching to associate vehicle-aware data for crowdsourced map data. The system is configured to perform the method, the method comprising the steps of: receiving perception data from a plurality of vehicles; processing the perceptual data to identify a first feature point observed by the first vehicle and a second feature point observed by the second vehicle, respectively; creating a feature descriptor for each of the first feature points and each of the second feature points using a multi-layer bitmap; determining a first feature point pair and a second feature point pair which are matched by comparing the histograms of the feature descriptors; associating the matched first feature point pair and second feature point pair; updating a map database by using the first feature point pair and the second feature point pair which are associated and matched; and operating the autonomous vehicle according to the updated map database.

Description

Technical Field

[0001] This disclosure generally relates to autonomous vehicles with perception systems, and more specifically, to a system and method for feature matching to correlate vehicle perception data collected by the perception systems. Background Technology

[0002] Modern vehicles capable of partial or full automation of driving are generally referred to as autonomous vehicles. Such autonomous vehicles have intelligent systems, also known as smart systems, such as Advanced Driver Assistance Systems (ADAS) and / or Automated Driving Systems (ADS) located on the vehicle and used to enhance or automate the functions of various vehicle systems.

[0003] The intelligent system has one or more control modules configured to receive and process information collected by external vehicle sensors. External vehicle sensors include, but are not limited to, cameras, radar, and lidar, which may be mounted on the exterior of the vehicle or inside the vehicle's interior and are configured to collect information about the vehicle's external surroundings. The collected information is also referred to as perception data. The one or more control modules process the perception data to detect and identify objects around the vehicle, including but not limited to surrounding vehicles, pedestrians, road configurations, traffic signs, landmarks, and road markings.

[0004] The detection and identification of objects are used by intelligent systems to partially or fully automate vehicles. Since perception data collected from a single vehicle equipped with an intelligent system is susceptible to sensor errors, it is possible to aggregate and process perception data collected by multiple vehicles equipped with intelligent systems for a section of road or area to achieve higher accuracy. Therefore, while current intelligent systems have achieved their intended purpose, a new and improved system and method are needed to perform feature matching to correlate perception data collected by multiple vehicles, thereby improving accuracy. Summary of the Invention

[0005] According to several aspects, a method for feature matching to associate vehicle perception data is provided. The method includes: receiving perception data from a plurality of vehicles, wherein the plurality of vehicles includes a first vehicle and a second vehicle; processing the perception data to identify a first feature point observed by the first vehicle and a second feature point observed by the second vehicle, respectively; creating a feature descriptor for each first feature point and each second feature point; comparing the feature descriptors for each first feature point and each second feature point to determine matching pairs of first and second feature points; associating the matching pairs of first and second feature points; updating a map database using the associated matching pairs of first and second feature points; and operating an autonomous vehicle based on the updated map database.

[0006] In an additional aspect of this disclosure, creating a feature descriptor for each of the first feature points and each of the second feature points includes: drawing the first feature point on a first bitmap and creating a first feature descriptor for each of the first feature points; and drawing the second feature point on a second bitmap and creating a second feature descriptor for each of the second feature points.

[0007] In another aspect of this disclosure, comparing the feature descriptors of each first feature point and each second feature point to determine a matching pair of first and second feature points further includes: applying a first spatial histogram to at least one of the first feature descriptors and applying a second spatial histogram to at least one of the second feature descriptors that is near at least one of the first feature descriptors; determining a matching first and second feature descriptors based on a predetermined feature distance threshold between the first and second spatial histograms; and determining a matching pair of first and second feature points based on the matching first and second feature descriptors.

[0008] In another aspect of this disclosure, at least one of the first bitmap and the second bitmap is a multi-layer bitmap. Each layer includes different attributes of the feature points.

[0009] In another aspect of this disclosure, determining a matching pair of first and second feature points further includes: identifying adjacent first and second feature points; determining a mismatched first and second feature descriptor based on the mutual exclusion property between the first and second spatial histograms; and filtering out the mismatched first feature descriptor.

[0010] In another aspect of this disclosure, before determining matching first feature points and second feature point pairs based on matching first and second feature descriptors, mismatched first feature descriptors are filtered out.

[0011] In another aspect of this disclosure, the method further includes determining an outlier pair of the matching first and second feature points; and filtering outlier pairs before associating the matching first and second feature point pairs.

[0012] In another aspect of this disclosure, determining the pair of matching exterior points of the first and second feature points includes: identifying a first observation region and a second observation region; randomly selecting a first pair of matching feature points in the first observation region and a second pair of matching feature points in the second observation region, wherein the first pair of matching feature points includes feature point P. 1,1 P 2,1 Furthermore, the feature points matched in the second pair of feature points include feature point P. 1,1 P 2,1 By translating P 2,1 With overlapping P 1,1 And fix the overlapping feature points P 1,1 P 2,1 To determine the translation vector; by using fixed overlapping feature points P 1,1 P 2,1 Simultaneous pivoting and rotation of feature point P 2,2 With overlapping P 1,2 To determine the angle of rotation; and to apply the translation vector and the angle of rotation to the remaining pairs of matched feature points.

[0013] In another aspect of this disclosure, the method also includes determining a score for the solution based on the number of interior points of the feature pair and the GPS variance from translation and rotation.

[0014] In another aspect of this disclosure, the method is executed by a controller located in one of the intelligent vehicle and a cloud-based server.

[0015] According to several aspects, a system for feature matching to correlate vehicle perception data is provided. The system includes: a server communication system configured to receive perception data from multiple vehicles; a map database; and a server controller in electrical communication with the server communication system and the map database. The server controller is configured to: receive perception data from the multiple vehicles, including a first vehicle and a second vehicle; process the perception data to identify a first feature point observed by the first vehicle and a second feature point observed by the second vehicle, respectively; create feature descriptors for each first feature point and each second feature point; determine matching pairs of first and second feature points, including comparing the feature descriptors of each first and second feature point; correlate the matching pairs of first and second feature points; update the map database using the correlated matching pairs of first and second feature points; and transmit the updated map database to at least one of the multiple vehicles for operation based on the updated map database.

[0016] In an additional aspect of this disclosure, creating a feature descriptor for each first feature point and each second feature point includes: drawing the first feature point on a first bitmap and creating a first feature descriptor for each first feature point; and drawing the second feature point on a second bitmap and creating a second feature descriptor for each second feature point. At least one of the first bitmap and the second bitmap is a multi-layer bitmap, and each layer includes different attributes of the feature points.

[0017] In another aspect of this disclosure, determining a matching pair of first and second feature points further includes: applying a first spatial histogram to at least one of the first feature descriptors and applying a second spatial histogram to at least one of the second feature descriptors that is near at least one of the first feature descriptors; determining a matching pair of the first and second feature descriptors based on a feature distance threshold between the first and second spatial histograms; and determining a matching pair of the first and second feature points based on the matching first and second feature descriptors.

[0018] In another aspect of this disclosure, determining a matching pair of first and second feature points further includes: identifying adjacent first and second feature points; determining a mismatched first and second feature descriptor based on the mutual exclusion property between the first and second spatial histograms; and filtering out the mismatched first feature descriptor.

[0019] In another aspect of this disclosure, the server controller is also configured to determine the outlier pairs of the matching first and second feature points; and to filter outlier pairs before associating the matching first and second feature point pairs.

[0020] In an additional aspect of this disclosure, determining the pair of exterior points of the matching first and second feature points includes: identifying a first observation region and a second observation region; selecting a first pair of matching feature points in the first observation region and a second pair of matching feature points in the second observation region, wherein the first pair of matching feature points includes feature point P. 1,1 P 2,1 And the second pair of matched feature points includes feature point P. 1,1 P 2,1 By translating P 2,1 With overlapping P 1,1 And fix the overlapping feature points P 1,1 P 2,1 To determine the translation vector; by using fixed overlapping feature points P 1,1 P 2,1 Simultaneous pivoting and rotation of feature point P 2,2 With overlapping P 1,2To determine the angle of rotation; and to apply the translation vector and the angle of rotation to the remaining pairs of matched feature points.

[0021] In another aspect of this disclosure, the server controller is also configured to determine the score of the solution based on the number of interior points of the feature pair and the GPS variance from translation and rotation.

[0022] According to several aspects, a system includes a processor and a non-transitory computer-readable medium. Instructions for feature matching to correlate vehicle perception data are stored on the non-transitory computer-readable medium. When executed by the processor, the instructions cause the processor to receive perception data about a portion of a vehicle path collected by a plurality of vehicles; create feature points for each of the plurality of vehicles by processing the data collected by each of the plurality of vehicles separately; create feature descriptors for each of the feature points of each of the plurality of vehicles using a multi-layer bitmap data structure; create lane line point associations by matching common feature descriptors and filtering out outliers; update map data using the lane point associations; and transmit the updated map data to at least one of the plurality of vehicles.

[0023] In an additional aspect of this disclosure, creating a feature descriptor for each of the feature points includes: drawing a plurality of first feature points on a first bitmap and creating a first feature descriptor for each first feature point; and drawing a plurality of second feature points on a second bitmap and creating a second feature descriptor for each second feature point.

[0024] In another aspect of this disclosure, the instructions further cause the processor to compare a first spatial histogram with respect to at least one of the first feature descriptors with a second spatial histogram with respect to at least one of the second feature descriptors; determine matching first and second feature descriptors based on a feature distance threshold between the first and second spatial histograms; and determine matching first and second feature points based on the matching first and second feature descriptors.

[0025] Further applicability will become apparent from the description provided herein. It should be understood that the specification and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0026] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0027] Figure 1 This is a functional block diagram of a vehicle with an intelligent system according to an exemplary embodiment;

[0028] Figure 2It is an illustration of a plan view of multiple vehicles with intelligent systems traveling on a road according to an exemplary embodiment;

[0029] Figure 3 According to exemplary embodiments Figure 2 An illustration of perception data collected by multiple vehicles;

[0030] Figure 4 This is a block flowchart of a method for feature matching to associate vehicle perception data according to an exemplary embodiment;

[0031] Figure 5 This is according to an exemplary embodiment. Figure 4 A detailed flowchart of the box;

[0032] Figure 6A This is an illustration of a multi-layer bitmap according to an exemplary embodiment;

[0033] Figure 6B This illustrates the following according to an exemplary embodiment: Figure 6A A spatial histogram of pixel intensity of a fragment in a multi-layered bitmap;

[0034] Figure 7A According to exemplary embodiments Figure 4 The detailed flowchart of another box;

[0035] Figure 7B This is an illustration of perception data collected by a first vehicle and a second vehicle according to an exemplary embodiment;

[0036] Figure 8A This is according to an exemplary embodiment. Figure 4 A detailed flowchart of another box;

[0037] Figure 8B This is another illustration of perception data collected by the first vehicle and the second vehicle according to an exemplary embodiment;

[0038] Figure 9A This is an illustration of selected matching pairs of feature points according to an exemplary embodiment;

[0039] Figure 9B This is according to an exemplary embodiment. Figure 9A Illustration of relatively good matching pairs of feature points;

[0040] Figure 10A It is an illustration of other selected matching pairs of feature points according to an exemplary embodiment; and

[0041] Figure 10B This is according to an exemplary embodiment. Figure 10A An illustration of relatively poor matching pairs of feature points. Detailed Implementation

[0042] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses. Illustrated embodiments are disclosed with reference to the accompanying drawings, in which the same reference numerals denote corresponding parts in several drawings. The drawings are not necessarily drawn to scale, and some features may be enlarged or minimized to show detail of particular features. The specific structural and functional details disclosed are not intended to be construed as limiting, but rather as a representative basis for teaching those skilled in the art how to practice the disclosed concepts.

[0043] As used herein, the terms “module,” “component module,” “control module,” or “controller” mean any hardware, software, firmware, electronic control component, processing logic, and / or processor device, individually or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors and memories (shared, dedicated, or grouped) that execute one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0044] Embodiments of this disclosure are described herein according to functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, or lookup tables, which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be combined with any number of system practices, and the systems described herein are merely exemplary embodiments of this disclosure.

[0045] The connecting lines shown in the various figures included herein are intended to illustrate exemplary functional relationships and / or physical couplings between various components. Conventional techniques may be used for signal processing, data transmission, signaling, control, and other functional aspects of the system (and its various operating components) may not be described in detail herein. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0046] Although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, parts, and / or sections, these elements, components, parts, and / or sections should not be limited by these terms unless otherwise stated. These terms are used to distinguish one element, component, part, and / or section from another element, component, part, and / or section.

[0047] In crowdsourced map construction, perception data collected by multiple smart vehicles over a road or area is processed to detect and identify objects, including lane lines, stop signs, etc. The location and identification of objects are updated in the crowdsourced map data. As a smart vehicle travels through the road or area, the updated crowdsourced map data is transmitted to the smart vehicle to improve vehicle operation. Crowdsourced perception data may exhibit minor differences in the detected object locations due to variables such as external sensor capabilities, the smart vehicle's position on the road, and GPS accuracy. Therefore, a common step in crowdsourced map construction is to correlate perception data collected by one vehicle with that collected by another. The following disclosures provide systems and methods for feature matching to correlate crowdsourced perception data.

[0048] Figure 1 This is a functional block diagram of a vehicle 100 having an intelligent system 102 (e.g., ADAS and / or ADS) capable of operating according to SAE J3016 driving automation levels from L0 (no driving automation) to L5 (fully automated driving). The vehicle 100 is also referred to as an autonomous vehicle 100 or an intelligent vehicle 100. The vehicle 100 typically includes a body 106, front wheels 108, and rear wheels 110. The body 106 essentially surrounds the vehicle systems and components of the vehicle 100. The front wheels 108 and rear wheels 110 are each rotatably coupled to the body 106 near a corresponding corner. Although the coupled vehicle 100 is shown as a passenger car, it is conceivable that the coupled vehicle 100 could be another type of road vehicle, such as a pickup truck, sports coupe, SUV, RV, or motorcycle.

[0049] As shown in the figure, vehicle 100 typically includes an intelligent system 102 such as ADAS and / or ADS, a propulsion system 120, a transmission system 122, a steering system 124, a braking system 126, a perception system 128, a vehicle communication system 130, and a Global Navigation Satellite System (GNSS) 132. Vehicle systems 120, 122, 124, 126, 128, 130, and 132 communicate with a control module 134 of the intelligent system 102. Vehicle 100 may also include various vehicle actuators 133 that communicate with the control module 134 and with selected vehicle systems 120, 122, 124, 126, 128, 130, and 132. The various vehicle actuators 133 are configured to selectively operate components of vehicle systems 120, 122, 124, 126, 128, 130, and 132 based on commands from the control module 134.

[0050] The intelligent system 102 includes a vehicle control module 134 that communicates with one or more vehicle systems 120, 122, 124, 126, 128, 130, 132 and vehicle actuators 133 via a controller area network (CAN) and / or Ethernet 137. The vehicle control module 134 can be configured to implement a method (method 400) for feature matching to correlate vehicle perception data, as described in detail below. Method 400 can be implemented on the vehicle control module 134 on a single vehicle 100 to process perception data collected by the single vehicle 100 from multiple passages along the same road segment. Method 400 can also be implemented on a cloud server 150 to process perception data collected by multiple vehicles 100 along a road segment.

[0051] Control module 134 includes at least one processor 144 and a non-transitory computer-readable storage device or medium 146. The non-transitory computer-readable storage device or medium 146 includes machine-readable instructions that, when executed by processor 144, cause processor 144 to perform the method 400 described below and operate vehicle 100 in a partially or fully automated driving mode. Processor 144 may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with control module 134, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or a device generally used for executing instructions. Vehicle computer-readable storage device or medium 146 may include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operational variables when processor 144 is powered off. The vehicle computer-readable storage device or medium 146 of the control module 134 may be implemented using multiple memory devices, such as PROM (programmable read-only memory), EPROM, EEPROM, flash memory, or other electrical, magnetic, optical, or combined memory devices capable of storing data, some of which represents executable instructions used by the control module 134 when controlling the intelligent vehicle 100.

[0052] The perception system 128 is configured to collect information and perception data about the external environment surrounding the vehicle 100. In a non-limiting exemplary embodiment, the perception system 128 may include one or more cameras 140A, 140B, 140C arranged to capture images and / or video of the environment adjacent to all sides of the vehicle 100. The one or more cameras 140A, 140B, 140C may include a front-view camera 140A, a rear-view camera 140B, and / or at least one side-view camera 140C. The one or more cameras 140A, 140B, 140C may have various image sensors, including, for example, charge-coupled device (CCD) sensors, complementary metal-oxide-semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors. The one or more cameras 140A, 140B, 140C may have various lens types, including, for example, wide-angle lenses and / or narrow-angle lenses, also within the scope of this disclosure.

[0053] GNSS 132 is configured to determine the geographic location of vehicle 100. In an exemplary embodiment, GNSS 132 is a Global Positioning System (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from multiple satellites, and the GPS controller calculates the geographic location of vehicle 100 based on the signals received by the GPS receiver antenna. In an exemplary embodiment, GPS also includes a map. The map contains information about infrastructure such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, the geographic location of vehicle 100 is contextualized using map information.

[0054] The vehicle communication system 130 is used by the vehicle control module 134 to communicate with other systems outside the vehicle 100. For example, the vehicle communication system 130 includes the ability to communicate with vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems such as remote call centers, and / or personal devices. Generally, the term vehicle-to-everything communication (“V2X” communication) refers to communication between the vehicle 100 and any remote system (e.g., vehicles, infrastructure, and / or remote systems). In some embodiments, the vehicle communication system 130 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication (e.g., using GSMA standards such as SGP.02, SGP.22, SGP.32, etc.). Therefore, the vehicle communication system 130 may also include an embedded universal integrated circuit card (eUICC) configured to store at least one cellular connectivity profile, such as an embedded subscriber identity module (eSIM) profile.

[0055] The vehicle communication system 130 is also configured to communicate via a personal area network (e.g., Bluetooth) and / or near-field communication (NFC). However, additional or alternative communication methods, such as Dedicated Short-Range Communication (DSRC) channels and / or mobile telecommunications protocols based on 3GPP standards, are also considered within the scope of this disclosure. A DSRC channel refers to a unidirectional or bidirectional short- to mid-range wireless communication channel designed specifically for automotive use, along with a set of corresponding protocols and standards. 3GPP refers to a partnership between several standards organizations that develop mobile telecommunications protocols and standards. 3GPP standards are structured as “versions.” Therefore, communication methods based on 3GPP versions 14, 15, 16, and / or future 3GPP versions are considered within the scope of this disclosure. Thus, the vehicle communication system 130 may include one or more antennas and / or communication transceivers for receiving and / or transmitting signals, such as cooperative sensing messages (CSM). The vehicle communication system 130 is configured to wirelessly transmit information between vehicle 100 and another vehicle. Furthermore, the vehicle communication system 130 is configured to wirelessly transmit information between the vehicle 100 and infrastructure or other vehicles. It should be understood that, without departing from the scope of this disclosure, the vehicle communication system 130 may be integrated with the vehicle control module 134 (e.g., integrated on the same circuit board as the vehicle control module 134, or otherwise integrated as part of the vehicle control module 134).

[0056] Continue to refer to Figure 1 The cloud server system 150 includes a server controller 152 that electrically communicates with the map database 154 and the server communication system 156. In a non-limiting example, the cloud server system 150 is located in a server farm or data center, etc., and is connected to the Internet using the server communication system. The server controller 152 includes at least one server processor and a server non-transitory computer-readable storage device or server medium. The description of the type and configuration given above for the vehicle control module 134 also applies to the server controller. In some examples, the server controller 152 may differ from the vehicle control module 134 in that the server controller 152 is capable of having higher processing speeds, including more memory, and / or including more inputs / outputs, etc. In a non-limiting example, the server processor and server medium of the server controller 152 are structurally and / or functionally similar to the processor and medium of the vehicle control module, as described above.

[0057] Map database 154 is used to store map data about roads, including, for example, lane line map data, as will be discussed in more detail below. Server communication system 156 is used to communicate with external systems (e.g., vehicle control module 134) via vehicle communication system 130. In a non-limiting example, server communication system 156 is structurally and / or functionally similar to vehicle communication system 130 of the vehicle system, as described above.

[0058] Figure 2 This is a non-limiting illustration of a scenario where a first intelligent vehicle 100A and a second intelligent vehicle 100B are traveling on a section of road 200 defined between a first lane marking 202 and a second lane marking 204. For simplicity, the first intelligent vehicle 100A and the second intelligent vehicle 100B are referred to as first vehicle 100A and second vehicle 100B, respectively. The position trajectory of each of the first and second vehicles is determined by using GNSS on the respective vehicles 100A and 100B. In one example, the perception system 128 of the respective vehicles 100A and 100B captures multiple images of the road 200 and areas of the adjacent road 200, including any landmarks. The control module 134 of the respective vehicles 100A and 100B uses computer vision algorithms to identify feature points P1, P2, and P3 in the multiple images of the road. Feature points P1, P2, and P3 may represent path markers, landmarks, traffic signs, road configurations, and any other items typically found on roads. The communication system 130 of each vehicle 100A and 100B is configured to upload the corresponding perception data and the vehicle's global location, and the timestamp is collected to the cloud server system 150.

[0059] Figure 3 This is an illustration of the perception data collected by each vehicle. The trajectory of the first vehicle 100A, determined by GNSS 132, is represented by reference numeral V1 (trajectory V1). The trajectory of the second vehicle 100B, determined by GNSS 132, is represented by reference numeral V2 (trajectory V2). The positions of feature points P1, P2, and P3 observed by the first vehicle 100A are respectively represented by reference numeral P1, P2, and P3. 1,1 P 2,1 P 1,3 The positions of feature points P1, P2, and P3 observed by the second vehicle 100B are indicated by the attached diagram label P. 2,1 P 2,2 P 2,3 express.

[0060] The offsets of feature points P1, P2, and P3 sensed by the first vehicle 100A and the second vehicle 100B are partly attributed to GNSS inaccuracies, sensor limitations, weather, obstacles, the relative positions of vehicles 100A and 100B when collecting the sense data, and one or more other influencing factors. The sense data collected by the first vehicle 100A and the second vehicle 100B is uploaded to a cloud server system 150. The cloud server system 150 is configured to correlate and align the sense data collected by multiple vehicles to improve the accuracy of the map database. The map database can be downloaded to intelligent vehicles to enhance and / or automate various vehicle functions.

[0061] Figure 4 This is a block flowchart of a method (method 400) for feature matching to correlate vehicle perception data in order to align and reconcile perception data collected by multiple vehicles. Method 400 uses computer vision features to align two semantic maps, or to align semantic perception data from two intelligent vehicles. A semantic map refers to a map or map data generated by processing semantic perception data. Semantic perception data refers to data collected by vehicle sensors on one or more vehicles. Method 400 includes representing landmarks (lane lines, road signs, etc.) as feature points using computer vision features, defining feature point descriptors, using spatial histograms to represent spatial relationships between neighboring feature points (also called adjacent feature points), implementing a matching algorithm to align feature points in perception data collected by multiple intelligent vehicles, and using a random sample consensus (RANSAC) algorithm to eliminate outliers in matched feature point pairs.

[0062] In box 402, multiple intelligent vehicles collect perception data about a portion of the road and / or area. The perception data, along with timestamps and vehicle trajectories, is transmitted to cloud server system 150. Method 400 proceeds to box 404.

[0063] In box 404, feature points are created. The cloud server system 150 processes the perceived data to create feature points representing the observations of each vehicle 100A, 100B. Feature points can represent landmarks, road signs, traffic signals, and other distinguishing features. Observed lane lines can be converted into feature points. In a non-restrictive example, in a semantic map or in the vehicle's perceived dataset, in the intelligent vehicle's own coordinate system, lane lines can be represented by the polynomial y = ax². 2 +bx+c represents the expression. For each lane line, the polynomial representation is converted into feature points: (x1,y(x1)),(x2,y(x2)),(x3,y(x3)). It should be understood that any method used to determine the mathematical equations describing one or more lane lines is within the scope of this disclosure.

[0064] In another exemplary embodiment, to determine the lane line map, server controller 152 may use a hill-climbing algorithm, such as that described in U.S. Application No. 17 / 930,503, entitled “HILL CLIMBING ALGORITHM FOR CONSTRUCTINGALANE LINE MAP,” filed September 8, 2022, the entire contents of which are incorporated herein by reference. It should be understood that any method used to determine the mathematical equations describing one or more lane lines is within the scope of this disclosure. Method 400 proceeds to block 406.

[0065] In box 406, create a feature descriptor. A multi-layered bitmap can be used to create a feature descriptor for each of the stated feature points. The created feature descriptor may include several pieces of information, such as: 1) the lane line attributes (color, type, location, etc.); 2) its relative positioning to neighboring landmarks (lane lines).

[0066] Figure 5 This is the detailed flowchart of box 406. Figure 6A This is an illustration of a multi-layer bitmap 500 (also referred to as bitmap 500), used as an exemplary illustration of the detail block flowchart of description box 406. Reference Figure 5 and Figure 6A The first lane line 502 and the second lane 504 observed by the intelligent vehicle are plotted on a multi-layer bitmap. Each layer of bitmap 500 represents certain attributes of lane lines 502 and 504, such as color, type, and position. In a non-limiting example, each of layers Y, W, and D of bitmap 500 represents certain attributes of lane lines 502 and 504. For example, layer Y represents a yellow lane line, layer W represents a white lane line, and layer D represents a dashed lane line. Method 400 proceeds to block 406B.

[0067] In block 406B, a noise-eliminating kernel density estimation (KDE) or Gaussian filter can be applied to indicate the uncertainty in lane line position, thereby eliminating noise. Method 400 proceeds to block 406C.

[0068] In box 406C, a plurality of pixels contained within a predetermined boundary shape surrounding each of the said feature points P1, P2, P3 are selected as feature descriptors. In other words, the feature descriptor is defined by all the pixels contained within the predetermined boundary shape. In a non-limiting example, the predetermined boundary shape can be a rectangle 506, a circle 508, a square 510, or even a row of pixels. A row of pixels can help reduce computational overhead. The predetermined boundary shape should be large enough to reach neighboring landmarks, also known as adjacent landmarks. In a non-limiting example, the predetermined boundary shape includes a lateral distance of 5 to 8 meters. Method 400 proceeds to box 406D.

[0069] In box 406D, a spatial histogram is applied to the selected pixels to further reduce the data size of the feature descriptor. Figure 6B This is an exemplary spatial histogram showing the intensity of pixels at a lateral distance X along the boundary shape 510. The horizontal axis represents the lateral distance X, and the vertical axis represents the pixel intensity; larger numbers indicate greater intensity. The first spatial histogram of a first feature descriptor of a first feature point observed by a first intelligent vehicle can be compared with the second spatial histogram of a second feature descriptor of a second feature point observed by a second intelligent vehicle to determine whether the first feature point matches the second feature point. In other words, the spatial histogram is used to determine whether the first feature point observed by the first vehicle is the same as the second feature point observed by the second vehicle.

[0070] The feature distance between two feature points within a feature descriptor can be determined. The feature distance is an objective score that summarizes the relative differences between two objects (feature points) in the problem domain. An example of determining the feature distance uses the following formula:

[0071]

[0072] The larger the number, the higher the probability that two individually observed feature points are the same feature point. Other feature distance metrics can also be used, such as Euclidean distance, cosine similarity, Mahalanobis distance, etc. Method 400 proceeds to box 408.

[0073] In box 408, the feature matching algorithm is implemented. The feature matching algorithm finds a match between the first vehicle-perceived landmark (feature point) and the second vehicle-perceived landmark (feature point). Figure 7A This is the detailed flowchart of box 408.

[0074] Figure 7B This is an illustration of the perception data collected by each of vehicles 100A and 100B, serving as an exemplary illustration of the detailed block flowchart for description box 408. The trajectory V1 of the first vehicle 100A is determined by GNSS 132 of the first vehicle 100A. The trajectory V2 of the second vehicle 100B is determined by GNSS 132 of the second vehicle. Feature points within boundary D observed by the first vehicle 100A are indicated by reference numeral P. 1,1 P 2,1 P 1,3 The feature points observed by the second vehicle 100B are indicated by the reference numeral P. 2,1 The feature points within boundary E as observed by the first vehicle are indicated by the reference numeral P. 1,E The feature points, as indicated and observed by the second vehicle, are designated by the reference numeral P. 2,E express.

[0075] refer to Figure 7A and Figure 7B Both, in box 408A, identify each of the feature points observed from multiple intelligent vehicles 100A, 100B. Figure 7B The image shows some feature points P of the first vehicle. 1,1 P 2,1 P 1,3 P 1,E And trajectory V1, and certain feature points P of the second vehicle 2,1 P 2,E And trajectory V2. Method 400 proceeds to box 408B.

[0076] In box 408B, for each feature point identified from the perception data of one vehicle, find or determine neighboring feature points from another vehicle. For example, feature point P is observed by the second vehicle 100B. 2,1 Feature point P 2,1 Having multiple adjacent feature points P observed by the first vehicle 100A 1,1 P 2,1 P 1,3 Adjacent feature points can be defined as nearby feature points (landmarks) within a predetermined boundary region D. Method 400 proceeds to box 408C.

[0077] In box 408C, nearby feature points, also known as adjacent feature points, are filtered out based on dissimilarity or mutually exclusive attributes (such as differences in color, type, distance, etc.). For example: Feature point P 1,3 The color may be yellow, and the feature point P may be white. 2,1 The colors do not match, therefore, feature point P 1,3 With P 2,1 Mismatch, and feature point P 1,3 It was filtered out. Method 400 proceeds to box 408D.

[0078] In box 408D, calculate the remaining pairs relative to feature point P. 2,1 The feature distance. For example: calculate the feature distance (P) of a feature distance pair. 2,1 P 1,1 ) and feature distance (P 2,1 P 1,2 ).

[0079] In box 408E, a pair of feature points with a distance higher than a predetermined feature distance threshold is selected as a feature match, meaning that the pair of feature points are identical feature points. For example, suppose the feature distance (P) 2,1 P 1,1 The value of ) is greater than the feature distance (P) 2,1 P 1,2 If so, then select feature point P in box 408F. 1,1As P 2,1 Feature matching (assuming a larger value means a more likely match calculated based on a specific feature distance). Method 400 returns to box 408A and repeats for each feature point observed by the selected vehicle until all feature point pairs are identified. Method 400 proceeds to box 410.

[0080] refer to Figure 7B A pair of matched feature points (P) 1,2 P 2,1 A feature pair that is considered a good match based on local information (e.g., color, type, relative position to other features, etc.) may be a mismatched feature point pair from a global perspective. For example, the feature point pair P shown in the boundary region E... 1,E and P 2,E Outline feature pairs may be well-matched based on their relative positions in terms of color, type, and underlying landmarks. However, if outline feature pairs are used for global alignment, trajectories V1 and V2 will be misaligned. Outline feature pairs can lead to incorrect global registration / alignment among all locally well-matched feature pairs. Methods based on the Random Sample Consensus (RANSAC) algorithm can be used to detect and remove outline feature matches to address lane line matching problems.

[0081] Figure 8A This is a detailed block flowchart of the method for detecting and removing outlier feature point matches (block 410). The combination of detecting and removing outlier feature point matches is also called filtering or filtering out outlier feature point matches. The remaining inlier feature point pairs are selected as the final matching feature points.

[0082] Figure 8B This is a diagram of the perception data collected from each of vehicles 100A and 100B. Figure 8B An exemplary illustration of the detail block flowchart used as description block 410. Figure 8B The trajectory V1 of the first vehicle 100A, the trajectory V2 of the second vehicle 100B, and feature point P observed by the first vehicle are shown. 1,1 and feature point P observed by the second vehicle 2,1 The first observation area R1, and the feature point P observed by the first vehicle. 1,2 and feature point P observed by the second vehicle 2,2 The second observation area is R2. Feature point P is determined using the flowchart in box 408. 1,1 and feature point P 2,1 To match feature pairs. Based on feature point P of box 408. 1,2 and feature point P 2,2 They were also identified as matching feature pairs.

[0083] refer to Figure 8A and Figure 8B In box 410A, for each vehicle 100A and 100B, two different observation regions are randomly selected, and matching feature point pairs are selected in each randomly selected region. As an example, a first observation region R1 and a second observation region R2 are randomly selected, and a matching feature pair P is randomly selected from the first observation region R1. 1,1 and P 2,1 And randomly select matching feature pairs P from the second observation region R2. 1,2 and P 2,2 Method 400 proceeds to box 410B.

[0084] In box 410B, feature point P is made possible by translating trajectory V2 relative to trajectory V1. 2,1 With feature point P 1,1 Overlap to align the two selected matching feature pairs. Fix the trajectory curves V1 and V2 at the overlapping feature point P. 1,1 P 2,1 At that location, and rotated so that feature point P 2,2 Rotate to point P 1,2 Overlap. Record the translation vector and rotation angle. This is achieved by applying the above translation vector to all points in V2 and around a fixed overlapping feature point P. 1,1 P 2,1 Rotate all translated feature points to transform the entire point cloud in V2. Feature distance pairs: Feature distance (P) 2,1 P 1,1 ), Feature distance (P) 2,1 P 1,2 Method 400 proceeds to box 410C.

[0085] As shown in the illustrative example in box 410B, Figure 9A This shows a matching pair P randomly selected from the first observation region R1. 1,1 and P 2,1 And randomly select P in the second observation region R2 1,2 and P 2,2 . Figure 9B It shows that the matching feature pairs are translated such that P 2,1 With P 1,1 Overlap, fixed overlapping feature point P 2,1 With P 1,1 And rotate trajectory V2 relative to trajectory V1 until feature point P. 2,2 With feature point P 1,2 overlapping. Figure 9B A relatively good global match with 12 inlier matching pairs is shown.

[0086] Figure 10A This shows a matching pair P randomly selected from the first observation region R1.1,1 and P 2,1 And select P in the second observation region R2 1,3 and P 2,3 . Figure 10B Trajectory V2 is shown, including the translation of matching feature pairs such that P 2,1 With P 1,1 Overlap, fixed P 2,1 With P 1,1 And rotate the curve until P 2,3 With P 1,3 overlapping. Figure 10B The diagram shows a relatively poor global match with 7 interior point matching pairs, as demonstrated by the intersection of trajectory V1 and trajectory V2.

[0087] In box 410C, the score of the solution is evaluated for each translation vector and rotation angle. The distance to each feature pair is calculated after the transformation, and the number of matching feature pairs within a threshold distance (e.g., 0.2 meters) is counted. The transformed GPS distance to each point in V2 is calculated, yielding the GPS offset variance. The percentage of inlier feature pairs in V1 and V2 is calculated separately. Method 400 proceeds to box 410D.

[0088] In box 410D, the combined score that yields the highest feature count and lowest GPS offset variance within 0.2 meters is calculated as the solution. The counts and GPS variance are normalized to the range [0,1], and the GPS variance is inverted to ensure that the lower the value, the better. For example, Combined_scores = (count_weight * normalized_count) + (GPS_variance_weight * normalized_variance). Method 400 proceeds to box 410E.

[0089] In box 410E, the inlier feature point pair is selected as the final matching feature pair between the first vehicle and the second vehicle. Method 400 proceeds to box 412.

[0090] Return to reference Figure 4 In box 412, the matching feature pairs of the best solution are used to associate feature points between the first vehicle and the second vehicle. Method 400 proceeds to box 414.

[0091] In box 414, the map data is updated and optimized using associated data and transmitted to at least one smart vehicle. Method 400 proceeds to box 416.

[0092] In box 416, at least one intelligent vehicle performs the functions of the intelligent system based at least in part on the updated and optimized map data received in box 414. In a non-limiting example, the intelligent system of at least one vehicle may utilize the updated and optimized map data to keep the at least one vehicle in a given lane or to operate the at least one vehicle at least partially autonomously.

[0093] The description in this disclosure is merely exemplary in nature, and variations thereof without departing from the spirit and scope of this disclosure are intended to fall within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for feature matching to correlate vehicle perception data, comprising: Sensing data is received from multiple vehicles, including a first vehicle and a second vehicle. The perceived data is processed to identify a first feature point observed by the first vehicle and a second feature point observed by the second vehicle, respectively. Create a feature descriptor for each first feature point and each second feature point; Compare the feature descriptors of each first feature point and each second feature point to determine matching pairs of first and second feature points; Associate the matched first and second feature point pairs; The map database is updated using the associated matched first and second feature point pairs; as well as The autonomous vehicle is operated based on the updated map database.

2. The method according to claim 1, wherein, Creating a feature descriptor for each first feature point and each second feature point includes: Plot the first feature point on the first image and create a first feature descriptor for each first feature point; and The second feature point is plotted on the second bitmap, and a second feature descriptor is created for each second feature point.

3. The method according to claim 2, wherein, The comparison of the feature descriptors for each of the first feature points and each of the second feature points further includes: A first spatial histogram is applied to at least one of the first feature descriptors, and a second spatial histogram is applied to at least one of the second feature descriptors that are near at least one of the first feature descriptors; Based on a predetermined feature distance threshold between the first spatial histogram and the second spatial histogram, a matching first feature descriptor and a second feature descriptor are determined; and Based on the matched first and second feature descriptors, the matched first feature point and second feature point pair are determined.

4. The method according to claim 2, wherein, At least one of the first bitmap and the second bitmap is a multi-layer bitmap, and each layer includes different attributes of the feature points.

5. The method according to claim 3, wherein, Determining the matching first and second feature point pairs also includes: Identify adjacent first and second feature points; Based on the mutual exclusion property between the first spatial histogram and the second spatial histogram, a mismatched first feature descriptor and a mismatched second feature descriptor are determined; and Filter out the first feature descriptors that do not match.

6. The method according to claim 5, wherein, Before determining the matching first feature point and second feature point pair based on the matching first feature descriptor and second feature descriptor, the mismatched first feature descriptor is filtered out.

7. The method according to claim 1, further comprising: Determine the pair of exterior points of the first and second feature points that match; as well as The outlier pairs are filtered out before the first and second feature point pairs of the matching are associated.

8. The method according to claim 7, wherein, Determining the pair of outliers of the first and second matching feature points includes: Identify the first and second observation regions; Randomly select a first pair of matching feature points in the first observation region and a second pair of matching feature points in the second observation region, wherein the first pair of matching feature points includes feature point P. 1,1 P 2,1 And the second pair of matched feature points includes feature point P. 1,2 P 2,2 ; By translating P 2,1 With overlapping P 1,1 And fix the overlapping feature points P 1,1 P 2,1 To determine the translation vector; By using fixed overlapping feature points P 1,1 P 2,1 Simultaneous pivoting and rotation of feature point P 2,2 With overlapping P 1,2 To determine the angle of rotation; and The translation vector and the rotation angle are applied to the remaining pairs of matched feature points.

9. The method according to claim 8, further comprising: The score of the solution is determined based on the number of interior points of the feature pair and the GPS variance from translation and rotation.

10. The method according to claim 1, wherein, The method is executed by a controller located in either the intelligent vehicle or a cloud-based server.

Images