Camera image pixel velocity segmentation using radar doppler measurements

Abstract

A vehicle includes a camera configured to generate images along the vehicle's path. A radar sensor is configured to transmit radio frequency (RF) signals and receive radar echoes along the vehicle's path. A velocity segmentation module is configured to: generate radial velocities of the radar echoes; divide the radar echoes into R clustered radar echoes, where R is a positive integer; project the R clustered radar echoes onto an image; identify R moving objects corresponding to the R clustered radar echoes by segmenting R groups of image pixels having image features similar to those corresponding to the pixels of the R clustered radar echoes; and calculate R radial velocities of the R clustered radar echoes respectively in response to the radial velocities of the R clustered radar echoes for each of the R moving objects.

Description

[0001] introduction The information provided in this section is for the purpose of presenting the overall context of this disclosure. The work of the currently named inventors—to the extent described in this section—and aspects of this description that may otherwise not qualify as prior art at the time of filing are neither expressly nor implicitly considered to be prior art to this disclosure.

[0002] This disclosure relates to driver assistance systems for vehicles, and more specifically to velocity segregation of camera image pixels measured using radar Doppler.

[0003] Vehicles encompass various levels of driver assistance (such as fully or partially autonomous vehicles). These vehicles rely on data generated by sensors (such as radio detection and ranging (radar) sensors, light detection and ranging (LiDAR) sensors), and / or images generated by cameras. When activated, the driver assistance system responds to the data generated by the sensors to control steering, acceleration, and / or braking. Summary of the Invention

[0004] A vehicle includes a camera configured to generate an image of the vehicle's path. A radar sensor is configured to transmit radio frequency (RF) signals and receive radar echoes in the vehicle's path. A velocity segmentation module is configured to: generate radial velocities of the radar echoes; divide the radar echoes into R clustered radar echoes, where R is a positive integer; project the R clustered radar echoes onto the image; identify R moving objects corresponding to the R clustered radar echoes by segmenting R groups of image pixels having image features similar to those corresponding to the pixels of the R clustered radar echoes; and calculate R radial velocities of the R clustered radar echoes respectively in response to the radial velocities of the R clustered radar echoes for each of the R moving objects.

[0005] Among other features, the driver assistance system is configured to control at least one of the vehicle's speed and direction of travel in response to R radial velocities of R moving objects. The speed segmentation module identifies the R moving objects by recognizing R groups of segmented pixels having image features similar to those of pixels corresponding to the R clustered radar echoes.

[0006] Among other features, the velocity segmentation module uses a neural network to identify R groups of segmented pixels. The neural network generates feature vectors for each pixel in the image. The velocity segmentation module identifies selected pixels for the R groups of segmented pixels based on these feature vectors. The velocity segmentation module calculates R radial velocities for the R groups of segmented pixels based on the average radial velocity of radar echoes in R clustered radar echoes.

[0007] Among other features, the speed segmentation module removes outliers from R clustered radar echoes. The speed segmentation module selectively classifies radar echoes into either static or dynamic categories based on the vehicle's own speed. The speed segmentation module selects dynamic radar echoes for clustering and does not select static radar echoes for clustering.

[0008] A method for sensing objects in a vehicle path includes: generating an image of the vehicle path; transmitting radio frequency (RF) signals and receiving radar echoes in the vehicle path; generating radial velocities of the radar echoes; grouping the radar echoes into R clustered radar echoes, where R is an integer greater than zero; projecting the R clustered radar echoes onto the image; identifying R moving objects corresponding to the R clustered radar echoes by segmenting R groups of image pixels having image features similar to those corresponding to pixels in the R clustered radar echoes; and calculating R radial velocities of the R clustered radar echoes in response to the radial velocities of the R clustered radar echoes for each of the R moving objects.

[0009] Among other features, the method includes controlling at least one of the vehicle's speed and direction of travel in response to R radial velocities of R moving objects. The method includes identifying the R moving objects by recognizing R groups of segmented pixels, each having image features similar to pixels corresponding to the R clustered radar echoes. The method includes using a neural network to identify the R groups of segmented pixels.

[0010] Among other features, the neural network generates feature vectors for the pixels of the image. The method includes identifying selected pixels for R groups of segmented pixels based on the feature vectors. The method also includes calculating R radial velocities for the R groups of segmented pixels based on the average radial velocity of radar echoes in R clustered radar echoes. Finally, the method includes removing outliers from the R clustered radar echoes.

[0011] Among other features, the method includes selectively classifying radar echoes into either static or dynamic categories based on the vehicle's ego speed. The method also includes selecting dynamic radar echoes for clustering and not selecting static radar echoes for clustering.

[0012] Further areas of applicability of this disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0013] This disclosure will be more fully understood from the detailed description and the accompanying drawings, in which: Figure 1The present disclosure is a functional block diagram of an example vehicle, which includes a radar sensor, a camera, and a controller including a speed segmentation module. Figure 2 The illustration shows an example of velocity separation of camera image pixels measured using radar Doppler. Figure 3 This is a flowchart illustrating an example of a method for performing velocity separation of camera image pixels using radar Doppler measurements, and for autonomously operating a vehicle using that information, according to this disclosure; and Figures 4 to 6 These are images illustrating various example scenarios using camera image pixels velocity separation based on radar Doppler measurements according to this disclosure.

[0014] In the accompanying drawings, reference numerals may be used repeatedly to identify similar and / or identical elements. Detailed Implementation

[0015] Although the following disclosure relates to driver assistance systems for passenger vehicles, the systems and methods described herein can be used in other types of vehicles.

[0016] Advanced Driver Assistance Systems (ADAS) control vehicle driving in a partially or fully autonomous mode using a recurring cycle comprising perception, prediction, and planning phases. During the perception phase, the vehicle's sensors provide sensor data, and that data is processed to identify the positions of objects in the vehicle's path. During the prediction and planning phase, the ADAS predicts where the vehicle and objects will be located in subsequent sampling intervals, evaluates various vehicle control inputs relative to the vehicle's desired path, and selects the control input for that sampling interval.

[0017] Driver assistance systems determine how to adjust actuators (e.g., brake, throttle position, and / or steering wheel angle) to control the vehicle's speed, acceleration, braking, and / or direction of travel. Sensors include one or more radio detection and ranging (radar) sensors, one or more light detection and ranging (LiDAR) sensors, and / or one or more cameras that generate different types of perception data.

[0018] A radar sensor emits RF signals and receives echoes (e.g., radar point clouds) from objects in the vehicle's path. Doppler is a radar measurement at each reflection point, proportional to the radial velocity of the reflection (radar echo). A camera generates an image of the vehicle's path. The ego-vehicle velocity Doppler component is removed, and the radial velocity is generated from the radar echoes in the radar point cloud.

[0019] Furthermore, radar echoes are also used to determine the distance between a vehicle and an object that causes reflections from the vehicle. Compared to lidar sensors, although radar sensors provide sparse data, they have several advantages. Radar sensors have a longer detection range, are resistant to inclement weather (such as rain or snow), and produce more useful Doppler measurements (due to the use of RF signals compared to the light used in lidar sensors).

[0020] Radar points that are geographically close together are grouped into clusters. The clustered radar echoes are projected onto the corresponding images from the camera. Image features are generated based on the RGB values ​​of the pixels corresponding to the locations of the clustered and projected radar points. Neighboring pixels with similar features are identified (or segmented) to form pixel groups around the clustered and projected radar echoes.

[0021] For example, nearby image pixels may correspond to vehicles, such as cars, motorcycles, or trucks. A radial velocity (based on the radial velocity of the corresponding clustered and projected radar points) is determined for each group of segmented pixels. In some examples, the radial velocity calculated for each group of segmented pixels is equal to the average of the radial velocities of the corresponding clustered and projected radar points.

[0022] For example, a driver assistance system receives radar echoes, clusters the echoes into first and second clusters, and projects the first and second clustered radar echoes onto corresponding pixels in an image. The first and second groups of image pixels—each possessing features similar to their corresponding pixels in the clustered radar echoes—are identified (e.g., using a neural network) and segmented. More specifically, the neural network generates a feature vector or descriptor for each pixel. The feature vectors are then used to find similarities between image pixels. In other words, image pixels with similar features (obtained from the neural network) are segmented into the same cluster. For example, the first and second groups of segmented image pixels could correspond to a first vehicle and a second vehicle in a vehicle path, respectively.

[0023] The radial velocities corresponding to the first and second cluster echoes are used to generate the total radial velocity of the first and second group of segmented image pixels, respectively. For example, the first and second group of segmented image pixels can be assigned the average radial velocity of the radar echoes corresponding to the first and second cluster radar echoes. It can be understood that outliers can be removed before averaging or applying another function.

[0024] Information generated during the perception phase is used by the driver assistance system in the prediction and planning phases. Continuing the example above, a first vehicle may be 10 meters ahead of the first vehicle and moving away from it at a speed of 6 m / s, while a second vehicle may be 18 meters ahead of the first vehicle and moving towards it at a speed of -5 m / s. Using this information, the driver assistance system can predict that in the next sample(s), the first vehicle will move further away from the first vehicle (and its possible location), and in the next sample(s), the second vehicle will move closer to the first vehicle (and its possible location). The driver assistance system uses this information when evaluating various control inputs to the vehicle.

[0025] Now for reference Figure 1 The vehicle 100 includes a controller 110 that communicates with a Global Positioning System (GPS) / compass 112. The GPS / compass 112 is configured to determine the position and direction of travel of the vehicle 100. The controller 110 communicates with one or more radio detection and ranging (radar) sensors 114, which are configured to transmit RF signals and receive radar echoes from objects in the path of the vehicle 100. The radar echoes from the one or more radar sensors 114 are stored in a radar point cloud 144.

[0026] The controller 110 can also communicate with one or more light detection and ranging (LiDAR) sensors 118, which are configured to emit light and detect LiDAR echoes. LiDAR echo points from the one or more LiDAR sensors 118 are stored in a LiDAR point cloud 146.

[0027] The controller 110 includes one or more cameras 122 configured to generate images of the path of the vehicle 100. The vehicle 100 includes one or more vehicle control devices 124, such as an accelerator pedal, a brake pedal, a steering wheel, etc.

[0028] The controller 110 includes a driver assistance module 132 configured to perform partial and / or fully autonomous control of the vehicle 100. For example, the driver assistance module 132 is configured to adjust actuators associated with the vehicle control device 124 based on the output of one or more radar sensors 114, GPS / compass 112, one or more lidar sensors 118 and / or one or more cameras 122.

[0029] The controller 110 includes a velocity segmentation module 138 configured to perform processing on the radar echoes and images as described above. The velocity segmentation module 138 includes a radar processing module 140 configured to calculate the range and radial velocity of the radar echoes in the radar point cloud 144 generated by the radar sensor 114. The radar processing module 140 is configured to perform Doppler calculations on the radar echo data, remove ego-vehicle velocity Doppler components, perform filtering on Doppler calculations exceeding a predetermined threshold, perform radar point clustering, and / or other functions.

[0030] The velocity segmentation module 138 includes an image processing module 136 configured to cluster radar points and project them onto an image. The image processing module 136 compares image features of pixels near the pixels corresponding to the clustered and projected radar points. The image processing module 136 removes outliers from the image features of each cluster.

[0031] Image processing module 136 segments image pixels that have features similar to those of the pixels corresponding to the clustered and projected radar echoes, and assigns Doppler velocities to the segmented image pixels. In some examples, image processing module 136 uses neural network 137 to segment relevant pixels.

[0032] The image processing module 136 outputs one or more Doppler segmented images with radial velocity and distance to the driver assistance module 132. The driver assistance module 132 uses the Doppler segmented images with radial velocity and distance to operate the vehicle 100 in a partially or fully autonomous driving mode.

[0033] Now for reference Figure 2 The diagram illustrates the velocity separation of camera image pixels using radar Doppler measurements. At 160, a radar point cloud is shown in a plan view or bird's-eye view orientation. At 164, radar echo points in the point cloud are clustered (e.g., grouping radar points in similar locations (as shown by the dashed circles in the plan view at 166)). At 172, the clustered radar echoes are projected onto images generated by one or more cameras.

[0034] At 176, nearby pixels are identified and segmented (e.g., forming segmented pixel groups) that have image features similar to those of pixels corresponding to the clustered and projected radar echoes, as shown at 178. At 182 and 184, a radial velocity is assigned to each segmented pixel group (e.g., an averaged radial velocity, or by applying another function). The driver assistance module uses the segmented image with radial velocity and distance when making decisions regarding actuator control associated with one or more vehicle control devices.

[0035] Now for reference Figure 3This illustrates a method for performing velocity separation of camera image pixels using radar Doppler measurements. At 210, the ego velocity Doppler component is removed from the Doppler measurement. For example, a stationary object is static or does not move, and its radar echo can be eliminated for this analysis. At 214, dynamic radar reflection points with radial velocities above a predetermined threshold are filtered. At 218, closely spaced radar points are grouped into clustered radar echoes. In other words, radar echo points from the radar point cloud whose locations are close to each other are grouped together to form clusters. Radar echo points near other radar echoes not located in a cluster are either associated with a different cluster or are not associated with a cluster (e.g., individual radar echoes).

[0036] At position 222, the radar echo clusters are projected onto the corresponding image from the camera. In other words, the radar echo clusters are projected onto the locations of the objects that generated the radar echoes in the image.

[0037] At position 226, image features from image pixels corresponding to the projection point clusters are identified. For example, the image is fed into a neural network that generates multiple feature vectors based on the image's red, green, and blue (RGB) pixel data. These feature vectors are then used to identify nearby pixels with similar feature vectors.

[0038] Neural network identifiers have feature vectors that are similar to those of the pixels corresponding to the clustered and projected radar echoes. In other words, the RGB pixels corresponding to a vehicle will have feature vectors that are more similar to those of the other background pixels surrounding the road, sky, sidewalk, or objects that cause reflections.

[0039] At position 230, outliers in the image features of each cluster are removed. For example, radar echoes corresponding to static objects may be located near or within the segmented pixel group.

[0040] At 234, RGB pixels with features similar to those at the projected cluster point are segmented and combined with image features already associated with that projected cluster point. The combined image is assigned Doppler values ​​(based on the radial velocity of the segmented cluster point (whose outliers are removed at 230)). At 238, the annotated image with point clustering and average radial velocity is used by the driver assistance system at 242 to adjust actuators associated with one or more vehicle control devices.

[0041] Now for reference Figure 4-6 This illustrates an example scenario using radar Doppler measurements and camera image pixel velocity separation. Figure 4In the process, radar point echoes are processed, clustered, and projected onto the image. Three groups of image pixels with dynamic echoes are identified at points D1, D2, and D3. Static echoes S are ignored. An outlier O is located in D3 and is removed from the calculation.

[0042] exist Figure 5 In the image, radar point echoes are processed, clustered, and projected onto the image. At points D4 and D5, two groups of image pixels with dynamic echoes are identified. Static echoes S are ignored. Figure 6 In the image, radar point echoes are processed, clustered, and projected onto the image. At D6, D7, and D8, three groups of image pixels with dynamic echoes are identified. Static echoes S are ignored.

[0043] Segmenting image pixels based on Doppler velocity (or radial velocity of reflected echoes) adds valuable information to improve object detection, prediction, tracking, and driving planning. Standard images from cameras lack direct motion information. The driver assistance system according to this disclosure provides high-resolution motion segmentation in camera images using sparse radar Doppler measurements.

[0044] The foregoing description is merely illustrative in nature and is by no means intended to limit this disclosure, its application, or its use. The broad teachings of this disclosure can be implemented in various forms. Therefore, while this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon examination of the drawings, specification, and the following claims. It should be understood that one or more steps within the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of this disclosure may be implemented in and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other remain within the scope of this disclosure.

[0045] Spatial and functional relationships between components (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms including “connection,” “joint,” “coupled,” “proximity,” “closest,” “above,” “under,” and “set.” Unless explicitly described as “direct,” when describing the relationship between the first and second components in the above disclosure, the relationship can be a direct relationship in which no other intervening components exist between the first and second components, or an indirect relationship in which one or more intervening components (either spatially or functionally) exist between the first and second components. As used herein, the phrase “at least one of A, B, and C” should be interpreted as meaning the logic of using a non-exclusive logical “OR” (A or B or C) and should not be interpreted as meaning “at least one of A, at least one of B, and at least one of C.”

[0046] In the diagrams, the direction of the arrows, as indicated by the arrowhead, typically indicates the flow of information (such as data or instructions) of interest for that diagram. For example, when components A and B exchange various information, but the information transmitted from component A to component B is relevant to this diagram, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B may send a request for or confirmation of receipt of that information to component A.

[0047] In this application, including the following definitions, the term "module" or "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or some or all of the foregoing, such as in a system-on-a-chip.

[0048] A module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow for load balancing. In a further example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0049] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuitry" covers a single processor circuitry that executes some or all of the code from multiple modules. The term "group processor circuitry" covers processor circuitry that executes some or all of the code from one or more modules in conjunction with additional processor circuitry. References to multiple processor circuitry cover multiple processor circuitry on a discrete die, multiple processor circuitry on a single die, multiple cores of a single processor circuitry, multiple threads of a single processor circuitry, or a combination of the foregoing. The term "shared memory circuitry" covers a single memory circuitry that stores some or all of the code from multiple modules. The term "group memory circuitry" includes memory circuitry that stores some or all of the code from one or more modules in conjunction with additional memory.

[0050] The term "memory circuit" is a subset of the term "computer-readable medium." As used herein, the term "computer-readable medium" does not cover non-transitory electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask-mode read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0051] The apparatus and methods described in this application can be implemented, in part or in whole, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The function blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of a skilled technician or programmer.

[0052] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. A computer program may also include or depend on stored data. A computer program may encompass a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0053] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from source code; (iv) source code for execution by an interpreter; and (v) source code for compilation and execution by a just-in-time (JIT) compiler, etc. As an example only, source code may be written using syntax from languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, etc. Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK and

Claims

1. A vehicle comprising: The camera is configured to generate images of the vehicle's path; Radar sensors are configured to transmit radio frequency (RF) signals and receive radar echoes in the path of the vehicle; and The speed segmentation module is configured as follows: The radial velocity that generates the radar echo; The radar echoes are divided into R clusters of radar echoes, where R is an integer greater than zero; Project the R clustered radar echoes onto the image; R moving objects corresponding to the R clustered radar echoes are identified by segmenting R groups of image pixels that have image features similar to those of the pixels corresponding to the R clustered radar echoes. as well as The R radial velocities of the R clustered radar echoes are calculated respectively in response to the radial velocities of the R clustered radar echoes for each of the R moving objects.

2. The vehicle of claim 1, further comprising a driver assistance system configured to control at least one of the vehicle's speed and direction of travel in response to the R radial velocities of the R moving objects.

3. The vehicle according to claim 1, wherein the speed segmentation module identifies the R moving objects by identifying R groups of segmented pixels having image features similar to those of pixels corresponding to the R clustered radar echoes.

4. The vehicle of claim 3, wherein the speed segmentation module uses a neural network to identify the R groups of segmented pixels.

5. The vehicle of claim 4, wherein the neural network generates feature vectors for the pixels of the image.

6. The vehicle of claim 5, wherein the speed segmentation module identifies selected pixels for the R group of segmented pixels based on the feature vector.

7. The vehicle according to claim 4, wherein the speed segmentation module calculates the R radial velocities of the R groups of segmented pixels based on the average radial velocity of the radar echoes in the R clustered radar echoes.

8. The vehicle of claim 1, wherein the speed segmentation module removes outliers from the R clustered radar echoes.

9. The vehicle of claim 1, wherein the speed segmentation module selectively classifies radar echoes into one of static and dynamic states based on the vehicle's own speed.

10. The vehicle of claim 9, wherein the speed segmentation module selects dynamic radar echoes for clustering and does not select static radar echoes for clustering.

Images