Bank-based sampling of radar detection results

Abstract

A technique of identifying a feature of an object in a physical environment of a set of radar sensors includes accessing a plurality of bin identifiers assigned to respective radar detection results obtained by the set of radar sensors, the bin identifiers specifying respective bins. The binning has at least (i) a first dimension representing a range of distance values and (ii) a second dimension representing a range of Doppler values. The technique further includes selecting a radar detection result based on the binning identifier, the selecting producing a plurality of samples, and processing the plurality of samples to determine one or more characteristics of the object.

Description

Technical Field

[0001] This disclosure is generally directed to radar (radio detection and range) technology, and more specifically to radar processing using bin-based sampling. Background Technology

[0002] Radar systems are widely used in vehicles, robots, and other technologies that perform environmental sensing. For example, vehicles can be equipped with millimeter-wave (mmWave) radar sensors, which can locate and orient themselves towards objects in the detection environment. The sensors emit high-frequency electromagnetic signals and receive the returned radar detection results, such as reflections and other signal content. The signal processing circuitry digitizes the radar detection results and compares them with the emitted signals to generate a radar point cloud, i.e., a three-dimensional spatial map of the vehicle's environment.

[0003] Points in a radar point cloud represent radar detection results and have certain attributes. Examples of these attributes include location (e.g., xyz coordinates), time, distance (also known as "range"), Doppler (radial velocity toward or away from the sensor), and radar cross section (RCS), which indicates the intensity of the detection result and suggests the material of the reflector, such as metal, wood, or plastic.

[0004] In some implementations, the computer uses multiple encoders to process radar detection results, where each encoder performs a sampling operation on the radar detection results, followed by a grouping operation to identify groups of radar detection results around the sample. The samples and groups are then processed to identify features of objects in the environment. Summary of the Invention

[0005] According to one aspect of the invention, an apparatus is provided, comprising a control circuit system including a set of processors coupled to a memory, the control circuit system being configured and arranged to perform a method for identifying features of an object in a physical environment by a set of radar sensors, the method comprising: accessing a plurality of bin identifiers of corresponding radar detection results obtained by the set of radar sensors, the bin identifiers specifying corresponding bins, the bins having at least (i) a first dimension representing a range of distance values ​​and (ii) a second dimension representing a range of Doppler values; selecting radar detection results based on the bin identifiers, the selection generating a plurality of samples; and processing the plurality of samples to determine one or more features of the object.

[0006] According to one or more embodiments, a specific bin in the bin includes multiple radar detection results, and selecting the radar detection results includes selecting radar detection results from the specific bin using a pseudo-random process.

[0007] According to one or more embodiments, selecting the radar detection result further includes using the pseudo-random process to select at least a second radar detection result from the specific bin.

[0008] According to one or more embodiments, selecting the radar detection result further includes selecting at least one radar detection result from each sub-box containing at least one radar detection result.

[0009] According to one or more embodiments, the method further includes determining a desired number of samples from the plurality of samples, wherein at least one radar detection result is selected from each bin to generate a set of samples exceeding the desired number of samples, and wherein the method further includes pseudo-randomly removing samples from the set of samples such that only the desired number of samples remain.

[0010] According to one or more embodiments, processing the plurality of samples includes: forming a radar detection result group for a specific sample among the plurality of samples, at least in part based on the bin identifier of the specific sample and the bin identifier of the radar detection result; and providing the group as input to a neural network.

[0011] According to one or more embodiments, the method further includes receiving the plurality of bin identifiers from the set of radar sensors.

[0012] According to one or more embodiments, the method further includes: receiving range attributes and Doppler attributes of the corresponding radar detection results from the set of radar sensors; and assigning the plurality of bin identifiers to the corresponding radar detection results based at least in part on the range attributes and Doppler attributes of the corresponding radar detection results.

[0013] According to a second aspect of the invention, a method is provided for identifying features of an object in a physical environment by a set of radar sensors, comprising: generating data representing radar detection results obtained by the set of radar sensors, the data including (i) distance values ​​detected by the set of radar sensors and (ii) Doppler values ​​detected by the set of radar sensors; assigning bin identifiers to the radar detection results based at least in part on the distance values ​​and the Doppler values, the bin identifiers specifying corresponding bins, the bins having at least (i) a first dimension representing a range of distance values ​​and (ii) a second dimension representing a range of Doppler values; and sampling the radar detection results from the bins based on the bin identifiers to generate a plurality of samples, and then processing the samples to determine one or more features of the object.

[0014] According to one or more embodiments, sampling the radar detection results includes selecting at least one radar detection result from each bin that includes at least one radar detection result.

[0015] According to one or more embodiments, the size of the bins is determined at least in part based on the desired number of samples among the plurality of samples.

[0016] According to one or more embodiments, the size of the sub-bin is determined at least in part based on the density of radar detection results.

[0017] According to one or more embodiments, the size of the sub-bin is also dynamically adjusted, at least in part, based on changes in the density of the radar detection results.

[0018] According to one or more embodiments, the bin has at least one additional dimension representing any one of (i) elevation angle, (ii) azimuth angle and (iii) time, and wherein assigning bin identifiers to the radar detection results is also based on at least one of (i) elevation angle, (ii) azimuth angle and (iii) time.

[0019] According to one or more embodiments, sampling the radar detection results includes performing sampling operations on multiple sub-boxes in parallel.

[0020] According to a third aspect of the invention, a radar system is provided, comprising a set of radar sensors and a computerized device communicatively coupled to the set of radar sensors, the radar system being configured and arranged to: transmit a plurality of radar signals into a physical environment of the radar system; receive radar reflections of the radar signals returning from the physical environment; generate, based on the radar signals and the radar reflections, data representing radar detection results of objects in the physical environment, the data including (i) range values ​​detected by the set of radar sensors and (ii) Doppler values ​​detected by the set of radar sensors; assign bin identifiers to the radar detection results at least in part based on the range values ​​and the Doppler values, the bin identifiers representing bins covering (i) a corresponding range of range values ​​and (ii) a corresponding range of Doppler values, wherein the computerized device is configured and arranged to sample the radar detection results from the bins based on the bin identifiers, the sampling generating a plurality of samples, and processing the plurality of samples to determine one or more features of the object.

[0021] According to one or more embodiments, the computerized device, which is constructed and arranged to sample the radar detection results, is also constructed and arranged to select at least one radar detection result from each bin containing at least one radar detection result.

[0022] According to one or more embodiments, at least one of (i) the set of radar sensors and (ii) the computerized device is further constructed and arranged to determine the size of the bins based at least in part on the desired number of samples among the plurality of samples.

[0023] According to one or more embodiments, the bin has a first dimension representing distance, a second dimension representing Doppler, and at least one additional dimension representing any one of (i) elevation angle, (ii) azimuth angle, and (iii) time, and wherein the computerized device is further configured and arranged to assign the bin identifier based on at least one of (i) elevation angle, (ii) azimuth angle, and (iii) time.

[0024] According to one or more embodiments, the computerized device, which is constructed and arranged to sample the radar detection results based on the bin identifier, is also constructed and arranged to perform sampling operations on multiple bins in parallel. Attached Figure Description

[0025] The foregoing and other features and advantages will become apparent from the following description of specific embodiments, as illustrated in the accompanying drawings, in which similar reference numerals refer to the same or similar parts in different views. The drawings are not necessarily drawn to scale, but are intended to illustrate the principles of various embodiments.

[0026] Figure 1 This is a block diagram of an example radar system in which improved techniques can be implemented.

[0027] Figure 2 The illustration is based on one or more embodiments and can be provided by Figure 1 The flowchart shows an example activity performed by the radar system.

[0028] Figure 3 This is an example range-Doppler plot of radar detection results based on one or more embodiments, showing the example bin in which the radar detection results are located.

[0029] Figure 4 It is a partial map of an example distance-Doppler map, extended according to one or more embodiments to include at least one additional dimension of elevation, azimuth, and time.

[0030] Figure 5 This is a series of tables illustrating examples of binning-based sampling techniques according to one or more embodiments.

[0031] Figure 6 This is a partial map of an example distance-Doppler image according to one or more embodiments, illustrating an example of a binning-based grouping technique.

[0032] Figure 7 This is a partial plot of an example distance-Doppler plot based on one or more embodiments, illustrating an example of parallel processing techniques.

[0033] Figure 8This is a flowchart illustrating an example method for setting bin sizes according to one or more embodiments.

[0034] Figure 9 This is a flowchart illustrating an example method for identifying object features according to one or more embodiments.

[0035] Figure 10 This is a graph showing an example comparison between bin-based sampling and farthest-point sampling. Detailed Implementation

[0036] Furthest Point Sampling (FPS) has become the de facto standard for encoder sampling in radar systems because it can represent the radar scene with a high degree of uniformity. A "radar scene" refers to a radar point cloud or any modified version thereof, such as a labeled point cloud, a segmented point cloud, or any portion of a larger point cloud. FPS is performed by randomly selecting a first point in the scene, then selecting the second point furthest from the first point—that is, the second point among all other points in the scene that has the largest Euclidean distance to the first point (e.g., in xyz coordinates). Then, a third point is selected as the sample furthest from the first and second points. This process continues, selecting additional points as the furthest from all previously selected points. This process continues until the desired number of points is selected. Then, grouping is performed, identifying points within the desired Euclidean distance of each sampled point.

[0037] While FPS offers numerous advantages, its efficiency can be low due to the significant range calculations required. Computers commonly used in radar applications have limited processing power, and the numerous range calculations required by FPS consume a large portion of maximum throughput. Some of these computers employ coprocessors, such as vector processors, graphics processing units (GPUs), or tensor processing units (TPUs), to accelerate computation. However, due to compatibility issues when preparing such calculations for coprocessor execution, or the need for additional workloads, range calculations are often excluded from accelerated processing. This can overload the computer. Therefore, a more computationally efficient sampling method is needed.

[0038] The aforementioned requirements are addressed at least in part by improving the sampling technique for radar detection results in radar scenarios. This technique involves assigning bin identifiers (BIDs) to radar detection results, at least in part, based on the range (range) and Doppler (radial velocity) values ​​associated with the results. Each BID uniquely identifies a corresponding bin with at least a range dimension and a Doppler dimension. The technique also includes selecting detection results based on BIDs, such as selecting one detection result per BID, selecting two detection results per BID, and so on. This allows for uniform sampling across the entire radar scenario. Advantageously, sampling does not require the extensive range calculations required by FPS, or even any range calculations at all, thus reducing the computer workload. Less powerful computers can be used, or additional useful tasks can be loaded onto existing computers. Furthermore, the improved technique is easily parallelized because selecting samples from one bin can be done independently of selecting samples from other bins. The improved technique offers further advantages during grouping, as groups can be formed based on BIDs rather than range calculations.

[0039] Embodiments of the improved technique will now be described. It should be understood that such embodiments are provided as examples to illustrate certain features and principles, and are not intended to be limiting.

[0040] Figure 1 An example radar system 100 in which improved techniques can be practiced is shown. Here, radar system 100 includes a set of radar sensors 110 (i.e., one or more radar sensors) that are communicatively coupled to a computerized device 140, for example via cable or wireless technology, such as Bluetooth, Wi-Fi, Wireless Controller Area Network (CAN), etc. As an example, a single radar sensor 110 is shown, but any number of radar sensors 110 may be provided.

[0041] Radar sensor 110 includes a transceiver 112, such as an antenna configured to transmit and / or receive radio frequency (RF) signals, and a processing circuitry 114. In some arrangements, transceiver 112 is configured to both transmit radar signal 132 and receive radar signal 134, for example, signals reflected from object 136 in the physical environment 102 of sensor 110. In other arrangements, separate antennas are provided for transmission and reception. For illustrative purposes, processing circuitry 114 is assumed to be common to all radar sensors, including all transmitting and receiving antennas, but other arrangements are possible. All such arrangements are contemplated within the scope of this disclosure.

[0042] The processing circuitry system 114 includes various hardware and software components configured to generate and receive RF signals, perform signal processing, and extract attributes. These components are described herein as a signal processor 116 and a point cloud generator 118. The signal processor 116 is arranged to perform both analog and digital signal processing. The point cloud generator 118 is arranged to identify radar detection results 122, determine attributes 124 of the detection results 122, and provide the detection results 122 as points in a radar point cloud 126, which is a spatial representation of the detection results 122. Attributes 124 typically include range, Doppler, RCS, and time. Depending on the transceiver configuration, attributes 124 may also include elevation and / or azimuth. In some examples, the point cloud generator 118 is naturally configured to measure spatial position using polar coordinates (e.g., range, elevation, and azimuth) and convert these polar coordinates to Cartesian coordinates, e.g., coordinates in xyz space.

[0043] According to one or more embodiments, the processing circuitry 114 also includes a BID allocator 120. The bin identifier (BID) allocator 120 may be implemented as part of, for example, a software architecture, firmware, hardware device such as an application-specific integrated circuit (ASIC), or any combination thereof, running on a processor (not shown) of the processing circuitry 114. The BID allocator 120 is configured to assign BID 130 to radar detection results 122, such as those represented in radar point clouds 126 or other radar scenarios.

[0044] According to one or more embodiments, the BID assigner 120 assigns BID 130 to the radar detection results 122 based on the distance (range) and Doppler attributes determined by the point cloud generator 118. For example, the entire range of expected possible range values ​​is divided into uniformly sized range ranges, such as 1-meter ranges, 5-meter ranges, 10-meter ranges, etc. Similarly, the entire range of expected Doppler values ​​is divided into uniformly sized Doppler ranges, such as 5 km / h ranges, 10 km / h ranges, 15 km / h ranges, etc. Alternatively, non-uniformly sized ranges can be provided.

[0045] In this example, each range of distance values ​​is associated with multiple ranges of Doppler values, and each range of Doppler values ​​is associated with multiple ranges of distance values. Thus, the distance-Doppler map ( Figure 3The distance-Doppler range can be constructed as a two-dimensional representation in a distance-Doppler space, where the intersection of the distance and Doppler ranges forms corresponding two-dimensional bins. For example, each bin corresponds to a corresponding range of distance values ​​and a corresponding range of Doppler values. Preferably, the size of the bins in each dimension exceeds the resolution of the sensor 110 in the corresponding dimension. For example, each bin is large enough to accommodate 10, 100, or 1000 different distance values ​​and 10, 100, or 1000 different Doppler values. These are merely examples provided for illustrative purposes.

[0046] According to one or more embodiments, the BID assigner 120 assigns the same BID 130 to all radar detections 122 found within the same bin in range-Doppler space. All radar detections 122 assigned to the same bin and therefore having the same BID 130 have substantially similar range and Doppler values.

[0047] BID distributor 120 is shown in dashed lines because its presence in sensor 110 is optional. For example, BID distributor 120 could be placed in computerized device 140 as an alternative. However, since distance values ​​are readily available within sensor 110, it is generally preferred to place BID distributor 120 within sensor 110. For example, if placed within computerized device 140, BID distributor 120 might need to convert distance values ​​from xyz coordinates to polar coordinates, or explicitly request distance values ​​from sensor 110. These requirements are eliminated if BID distributor 120 is placed within sensor 110.

[0048] like Figure 1 It is also shown that the computerized device 140 includes a set of processors 142 and a memory 150. In this example, the set of processors 142 includes one or more processing chips and / or assemblies, such as one or more central processing units (CPUs), vector processors, GPUs, TPUs, etc. The memory 150 may include volatile memory, such as random access memory (RAM), and non-volatile memory, such as one or more read-only memory (ROM), disk drives, solid-state drives, etc. The set of processors 142 and the memory 150 together form a control circuitry system that is constructed and arranged to perform the various methods and functions described herein. Furthermore, the memory 150 includes various software structures implemented in the form of executable instructions. When the set of processors 142 executes the executable instructions, the set of processors 142 performs operations defined by the software structures. Although certain software structures are specifically shown and described, it should be understood that the memory 150 typically includes many other software components not shown, such as operating systems, various applications, processes, and background programs.

[0049] According to one or more embodiments, the memory 150 of the computerized device 140 "includes" a tracker 160, a segmenter 170, a set of encoders 180, and a classification head 190, i.e., implemented by executing software instructions. The tracker 160 is configured to, for example, identify objects in the radar point cloud 126 based on similarity in position and rate, and assign object identifiers to detected objects. In some examples, the tracker 160 is also configured to construct virtual bounding boxes around the identified objects and track the identified objects over time.

[0050] Segmenter 170 is configured to divide radar point cloud 126 into multiple segmented point clouds 172, for example, each object identified by tracker 160 has one segmented point cloud 172. Trackers and segmenters are known in the art and do not require further description.

[0051] The set of encoders 180 typically includes multiple encoders 180a to 180n arranged in cascade, wherein a single encoder is referred to herein as 180x. Each encoder 180x is constructed and arranged to perform sampling and grouping, and to produce point features representing aspects of objects in the physical environment 102. Typically, the set of encoders 180 includes at least three encoders, but any number of encoders, including a single encoder, can be provided.

[0052] The classification head 190 is configured to receive point features from the set of encoders 180 and provide an output in the form of a category label 192. For example, the category label 192 can identify the detected object 136 as a pedestrian, two-wheeled vehicle, car, or truck. In this example, the classification head 190 is implemented using a neural network, such as a convolutional neural network.

[0053] Vehicles, robots, or other devices equipped with radar system 100 can use category label 192 in any suitable manner. For example, a vehicle equipped with radar system 100 can display a driver's screen in which category label 192 is presented as a corresponding icon placed relative to the description of the vehicle itself, thereby providing a visual representation of the vehicle's surrounding environment as determined by radar system 100. As another example, other devices such as vehicles or robots can detect objects 136 in the physical environment 102 based on category label 192 and trigger appropriate responses based on such detections. The responses can be independent of operator input.

[0054] According to one or more embodiments, encoder 180x in the set of encoders 180 includes a BID-based sampler 182, a BID-based grouper 184, and a neural network 186. The BID sampler 182 is configured to perform sampling based on BID 130, rather than sampling radar detection results 122 using farthest point sampling (FPS), which involves numerous distance calculations and can be computationally expensive. For example, the BID-based sampler 182 is configured to access BID 130 assigned to radar detection results 122 in the radar scene and select radar detection results 122 from BID 130 in an unbiased manner, such as by selecting one radar detection result 122 per BID, selecting two radar detection results 122 per BID, and so on. Because different BIDs 130 correspond to different distances and Doppler ranges, selecting samples based on BID 130 results in uniform sampling of the radar scene. Such uniform sampling provides similarities to FPS but at a lower computational cost.

[0055] According to one or more embodiments, the BID grouper 184 is configured to group radar detection results 122 based on BID 130. For example, the BID grouper 184 may form a group by including all radar detection results 122 with the same BID 130 as the sample in one group. Thus, distance calculations for sampling and grouping are minimized or completely avoided. However, it should be understood that one or more embodiments may provide the BID grouper 184 with a conventional grouper, such as a grouper that uses distance calculations to determine group members. Therefore, the BID grouper 184 can be considered optional.

[0056] Within encoder 180x, samples and groups are provided as input to neural network 186, which is configured to generate point features based on the samples and groups. For example, point features could be features of object 136. Such features may correspond to physically observable characteristics, but they may also (or alternatively) reflect correlations, convolutions, and / or other combinations between various attributes of radar detection results 122.

[0057] Figure 2 Example method 200 for operating a radar system is shown, for example Figure 1 The radar system 100 is depicted in the diagram. For example, the various actions of method 200 can be performed by the set of sensors 110 or by the computerized device 140, or by both the set of sensors 110 and the computerized device 140.

[0058] At 210, the set of sensors 110 is operated to transmit RF signal 132 and receive RF signal 134. The received signal 134 may include a reflection of the transmitted RF signal 132 from an object 136 in the physical environment 102.

[0059] At 220, data indicating radar detection results 122 and associated attributes 124 are generated. For example, the set of sensors 110 identifies radar detection results 122 and assigns attributes 124 of range, Doppler, time, and RCS to the radar detection results 122. In some examples, the set of sensors 110 also assigns attributes 124 of elevation and / or azimuth. The set of sensors 110 arranges the radar detection results 122 and attributes 124 into a radar point cloud 126, which can be provided, for example, as a data structure such as an array or tensor.

[0060] At 230, a bin identifier (BID) 130 is assigned to the radar detection result 122. The assigned BID 130 specifies a bin, which has at least a first dimension representing the range of range values ​​and a second dimension representing the range of Doppler values. Preferably, the task of assigning BID 130 to the detection result 122 is performed by the set of sensors 110, but alternatively, it can also be performed by a computerized device 140.

[0061] At 240, the first encoder 180a samples the radar detection results 122 based on the assigned BID 130. For example, the encoder 180a running on the computerized device 140 selects multiple samples from the radar detection results 122 in the radar scene by selecting radar detection results across different BIDs 130. Preferably, the selection is performed in an unbiased manner, such as selecting one radar detection result per BID, two radar detection results per BID, and so on. In the case where the same BID 130 is assigned to multiple radar detection results 122 (i.e., multiple detection results 122 are located in the same bin), the selection of one or more detection results for BID 130 can be performed pseudo-randomly. For example, suppose the same BID 130 is assigned to ten different detection results 122. Such detection results can be placed in a list, and the list can be shuffled, for example, using shuffling methods common in certain computer languages, such as Python. Then, the first detection result in the shuffled list can be taken as the sample pseudo-randomly selected for BID 130. If N samples are required per BID, the first N detection results from the shuffled list can be taken as N pseudo-randomly selected samples. It should be understood that the shuffling method is merely one example, as there are many other ways to pseudo-randomly select items from a list, and this disclosure is intended to cover any of them. Furthermore, embodiments are not limited to pseudo-random sampling of radar detection results within the same bin. For example, samples can be obtained in other ways, such as by sorting the radar detection results 122 according to a specific dimension (e.g., range, Doppler, elevation, azimuth, time, etc.) and selecting the detection results with the maximum value along the sorting dimension.

[0062] At 250, the first encoder 180a groups radar detection results 122 around a sample from a plurality of samples based on the assigned BID 130. For example, assuming ten different detection results 122 are assigned to the same BID 130, as described above, those ten detection results can form a group. The group can be associated with a sample pseudo-randomly selected for said BID 130.

[0063] At position 260, encoder 180a processes samples and groups to identify features in the radar scene. For example, encoder 180a applies multiple samples and associated groups to neural network 186 to generate point features.

[0064] At 270, the computerized device 140 determines whether the current encoder 180a is the last encoder among encoders 180a to 180n. If not, then method 200 returns to step 240 to continue sampling (240), grouping (250), and processing (260) for the next encoder 180b. Method 200 continues in this manner until sampling, grouping, and processing of the last encoder 180n are completed. At this point, the operation continues to 280, where the classification head 190 processes the accumulated results from encoders 180a to 180n and assigns a category label 192. This category label can identify the type of detected object, such as a pedestrian, a car, or some other type of object.

[0065] Figure 3 An example range-Doppler map 300 according to one or more embodiments is shown. As shown, the range-Doppler map 300 has two dimensions: a first dimension representing distance 310, for example, with values ​​ranging from 0 to 100 meters, and a second dimension representing Doppler 320, for example, with values ​​ranging from -50 km / h to +50 km / h. These ranges are merely examples. Various radar detection results 122 are represented as points on the range-Doppler map 300, where each radar detection result 122 has a distance value and a corresponding Doppler value. For ease of understanding, a sparse arrangement of the radar detection results 122 is depicted; however, it should be understood that the fill density of the radar detection results 122 in the range-Doppler map is generally higher than shown.

[0066] The distance-Doppler map 300 comprises multiple bins 330, each representing a range of distance values, such as a 10-meter range, and a corresponding range of Doppler values, such as a 10 km / h range. In the depicted example, each bin 330 covers a rectangular area. A total of 100 bins are shown.

[0067] According to one or more embodiments, each bin 330 has a unique BID 130. For example, the first bin 330 (lower left) has BID 130 0, and the BIDs increase sequentially from left to right and from bottom to top, up to a maximum value of 99 in the upper right.

[0068] The specific method of assigning BID 130 can vary. For example, BID 130 can increase from top to bottom or from right to left, instead of from bottom to top and from left to right as shown. They can increase in increments other than one. However, it is preferred to assign BID 130 to bins 330 in a deterministic manner, such as according to some convention or formula, making it possible to determine the BID of neighboring bins 330 based on the BID of the current bin.

[0069] although Figure 3 BID 130 is shown as being assigned to bin 330, but this is merely for illustration, as in practice BIDs are typically assigned to radar detection results 122. For example, a BID 130 may be assigned to a radar detection result 122 based on its range and Doppler value. As a specific example, if the range of detection result 122 is 43 m and its Doppler value is -47 km / h, then detection result 122 could be assigned BID 4, since BID = 4 covers the two-dimensional area including detection result 122. Therefore, it is evident that detection results 122 falling within the same bin 330 have the same BID 130, while detection results 122 falling within different bins have different BIDs 130.

[0070] Figure 4 This is shown as a portion of the extended range-Doppler map 300, including an additional dimension. For example, the third dimension 410 represents elevation, azimuth, or time. Any number of additional dimensions can be provided. Each additional dimension of the range-Doppler map 300 adds a dimension to the bins 330, typically multiplying the total number of bins 330, and thus multiplying the total number of possible BIDs 130. For example, if the third dimension 410 corresponds to elevation, then assigning a BID to the detection result 122 is based not only on distance and Doppler but also on elevation. If a fourth dimension, azimuth, is added, then assigning a BID to the detection result 122 is based not only on distance, Doppler, and elevation but also on azimuth, and so on.

[0071] Figure 5 An example arrangement for sampling radar detection results 122 is shown according to one or more embodiments. For example, such sampling may be performed by... Figure 1 The BID sampler 182 is executed.

[0072] Starting with the left figure, Table 510 shows the range and Doppler value of radar detection result 122, along with the associated BID 130. Each row 502 in Table 510 represents a different detection result 122. In total, row 502 represents 32 detection results 122, which is... Figure 3 The total number of detection results 122 shown.

[0073] The probe results 122 in Table 510 are sampled (520) to produce the sampled results shown in Table 530. It can be observed that Table 510 (above) includes multiple probe results 122 for certain BIDs (e.g., 6, 52, 68, and 78). However, here, in this example, sampling 520 pseudo-randomly selects a single probe result 122 for such BIDs to be included in Table 530. The result of sampling 520 is a set of samples 532, comprising a total of 26 samples.

[0074] In some examples, the BID sampler 182 ( Figure 1 The sample set 532 is configured to select a desired number of samples 560, for example, 20 samples. To achieve the desired number of samples 560, the set of samples 532 is reduced (540) to the desired number of samples 560 by pseudo-randomly removing samples from the set of samples 532 until the desired number 560 is reached. An example of the reduction result is shown in Table 550. It should be understood that the reduction 540 should be performed pseudo-randomly to avoid bias in the remaining samples.

[0075] If the BID sampler 182 acquires fewer samples than the required number 560 (e.g., selecting one probe result per BID), then a second probe result can be selected from each bin 330 with a second probe result, and this process continues in a similar manner if a larger number of probe results is needed, until the required number 560 is reached. If adding probe results 122 per BID causes the total number of samples to exceed the required number 560, then samples can be pseudo-randomly removed, as described above, to reach the required number 560. If adding probe results 122 still does not reach the required number 560, then samples can be pseudo-randomly repeated to achieve the required number 560.

[0076] Figure 6 Distance-Doppler graph 300 is shown according to one or more embodiments. Figure 3 This section, as part of the document, demonstrates example grouping methods. For instance, such grouping can be achieved by... Figure 1 The BID grouper 184 is executed.

[0077] For example, suppose a radar detection result 122a has been selected as the sample to be grouped. The grouping performed by the BID grouper 184 is not based on calculating distances to nearby points and selecting points within a specific distance threshold for inclusion in the group; instead, it selects only all radar detection results 122 that share the same BID 130 as sample 122a. In the example shown, detection result 122a has been assigned BID 22, and only one other detection result 122b has been assigned the same BID. Therefore, a group can be formed for sample 122a by including both detection results 122a and 122b, which share the same BID.

[0078] According to one or more embodiments, groups can be formed not only by bins having the same BID 130 as the sample, but also by adjacent bins 330 in the distance-Doppler map 300. Thus, for example, unlike the group described above which only contains two detection results 122a and 122b, groups can be formed not only by bins having BID 22, but also by adjacent bins, for example, bins having BIDs 11, 12, 13, 21, 23, 31, 32, and 33. It should be understood that grouping can be performed flexibly, and a larger number of bins can be included as needed. The examples shown are merely illustrative.

[0079] Figure 7 An example arrangement for parallel sampling is shown according to one or more embodiments. For example, such sampling may be performed by... Figure 1 The BID sampler 182 is executed. Similarly, only the distance-Doppler graph 300 is shown. Figure 3 Part of ).

[0080] Here, multiple worker units 710, 720, 730, 740, 750, 760, 770, 780, and 790 are assigned to corresponding BIDs 11, 12, 13, 21, 22, 23, 31, 32, and 33 for performing sampling (and optional grouping). Worker units 710, 720, 730, 740, 750, 760, 770, 780, and 790 can be corresponding threads, CPU cores, or any other processing structure that can work in parallel.

[0081] As previously described, sampling may involve pseudo-randomly selecting one or more probe results 122 from each bin 330 for each BID 130. Therefore, in this example, the task of sampling from one bin does not depend on the task of sampling from any other bin. Thus, sampling from different bins can be performed independently, in parallel, and simultaneously. It should be understood that the option of parallel processing using the previous FPS method is limited because each sample depends on all previous samples, hindering the ability to independently assign different samples to different work units.

[0082] Figure 8 An example method 800 for setting bin sizes according to one or more embodiments is shown. For example, method 800 may be performed by the set of sensors 110 and / or by a computerized device 140 (…). Figure 1 ) Implement.

[0083] At 810, at least in part based on the required number of samples and / or one or more aspects of the radar scene, such as range-Doppler space ( Figure 3 The density of the radar detection results 122 in the radar system determines the size of the bins 330. For example, if the encoder 180x is programmed to acquire 100 samples of a radar scene, the radar system 100 can divide the range-Doppler space into 100 uniformly sized bins 330. Each bin 330 then extends to cover one-tenth of the entire range range and one-tenth of the entire Doppler range. However, if the radar scene is expected to include different areas with varying densities of the radar detection results 122, smaller bins can be used, such as bins extending to cover one-twentieth or one-fiftieth of the entire range in each dimension. In this way, the acquired samples can more accurately reflect the shape of smaller features. In some examples, the BID 130 can be assigned at a fine-grained level, but adjacent bins 330 can be merged into larger bins to sample according to various aspects of the radar scene, such as density. Many variations are envisioned.

[0084] At point 820, the size of bin 330 is dynamically adjusted based on changes in the radar scene, such as changes in the density or other features of the radar detection result 122. For example, the set of sensors 110 can monitor the characteristics of the radar point cloud 126 and adjust the size of bin 330 to achieve a more optimized balance between bin size and sampling performance, which can be evaluated based on classification quality, computational efficiency, or other factors.

[0085] Figure 9 An example method 900 is shown to identify features of objects in a physical environment 102 of a set of radar sensors 110, and an overview of some of the aforementioned features is provided. For example, method 900 typically consists of combining... Figure 1The software structure described in the radar system 100 is executed, the software structure residing in the memory 150 of the computerized device 140 and run by the set of processors 142. The various actions of method 900 can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than that shown can be constructed, which may include performing several actions simultaneously.

[0086] At 910, data representing the radar detection results 122 obtained by the set of radar sensors 110 is generated. For example, the data can be provided in a data structure representing a radar point cloud 126 or other radar scene. The data includes (i) the range value 310 detected by the set of radar sensors 110, and (ii) the Doppler value 320 detected by the set of radar sensors 110.

[0087] At 920, a bin identifier (BID) 130 is assigned to the radar detection result 122, at least in part based on the range value 310 and the Doppler value 320. BID 130 specifies a corresponding bin 330. Bin 330 has at least (i) a first dimension representing the range of the range value 310, and (ii) a second dimension representing the range of the Doppler value 320. In some examples, the set of sensors 110 is assigned BID 130. In other examples, a computerized device 140 is assigned BID 130.

[0088] At position 930, based on BID 130, 520 samples are taken from the radar detection result 122 from sub-bin 330. Figure 5 This can be used to generate multiple samples, such as the set of samples 532 shown in Table 530, or to reduce the number of samples 540 or otherwise adjust the number of samples as shown in Table 550.

[0089] At position 940, multiple samples are processed to determine one or more features of objects in the radar scene. For example, encoder 180x can form groups around corresponding samples among the multiple samples, and the samples and groups can be input into neural network 186 for extracting point features. Point features can be applied to continuous encoder 180x and / or classification head 190 to generate object labels 192, such as labels identifying objects as pedestrians, two-wheeled vehicles, cars, trucks, etc.

[0090] In some examples, method 900 may be embodied as a computer program product comprising one or more non-transitory computer-readable storage media 950, such as magnetic disks, magnetic tapes, laser disks, DVDs, optical discs, flash drives, solid-state drives, secure digital (SD) chips or devices, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Any number of computer-readable media may be used. The media may be encoded with instructions that, when executed on one or more computers or other processors, perform one or more processes described herein. Such media may be considered an article of manufacture or a machine and may be transported from one machine to another.

[0091] Figure 10 Figure 1000 illustrates an example comparison between bin-based sampling and FPS according to one or more embodiments. Figure 1000 shows the relationship between the average sampling run time, measured in seconds, and the point cloud size, measured in the number of radar detection results 122. The tests were performed based on a standard dataset, and each point depicted in Figure 1000 is the average of ten different runs. Curve 1010 depicts the results of bin-based sampling, and curve 1020 depicts the results of FPS.

[0092] Bin-based sampling has the consistent advantage across all point cloud sizes. It can be observed that the complexity of bin-based sampling is O(m), where m is the number of bins. This is typically lower than the number of points in the point cloud. In contrast, the complexity of FPS is O(nk), where k is the number of samples and n is the total number of radar detection results. Therefore, when k is close to n, the complexity of FPS is O(n^2). 2 ).

[0093] The performance benefits of bin-based sampling do not come at the expense of classification accuracy. Macro-average F1 scores were calculated for four training categories: vehicles, pedestrians, two-wheelers, and static objects, with virtually no observable difference between the two sampling techniques (scores were both between 87% and 88%). Therefore, bin-based sampling provides clear performance benefits without sacrificing quality.

[0094] An improved technique for sampling radar detection results 122 in a radar scene has been described. The technique includes assigning bin identifiers (BIDs) 130 to the radar detection results 122, at least in part, based on range (range) values ​​310 and Doppler (radial velocity) values ​​320 associated with them. The BID 130 uniquely identifies a corresponding bin 330 that has at least a range dimension and a Doppler dimension. The technique also includes selecting detection results 122 based on the BID 130, such as selecting one detection result per BID, selecting two detection results per BID, and so on. This allows for uniform sampling of the entire radar scene. Advantageously, sampling does not require the extensive range calculations required by FPS, or even any range calculations at all, thus reducing the computer workload. Less powerful computers can be used, or additional useful tasks can be loaded onto existing computers.

[0095] Some embodiments are directed to an apparatus including a control circuitry system comprising a set of processors coupled to a memory. The control circuitry system is configured and arranged to perform a method for identifying features of objects in a physical environment by a set of radar sensors. The method includes accessing a plurality of bin identifiers assigned to corresponding radar detection results obtained from the set of radar sensors, the bin identifiers specifying corresponding bins, each bin having at least (i) a first dimension representing a range of range values ​​and (ii) a second dimension representing a range of Doppler values. The method further includes selecting radar detection results based on the bin identifiers, the selection generating a plurality of samples, and processing the plurality of samples to determine one or more features of the object.

[0096] According to one or more other embodiments, a specific bin in the bin includes multiple radar detection results, and selecting radar detection results includes selecting radar detection results from the specific bin using a pseudo-random process.

[0097] According to one or more other embodiments, selecting radar detection results further includes using a pseudo-random process to select at least a second radar detection result from the specific bin.

[0098] According to one or more other embodiments, selecting radar detection results further includes selecting at least one radar detection result from each sub-box containing at least one radar detection result.

[0099] According to one or more other embodiments, the method further includes determining a desired number of samples from the plurality of samples. According to such embodiments, at least one radar detection result is selected from each bin to generate a set of samples exceeding the desired number of samples, and the method further includes pseudo-randomly removing samples from the set of samples such that only the desired number of samples remain.

[0100] According to one or more other embodiments, processing the plurality of samples includes forming a radar detection result group for a specific sample among the plurality of samples, at least in part based on the bin identifier of the specific sample and the bin identifier of the radar detection result, and providing the group as input to a neural network.

[0101] According to one or more other embodiments, the method further includes receiving a plurality of bin identifiers from the set of radar sensors.

[0102] According to one or more other embodiments, the method further includes receiving range and Doppler attributes of corresponding radar detection results from the set of radar sensors, and assigning a plurality of bin identifiers to the corresponding radar detection results based at least in part on the range and Doppler attributes of the corresponding radar detection results.

[0103] An additional embodiment is a method for identifying features of objects in a physical environment using a set of radar sensors. The method includes generating data representing radar detection results obtained by the set of radar sensors, the data including (i) range values ​​detected by the set of radar sensors and (ii) Doppler values ​​detected by the set of radar sensors. The method further includes assigning bin identifiers to the radar detection results based at least in part on the range values ​​and Doppler values, the bin identifiers specifying corresponding bins, each bin having at least (i) a first dimension representing a range of range values ​​and (ii) a second dimension representing a range of Doppler values. The method further includes sampling the radar detection results from the bins based on the bin identifiers to generate multiple samples, and then processing the samples to determine one or more features of the object.

[0104] According to one or more other embodiments, sampling radar detection results includes selecting at least one radar detection result from each bin that includes at least one radar detection result.

[0105] According to one or more other embodiments, the method further includes determining the bin size based at least in part on the desired number of samples among the plurality of samples.

[0106] According to one or more other embodiments, determining the bin size is also based at least in part on the density of radar detection results.

[0107] According to one or more other embodiments, the method further includes dynamically adjusting the bin size based at least in part on changes in radar detection density.

[0108] According to one or more other embodiments, the bin has at least one additional dimension representing any one of (i) elevation angle, (ii) azimuth angle and (iii) time, and the bin identifier is assigned to the radar detection results based on at least one of (i) elevation angle, (ii) azimuth angle and (iii) time.

[0109] According to one or more other embodiments, sampling radar detection results includes performing sampling operations on multiple sub-boxes in parallel.

[0110] Another embodiment is directed to a radar system comprising a set of radar sensors and a computerized device communicatively coupled to the set of radar sensors. The radar system is configured to transmit multiple radar signals into a physical environment, receive radar reflections of the radar signals returning from the physical environment, and generate data representing radar detection results of objects in the physical environment based on the radar signals and radar reflections, the data including (i) range values ​​detected by the set of radar sensors and (ii) Doppler values ​​detected by the set of radar sensors. The radar system is also configured to assign bin identifiers to the radar detection results, at least in part, based on the range values ​​and Doppler values, the bin identifiers representing bins covering (i) a corresponding range value range and (ii) a corresponding Doppler value range. Additionally, the computerized device is configured to sample the radar detection results from the bins based on the bin identifiers, the sampling generating multiple samples, and processing the multiple samples to determine one or more features of the object.

[0111] According to one or more other embodiments, the computerized device configured and arranged to sample radar detection results is also configured and arranged to select at least one radar detection result from each bin containing at least one radar detection result.

[0112] According to one or more other embodiments, at least one of (i) the set of radar sensors and (ii) the computerized device is also constructed and arranged to determine the bin size based at least in part on the desired number of samples among the plurality of samples.

[0113] According to one or more other embodiments, the bins have a first dimension representing distance, a second dimension representing Doppler, and at least one additional dimension representing any one of (i) elevation angle, (ii) azimuth angle, and (iii) time. The computerized device is also configured and arranged to assign bin identifiers based on at least one of (i) elevation angle, (ii) azimuth angle, and (iii) time.

[0114] According to one or more other embodiments, the computerized device configured to sample radar detection results based on bin identifiers is also configured to perform sampling operations on multiple bins in parallel.

[0115] Some embodiments have been described, and many alternative embodiments or variations can be made. For example, although bin-based sampling has been described above as applying to all encoders 180, i.e., encoders 180a to 180n, this is merely an example. Alternatively, some encoders 180 may continue to use FPS or other techniques, especially where encoder 180 samples a smaller set of radar detection results 122. Therefore, while bin-based sampling is preferred for all encoders 180, it is not necessary for all encoders 180 to do so.

[0116] Furthermore, although the embodiments have been described in conjunction with the encoder 180 that performs feature detection, other embodiments can be constructed in the context of other functions, such as tracking, segmentation, or any function involving uniform sampling 520 of radar detection results 122 in a radar scene.

[0117] Furthermore, although features have been shown and described with reference to specific embodiments herein, such features may be included and thereby incorporated in any of the disclosed embodiments and variations thereof. Therefore, it should be understood that features disclosed in connection with any embodiment are included in any other embodiment.

[0118] As used throughout this document, the words “comprising,” “including,” “containing,” and “having” are intended to describe in an open-ended manner certain items, steps, elements, or aspects of something. Furthermore, as used herein, and unless specifically stated otherwise, the word “set” means one or more of something. This is true whether the phrase “set of” is followed by a singular or plural object, and whether it is combined with a singular or plural verb. Moreover, a “set” of elements can describe fewer than all existing elements. Therefore, there may be additional elements of the same kind that are not part of the set. Additionally, ordinal expressions, such as “first,” “second,” “third,” etc., may be used herein as adjectives for identification purposes. Unless explicitly indicated, these ordinal expressions are not intended to imply any ordering or sequence. Thus, for example, a “second” event may occur before or after a “first” event, or even if the first event did not occur. Furthermore, the designation of a particular element, feature, or action as “first” herein should not be construed as requiring the existence of a “second” or other such element, feature, or action. Rather, a “first” item may be unique. Additionally, unless explicitly stated otherwise, “based on” is intended to be non-exclusive. Therefore, unless explicitly stated otherwise, “based on” should be interpreted as meaning “at least partially based on.” Furthermore, although the term “user” as used herein may refer to a human being, the term is also intended to encompass non-human entities such as robots, automated programs, and other computer-implemented programs and technologies. Although certain embodiments are disclosed herein, it should be understood that these embodiments are provided as examples and should not be construed as limiting.

[0119] Therefore, those skilled in the art will understand that various changes in form and detail may be made to the embodiments disclosed herein without departing from the scope of the appended claims.

Claims

1. An apparatus, comprising: including control circuitry including a set of processors coupled to a memory, the control circuitry constructed and arranged to perform a method of identifying a characteristic of an object in a physical environment of a set of radar sensors, the method including: accessing a plurality of bin identifiers assigned to respective radar detections by the set of radar sensors, the bin identifiers specifying respective bins, a bin having at least (i) a first dimension representing a range of distance values and (ii) a second dimension representing a range of Doppler values; selecting radar detections based on the bin identifiers, the selecting resulting in a plurality of samples; and processing the plurality of samples to determine one or more characteristics of the object.

2. The apparatus of claim 1, wherein, A particular bin of the bins includes a plurality of radar detections, and wherein selecting the radar detections includes selecting radar detections from the particular bin using a pseudo-random process.

3. The apparatus of claim 2, wherein, Selecting the radar detections further includes selecting at least a second radar detection from the particular bin using the pseudo-random process.

4. The apparatus of claim 1, wherein, Selecting the radar detections further includes selecting at least one radar detection from each bin containing at least one radar detection.

5. The apparatus of claim 4, wherein, The method further includes determining a required number of samples in the plurality of samples, wherein selecting at least one radar detection from each bin results in a set of samples having a number exceeding the required number of samples, and wherein the method further includes pseudo-randomly removing samples from the set of samples such that only the required number of samples remain.

6. The apparatus of claim 1, wherein, Processing the plurality of samples includes: for a particular sample in the plurality of samples, forming a group of radar detections based at least in part on a bin identifier of the particular sample and bin identifiers of the radar detections; and providing the group as input to a neural network.

7. The apparatus of claim 1, wherein, The method further includes receiving the plurality of bin identifiers from the set of radar sensors.

8. The apparatus of claim 1, wherein, The method further includes: receiving distance attributes and Doppler attributes of the respective radar detections from the set of radar sensors; and assigning the plurality of bin identifiers to the respective radar detections based at least in part on the distance attributes and Doppler attributes of the respective radar detections.

9. A method of identifying characteristics of an object in a physical environment of a set of radar sensors, characterized in that, including: generating data representing radar detections by the set of radar sensors, the data including (i) distance values detected by the set of radar sensors and (ii) Doppler values detected by the set of radar sensors; assigning bin identifiers to the radar detections based at least in part on the distance values and the Doppler values, the bin identifiers specifying respective bins, a bin having at least (i) a first dimension representing a range of distance values and (ii) a second dimension representing a range of Doppler values; and based on the bin identifiers, sampling radar detections from the bins to result in a plurality of samples, the samples thereafter being processed to determine one or more characteristics of the object.

10. A radar system, characterized by including a set of radar sensors and a computerized device communicatively coupled to the set of radar sensors, the radar system constructed and arranged to: transmitting a plurality of radar signals into a physical environment of the radar system; receiving radar reflections of the radar signals back from the physical environment; generating, based on the radar signals and the radar reflections, data representing radar detections of objects in the physical environment, the data including (i) range values detected by the set of radar sensors and (ii) Doppler values detected by the set of radar sensors; assigning, based at least in part on the range values and the Doppler values, bin identifiers to the radar detections, the bin identifiers representing bins covering (i) ranges of corresponding range values and (ii) ranges of corresponding Doppler values, wherein the computerized device is structured and arranged to sample radar detections from the bins based on the bin identifiers, the sampling resulting in a plurality of samples, and to process the plurality of samples to determine one or more characteristics of the objects.

Images