Fuzzy labeling of low-level electromagnetic sensor data

Abstract

This document describes techniques and systems for fuzzy labeling of low-level electromagnetic sensor data. Sensor data in the form of an energy spectrum is obtained, and points within an estimated geographic boundary of a scatterer represented by a tail are labeled with a value of one. Remaining points of the energy spectrum are labeled with a value between zero and one, where each respective remaining point is farther from the geographic boundary, the smaller the value. The fuzzy labeling process can utilize deeper information available from the energy distribution in the energy spectrum. A model can be trained to effectively label energy spectrum plots in this manner. This can result in lower computational cost than other labeling methods. Furthermore, false detections by the sensor can be reduced, resulting in more accurate detection and tracking of objects.

Description

Background Technology

[0001] The availability of labeled data is a prerequisite for supervised learning tasks used to train neural network models. The successful deployment of machine learning systems largely depends on robust labeled datasets. In contrast to vision and natural language processing, automotive electromagnetic sensors lack sufficient high-quality labeled data, thus limiting the range of supervised learning tools available for electromagnetic sensor data processing. Summary of the Invention

[0002] This document describes a technique and system for fuzzy labeling of low-level electromagnetic sensor data. Sensor data in the form of an energy spectrum is obtained, and points within the estimated geographic boundary of the scatterer, represented by a smear, are labeled with a value of one. Remaining points in the energy spectrum are labeled with values ​​between zero and one, with the value decreasing as each corresponding remaining point moves further from the geographic boundary. The fuzzy labeling process can utilize deeper information available from the energy distribution in the energy spectrum. Models can be trained to efficiently label energy spectra in this manner. This can result in lower computational costs compared to other labeling methods. Furthermore, false detections by the sensor can be reduced, leading to more accurate object detection and tracking.

[0003] This invention presents a simplified concept related to the fuzzy labeling of low-level electromagnetic sensor data, which is further described in the detailed description and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Although described primarily in the context of automotive radar sensors, the techniques for fuzzy labeling of low-level electromagnetic sensor data can be applied to other applications that require robustly labeled data associated with electromagnetic sensors for training models. Furthermore, these techniques can also be applied to other sensor systems that provide energy spectra as low-level data. Attached Figure Description

[0004] This document describes in detail one or more aspects of the obscure labeling of low-level electromagnetic sensor data with reference to the following figures, wherein the same numbers are used throughout the figures to indicate similar components:

[0005] Figure 1 An example training environment for fuzzy labeling of low-level electromagnetic sensor data according to the techniques of this disclosure is shown;

[0006] Figure 2 An example environment in which the technique according to this disclosure can be applied to fuzzy labeling of low-level electromagnetic sensor data is shown;

[0007] Figure 3An example vehicle according to the technology of this disclosure is shown, the example vehicle including a system configured to utilize a model trained using fuzzy labels with low-level electromagnetic sensor data;

[0008] Figure 4 A graph of the energy spectrum labeled using fuzzy markers of low-level electromagnetic sensor data according to the technique of this disclosure is shown;

[0009] Figure 5 The LiDAR bounding box and radar bounding box, projected onto a radar image, are shown according to the technique of this disclosure for blurry markings of low-level electromagnetic sensor data.

[0010] Figure 6 An example method for fuzzy labeling of low-level electromagnetic sensor data according to the technology of this disclosure is shown. Detailed Implementation

[0011] Overview

[0012] Labeling electromagnetic sensor (e.g., radar) data individually can be rare due to the spectral tails of targets in different dimensions (e.g., azimuth, elevation) (e.g., radar leakage in sideangular bins) and the difficulty of separating and segmenting objects. For example, a finite-sized scatterer (e.g., an object) produces a radar echo of a sinusoidal function. The energy distribution of this function along a specific dimension (e.g., azimuth) can show a peak at the location of the scatterer, and sidelobes with decreasing intensity on either side of the peak. Therefore, segmenting objects or identifying their boundaries in radar echoes can be challenging. In contrast, light detection and ranging (LiDAR) systems produce distinct 3D point clouds, and cameras produce objects with sharp boundaries in two angular dimensions. Object segmentation and labeling are much easier to achieve using LiDAR and cameras.

[0013] Some current methods for labeling radar data can use rigid markers generated by a LiDAR or camera, which collects data simultaneously with the radar system. Object markers can be generated using LiDAR or camera images and then projected onto the radar data to define rigid boundaries for various labeled objects.

[0014] Some existing methods for labeling LiDAR and camera images segment objects using their geometric boundaries and apply a binary label to that pixel in the image based on whether it is inside or outside the boundary. Pixels inside the boundary can be labeled as 1 (positive) or 0 (negative). Due to radar spectral tailing, determining the geometric boundaries of objects solely from radar data is difficult, if not impossible. Furthermore, useful information in the extended radar reflection spectrum may be lost if the geometric boundaries derived from LiDAR or camera images are projected onto radar data and binary labels are applied. Similarly, geometric information, crucial for detecting and classifying moving objects, is absent in the distance rate of change dimension. If radar data is labeled in a binary manner (only positive and negative), a neural network is trained to learn rigid boundaries, for example, in the following way.

[0015]

[0016] This is not easily discernible in radar data and is even more difficult for neural networks to learn.

[0017] In contrast, this document describes a fuzzy labeling system for radar data that segments and labels objects in radar signal space (e.g., radar spectrograms, radar images) at the pixel level. In this context, "fuzzy labeling" refers to a non-binary labeling method that assigns a label between 0 and 1 to pixels in radar echoes (e.g., radar spectral tails) outside the geometric boundaries of objects determined from LiDAR or radar data. Each pixel is not labeled as an independent data point. The properties of a pixel and its label are spatially and temporally related.

[0018] As described in this paper, the fuzzy labeling system is based on radar signals (e.g., radar echoes) and takes into account the soft boundaries of objects seen in radar images. This fuzzy labeling process uses low-level radar data (e.g., time series, uncompressed data cubes, lossless Fast Fourier Transform (FFT)) that leverages in-depth information obtainable from the distribution of energy corresponding to a target across ranges of distance, rate of change of distance, azimuth, and elevation. The fuzzy labeling system can locate objects in each of the four dimensions and segment the energy distribution, containing useful information corresponding to the object for detection, tracking, classification, etc. This process maximizes the information extracted from the radar signals and can be used as input to a neural network that can be trained to detect and classify moving and stationary objects. In this way, false positives can be reduced during object detection and tracking without compromising the dynamic range of the system. Furthermore, as described in this paper, models trained using the fuzzy labeling system can adapt to different driving scenarios, including those not included in the training, while maintaining the performance of the trained model. Although the fuzzy labeling system is described in the context of vehicle radar sensors, it can be used in other applications with other electromagnetic sensors.

[0019] Example Environment

[0020] Figure 1 An example training environment 100 for fuzzy labeling of low-level electromagnetic sensor data according to the techniques of this disclosure is shown. In this example, a first sensor on vehicle 102 is described as a LiDAR sensor 104, and a second sensor on vehicle 102 is described as a radar sensor 106. However, the first sensor can be any imaging sensor, such as an optical camera or a thermal camera. Similarly, the second sensor can be any electromagnetic sensor or other sensors (e.g., sonar) that include spectral spread due to energy leakage in their data.

[0021] Example training environment 100 may be a controlled environment in which a vehicle 102, including a LiDAR sensor 104 and a radar sensor 106, collects sensor data about the training environment 100. Training environment 100 includes one or more objects, such as object 108 and object 110.

[0022] LiDAR sensor 104 collects LiDAR data including the geometric locations of objects 108 and 110. The geometric locations may include specific geometric boundaries of objects 108 and 110, which can be determined in the LiDAR point cloud. Radar sensor 106 collects low-level radar data that may have higher resolution in some dimensions (e.g., range, Doppler) and lower resolution in other dimensions (e.g., azimuth, elevation). This lower resolution can be represented as an energy spectrum tail when the radar data is represented as an energy spectrum map. The energy spectrum map may be two-dimensional, representing dimensions such as range and time, or elevation and time.

[0023] The fuzzy-labeled model training system 112 can acquire LiDAR data and low-level radar data from vehicle 102 via over-the-air (OTA) or other methods. In other aspects, the fuzzy-labeled model training system 112 can reside within vehicle 102. LiDAR data can be input into the fuzzy-labeled model training system 112 as LiDAR input data 114. LiDAR input data 114 serves as ground truth for training the model. Similarly, radar data can be input into the fuzzy-labeled model training system 112 as radar input data 116. The fuzzy-labeled model training system 112 can include a combination of hardware components and software components executing thereon. For example, the non-transient computer-readable storage medium (CRM) of the fuzzy-labeled model training system 112 can store machine-executable instructions that, when executed by the processor of the fuzzy-labeled model training system 112, cause the fuzzy-labeled model training system 112 to output a fuzzy-labeled training model 128 based on low-level electromagnetic sensor data. The fuzzy labeling model training system 112 also includes a LiDAR data processing module 118, a radar image generator 120, a bounding box generator 122, a fuzzy labeling module 124, and a machine learning module 126. In other examples, it can use... Figure 1 The different component arrangements or numbers shown are used to perform operations associated with the fuzzy labeling model training system 112.

[0024] The LiDAR data processing module 118 can identify geographic locations, specifically, the geographic boundaries of objects 108 and 110. The LiDAR data processing module can then output this geographic boundary information to the bounding box generator 122.

[0025] Radar image generator 120 can generate an energy spectrum map based on radar input data 116. For example, the energy spectrum map can be a range-time map generated by selecting a specific azimuth angle and Doppler bin and collapsing the elevation dimension. This can generate amplitudes of various ranges over time to create the range-time map. Similarly, other dimensions can be represented in the energy spectrum map. The energy spectrum map can also be called a radar image because it can be analyzed and manipulated like a camera image. Spectral tails (e.g., radar spectral tails) can be observed on a radar image as high-intensity pixels (e.g., bright pixels), with pixels surrounding the high-intensity pixel losing their intensity as they diffuse away from it. Spectral tails can be caused by radar leakage in the side-angle bin and can contribute to the noise level of radar sensor 106. The energy spectrum map can be represented as a radar image, similar to a camera image, with pixels that can be analyzed based on various characteristics (e.g., position and intensity within the image). The radar image generated by radar image generator 120 can be output to bounding box generator 122.

[0026] Bounding box generator 122 can receive the geographic boundaries of objects 108 and 110, and generate a first bounding box based on these boundaries. Similarly, bounding box generator 122 can determine radar trails that may be associated with objects 108 and 110, and generate a second bounding box covering the associated radar trails. Bounding box generator 122 can project the first and second bounding boxes onto a radar image. Based on the projections of the first and second bounding boxes onto the radar image, a first portion of the radar trail can be identified. The first portion is the union of the first and second bounding boxes. The second portion of the radar trail covers the remaining portion of the radar trail not included in the first portion.

[0027] The fuzzy labeling module 124 can label the first part of the radar trail with the highest value (e.g., a value of one on a scale from zero to one). The fuzzy labeling module 124 can analyze the second part of the radar trail and label pixels in the second part with values ​​between the lowest and highest values. The labeling pattern for pixels in the second part can typically be based on their distance from the first part, from higher to lower values. For example, pixels in the second part can be labeled from closest to farthest from the first part with patterns such as 0.9, 0.8, 0.75, 0.5, 0.4, 0.2, 0.1. The decreasing pattern can be linear, exponential, Gaussian, or any other continuous function. Any pixel in the radar image not associated with the first or second part of the radar trail can be labeled with the lowest value (e.g., zero).

[0028] The tags generated by the fuzzy tagging module 124 can be used as tags for the machine learning module 126. Because the fuzzy tags are image-based (e.g., radar images), the machine learning model can use neural networks that are helpful for image processing. One example of such a neural network is a convolutional neural network (CNN), but other neural networks can also be used. The machine learning module 126 outputs a trained model 128, which can then be deployed to the automotive system. Because the trained model 128 has been trained using machine learning techniques in a controlled environment, it can be very efficient in terms of computation and memory. This efficiency allows the trained model 128 to be used in automotive applications with limited computing resources. Furthermore, the trained model 128 can result in fewer false positives detected by the radar system when used in uncontrolled environments.

[0029] In some aspects, the fuzzy labeling model training system 112 can reside in vehicle 102, and the trained model 128 can be continuously retrained in an uncontrolled environment using sensor data received by vehicle 102. Furthermore, the retrained model can be deployed to other vehicles. These other vehicles may also include other fuzzy labeling model training systems 112. Each respective vehicle can upload its corresponding retrained model and / or its corresponding sensor data to the cloud. In these aspects, the trained model 128 can be retrained with very large datasets.

[0030] Figure 2 An example environment 200, in which low-level electromagnetic sensor data can be applied according to the techniques of this disclosure, is illustrated. In the depicted environment 200, a vehicle 202 travels on a road by relying at least in part on the output from a radar system 204. Although shown as a car, the vehicle 202 could represent other types of motorized vehicles (e.g., trucks, motorcycles, buses, tractors, semi-trailers), non-motorized vehicles (e.g., bicycles), rail vehicles (e.g., trains), water vehicles (e.g., boats), aircraft (e.g., airplanes), or spacecraft (e.g., satellites), etc.

[0031] Radar system 204 can be mounted on, integrated with, or mounted on any mobile platform, including mobile machinery or robotic equipment. For example, components of radar system 204 can be located at the front, rear, top, bottom, or side of vehicle 202, within the bumper, integrated into side mirrors, formed as part of headlights and / or taillights, or at any other internal or external location where object detection is required. Vehicle 202 may include multiple radar systems 204 (such as a first radar system and a second radar system) to provide a custom field of view 206 encompassing a specific area of ​​interest outside vehicle 202. Acting as part of the perception system of vehicle 202, radar system 204 assists in driving vehicle 202 by enabling advanced safety or autonomous driving features. Vehicle subsystems may rely on radar system 204 to detect whether any object (e.g., object 208) is present in the environment 200 within a specific field of view (FOV) 206.

[0032] Radar system 204 is configured to detect object 208 within a radiation field of view 206. For example, object 208 may be a stationary or moving object and may include one or more materials that reflect radar signals. Object 208 may be another vehicle, traffic sign, obstacle, animal, pedestrian, or any other object or debris.

[0033] Radar system 204 may include a combination of hardware components and software components executing thereon. For example, a non-transient computer-readable storage medium (CRM) of radar system 204 may store machine-executable instructions that, when executed by a processor of radar system 204, cause radar system 204 to output information about objects detected in field of view 206. As an example, radar system 204 includes signal processing components that may include a radar monolithic microwave integrated circuit (MMIC) 210, a training model processor 212, and a radar processor 214. Radar MMIC 210, training model processor 212, and radar processor 214 may be physically separate components, or their functionality may be included within a single integrated circuit. Similarly, in some aspects, other processors may be present. In this example, radar system 204 also includes a fuzzy marking module, a thresholding module 218, and an output manager 220. In other examples, [the following may be used]... Figure 2 The different component arrangements or numbers shown perform the operations associated with radar system 204. These components receive radar signals to generate detection 222 and refined radar image 224 (e.g., a radar image with reduced false positives). Detection 222 and refined radar image 224 can be used to update object tracking and classify objects.

[0034] For example, radar MMIC 210 can receive low-level radar signals emitted by radar system 204 and reflected from object 208. These low-level radar signals can be raw digitized signals or signals that have been preprocessed (e.g., lossless FFT, uncompressed data cube) without data loss. The low-level radar signals can be input into a blur labeling module 216 executed by training model processor 212. A low-level radar image can be generated based on the low-level radar data, and pixels of the low-level radar image can be labeled based on training model 128. The labeled data (e.g., labeled pixels) can be output to a thresholding module 218 executed on radar processor 214. The thresholding module 218 applies a threshold to the labeled data and generates a refined radar image 224 that includes only labeled data greater than the threshold. Similarly, detection 222 can be determined based on labeled data greater than the threshold. Output manager 220 can output detection 222 and refined radar image 224 to other systems of vehicle 202 for automotive and security applications. In this way, the output detection 222 and refined radar image 224 can include relevant information included in the low-level radar data, but reduce the number of false detections that the radar system 204 may have reported without using the training model 128.

[0035] Example vehicle configuration

[0036] Figure 3 An example vehicle 202-1 according to the technology of this disclosure is shown, which includes a system configured to utilize a model trained with fuzzy labels using low-level electromagnetic sensor data. Vehicle 202-1 is an example of vehicle 202. Included in vehicle 202-1 is radar system 204-1, which is an example of radar system 204. Vehicle 202-1 further includes a communication link 302, which radar system 204-1 can use to communicate with other vehicle-based systems 306. Communication link 302 can be a wired or wireless link, and in some cases includes a communication bus (e.g., a CAN bus). Other vehicle-based systems 306 perform operations based on information received from radar system 204-1 via link 302 (such as data output from radar system 204-1, including information indicating one or more objects identified and tracked in the FOV).

[0037] Similar to radar system 204, radar system 204-1 includes radar MMIC 210-1, training model processor (e.g., embedded processor for machine learning models) 212-1, and radar processor 214-1. Radar MMIC 210-1 includes one or more transceivers / receivers 308, timing / control circuitry 310, and analog-to-digital converters (ADCs) 312.

[0038] Radar system 204-1 further includes a non-transient computer-readable storage medium (CRM) 314 (e.g., memory, long-term storage, short-term storage) that stores instructions for radar system 204-1. CRM 314 stores a fuzzy labeling module 216-1, a thresholding module 218-1, and an output manager 220-1. Other instructions related to the operation of radar system 204-1 may also be stored in CRM 314. Components of radar system 204-1 communicate via link 316. For example, training model processor 212-1 receives low-level radar data 318 from MMIC 210-1 via link 316 and receives instructions from CRM 314 to execute fuzzy labeling module 216-1. Radar processor 214-1 receives labeled radar image 320 (e.g., fuzzy labeled radar image) from training model processor 212-1. The radar processor 214-1 also receives instructions from the CRM 314 via link 316 to execute the threshold module 218-1 and the output manager 220-1.

[0039] The fuzzy labeling module 216-1 generates a low-level radar image based on low-level radar data 318 and executes a model trained to perform fuzzy labeling of low-level electromagnetic sensor data (e.g., training model 128). The training model labels pixels of the low-level radar image from zero to one based on the training techniques described herein. The output of the fuzzy labeling module 216-1 is the labeled radar image 320. The training model can be updated periodically via over-the-air (OTA) updates or other methods.

[0040] Thresholding module 218-1 receives the marked radar image 320 and applies a threshold to the marked pixels. Thresholding module 218-1 outputs a thinned radar image 322. The thinned radar image 322 includes pixels in the marked radar image that are larger than the threshold. The thinned radar image 322 is made available by output manager 220-1 to other vehicle-based systems 306. Detection based on the thinned radar image 322 is also made available to other vehicle-based systems 306.

[0041] Other vehicle-based systems 306 may include an autonomous control system 306-1, a safety system 306-2, a positioning system 306-3, a vehicle-to-vehicle system 306-4, an occupant interface system 306-5, a multi-sensor tracker 306-6, and other systems not shown. Objects in the FOV can be inferred and classified based on the refined radar image 322 output to other vehicle-based systems 306. In this way, in response to radar data generated by radar system 204-1 combining and analyzing received signals, other vehicle-based systems 306 can receive indications of one or more objects detected by radar system 204-1. Other vehicle-based systems 306 may perform driving functions or other operations, which may include using the output from radar system 204-1 to assist in determining driving decisions. For example, vehicle control system 306-1 may provide automatic cruise control and monitor radar system 204-1 to obtain output indicating the presence of objects in the FOV, for example, to reduce speed and prevent collisions with objects in the path of vehicle 202-1. When data obtained from radar system 204-1 indicates that one or more objects are passing in front of vehicle 202-1, safety system 306-2 or occupant interface system 306-5 can provide an alarm or perform specific actions.

[0042] Example Implementation

[0043] Figure 4 A graph of the energy spectrum 402 labeled using a blurred label of low-level electromagnetic sensor data according to the technology of this disclosure is shown. Figure 4 The fuzzy labeling logic in the range dimension with a linearly decreasing pattern is demonstrated. However, any radar dimension can be demonstrated.

[0044] Energy spectrum 402 can be a typical energy spectrum of a finite range from a scatterer (e.g., an object). The reflection center of energy spectrum 402 represents the intensity peak of energy spectrum 402. The geometric boundary 406 of the scatterer may be indistinguishable in energy spectrum 402 and can be determined based on data from another sensor projected onto energy spectrum 402 (e.g., LiDAR data projected onto a radar image).

[0045] Some conventional labeling techniques can use binary labeling to label the energy spectrum 402. For example, after estimating the geometric boundary 406 of the scatterer on the energy spectrum 402, a binary label can be used to label the labeled portion 408 representing the geometric boundary 406 and the region within that geometric boundary. The labeled portion 410 of the energy spectrum 402 located outside the geometric boundary 406 may be unlabeled (e.g., labeled with a value of zero).

[0046] Conversely, using fuzzy labeling logic, the reflection center 404 of the energy spectrum 402 can be labeled with a 1. The fuzzy label 412 decreases from 1 to zero. Instead of labeling only the geometric boundary 406, the fuzzy label labels the entire energy spectrum 402 and takes into account the soft boundaries of scatterers included in the energy spectrum 402. Using fuzzy labeling logic in this way makes it possible to consider all information in the radar signal. Fuzzy labeling logic can provide machine learning models (e.g., artificial neural networks) that utilize less ambiguous labeling and produce more robust models for radar sensors, traffic scenarios, and object (target) types. Furthermore, fuzzy labeling can produce fewer false positives than traditional labeling techniques.

[0047] Figure 5 The diagram illustrates LiDAR bounding boxes 502 and 504, representing blurred markings for low-level electromagnetic sensor data according to the present disclosure, projected onto radar image 506. The radar image shows the radiation area reflected by objects and received by the radar sensor.

[0048] The LiDAR point cloud 508 detects several objects including object 510. The LiDAR bounding box 502 of object 510 is projected onto radar image 506 and represents the geometric position (e.g., geometric boundary) of object 510 on radar tail 512 in radar image 506.

[0049] Radar bounding box 504 encompasses radar tail 512. Radar bounding box 504 includes a first portion 504-1 and a second portion 504-2 of radar tail 512. The first portion 504-1 corresponds to the portion of the radar tail that serves as the geometric location of object 510. The first portion 504-1 is determined by finding the union of LiDAR bounding box 502 and radar bounding box 504. The second portion 504-2 includes the remaining portion of the radar tail outside the geometric location of object 510. For the fuzzy labeling of radar tail 512, pixels in the first portion 504-1 can be labeled with a value of one. Pixels in the first portion 504-2 can be labeled with a value between zero and one, with a higher value for pixels closer to the first portion 504-1 and a smaller value for each corresponding pixel farther away from the first portion 504-1. This fuzzy labeling system allows for the continuous and smooth labeling of all pixels in radar tail 512, taking into account all the information contained within radar tail 512.

[0050] Example Method

[0051] Figure 6An example method 600 for blurry labeling of low-level electromagnetic sensor data according to the technology of this disclosure is illustrated. The blurry labeling is used to train a machine learning model to efficiently label electromagnetic images using fewer computational resources. Data can be collected in a controlled training environment where objects are specifically placed at particular geometric locations. At step 602, the geometric location (e.g., geometric boundaries) of the object is identified based on first sensor data obtained from a first sensor. The first sensor can be any imaging sensor that can provide a clear outline of the object in its field of view, such as a LiDAR or a camera.

[0052] At step 604, an energy spectral tail corresponding to an object is identified on a spectrogram derived from the second sensor data. The spectrogram (e.g., a radar image) can be derived from low-level data obtained from an electromagnetic sensor (such as a radar sensor). The spectrogram can be a two-dimensional plot representing dimensions of the electromagnetic sensor, such as range and time dimensions. The energy spectral tail, or radar tail, can represent objects and energy leakage (e.g., noise) in the side lobe of the second sensor data (e.g., data derived from the sidelobes of a second sensor antenna). By collapsing the elevation information in the second sensor data, the spectrogram can include range and time dimensions.

[0053] At step 606, a first portion of the energy spectrum tail is identified. This first portion corresponds to the geometric location of the object. The first portion can be identified by projecting the geometric location identified in the first sensor data onto the spectrogram. Specifically, a bounding box can be generated around the geometric location in the first sensor data image, and this bounding box can be projected onto the spectrogram to identify the first portion of the energy spectrum tail.

[0054] At step 608, each pixel in the first portion of the energy spectrum tail is labeled with a value of 1. Because this first portion covers the geometric boundaries of the object, these labels can be used as ground truth values ​​for training the model.

[0055] At step 610, each pixel in the second part of the energy spectrum tail is labeled with a value between zero and one. The second part includes all pixels in the energy spectrum tail that were not included in the first part. That is, both the first and second parts include all pixels in the energy spectrum tail. Pixels in the second part that are closer to the first part are labeled with higher values. As the position of each corresponding pixel is further away from the first part, the value of that pixel decreases. For example, a pixel in the second part that is close to the first part can have a value of 0.9. The next closest pixel can have a value of 0.8, and so on. The decreasing pattern of pixel values ​​can be linear, exponential, Gaussian, or any other continuous function. In some cases, the energy spectrum tail can correspond to multiple objects. In these cases, these objects may be indistinguishable in the energy spectrum tail. However, the labeling of pixels is the same as when the energy spectrum tail corresponds to only a single object.

[0056] At step 612, the model is trained based on the labels of the first and second portions of the energy spectrum tail to label pixels in the spectrogram used in real-world applications. Because the spectrogram is essentially an image, machine learning techniques used for training image classification models can be used. For example, the model could be a convolutional neural network that efficiently classifies pixels in an image. Furthermore, substantial computational resources can be used to train the model, as training can be performed outside the automotive system (e.g., in a laboratory). The resulting model executable software can be very inexpensive in terms of computational resources, making it well-suited for automotive applications. Using this method of training the model with fuzzy labels, all data from the received electromagnetic signal can be considered. The output of the trained model can be used to accurately detect objects while minimizing false detections.

[0057] Additional examples

[0058] Example 1: A method comprising: identifying the geometric location of at least one object based on first sensor data obtained from a first sensor, the second sensor being an electromagnetic sensor; identifying an energy spectral tail corresponding to the object on a spectrogram derived from second sensor data obtained from a second sensor, the second sensor being an electromagnetic sensor; identifying a first portion of the energy spectral tail corresponding to the geometric location of the object; labeling each pixel in the first portion of the energy spectral tail with a value of one; labeling each pixel in a second portion of the energy spectral tail, including all pixels not included in the first portion, with a value between zero and one, the value decreasing the further each corresponding pixel is from the first portion of the energy spectral tail; and training a model by machine learning and based on the labels of each pixel in the first portion and each pixel in the second portion to label the spectrogram for detecting and tracking the object.

[0059] Example 2: The method of Example 1 further includes marking each pixel in the spectrogram that is not included in the energy spectrum tail with a value of zero.

[0060] Example 3: A method for any of the previous examples, where the model is a convolutional neural network.

[0061] Example 4: A method in any of the previous examples, where the first sensor is either a light detection and ranging (LiDAR) sensor or a camera.

[0062] Example 5: A method in any of the previous examples, where the second sensor is a radar sensor.

[0063] Example 6: A method in any of the previous examples, where the energy spectrum tail includes radar energy leakage in the side corner compartment.

[0064] Example 7: A method from any of the previous examples, where deriving the spectrogram involves generating a radar image by selecting an azimuth angle and a Doppler chamber.

[0065] Example 8: A method from any of the previous examples, wherein generating a radar image further includes collapsing elevation information from the second sensor data, such that the radar image includes both distance and time dimensions.

[0066] Example 9: A method in any of the previous examples, wherein identifying the geometric location of an object includes determining a first bounding box, which represents the geometric boundary of the object.

[0067] Example 10: A method in any of the previous examples, wherein identifying the first portion of the energy spectrum tail includes projecting a first bounding box onto the spectrogram.

[0068] Example 11: A method in any of the previous examples, wherein identifying the energy spectrum tail includes determining a second bounding box, the second bounding box including a first portion of the energy spectrum tail and a second portion of the energy spectrum tail.

[0069] Example 12: A method in any of the previous examples, where the second part of the energy spectrum tail is the portion of the energy spectrum tail that is included in the second bounding box but not included in the first bounding box.

[0070] Example 13: A method from any of the previous examples where the geometric boundaries of the object are used as ground truth data to train a model for each pixel of the labeled spectrogram.

[0071] Example 14: A method from any of the previous examples, where the energy spectrum tail corresponds to multiple objects that are close to each other, such that each corresponding object among the multiple objects is indistinguishable from the other objects in the energy spectrum tail.

[0072] Example 15: A system comprising: at least one processor configured to: acquire first sensor data from a first sensor, the first sensor data being based on a controlled environment including at least one object; acquire second sensor data from a second sensor, the second sensor data being based on the controlled environment, the second sensor being an electromagnetic sensor; identify the geometric location of the object based on the first sensor data; generate a spectrogram displaying a region of radiation reflected by the object in the controlled environment based on the second sensor data; identify an energy spectral tail on the spectrogram corresponding to the radiation reflected by the object; identify a first portion of the energy spectral tail corresponding to the geometric location of the object; label each pixel in the first portion of the energy spectral tail with a value of one; label each pixel in a second portion of the energy spectral tail, including all pixels not included in the first portion, with a value between zero and one, the value decreasing the further each corresponding pixel is from the first portion of the energy spectral tail; and train a model by machine learning and based on the labels of each pixel in the first portion and each pixel in the second portion to label the spectrogram for detecting and tracking objects.

[0073] Example 16: A system in any of the preceding examples, wherein: the first sensor is either a light detection and ranging (LiDAR) sensor or a camera; and the second sensor is a radar sensor.

[0074] Example 17: A system in any of the preceding examples, wherein the processor is configured to generate a spectrogram by at least the following: generating a radar image by selecting an azimuth angle and a Doppler chamber; and collapsing elevation information in the second sensor data such that the radar image includes a range and time dimension.

[0075] Example 18: A system in any of the previous examples, wherein the energy spectrum tail includes radar energy leakage in the side compartment determined in the second sensor data.

[0076] Example 19: A system in any of the preceding examples, wherein the processor is further configured to: identify the geometric location of an object by determining at least a first bounding box, the first bounding box representing the geometric boundary of the object; identify an energy spectrum tail by determining at least a second bounding box, the second bounding box comprising all pixels of the energy spectrum tail on the spectrogram; and identify a first portion of the energy spectrum tail by projecting at least the first bounding box onto the spectrogram, the first portion being the union of the first and second bounding boxes.

[0077] Example 20: A system comprising: at least one processor configured to: receive sensor data from an electromagnetic sensor associated with at least one object, based on a controlled environment; generate a spectrogram showing a region of radiation reflected by the object, based on the sensor data; generate a labeled sensor data image based on the spectrogram by a trained model, the trained model being trained to label the sensor data image at least in the following manner: identifying an energy spectral tail on the spectrogram corresponding to the radiation reflected by the object; identifying a first portion of the energy spectral tail corresponding to the geometric position of the object; labeling each pixel in the first portion of the energy spectral tail with a value of one; and labeling each pixel in a second portion of the energy spectral tail, including all pixels not included in the first portion, with a value between zero and one, the value decreasing the further each corresponding pixel is from the first portion of the energy spectral tail; and outputting the labeled sensor data image for detecting and tracking the object using the electromagnetic sensor.

[0078] Conclusion

[0079] While various embodiments of this disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that this disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the spirit and scope of this disclosure as defined by the following claims. Problems associated with tagging sensor data may occur in other systems. Therefore, although described as a method for fuzzy tagging of low-level electromagnetic sensor data, the techniques described above can be applied to other systems that would benefit from fuzzy tagging. Furthermore, these techniques can also be applied to other sensors that output energy spectra.

[0080] Unless the context explicitly states otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method for fuzzy marking, the method comprising: Based on first sensor data obtained from a first sensor, the geometric position of at least one object is identified, wherein the first sensor is an imaging sensor; The energy spectral tail corresponding to the object is identified on the spectrum derived from the second sensor data obtained from the second sensor, which is an electromagnetic sensor; The first portion of the energy spectrum tail, corresponding to the geometric position of the object, is identified; In the first portion of the energy spectrum tail, each pixel is labeled with a value of 1; In the second part of the energy spectrum tail, which includes all pixels not included in the first part, each pixel is labeled with a value between zero and one, with the value being smaller the further each corresponding pixel is from the first part of the energy spectrum tail. as well as The model is trained using machine learning and based on the labels of each pixel in the first part and each pixel in the second part to label the spectrogram for detecting and tracking objects.

2. The method of claim 1, further comprising marking each pixel in the spectrogram that is not included in the energy spectrum tail with a value of zero.

3. The method as described in claim 2, characterized in that, The model is a convolutional neural network.

4. The method as described in claim 1, characterized in that: The first sensor is either a light detection and ranging (LiDAR) sensor or a camera; and The second sensor is a radar sensor.

5. The method as described in claim 4, characterized in that, The energy spectrum trail includes radar energy leakage in the side corner compartment.

6. The method as described in claim 4, characterized in that, Exporting the spectrogram involves generating a radar image by selecting an azimuth angle and a Doppler chamber.

7. The method as described in claim 6, characterized in that, Generating the radar image further includes collapsing the elevation angle information in the second sensor data, so that the radar image includes a distance and time dimension.

8. The method as described in claim 1, characterized in that, Identifying the geometric location of the object includes determining a first bounding box, which represents the geometric boundary of the object.

9. The method as described in claim 8, characterized in that, Identifying the first portion of the energy spectrum tail includes projecting the first bounding box onto the spectrogram.

10. The method as described in claim 9, characterized in that, Identifying the energy spectrum tail includes defining a second bounding box, the second bounding box including the first portion of the energy spectrum tail and the second portion of the energy spectrum tail.

11. The method as described in claim 10, characterized in that, The second part of the energy spectrum tail is the portion of the energy spectrum tail that is included in the second bounding box but not included in the first bounding box.

12. The method as described in claim 8, characterized in that, The geometric boundaries of the object are used as ground truth data to train a training model that labels each pixel of the spectrogram.

13. The method as described in claim 1, characterized in that, The energy spectrum tail corresponds to a plurality of objects that are close to each other, such that each corresponding object among the plurality of objects is indistinguishable from the other objects in the energy spectrum tail.

14. A system for fuzzy marking, the system comprising: At least one processor, said at least one processor being configured to: First sensor data is obtained from a first sensor, the first sensor data being based on a controlled environment including at least one object, the first sensor being an imaging sensor; Second sensor data is obtained from a second sensor, which is based on the controlled environment; the second sensor is an electromagnetic sensor. Based on the data from the first sensor, the geometric position of the object is identified; Based on the data from the second sensor, a spectrum diagram showing the radiation region reflected by objects in the controlled environment is generated; On the spectrum diagram, the energy spectral tail corresponding to the radiation reflected by the object is identified; The first portion of the energy spectrum tail, corresponding to the geometric position of the object, is identified; In the first portion of the energy spectrum tail, each pixel is labeled with a value of 1; In the second part of the energy spectrum tail, which includes all pixels not included in the first part, each pixel is labeled with a value between zero and one, with the value being smaller the further each corresponding pixel is from the first part of the energy spectrum tail. as well as The model is trained using machine learning and based on the labels of each pixel in the first part and each pixel in the second part to label the spectrogram for detecting and tracking objects.

15. The system as described in claim 14, characterized in that, The model is a convolutional neural network.

16. The system as described in claim 14, characterized in that: The first sensor is either a light detection and ranging (LiDAR) sensor or a camera; and The second sensor is a radar sensor.

17. The system as claimed in claim 15, characterized in that, The processor is configured to generate the spectrogram in at least the following ways: Radar images are generated by selecting the azimuth angle and Doppler beam; and The elevation angle information in the second sensor data is collapsed so that the radar image includes both distance and time dimensions.

18. The system as described in claim 15, characterized in that, The energy spectrum tail includes radar energy leakage in the side compartment identified in the second sensor data.

19. The system as described in claim 15, characterized in that, The processor is further configured to: The geometric location of the object is identified by at least determining a first bounding box, the first bounding box representing the geometric boundary of the object; The energy spectrum tail is identified by defining at least a second bounding box, the second bounding box including all the pixels of the energy spectrum tail on the spectrogram; as well as The first portion of the energy spectrum tail is identified by projecting at least the first bounding box onto the spectrogram; the first portion is the union of the first bounding box and the second bounding box.

20. A system for fuzzy marking, the system comprising: At least one processor, said at least one processor being configured to: Based on a controlled environment, sensor data is received from an electromagnetic sensor associated with at least one object; Based on the sensor data, a spectrum diagram showing the radiation region reflected by the object is generated; A labeled sensor data image is generated by a trained model based on the spectrogram, the trained model being trained to label the sensor data image in at least the following ways: On the spectrum diagram, the energy spectral tail corresponding to the radiation reflected by the object is identified; The first portion of the energy spectrum tail, corresponding to the geometric position of the object, is identified; In the first portion of the energy spectrum tail, each pixel is labeled with a value of 1; as well as In the second part of the energy spectrum tail, which includes all pixels not included in the first part, each pixel is labeled with a value between zero and one, with the value being smaller the further each corresponding pixel is from the first part of the energy spectrum tail. as well as The marked sensor data image is output for use in detecting and tracking objects using the electromagnetic sensor.

Images