Generation and use of synthetic radar views of a scene

Abstract

A first general aspect of the present disclosure relates to a method for generating a representation of the radar propagation properties of a scene. The method comprises: obtaining a plurality of radar measurements of the scene, taken from a plurality of positions; rendering, based on the representation of the radar propagation properties of the scene, a plurality of synthetic radar measurements of the scene at the plurality of positions; and learning the representation of the radar propagation properties of a scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements of the scene.

Description

[0001] Description

[0002] title Radar views of a scene

[0003] State of the art

[0004] Radar technology is used in various applications to sense environments or to detect and locate objects. For example, millimeter-wave radar sensor systems (C.mmWave) are becoming increasingly common in vehicles and robots, for example, to assist drivers and for collision avoidance tasks.

[0005] Because the design, testing, and deployment of new radar systems in the real world can be costly, many rapid prototyping pipelines rely heavily on simulation. State-of-the-art radar simulation tools often require the user to manually specify the geometry and characteristics of a scene, including all material properties. While other sensor systems (e.g., LIDAR) can be used to sample an environment and generate a grid or voxel map, they cannot capture radar-specific material properties, which are critical for generating realistic radar samples. This therefore leads to (significantly) simplified scene or environment models in practice due to the difficulty of thoroughly examining a scene and generating (or annotating) a model by hand.In other state-of-the-art techniques, an explicit representation of a scene is directly generated based on radar measurements using machine learning approaches. These techniques often require extensive training data, do not generalize to different scenes, and may hallucinate features of the scene (i.e., encode features that are not present).

[0006] The techniques of the present disclosure are intended to address some of these problems. Summary of the Invention

[0007] A first general aspect of the present disclosure relates to a method for generating a representation of the radar propagation characteristics of a scene. The method comprises obtaining a plurality of radar measurements of the scene taken from a plurality of positions, rendering, based on the representation of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements of the scene at the plurality of positions, and learning the representation of the radar propagation characteristics of a scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements of the scene.

[0008] A second general aspect of the present disclosure relates to methods for generating one or more synthetic radar views of a scene. The method comprises obtaining a representation of the radar propagation properties of a scene generated according to the method of the first general aspect, obtaining a specification of one or more synthetic views of a scene, and rendering, based on the obtained representation of the radar reflectance properties of a scene, one or more synthetic radar measurements according to the obtained specification.

[0009] A third general aspect of the present disclosure relates to a method for locating an object in a scene. The method comprises obtaining a representation of the radar propagation characteristics of a scene defined according to the first general aspect, obtaining a radar measurement of the scene taken from an unknown position, and determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements generated using the representation.

[0010] A fourth general aspect of the present disclosure relates to a method for developing and / or testing a radar sensor system, a component thereof, or a system consuming the output data of a radar sensor system. The method comprises obtaining a representation of the radar reflectance characteristics of a scene, defined according to the first general aspect, rendering, based on the obtained representation of the radar propagation characteristics of a scene, one or more synthetic radar measurements according to the obtained specification, and using the one or more synthetic radar measurements for developing and / or verifying and / or testing a radar sensor system, a component thereof, or a system consuming the output data of a radar sensor system.

[0011] A fifth general aspect of the present disclosure relates to a representation of the radar propagation properties of a scene, the representation being generated according to the first general aspect.

[0012] A sixth general aspect of the present disclosure relates to a computer system configured to perform any of the methods of the first to fifth general aspects.

[0013] A seventh general aspect of the present disclosure relates to a computer program comprising instructions that, when executed by a computer system, cause the computer system to perform the steps of any of the methods of the first to fifth general aspects.

[0014] The techniques of the first to seventh aspects may, in some implementations, have one or more of the following advantages.

[0015] First, the techniques of the present disclosure may not require an explicit algorithm for recovering the representation of the radar propagation properties of a scene based on the plurality of radar measurements of the scene taken from a plurality of positions. This is the construction paradigm of many prior art techniques involving the construction of representations (also called models) of the radar propagation properties of a scene. Instead, according to the present disclosure, the representation is learned by implicitly generating synthetic radar measurements or views of the scene based on the representation. The parameters of the representation are adjusted in this process. This may result in a representation that implicitly encodes the radar propagation properties of the scene (i.e., an implicit scene representation).Thus, explicit modeling of the scene, taking geometry and material properties into account, may not be required in certain implementations (or only to a lesser extent, compared to some techniques that involve an explicit algorithm for recovering the representation of a scene's radar propagation properties based on the majority of radar measurements of the scene taken from a plurality of positions). This, in turn, can simplify the modeling task, but can also lead to a more accurate representation in some situations. In some state-of-the-art techniques, the difficulties and complications of the explicit modeling task result in the use of simplifying assumptions and approximations to make the modeling task manageable. This can lead to errors and artifacts in the synthetic views generated based on the corresponding models.The present approach has certain conceptual similarities to an approach called "Neural Radiance Fields" (NeRFs), which is used to learn implicit scene representations to generate synthetic images in the optical or visual domain. However, the radar domain has several distinct differences compared to the optical or visual domain, which prevent a direct transfer of solutions proposed for Neural Radiance Fields in the optical or visual domain to the radar domain.

[0016] Second (and related to the first point), the techniques of the present disclosure may, in some examples, enable the generation of more accurate synthetic measurements or views of a scene in the radar domain. These synthetic measurements or views may be used in various development and testing tasks of radar sensor systems or systems that consume the output data of a radar sensor system (e.g., rapid prototyping tasks) and may, in some situations, accelerate and / or improve the performance of the radar sensor systems thus developed or tested, or of the system that consumes the output data of a radar sensor system. Furthermore, the representations of the present disclosure may be used in various radar sensor system applications to monitor and control systems for which the radar sensor systems are employed (e.g., for or in a vehicle or robot).

[0017] In the present disclosure, several terms are used in a specific way: A "representation of the radar propagation properties of a scene" is a model of the scene in the radar domain. The representation may encode the radar propagation properties (e.g., transmittance and reflectance, or other propagation properties) so that a rendering technique can sample the representation to render a synthetic view of the scene. The representation may, but need not, encode or be based on a geometric representation of the scene.

[0018] The term "minimizing" includes, but is not limited to, finding an actual (global or local) minimum of a function that is the subject of the minimization step (e.g., an error function quantifying a difference between acquired radar measurements and synthetic radar measurements of a scene). Rather, minimization is the process of reducing a quantity (e.g., an error function) according to a predetermined criterion. The terms "training" and "learning" refer to the corresponding concepts in the machine learning domain. For example, training a representation or model may mean determining values ​​of parameters of the representation or model (to adapt the representation or model to a particular task, e.g., generating synthetic radar measurements in the present disclosure). The term "learning" describes the same act, but "seen through the eyes" of the representation or model.

[0019] A "scene" can be any environment and / or geographic area (i.e., a scene is not limited to a specific space, such as a single room) whose space is potentially or actually monitored or investigated by the radar sensor system. A scene can have an arbitrary spatial extent and configuration (e.g., a scene can span multiple rooms, floors, buildings, a portion of an indoor and / or outdoor environment, such as a street or other passageway, and / or environments).

[0020] The term “radar measurement” or “view” is used for both measured or acquired data and synthetically generated data.

[0021] A "vehicle" can be any device for transporting people or goods (e.g., a car, a bus, or a truck). Vehicles can be ground-based, sea-based, air-based, or space-based (e.g., airplanes, ships, spacecraft, etc.). A vehicle can be designed to operate at least partially autonomously.

[0022] Description of the drawings

[0023] Fig. 1 is a schematic drawing of exemplary elements involved in generating synthetic views of radar measurements according to the present disclosure.

[0024] Fig. 2 is a swim lane diagram illustrating various methods according to the present disclosure.

[0025] Fig. 3 shows a schematic representation of a rendering process for generating synthetic radar measurements according to the present disclosure.

[0026] Fig. 4(a) and Fig. 4(b) illustrate the concept of a Doppler representation of radar sensor measurement data according to the present disclosure. Fig. 5 shows several views of scenes measured and synthesized using the techniques of the present disclosure and several other techniques.

[0027] Detailed description

[0028] First, we discuss the technique for generating a representation of the radar propagation characteristics of a scene, including the preprocessing steps of the measurement data and the rendering steps for generating synthetic views. Figure 1 is a schematic drawing of exemplary elements involved in generating synthetic views of radar measurements, according to the present disclosure. Figure 2 is a swimlane diagram illustrating various methods according to the present disclosure.

[0029] A method for generating a representation of the radar propagation characteristics of a scene 12 includes obtaining 101 a plurality of radar measurements of the scene taken from a plurality of positions, rendering 102, based on the representation 15 of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements 21 of the scene 12 at the plurality of positions, and learning 103 the representation 15 of the radar propagation characteristics of a scene 12 by reducing a difference between the obtained radar measurements and the synthetic radar measurements 21 of the scene 12. Aspects of each of these steps and further steps are discussed below.

[0030] Obtaining 101 a plurality of radar measurements of the scene 12, taken from a plurality of positions, may comprise obtaining the plurality of radar measurements along a trajectory in and / or around the scene 12. The plurality of radar measurements may be acquired by any suitable radar sensor system 11 (e.g., by moving the radar sensor system 11 through and / or around the scene 12). In some examples, the plurality of radar measurements are acquired by a plurality of different radar sensor systems. In some examples, the plurality of radar measurements of the scene 12 are annotated with metadata. The metadata may comprise information indicating a pose of the corresponding radar sensor system 11 when acquiring the radar measurement of the plurality of radar measurements (e.g., a location in the scene 12 and / or a direction the radar sensor system is pointing).Additionally or alternatively, the metadata may include information regarding a relative velocity of the respective radar sensor system 11 and the scene 12 when the radar measurement is acquired. In some examples, the scene 12 is (essentially) static, so the relative velocity information includes a velocity of the radar sensor system 11 relative to the scene 12 when the radar measurement is obtained. As discussed below, this relative velocity information may be useful for increasing the angular resolution of the (measured and synthesized) views in certain situations.

[0031] The radar sensor system 11 may include a plurality of antennas (e.g., one or more transmit antennas and one or more receive antennas arranged in an array, such as two or more transmit antennas and two or more receive antennas). In some examples, the radar measurements include multiple channels, with each channel corresponding to an antenna of a plurality of antennas of the radar sensor system. In some examples, the radar sensor system 11 employs frequency-modulated continuous wave (FMCW) (e.g., a frequency of the waveform transmitted by the radar sensor system 11 is continuously and / or periodically changed). In this way, radar radiation reflected from different instances may have different frequencies (and range data may be obtained thereby, since the distance or separation of a reflecting surface is manifested in a frequency shift).

[0032] In some examples, the radar measurements are the result of a preprocessing procedure. In these examples, raw data measurements are obtained at the plurality of positions (the term "raw measurement data" is used in the present disclosure to indicate any measurement data prior to the output of a preprocessing pipeline; i.e., raw measurement data that has undergone a first preprocessing step of multiple preprocessing steps is still referred to as raw measurement data). The examples include preprocessing the raw data measurements to obtain the radar measurements used in the further processing steps described herein.

[0033] The preprocessing procedure may include one or more of the following steps.

[0034] In some examples (e.g., if an FMCW radar sensor system is used), range data (i.e., data indicating a distance from a reflecting surface to the radar sensor system) may be generated by computing a Fourier transform (e.g., a 1-dimensional FFT) of the raw data measurements. This may convert frequency shifts in the raw data measurements into time delays (which are proportional to distance). In some examples, the range coordinate comprises a plurality of range windows (e.g., 50 or more, or 100 or more range windows).

[0035] Additionally or alternatively, the preprocessing procedure may include calculating Doppler velocities by processing a time series of obtained raw measurement data. The Doppler velocities indicate a relative movement speed of the radar sensor system 11 and the scene 12. In some examples, the scene 12 is static (considered static). In these examples, the Doppler velocities indicate a speed of the radar sensor system 11 in the (static) scene 12. As the radar sensor system 11 moves through and around the scene, a phase of the reflected signal may change between different measurement windows. The Doppler velocity may be calculated based on a rate of this phase change.In some examples, preprocessing may include computing a Fourier transform over a time sequence of radar measurements obtained at different times as radar sensor system 11 moves through or around scene 12.

[0036] In some examples (e.g., for radar sensor systems 12 comprising a plurality of antennas), a further (azimuth and / or elevation) Fourier transform is computed over the raw measurement data obtained from different antennas (e.g., for multiple pairs of receive and transmit antennas, e.g., in particular, all possible pairs of transmit and receive antennas of the radar sensor system) to generate a plurality of azimuth and / or elevation windows in the radar measurement data (radar measurement data is often represented in a range dimension, indicating a distance between the radar sensor system and an object 13, and two angular components, referred to as elevation and azimuth, indicate an angle in two mutually perpendicular planes centered on the radar sensor system).

[0037] A resulting set of measurement data has a similar structure as shown for the synthetic measurement data 21 in Fig. 1. As can be seen, the measurement data is three-dimensional: The set of measurement data comprises a first dimension indicating the particular antenna or transmitter / receiver pair (Fig. 1 shows, by way of example, four windows drawn perpendicular to the plane of the paper). In a second dimension, each data set comprises a plurality of windows for the range data (drawn along the vertical axis) and, in a third dimension, a plurality of Doppler velocity windows (drawn along the horizontal axis). The measurements are divided along three dimensions, i.e., a range dimension, a Doppler velocity direction, and an antenna direction (e.g., after undergoing one or more of the preprocessing steps discussed above and / or below).A specific window is also referred to as a "radar pixel" in the present disclosure. Other structures of the radar measurement data may be used in different situations. For example, in some examples, the radar measurement data does not include a Doppler dimension. Determining the Doppler velocities and performing the corresponding preprocessing steps discussed above may have certain advantages in some situations. In particular, the resolution of the Doppler velocity data can (in theory) be arbitrarily high, since it can be increased by increasing the integration time of a radar measurement.

[0038] In some examples, the Fourier-transformed raw measurement data can be filtered to reduce the leakage effect (i.e., spectral leakage between adjacent windows along the corresponding axis). For example, a Hann filter can be applied to range data and / or Doppler velocity data.

[0039] The radar measurement data, preprocessed according to one or more of the steps discussed above, is then used in the generation process for generating a representation of the radar propagation characteristics of a scene according to the present disclosure. We will next discuss aspects of the representation and the learning technique for learning the representation of a scene. As discussed above, one element of the techniques of the present disclosure includes applying machine learning techniques to train the representation 15 to provide input data (regarding the radar propagation characteristics of a scene) to a rendering module 22 such that the generated synthetic radar measurements 21 of the scene 12 correspond (e.g., are as close as possible) to the received (acquired) radar measurement as discussed above. In other words, the received (acquired) radar measurement data, including the metadata (e.g.,comprising information that indicates a pose and / or a speed of the corresponding radar system 11 when detecting the radar measurement) is the ground truth on the basis of which the representation 15 is trained.

[0040] The representation 15 of the radar propagation properties of a scene can be modeled according to different design principles. Next, we discuss details regarding the representation.

[0041] The representation 15 may have any suitable form so that a rendering module 22 can be provided with the radar propagation properties 17 of the scene 12 necessary to render synthetic measurement data 21 (which can then be compared with the acquired measurement data to train the representation 15). The representation 15 has a plurality of parameters that can be adjusted in the training process for training the representation 15 to encode the radar propagation properties of the scene 12.

[0042] In some examples, the representation 15 may include a neural network 23 (e.g., a convolutional neural network or other deep neural network topology). In this case, the parameters that may be adjusted in the training process for training the representation 15 to encode the radar propagation characteristics of the scene may include the weights of the neural network 23 (and additionally, in some examples, one or more hyperparameters related to the neural network 23). Additionally or alternatively, the representation may include a spatial grid structure (e.g., a simple grid), an octree, a spatial hash table, an interpolated spatial grid, a collection of points or blobs with spherical, Gaussian, or other structure, or any combination of these elements (e.g., a spatial hash table with a neural network for resolving hash collisions).Depending on the nature of the configuration, the input and output parameters of the representation and training processes may vary.

[0043] In some examples, the representation 15 defines a field that encodes the radar propagation properties 17 (e.g., radar transmittance and radar reflectance) of the scene 12. In these examples, learning the representation 15 includes learning the radar propagation properties 17 (e.g., radar transmittance and radar reflectance) of the scene 12.

[0044] The representation can encode the radar propagation properties 12 of the scene 12 (e.g., the transmittance and the reflectance) in various different ways.

[0045] In some examples, the representation 15 models a transmittance and a reflectance for each point in the scene. The transmittance and reflectance values ​​may, in some examples, depend on the angle of incidence 16 of an incoming radar wave (e.g., the representation maps a six-dimensional vector comprising transmittance and reflectance to a first scalar reflectance value and a second scalar reflectance value). This may enable modeling of a variety of radar phenomena, such as partial occlusions, specularities, and phantom reflections.

[0046] In some examples, representation 15 encodes a function that outputs one or more radar propagation parameters (e.g., transmittance and / or reflectance 17) for a set of input parameters. The set of input parameters may include any parameter that characterizes one or more radar measurements (e.g., a set of parameters that specify the geometry of an observation point, such as a position in or relative to the scene and an orientation and / or radar sensor parameters).

[0047] In the previous section, reflectance and transmittance were discussed as exemplary parameters encoding radar propagation characteristics 17 of scene 12 in representation 15. In other examples, other parameters may also be used (or in addition to them). For example, in some examples, an absorbance may be used in addition to or instead of transmittance or reflectance.

[0048] In some examples, the representation 15 may employ a voxel-based approach to modeling the scene 12. For example, the scene may be modeled by a plurality of voxels, and the representation may encode radar propagation properties for each voxel (e.g., reflectance and transmittance, as discussed above). In other examples, the representation does not model the scene based on a geometric partitioning of the scene.

[0049] In some examples, representation 15 encodes the radar propagation properties of the scene as a function of view angle. For example, the representation may map angles of incoming radar waves in scene 12 to a set of spherical harmonic coefficients. Radar propagation properties may then be calculated based on a base value of the corresponding propagation property (e.g., transmittance or reflectance) and the spherical harmonic coefficients.

[0050] In the example of Fig. 1, the representation 15 is structured as follows. As can be seen, the representation 15 receives information regarding a plurality of range windows, a measurement position, and an incidence angle of a corresponding wave as input parameters. The representation outputs a set of spherical harmonic coefficients 24 for the range window information, as well as base values ​​25 for the transmittance and reflectance properties. Based on these base values ​​and the incidence angle, the representation can output reflectance and transmittance values ​​17. These reflectance and transmittance values ​​17 can be consumed by the rendering module 22 to generate the synthetic radar measurements of the scene.

[0051] We will then discuss in more detail aspects of the step of rendering 102, based on the representation of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements of the scene at the plurality of locations. Figure 3 shows a schematic representation of a rendering process for generating synthetic radar measurements 37 according to the present disclosure.

[0052] In some examples, rendering includes obtaining radar transmittance and reflectance values ​​35 (or any other radar propagation property of the scene) from the representation and propagating radar beams through the scene based on the obtained radar transmittance and reflectance values ​​35 to generate the synthetic radar measurements of the scene. Radar wave propagation has certain idiosyncrasies that complicate a direct application of the rendering model used in many Neural Radiance Fields (NeRF) approaches. In particular, in typical Neural Radiance Fields (NeRF) approaches, an image pixel is rendered by integrating samples along a (one-dimensional) ray. This is illustrated in Fig. 4(b) by an exemplary ray 41 for a particular radar pixel 44.In typical NeRF (Neural Radiance Fields) approaches, a model or representation of the scene outputs color and transparency values ​​that can be sampled along the ray and integrated to obtain the pixel color. Now, radar waves 42 propagate radially from the antenna. Therefore, depending on the detection capabilities of the radar sensor system 11, each pixel 44 in a radar image may correspond to a two-dimensional region of space in the scene. Therefore, rendering a pixel 44 of a radar image may include integrating over a two-dimensional region of space in the scene. In some examples, rendering may include using a predetermined rendering equation (e.g.,implemented in a rendering module 22, as discussed above), consumes the output data of the representation 15 (and optionally additional information regarding the view to be rendered and / or the radar sensor system for which the view is being rendered) and generates a synthetic radar measurement 21 (i.e., a view) of the scene. In some examples, rendering may include rendering a plurality of pixels that form the synthetic radar measurement 21 (i.e., the view) of the scene.

[0053] In some examples, rendering includes propagating radar waves 31 (e.g., beams) from a transmitter to a receiver at a particular location 32. Radar propagation properties 35 of the scene along the beams 31 (e.g., transmittance and reflectance) may be sampled from the representation 15 and used to calculate reflected and transmitted signals 36 along the beams. In some examples, rendering may include sampling beams along predetermined range windows. Furthermore, rendering may include integrating beams over a two-dimensional space in the scene according to the particular range window.

[0054] Additionally or alternatively, rendering may include scanning beams along predetermined azimuth or elevation windows. In some examples, rendering includes considering parameters of radar sensor system 11 (e.g., antenna beamforming gain or other parameters). The parameters of radar sensor system 11 may be self-learned in some examples.

[0055] In some examples, rendering may include generating synthetic radar measurements 21, 37 corresponding to the acquired radar measurements discussed above. For example, rendering may include rendering the synthetic radar measurements 21, 37 corresponding to the acquired radar measurements after one or more of the preprocessing steps discussed above (e.g., one or more synthetic radar measurements are generated by synthesizing the acquired radar measurements of the radar sensor system at specific locations).

[0056] As explained, in some examples, the radar measurements are represented as having a range value, a Doppler value, and an elevation / azimuth value. In other words, each radar pixel in a measurement or view is identified by a range window (i.e., identifying a distance to the radar sensor), a Doppler window, and an azimuth / elevation window. Figure 1 illustrates the range dimensions / windows 20a, the Doppler dimension / window 20b, and the azimuth / elevation or antenna dimension / window 20c for synthetic radar measurements 21. For some radar sensor systems 11 (e.g., certain mmWave radar sensor systems), resolution along the elevation and / or azimuth directions may be relatively coarse. As explained above, using the Doppler representation can reduce angular ambiguity for these systems (and other systems as well).In this case, the two-dimensional region of the scene projected onto a radar pixel may be a circle (see exemplary circle 43 in Fig. 4(b)). In some cases, rendering the radar pixel 44 of a synthetic radar measurement 21, 37 may involve integrating along this circle 43.

[0057] In some examples, rendering can be optimized to improve the computational efficiency of the process. For example, samples of the radar propagation properties of the representation can be reused to render multiple radar pixels. This can reduce how often the representation needs to be sampled. For example, in a representation that has a range dimension and a Doppler dimension, as introduced above, all windows with the same Doppler value can use the same samples of the radar propagation properties of the representation (e.g., transmittance and reflectance).

[0058] Additional details of an example rendering procedure are given in the paper by T. Huang et al. entitled “DART: Implicit Doppler Tomography for Radar Novel View Synthesis.”

[0059] Although some specific aspects of the rendering step were discussed above, the technique of the present disclosure is not limited in this regard. It should be appreciated that any rendering (or ray tracing) technique may be employed to generate synthetic radar measurements.

[0060] A suitable representation of the radar propagation properties of a scene for which synthetic radar measurements are to be rendered must be selected (i.e., input parameters necessary for the rendering equation must be sampled from the representation). As a result of the rendering step, synthetic radar measurements 21, 37 corresponding to the acquired radar measurements at the plurality of positions in the scene 12 can be generated.

[0061] Now, the techniques of the present disclosure include learning 103 the representation 15 of the radar propagation characteristics of the scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements 21, 37 of the scene 12. In other words, the representation is trained (i.e., its parameters are adjusted) such that the synthetic radar measurements 21, 37 rendered based on the output of the representation 15 resemble the corresponding acquired radar measurements of the scene 12. As mentioned above, the acquired radar measurements form the ground truth for the training process. As also explained above, the learning process does not directly learn a representation using the acquired radar measurements as input parameters. Rather, the radar measurements or views for the positions (e.g.,Location and direction) of the acquired radar measurements are synthesized, and the parameters of the representation can be learned based on a comparison of the acquired and synthesized radar measurements 21 , 37 .

[0062] The learning step may include any machine learning technique. For example, reducing a difference between the acquired radar measurements and the synthetic radar measurements 21, 37 of the scene 12 may include defining an error function (e.g., a difference measure between acquired and synthesized radar measurements) and reducing the error function (in an iterative manner). In some examples, reducing the difference may include applying a gradient descent technique (e.g., a stochastic gradient descent technique).

[0063] In some examples, the representation 15 includes a neural network 23. Any technique for learning the parameters of a neural network (e.g., a backpropagation algorithm) may be employed to reduce a difference between the obtained radar measurements and the synthetic radar measurements 21, 37 of the scene 12.

[0064] When a certain stopping criterion is met (e.g., a difference between acquired and synthesized radar measurements is below a certain threshold), training can be stopped. The trained representation of the radar propagation properties of the scene can then be used to synthesize radar measurements of scene 12. The representation encodes the radar propagation properties (e.g., transmittance and reflectance 17) of scene 12.

[0065] It should be emphasized that the representation 15 only encodes the scene 12 for which it was trained. However, unlike some prior art techniques that involve directly modeling or learning a scene from acquired radar measurements, the above technique may be scene-agnostic, meaning the representation may be trained to encode the radar propagation properties of other, different scenes. In some examples, after training, the representation 15 may undergo one or more post-processing steps. For example, the representation may be converted into a different form for a deployment stage. In some examples, the radar propagation properties (e.g., transmittance and reflectance) may be sampled at predefined points in the scene and stored in a data structure (e.g., a grid covering the scene).The process of generating synthetic radar views of the scene may then sample the radar propagation properties of the scene from the data structure (rather than, for example, propagating through a neural network in a forward direction). In other examples, the representation of the synthetic view may remain structurally unchanged in the training and generation phases (e.g., comprising a neural network through which data is propagated in a forward direction to sample the radar propagation properties of the scene (e.g., transmittance and reflectance).

[0066] In some examples, a method for generating one or more synthetic radar views of a scene includes obtaining 201 a representation of the radar propagation properties of a scene generated (e.g., trained) according to any of the techniques of the present disclosure, obtaining 202 a specification of one or more synthetic views of the scene, and rendering 203, based on the obtained representation of the radar reflectance properties of the scene, one or more synthetic radar measurements according to the obtained specification.

[0067] In general, the rendering process can involve the same steps as described above in the context of learning the representation parameters (except that the representation parameters are fixed in this phase). We will not repeat the details of the rendering process, but rather refer to the explanation above. Any technique described above in the context of the rendering process that is not specific to the training phase can also be used in a synthetic view generation phase. The synthetic radar views can be generated for one or more positions, orientations, and velocities of a radar sensor system monitoring the space.

[0068] In some examples, the techniques include generating a radar map of the scene based on the one or more synthetic radar measurements. For example, a tomographic map of a scene may be generated based on the synthetic radar measurements.

[0069] In some examples, rendering to generate synthetic views of the scene may be performed using a different set of radar sensor system parameters compared to the training phase. For example, a more capable radar sensor system may be used to generate the acquired radar measurements in the training phase, and a less capable radar sensor system may be used to generate the synthetic radar view (e.g., a radar sensor system that is being developed and / or tested).

[0070] Figure 5 shows multiple views of scenes (an open space comprising multiple objects in the top row and an office space in the bottom row) measured and synthesized using the techniques of the present disclosure and several other techniques. As can be seen, when using a range-Doppler representation as described above, the techniques of the present disclosure (second column) can synthesize radar measurement views that are closer to ground truth than other simulation techniques.

[0071] As also shown in Fig. 2, the one or more synthetic radar views (e.g., further processed into radar maps or other data structures) can be applied in different contexts.

[0072] In some examples, the one or more synthetic radar views generated as described in the present disclosure may be used in a development and / or testing process of a radar sensor system or a system that consumes the output data of a radar sensor system. For example, a radar sensor system or a component thereof, or a system that consumes the output data of a radar sensor system, may be developed and tested by synthesizing 301 radar measurement views of the radar sensor system or a component thereof using the representation of scenes according to the present disclosure and evaluating 302 the performance of the radar sensor system or the component thereof, or a system that consumes the output data of a radar sensor system.In some examples, the techniques may include developing and / or testing an algorithm for controlling a radar sensor system or a component thereof, or processing the output of a radar sensor system or a component thereof. For example, the radar sensor system may be deployed to monitor the environment of a system (e.g., a vehicle or a robot). Additionally or alternatively, a function of a system may be controlled based on the output of the radar sensor system. This function may include an obstacle detection or avoidance function, an environment sensing function, an environment mapping function, or more advanced functions built upon these functions (e.g., a control function of a system, such as a vehicle or a robot, or navigation in an environment).

[0073] In some examples, the techniques of the present invention include producing and / or implementing 303 a developed and / or tested algorithm for controlling a radar sensor system or a component thereof or processing the output of a radar sensor system or a component thereof (e.g., manufacturing a system comprising a radar sensor system and / or installing and / or storing software comprising the algorithm on a computer system).

[0074] In some examples, the synthetic radar views generated using the techniques of the present disclosure may themselves be used to train a system or component (e.g., a system or component for a vehicle or robot). The techniques of the present disclosure may also be used in applications other than those for developing and / or testing radar sensor systems or components thereof, or systems that consume the output data of a radar sensor system. In some examples, the techniques for generating synthetic views may be performed during operation of a system (e.g., a vehicle or robot) that includes or is coupled to a radar sensor system. The synthetic views generated during operation of the system (e.g.,Synthetic views generated by a radar sensor system (vehicle or robot) including or coupled to the radar sensor system may be used to monitor and / or control a function of the system. For example, the present disclosure also relates to a method for locating an object in a scene. The method includes obtaining a representation of the radar reflectance characteristics of a scene defined according to any of the techniques of the present disclosure. The method further includes obtaining a radar measurement of the scene taken from an unknown position and determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements generated using the representation. For example, a robot, vehicle, or other moving system may be located in the scene based on the comparison.

[0075] In other examples, the techniques for generating synthetic views may be performed during operation of a system (e.g., a vehicle or a robot) that includes or is coupled to a radar sensor system to monitor the operation of the radar sensor system. For example, a captured view of the scene based on the representation may be compared to a synthesized view to assess an operating condition and / or detect a failure condition of the radar sensor system (e.g., whether the captured views of the scene differ from the synthesized views in a predetermined manner). In the preceding sections, the techniques of the present disclosure were discussed primarily based on methods for implementing the techniques of the present disclosure.

[0076] The present disclosure relates to a computer system configured to perform any of the methods of the present disclosure. The computer system may comprise any hardware or software suitable for performing the corresponding techniques of the present disclosure. It is understood that the computer systems may differ depending on the particular instruction (e.g., depending on whether a computer system performs the representation learning techniques, generates synthetic radar views based on the learned representation, or uses the representation in the operation of a system). In various examples, the computer system may comprise a distributed

[0077] Computer system and / or a cloud computer system. The computer system may be a general-purpose computer system or may include adapted hardware for carrying out the techniques of the present disclosure. The present disclosure relates to a computer program comprising instructions that, when executed by a computer system, cause the computer system to perform the steps according to any of the methods of the present disclosure. The computer program may be stored on any suitable storage medium (e.g., an SSD storage medium). Additionally or alternatively, the

[0078] computer program encoded in a data signal.

Claims

Claims 1. A method for generating a representation (15) of the radar propagation properties (17) of a scene (12), comprising: Obtaining a plurality of radar measurements of the scene (12) taken from a plurality of positions; Rendering, based on the representation (15) of the radar propagation properties of the scene (12), a plurality of synthetic radar measurements (21; 37) of the scene (12) at the plurality of positions; and Learning the representation (15) of the radar propagation properties of the scene (12) by reducing a difference between the obtained radar measurements and the synthetic radar measurements (21; 37) of the scene (12).

2. The method of claim 1, wherein the representation (15) defines a field encoding the radar transmittance and radar reflectance properties of the scene (12), wherein learning the representation (15) comprises learning the radar transmittance and radar reflectance properties of the scene (12).

3. The method of claim 2, wherein rendering further comprises: Obtaining radar transmittance and radar reflectance values ​​from the plot (15); Propagating radar beams through the scene (12) based on the obtained radar transmittance and radar reflectance values ​​to generate the synthetic radar measurements (21; 37) of the scene (12).

4. The method of any one of claims 1 to 3, wherein obtaining a plurality of radar measurements comprises obtaining the plurality of radar measurements along a trajectory in and / or around the scene (12).

5. The method according to any one of claims 1 to 4, wherein the representation (15) comprises a neural network (23).

6. The method according to any one of claims 1 to 5, wherein the plurality of received and acquired radar measurements are represented in a space, taking into account a Doppler effect generated by a relative velocity of a radar sensor system (11) comprising the received plurality of radar measurements in the scene (12).

7. A method for generating one or more synthetic radar views (21; 37) of a scene (12), comprising: Obtaining a representation (15) of the radar propagation characteristics of the scene (12) generated according to the methods of any one of claims 1 to 6; Obtaining a specification of one or more synthetic views (21; 37) of the scene (12); Rendering, based on the obtained representation (15) of the radar reflection properties of the scene, one or more synthetic radar measurements (21; 37) according to the obtained specification.

8. The method of claim 7, further comprising generating a radar map of the scene based on the one or more synthetic radar measurements (21; 37).

9. The method according to any one of claims 7 and 8, wherein the rendering comprises: Obtaining radar transmittance and radar reflectance values ​​from the plot (15); Propagating radar beams through the scene (12) based on the obtained radar transmittance and radar reflectance values ​​to Generating the one or more synthetic views (21; 37) of the scene (12).

10. A method for locating an object (11) in a scene, comprising: Obtaining a representation (15) of the radar reflection properties of the scene (12) defined according to any one of claims 1 to 6; Obtaining a radar measurement of the scene (12) taken from an unknown position; Determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements (21; 37) generated using the representation (15).

11. A representation of the radar propagation properties of a scene, the representation being generated according to the methods of any one of claims 1 to 6.

12. A computer system configured to carry out one of the methods according to any one of claims 1 to 10.

13. A computer program comprising instructions which, when executed by a computer system, cause the computer system to perform the steps according to the methods of any one of claims 1 to 10.

Images