Creating and using synthetic radar views of a scene

Abstract

A first general aspect of the present disclosure relates to a method for generating a representation of the radar propagation characteristics of a scene. The method includes obtaining a plurality of radar measurements of the scene taken from a plurality of positions, rendering, based on the representation of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements of the scene at the plurality of positions, and learning the representation of the radar propagation characteristics of a scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements of the scene.

Description

Prior ArtRadar technology is used in various applications to detect environments or to detect and locate objects. For example, millimeter wave radar sensor ("mmWave") systems are becoming increasingly prevalent in vehicles and robots, e.g., for driver assistance as well as collision avoidance tasks.Because the construction, testing, and deployment of new radar systems in the real world may be costly, many rapid prototyping pipelines build increasingly on simulation. Prior art radar simulation tools often require the user to manually specify the geometry and characteristics of a scene, including all material properties. While other sensor systems (e.g., LIDAR) may be used to scan an environment and generate a grid or voxel map, they cannot detect radar specific material characteristics that are critical for generating realistic radar scans. This, due to the difficulty of thoroughly examining a scene and creating (or labeling) a model with the hand, results in (significantly) simplified scene or environment models in practice. In other prior art techniques, an explicit representation of a scene is directly generated based on radar measurements using machine learning approaches. These techniques often require extensive training data, cannot be generalized to different scenes, and can hallucinate features of the scene (i.e., encode features that are not present).The techniques of the present disclosure are intended to address some of these issues.Summary of the InventionA first general aspect of the present disclosure relates to a method for generating a representation of the radar propagation characteristics of a scene. The method includes obtaining a plurality of radar measurements of the scene taken from a plurality of positions, rendering based on the representation of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements of the scene at the plurality of positions, and learning the representation of the radar propagation characteristics of a scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements of the scene.A second general aspect of the present disclosure relates to methods for generating one or more synthetic radar views of a scene. The method comprises obtaining a representation of the radar propagation properties of a scene generated according to the method of the first general aspect, obtaining a specification of one or more synthetic views of a scene, and rendering, based on the obtained representation of the radar reflection properties of a scene, one or more synthetic radar measurements according to the obtained specification.A third general aspect of the present disclosure relates to a method of locating an object in a scene. The method comprises obtaining a representation of the radar propagation characteristics of a scene defined according to the first general aspect, obtaining a radar measurement of the scene taken from an unknown position, and determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements generated using the representation.A fourth general aspect of the present disclosure relates to a method for developing and / or testing a radar sensor system, a component thereof, or a system that consumes the output data of a radar sensor system. The method comprises obtaining a representation of the radar reflection characteristics of a scene defined according to the first general aspect, rendering based on the obtained representation of the radar propagation characteristics of a scene, one or more synthetic radar measurements according to the obtained specification, and using the one or more synthetic radar measurements for developing and / or testing and / or testing a radar sensor system, a component thereof, or a system that consumes the output data of a radar sensor system.A fifth general aspect of the present disclosure relates to a representation of the radar propagation properties of a scene, wherein the representation is generated according to the first general aspect.A sixth general aspect of the present disclosure relates to a computer system configured to execute any one of the methods of the first to fifth general aspects.A seventh general aspect of the present disclosure relates to a computer program comprising instructions which, when executed by a computer system, cause the computer system to carry out the steps according to any of the methods of the first to fifth general aspects.The techniques of the first to seventh aspects may have one or more of the following advantages in some implementations.First, the techniques of the present disclosure may not require an explicit algorithm to restore the representation of the radar propagation characteristics of a scene based on the plurality of radar measurements of the scene taken from a plurality of positions. This is the design paradigm of many prior art techniques that relate to the construction of representations (also called models) of the radar propagation characteristics of a scene. Instead, the representation according to the present disclosure is learned by implicitly generating synthetic radar measurements or views of the scene based on the representation. The parameters of the representation are adjusted in this process. This may result in a representation implicitly encoding the radar propagation characteristics of the scene (i.e., an implicit scene representation). In this way, explicit modeling of the scene taking into account geometry and material characteristics may not be required (or only to a lesser extent compared to some techniques that include an explicit algorithm for restoring the representation of the radar propagation characteristics of a scene based on the plurality of radar measurements of the scene taken from a plurality of positions) in certain implementations. This in turn may simplify the modeling task, but may also result in a more accurate presentation in some situations. In some prior art techniques, the difficulties and complications of the explicit modeling task result in using simplifying assumptions and approximations to make the modeling task manageable. This may result in errors and artifacts in the synthetic views generated based on the corresponding models. The present approach has certain conceptual similarities to an approach called Neural Radiation Fields (NeRFs) that is used to learn implicit scene representations to generate synthetic images in the optical or visual domain. However, the radar range has several distinct differences compared to the optical or visual range that prevents direct transmission of solutions proposed for neural radiance fields in the optical or visual range into the radar range.Second, (and related to the first point), the techniques of the present disclosure may enable more accurate synthetic measurements or views of a scene in the radar domain to be generated, in some examples. These synthetic measurements or views may be used in various radar sensor system or system development and testing tasks that consume radar sensor system output data (e.g., in rapid prototyping tasks), and in some situations may speed up and / or improve performance of the radar sensor systems so developed or tested or the system that consumes radar sensor system output data. Moreover, the representations of the present disclosure may be employed in various applications of radar sensor systems to monitor and control systems for which the radar sensor systems are employed (e.g., for or in a vehicle or robot).In the present disclosure, multiple terms are used in a particular manner:A "representation of the radar propagation characteristics of a scene" is a model of the scene in the radar domain. The representation may represent the radar propagation characteristics (e.g., transmittance and reflectance orThe other characteristics of the propagation may encode such that a rendering technique may scan the representation to render a synthetic view of the scene. The representation may, but need not, encode or be based on a geometric representation of the scene.The term "minimizing" includes, but is not limited to, finding an actual (global or local) minimum of a function that is the subject of the minimizing step (e.g., an error function that quantitates a difference between acquired radar measurements and synthetic radar measurements of a scene). Minimization is rather the process of reducing a quantity (e.g., an error function) according to a predetermined criterion. The terms "training" and "learning" refer to the corresponding concepts in the machine learning domain. For example, training a representation or model may mean to determine values of parameters of the representation or model (to adapt the representation or model to a particular task, e.g., generating synthetic radar measurements in the present disclosure). The term "learning" describes the same act, but "seen through the eyes" of the representation or model.A "scene" may be any environment and / or geographical area (i.e., a scene is not limited to a particular space, such as a single room), the space of which is potentially or actually monitored or examined by the radar sensor system. A scene may have arbitrary spatial extent and configuration (e.g., a scene may span multiple rooms, floors, buildings, a portion of an indoor and / or outdoor environment, such as a road or other passway and / or environments).The term "radar measurement" or "view" is used for measured or detected as well as synthetically produced data.A "vehicle" may be any device for transporting people or goods (e.g., a car, bus, or truck). Vehicles may be ground-based, sea-based, aerial-based, or space-based (e.g., aircraft, ships, spacecraft, etc.). A vehicle may be configured to operate at least partially autonomously.DESCRIPTION OF THE DRAWINGSFIG. 1 is a schematic drawing of example elements involved in generating synthetic views of radar measurements, in accordance with the present disclosure. FIG. 2 is a swimming trajectory diagram illustrating various methods according to the present disclosure. FIG. 3 is a schematic illustration of a rendering process for generating synthetic radar measurements in accordance with the present disclosure. FIGS. 4( a) and 4( b) illustrate the concept of Doppler mapping radar sensor measurement data according to the present disclosure. FIG. 5 shows multiple views of scenes measured and synthesized using the techniques of the present disclosure and several other techniques.DETAILED DESCRIPTIONFirst, we discuss the technique for generating a representation of the radar propagation characteristics of a scene, comprising the preprocessing steps of the measurement data and the rendering steps for generating synthetic views. FIG. 1 is a schematic drawing of example elements involved in generating synthetic views of radar measurements, in accordance with the present disclosure. FIG. 2 is a swimming trajectory diagram illustrating various methods according to the present disclosure.A method for generating a representation of the radar propagation characteristics of a scene 12 comprises obtaining 101 a plurality of radar measurements of the scene taken from a plurality of positions, rendering 102 based on the representation 15 of the radar propagation characteristics of the scene, a plurality of synthetic radar measurements 21 of the scene 12 at the plurality of positions, and learning 103 the representation 15 of the radar propagation characteristics of a scene 12 by reducing a difference between the obtained radar measurements and the synthetic radar measurements 21 of the scene 12.Obtaining 101 a plurality of radar measurements of the scene 12 taken from a plurality of positions may include obtaining the plurality of radar measurements along a trajectory in and / or around the scene 12. The plurality of radar measurements may be detected by any suitable radar sensor system 11 (e.g., by moving the radar sensor system 11 through and / or around the scene 12). In some examples, the plurality of radar measurements are detected by a plurality of different radar sensor systems. In some examples, the plurality of radar measurements of the scene 12 are labeled with metadata. The metadata may include information indicative of a pose of the corresponding radar sensor system 11 in detecting the radar measurement of the plurality of radar measurements (e.g., a location in the scene 12 and / or a direction in which the radar sensor system points). Additionally or alternatively, the metadata may include information regarding a relative speed of the corresponding radar sensor system 11 and the scene 12 when detecting the radar measurement. In some examples, the scene 12 is (substantially) static such that the relative velocity information includes a velocity of the radar sensor system 11 relative to the scene 12 when the radar measurement is obtained. As discussed below, this relative velocity information may be helpful in increasing the angular resolution of the (measured and synthesized) views in certain situations.The radar sensor system 11 may include a plurality of antennas (e.g., one or more transmit antennas and one or more receive antennas arranged in an array, for example, two or more transmit antennas and two or more receive antennas). In some examples, the radar measurements include multiple channels, each channel corresponding to one antenna of a plurality of antennas of the radar sensor system. In some examples, the radar sensor system 11 employs a frequency modulated continuous wave (FMCW) (e.g., a frequency of the waveform transmitted by the radar sensor system 11 is continuously and / or periodically changed). In this way, radar radiation reflected from different instances may have different frequencies (and range data may be obtained thereby because the range or distance of a reflective surface manifests itself in a frequency shift).In some examples, the radar measurements are the result of a pre-processing procedure. In these examples, raw data measurements are obtained at the plurality of locations (the term "raw measurement data" is used in the present disclosure to indicate any measurement data prior to the output of a preprocessing pipeline, i.e., raw measurement data that has undergone a first preprocessing step of multiple preprocessing steps is still referred to as raw measurement data). The examples include preprocessing the raw data measurements to obtain the radar measurements used in the further processing steps described herein.The preprocessing procedure may comprise one or more of the following steps.In some examples (e.g., if an FMCW radar sensor system is used), range data (i.e., data indicative of a distance of a reflective surface to the radar sensor system) may be generated by calculating a Fourier transform (e.g., a 1-dimensional FFT) of the raw data measurements. This can convert frequency shifts in the raw data measurements into time delays (which are proportional to distance). In some examples, the range coordinate includes a plurality of range windows (e.g., 50 or more, or 100 or more range windows).Additionally or alternatively, the preprocessing procedure may comprise calculating Doppler velocities by processing a time series of raw measurement data obtained. The Doppler speeds indicate a relative movement speed of the radar sensor system 11 and the scene 12. In some examples, the scene 12 is static (considered static). In these examples, the Doppler speeds indicate a speed of the radar sensor system 11 in the (static) scene 12. As the radar sensor system 11 moves through and around the scene, a phase of the reflected signal may change between different measurement windows. The Doppler velocity may be calculated based on a rate of this phase change. In some examples, the preprocessing may include calculating a Fourier transform over a time sequence of radar measurements obtained at different times as the radar sensor system 11 moves through or around the scene 12.In some examples (e.g., for radar sensor systems 12 including a plurality of antennas), another (azimuth and / or elevation) Fourier transform is calculated over the raw measurement data obtained from different antennas (e.g., for multiple pairs of receive and transmit antennas, e.g., particularly all possible pairs of transmit and receive antennas of the radar sensor system) to generate a plurality of azimuth and / or elevation windows in the radar measurement data (radar measurement data is often represented in a range dimension indicating a distance between the radar sensor system and an object 13, and two angle components, referred to as elevation and azimuth, indicate an angle in two mutually perpendicular planes centered on the radar sensor system).A resulting set of measurement data has a similar structure as shown for the synthetic measurement data 21 in FIG. 1. As can be seen, the measurement data is three-dimensional: the set of measurement data comprises a first dimension which indicates the particular antenna or the transmitter / receiver pair (FIG. 1 shows, by way of example, four windows drawn perpendicular to the paper plane). In a second dimension, each data set includes a plurality of windows for the range data (drawn along the vertical axis) and, in a third dimension, a plurality of Doppler velocity windows (drawn along the horizontal axis). The measurements are divided along three dimensions, i.e., a range dimension, a Doppler velocity direction, and an antenna direction (e.g., after one or more of the preprocessing steps discussed above and / or below have been traversed). A particular window is also referred to as a "radar pixel" in the present disclosure. Other structures of the radar measurement data may be used in different situations. For example, in some examples, the radar measurement data does not include a Doppler dimension.Determining the Doppler speeds and performing the respective preprocessing steps discussed above may have certain advantages in some situations. In particular, the resolution of the Doppler velocity data (in theory) may be arbitrarily high, as it can be increased by increasing an integration time of a radar measurement.In some examples, the Fourier-transformed raw measurement data may be filtered to reduce the leakage effect (i.e., spectral leakage between adjacent windows along the corresponding axis). For example, a Hann filter may be applied to range data and / or Doppler velocity data.The radar measurement data preprocessed according to one or more of the steps discussed above is then used in the generation process to generate a representation of the radar propagation characteristics of a scene according to the present disclosure. We will next discuss aspects of the representation and learning technique for learning the representation of a scene. As discussed above, an element of the techniques of the present disclosure includes applying machine learning techniques to train the representation 15 such that it provides input data (regarding radar propagation characteristics of a scene) to a rendering module 22 such that the generated synthetic radar measurements 21 of the scene 12 correspond to (e.g., come as close as possible to) the obtained (captured) radar measurement as discussed above. In other words, the obtained (acquired) radar measurement data comprising the metadata (e.g. comprising information indicating a pose and / or a speed of the corresponding radar system 11 when acquiring the radar measurement) is the ground truth on the basis of which the representation 15 is trained.The representation 15 of the radar propagation properties of a scene can be modeled according to different construction principles. Next, we discuss details regarding the illustration.The representation 15 may have any suitable shape such that a rendering module 22 may be provided with the radar propagation characteristics 17 of the scene 12 needed to render synthetic measurement data 21 (which may then be compared to the captured measurement data to train the representation 15). The representation 15 has a plurality of parameters that can be adjusted in the training process to train the representation 15 to encode the radar propagation characteristics of the scene 12.In some examples, the representation 15 may include a neural network 23 (e.g., a convolutional neural network or other deep neural network topology). In this case, the parameters that may be adjusted in the training process to train the representation 15 to encode the radar propagation characteristics of the scene may include the weights of the neural network 23 (and additionally, in some examples, one or more hyperparameters with respect to the neural network 23). Additionally or alternatively, the representation may include a spatial grid structure (e.g., a simple grid), an octree, a spatial hash table, an interpolated spatial grid, a collection of points or blobs with spherical, Gaussian, or other structure, or any combination of these elements (e.g., a spatial hash table with a neural network to resolve hash collisions). Depending on the nature of the configuration, the input and output parameters of the representation and training processes may be different.In some examples, the representation 15 defines a field that encodes the radar propagation characteristics 17 (e.g., radar transmittance and radar reflectance) of the scene 12. In these examples, learning of the representation 15 includes learning the radar propagation characteristics 17 (e.g., radar transmittance and radar reflectance) of the scene 12.The representation may encode the radar propagation characteristics 12 of the scene 12 (e.g., the transmittance and the reflectance) in various, different ways.In some examples, the representation 15 models a transmittance and a reflectance for each point in the scene. The transmittance and reflectance values may depend on the angle of incidence 16 of an incoming radar wave (e.g., the representation maps a six-dimensional vector including transmittance and reflectance to a first scalar value for reflectance and a second scalar value for reflectance), in some examples. This may allow modelling of a variety of radar phenomena, such as partial occlusions, reflections and phantom reflections.In some examples, the representation 15 encodes a function that outputs one or more radar propagation parameters (e.g., transmittance and / or reflectance 17) for a set of input parameters. The set of input parameters may include any parameter characterizing one or more radar measurements (e.g., a set of parameters indicative of the geometry of an observation point, such as a position in or with respect to the scene and an orientation and / or radar sensor parameters).In the previous section, reflectance and transmittance have been discussed as exemplary parameters that encode radar propagation characteristics 17 of scene 12 in representation 15. In other examples, other parameters may also (or in addition) be used. For example, in some examples, an absorbance may be used in addition to or in place of transmittance or reflectance.In some examples, the representation 15 may employ a voxel-based approach to modeling the scene 12. For example, the scene may be modeled by a plurality of voxels, and the representation may encode radar propagation characteristics for each voxel (e.g., reflectivity and transmittance, as discussed above). In other examples, the representation does not model the scene based on a geometric subdivision of the scene.In some examples, the representation 15 encodes the radar propagation characteristics of the scene as a function of angle of view. For example, the representation may map angles of incoming radar waves in the scene 12 to a set of spherical harmonic coefficients. Radar propagation characteristics may then be calculated based on a base value of the corresponding propagation characteristic (e.g., transmittance or reflectance) and the spherical harmonic coefficients.In the example of FIG. 1, the representation 15 is structured as follows. As can be seen, the display 15 receives information regarding a plurality of range windows of a measurement position and an angle of incidence of a corresponding shaft as input parameters. The representation outputs a set of spherical harmonic coefficients 24 for the range window information as well as baseline values 25 for the transmittance and reflectance characteristics. Based on these base values and the angle of incidence, the representation can output reflectance and transmittance values 17. These reflectivity and transmittance values 17 may be consumed by the rendering module 22 to generate the synthetic radar measurements of the scene.We will next discuss in more detail aspects of the rendering step 102 based on the representation of the radar propagation characteristics of the scene of a plurality of synthetic radar measurements of the scene at the plurality of positions. FIG. 3 is a schematic illustration of a rendering process for generating synthetic radar measurements 37 in accordance with the present disclosure.In some examples, rendering includes obtaining radar transmittance and radar reflectance values 35 (or any other radar propagation characteristic of the scene) from the representation and propagating radar beams through the scene based on the obtained radar transmittance and radar reflectance values 35 to generate the synthetic radar measurements of the scene. Radar wave propagation has certain characteristics that make direct application of the rendering model used in many Neural Radiation Fields (NeRF) approaches difficult. In particular, in typical neural radiation fields (NeRF) approaches, an image pixel is rendered by integrating samples along a (one-dimensional) ray. This is represented in FIG. 4(b) by an exemplary beam 41 for a particular radar pixel 44. In typical neural radiation fields (NeRF) approaches, a model or representation of the scene outputs color and transparency values that can be sampled and integrated along the beam to obtain the pixel color. Radar waves 42 now propagate radially starting from the antenna. Therefore, depending on the detection capabilities of the radar sensor system 11, each pixel 44 in a radar image may correspond to a two-dimensional region of space in the scene. Thus, rendering a pixel 44 of a radar image may include integrating over a two-dimensional region of space in the scene.In some examples, rendering may include using a predetermined rendering equation (e.g., implemented in a rendering module 22, as discussed above), which consumes output data of the representation 15 (and optionally additional information regarding the view to be rendered and / or the radar sensor system for which the view is rendered), and generates a synthetic radar measurement 21 (i.e., a view) of the scene. In some examples, rendering may include rendering a plurality of pixels that form the synthetic radar measurement 21 (i.e., the view) of the scene.In some examples, rendering includes propagating radar waves 31 (e.g., beams) from a transmitter to a receiver at a particular position 32. radar propagation characteristics 35 of the scene along the beams 31 (e.g., transmittance and reflectance) may be sampled from the representation 15 and used to calculate reflected and transmitted signals 36 along the beams. In some examples, rendering may include scanning beams along predetermined range windows. Moreover, rendering may include integrating rays across a two-dimensional space in the scene corresponding to the determined range window.Additionally or alternatively, rendering may comprise scanning beams along predetermined azimuth or elevation windows. In some examples, rendering includes considering parameters of radar sensor system 11 (e.g., antenna beamforming gain or other parameters). The parameters of the radar sensor system 11 may be themselves learned in some examples.In some examples, rendering may include generating synthetic radar measurements 21, 37 according to the detected radar measurements discussed above. For example, rendering may include rendering the synthetic radar measurements 21, 37 according to the detected radar measurements after one or more of the preprocessing steps discussed above (e.g., one or more synthetic radar measurements are generated by synthesizing the detected radar measurements of the radar sensor system at particular locations).As discussed, in some examples, the radar measurements are represented as having a range value, a Doppler value, and an elevation / azimuth value. In other words, each radar pixel in a measurement or view is identified by a range window (i.e., identifying a distance to the radar sensor), a Doppler window, and an azimuth / elevation window. FIG. 1 illustrates the range dimensions / windows 20 a, the Doppler dimension / windows 20 b, and the azimuth / elevation or antenna dimension / windows 20 cfor synthetic radar measurements 21. As discussed above, using the Doppler plot can reduce the angular ambiguity for these systems (and also for other systems). In this case, the two-dimensional region of the scene projected onto a radar pixel may be a circle (see example circle 43 in FIG. 4( b)). In some cases, rendering the radar pixel 44 of a synthetic radar measurement 21, 37 may include integrating along this circle 43.In some examples, rendering may be optimized to improve the computing efficiency of the process. For example, samples of the radar propagation characteristics of the representation may be reused to render multiple radar pixels. This can reduce how often the representation must be scanned. For example, in a representation having a range dimension and a Doppler dimension as introduced above, all windows having the same Doppler value may use the same samples of the radar propagation characteristics of the representation (e.g., transmittance and reflectance).Additional details of an exemplary rendering procedure are given in the paper by T. Huang et al., entitled "DRT: Implicit Doppler Tomography for Radar Novel View Synthesis.".Although some certain aspects of the rendering step have been discussed above, the technique of the present disclosure is not limited in this respect. It can be seen that any rendering (or ray tracing) technique can be used to generate synthetic radar measurements. An appropriate representation of the radar propagation characteristics of a scene for which synthetic radar measurements are to be rendered must be selected (i.e., input parameters necessary for the rendering equation must be sampled from the representation).As a result of the rendering step, synthetic radar measurements 21, 37 corresponding to the detected radar measurements may be generated at the plurality of positions in the scene 12.Now, the techniques of the present disclosure include learning 103 the representation 15 of the radar propagation characteristics of the scene by reducing a difference between the obtained radar measurements and the synthetic radar measurements 21, 37 of the scene 12. As mentioned above, the detected radar measurements form the ground truth for the training process. As also discussed above, the learning process does not directly learn a representation using the acquired radar measurements as input parameters. Rather, the radar measurements or views for the positions (e.g., location and direction) of the detected radar measurements are synthesized, and the parameters of the representation may be learned based on a comparison of the detected and synthesized radar measurements 21, 37.The learning step may include any machine learning technique. For example, reducing a difference between the obtained radar measurements and the synthetic radar measurements 21, 37 of the scene 12 may include defining an error function (e.g., a difference measure between detected and synthesized radar measurements) and reducing the error function (in an iterative manner). In some examples, reducing the difference may include applying a gradient technique (e.g., a stochastic gradient technique).In some examples, the representation 15 includes a neural network 23. any technique for learning the parameters of a neural network (e.g., a backpropagation algorithm) may be employed to reduce a difference between the obtained radar measurements and the synthetic radar measurements 21, 37 of the scene 12.If a particular stop criterion has been met (e.g., a difference between detected and synthesized radar measurements is below a particular threshold), training may be stopped. The trained representation of the radar propagation characteristics of the scene may then be used to synthesize radar measurements of the scene 12. The representation encodes the radar propagation characteristics (e.g., transmittance and reflectance 17) of the scene 12.It should be emphasized that the representation 15 encodes only the scene 12 for which it was trained. However, unlike some prior art techniques that involve directly modeling or learning a scene from acquired radar measurements, the above technique may be scene agnostic, i.e., the representation may be trained to encode the radar propagation characteristics of other, different scenes. In some examples, post-training representation 15 may undergo one or more post-processing steps. For example, the representation may be converted to a different form for a stage of use. In some examples, the radar propagation characteristics (e.g., transmittance and reflectance) may be sampled at predefined points in the scene and stored in a data structure (e.g., a grid overlaying the scene). The synthetic radar view generation process of the scene may then sample the radar propagation characteristics of the scene from the data structure (instead of, e.g., propagating through a forward direction neural network). In other examples, the representation in the training phase and the generation phase (e.g., comprising a neural network through which data is propagated in a forward direction to sample the radar propagation characteristics of the scene (e.g., transmittance and reflectance) of the synthetic view may remain structurally unchanged.In some examples, a method for generating one or more synthetic radar views of a scene includes obtaining 201 a representation of the radar propagation characteristics of a scene generated (e.g., trained) according to any of the techniques of the present disclosure, obtaining 202 a specification of one or more synthetic views of the scene, and rendering 203, based on the obtained representation of the radar reflection characteristics of the scene, one or more synthetic radar measurements according to the obtained specification.In general, the rendering process may include the same steps as described above in the context of learning the parameters of the representation (only that the parameters of the representation are fixed at this stage). We will not repeat the details of the rendering process, but rather refer to the discussion above. Any technique described above in the context of the rendering process and not specific to the training phase may also be used in a synthetic view generation phase. The synthetic radar views may be generated for one or more positions, orientations, and speeds of a radar sensor system observing the space.In some examples, the techniques include generating a radar map of the scene based on the one or more synthetic radar measurements. For example, a tomographic map of a scene may be generated based on the synthetic radar measurements.In some examples, rendering to generate synthetic views of the scene may be performed using a different set of radar sensor system parameters compared to the training phase. For example, a more powerful radar sensor system may be used to generate the detected radar measurements in the training phase, and a less powerful radar sensor system may be used to generate the synthetic radar view (e.g., a radar sensor system that should be developed and / or tested).FIG. 5 shows multiple views of scenes (a free space comprising multiple objects in the top row and an office space in the bottom row), measured and synthesized using the techniques of the present disclosure and multiple other techniques. As can be seen, using a range-doppler representation as described above, the techniques of the present disclosure (2nd column) can synthesize radar measurement views that come closer to ground truth than other simulation techniques.As also shown in FIG. 2, the one or more synthetic radar views (e.g., further processed to radar maps or other data structures) may be applied in different contexts.In some examples, the one or more synthetic radar views generated as described in the present disclosure may be used in a development and / or inspection process of a radar sensor system or a system that consumes the output data of a radar sensor system. For example, a radar sensor system or a component thereof or a system that consumes the output data of a radar sensor system may be developed and checked by synthesizing 301 radar measurement views of the radar sensor system or a component thereof using the representation of scenes according to the present disclosure and evaluating 302 the performance of the radar sensor system or the component thereof or a system that consumes the output data of a radar sensor system. In some examples, the techniques may include developing and / or testing an algorithm for controlling a radar sensor system or component thereof, or processing the output of a radar sensor system or component thereof. For example, the radar sensor system may be used to monitor the environment of a system (e.g., a vehicle or a robot). Additionally or alternatively, a function of a system may be controlled based on the output of the radar sensor system. This function may include an obstacle detection or avoidance function, an environment detection function, an environment mapping function, or advanced functions based on these functions (e.g., a control function of a system such as a vehicle or a robot or navigation in an environment).In some examples, the techniques of the present invention include producing and / or implementing 303 a developed and / or tested algorithm for controlling a radar sensor system or component thereof or processing the output of a radar sensor system or component thereof (e.g., fabricating a system comprising a radar sensor system and / or installing and / or storing software comprising the algorithm on a computer system).In some examples, the synthetic radar views generated using the techniques of the present disclosure may themselves be employed to train a system or component (e.g., a system or component for a vehicle or robot) The techniques of the present disclosure may also be used in applications other than those for developing and / or testing radar sensor systems or components thereof, or systems that consume the output data of a radar sensor system. In some examples, the techniques for generating synthetic views may be performed during operation of a system (e.g., a vehicle or robot) that includes or is coupled to a radar sensor system. The synthetic views generated during operation of the system (e.g., the vehicle or robot) including or coupled to the radar sensor system may be used to monitor and / or control a function of the system. For example, the present disclosure also relates to a method of locating an object in a scene. The method includes obtaining a representation of the radar reflection characteristics of a scene defined according to any of the techniques of the present disclosure. The method further comprises obtaining a radar measurement of the scene taken from an unknown position, and determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements generated using the representation. For example, a robot, vehicle, or other moving system may be located in the scene based on the comparison.In other examples, the techniques for generating synthetic views may be performed during operation of a system (e.g., a vehicle or robot) that includes or is coupled to a radar sensor system to monitor operation of the radar sensor system. For example, a captured view of the scene based on the representation may be compared to a synthesized view to assess an operating condition and / or detect a failure state of the radar sensor system (e.g., whether the captured views of the scene deviate from the synthesized views in a predetermined manner).In the preceding paragraphs, the techniques of the present disclosure have been discussed mainly based on methods for implementing the techniques of the present disclosure.The present disclosure relates to a computer system configured to perform any of the methods of the present disclosure. The computer system may include any hardware or software suitable for carrying out the corresponding techniques of the present disclosure. It should be appreciated that the computer systems may differ depending on the particular instruction (e.g., depending on whether a computer system is executing the learning techniques of the representation, generating synthetic radar views based on the learned representation, or using the representation in operation of a system). The computer system may be a distributed computer system and / or a cloud computer system in various examples. The computer system may be a general purpose computer system or may include hardware adapted to perform the techniques of the present disclosure.The present disclosure relates to a computer program comprising instructions which, when executed by a computer system, cause the computer system to carry out the steps according to any of the methods of the present disclosure. The computer program may be stored on any suitable storage medium (e.g., an SSD storage medium). Additionally or alternatively, the computer program can be encoded in a data signal.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureHuang, T. et al., entitled "DRT: Implicit Doppler Tomography for Radar Novel View Synthesis

[0051]

Claims

A method for generating a representation (15) of the radar propagation characteristics (17) of a scene (12), comprising: obtaining a plurality of radar measurements of the scene (12) taken from a plurality of positions; rendering, based on the representation (15) of the radar propagation characteristics of the scene (12), a plurality of synthetic radar measurements (21; 37) of the scene (12) at the plurality of positions; and learning the representation (15) of the radar propagation characteristics of the scene (12) by reducing a difference between the obtained radar measurements and the synthetic radar measurements (21; 37) of the scene (12).The method of claim 1, wherein the representation (15) defines a field encoding the radar transmittance and radar reflectance characteristics of the scene (12), wherein learning the representation (15) comprises learning the radar transmittance and radar reflectance characteristics of the scene (12).The method of claim 2, wherein rendering further comprises: obtaining radar transmittance and radar reflectance values from the representation (15); propagating radar beams through the scene (12) based on the obtained radar transmittance and radar reflectance values to generate the synthetic radar measurements (21; 37) of the scene (12).The method of any of claims 1 to 3, wherein obtaining a plurality of radar measurements comprises obtaining the plurality of radar measurements along a trajectory in and / or around the scene (12).Method according to one of Claims 1 to 4, wherein the representation (15) comprises a neural network (23).Method according to one of Claims 1 to 5, wherein the pluralities of obtained and detected radar measurements are displayed in a space, taking into account a Doppler effect which is generated by a relative speed of a radar sensor system (11) which comprises the obtained plurality of radar measurements in the scene (12).A method for generating one or more synthetic radar views (21; 37) of a scene (12), comprising: obtaining a representation (15) of the radar propagation properties of the scene (12) generated according to the methods of any of claims 1 to 6; obtaining a specification of one or more synthetic views (21; 37) of the scene (12); rendering, based on the obtained representation (15) of the radar reflection properties of the scene, one or more synthetic radar measurements (21; 37) according to the obtained specification.The method of claim 7, further comprising generating a radar map of the scene based on the one or more synthetic radar measurements (21; 37).The method of any of claims 7 and 8, wherein rendering comprises: obtaining radar transmittance and radar reflectance values from the representation (15); propagating radar beams through the scene (12) based on the obtained radar transmittance and radar reflectance values to generate the one or more synthetic views (21; 37) of the scene (12).A method of locating an object (11) in a scene comprising: obtaining a representation (15) of the radar reflectivity characteristics of the scene (12) defined according to any one of claims 1 to 6; obtaining a radar measurement of the scene (12) taken from an unknown position; determining the unknown position based on a comparison of the obtained radar measurement of the scene and one or more synthetic radar measurements (21; 37) generated using the representation (15).A representation of the radar propagation characteristics of a scene, the representation being generated according to the methods of any of claims 1 to 6.A computer system configured to perform any of the methods of any of claims 1 to 10.A computer program comprising instructions which, when executed by a computer system, cause the computer system to carry out the steps according to the methods of any one of claims 1 to 10.

Images