System for detecting objects in radar data frames using cyclic convolutional neural network

Abstract

The invention relates to a system for detecting objects in a scene from radar data frames of the scene, the frames being continuous in time. The object detection system has an encoder-decoder structure, the encoder E comprises one or more (Q) cascade stages, and each cascade stage comprises a convolutional neural network (CNN) and a recurrent neural network (RNN) which are arranged in series. And for each received new data frame rk, the encoder provides a group of spatial features Fk according to the current internal state (memory) and updates the internal state, and the internal state is formed by the internal state of each level of RNN network of the encoder. The decoder D uses the spatial features Fk provided at the moment k to provide output information relating to objects present in the scene.

Description

Technical Field

[0001] This invention relates generally to the fields of radar detection and recurrent convolutional neural networks. It is particularly applicable to driver assistance systems and even autonomous driving systems. Background Technology

[0002] Advanced driver assistance systems (ADAS) have seen significant growth in recent years. These systems help drivers of motor vehicles perceive their environment (obstacles, other vehicles, pedestrians, etc.) and can even enable the vehicle to react automatically in advance of the driver's reactions.

[0003] ADAS systems typically utilize various types of sensors mounted on the vehicle, such as cameras, lidar, and radar. Cameras provide rich semantic information about the vehicle's environment, while lidar can acquire high-resolution representations of objects present in that environment. However, cameras and lidar are sensitive to adverse weather conditions (fog, rain, snow) and poor lighting conditions (darkness), which can impair environmental perception. In contrast, radar overcomes these conditions because it operates in the millimeter-wavelength electromagnetic range, and radar beams can penetrate raindrops or fog droplets. Radar detection equipment can accurately calculate the distance to obstacles and their speed.

[0004] Using one-dimensional or two-dimensional antenna arrays can also determine the direction of arrival of reflected signals, thereby determining the angle from which obstacles are seen from the vehicle.

[0005] Typically, radar systems do not provide raw data (which is usually a power spectrum representing the environment in terms of range and angle or range and velocity). Instead, they directly apply additional signal processing steps to the power spectrum to detect potential targets. These additional steps include, for example, adaptive thresholding (constant false alarm rate (CFAR)), calculation of the range, angle of arrival, relative velocity (Doppler velocity) for potential targets, and / or target tracking.

[0006] Therefore, radar systems can generate target lists (each target, for example, associated with a point cloud) and associate each target with low-level information such as its position, azimuth, relative velocity, and radar cross-section (RCS). However, these target lists do not include classification information (e.g., pedestrians, obstacles, vehicles), making them difficult to use for driver assistance needs. Furthermore, it is impossible to determine whether multiple points in the same point cloud detected by such a radar system effectively belong to the same object, or whether the detected target is indeed the target to be monitored and not an unimportant element (e.g., a fixed element such as a tree, interference, etc.).

[0007] Radars used in driver assistance or autonomous driving applications are typically of the Frequency Modulated Continuous Wave (FMCW) type. As can be recalled, this type of radar transmits linearly frequency-modulated pulses (chirps) with a repetition period of P, such as... Figure 1 As shown, a linearly leveled pulse 110 is a linearly frequency modulated wavefunction. During reception, the received signal is mixed with the transmitted signal in 120, low-pass filtered in 130, converted to a digital signal in 140, and subjected to a Fast Fourier Transform (FFT) in 150. The FFT can be performed on a short time step (fast-time FFT or range FFT) 151 corresponding to the sampling frequency of the linearly frequency modulated pulse for range analysis (range spectrum); and for each range, it can be performed on a long time step (slow-time FFT) corresponding to the repetition period T for velocity analysis (range-Doppler spectrum) 152. When the radar system uses a transmit and receive antenna array (also known as a Multiple Input Multiple Output (MIMO) radar), the signals received by each antenna can be used to obtain the direction of arrival (DoA) (range-angle spectrum) 153 based on the range. This yields a third-order tensor of “low-level” radar data (or raw radar data), which, called RAD, can be projected onto a two-dimensional space (or view) to give RD (range, velocity), RA (range, angle), or AD (angle, velocity) frames that appear sequentially with a repeating period or a multiple thereof.

[0008] Furthermore, as mentioned above, the radar data at the FFT output can undergo post-processing 160 to provide a target list 170.

[0009] Various methods have been proposed in the prior art to detect objects from raw radar data (RD, RA, AD, and even RAD) (sometimes referred to as pre-CFAR in the literature) using deep neural networks. Most of these methods use two-dimensional convolutional neural networks (2D CNNs) that operate on a single data frame.

[0010] However, these methods failed to correctly detect similar objects (such as pedestrians or cyclists). Therefore, a method using a three-dimensional convolutional neural network (3DCNN) operating on multiple consecutive frames has been proposed in the literature. In these networks, convolutions are performed in both the temporal and spatial dimensions to reduce the size of the blocks in these dimensions and retain only the most relevant information.

[0011] For example, a description of such convolutional neural networks that operate on raw data can be found in Y. Wang et al.’s article entitled “RODNet: a real-time radar object detection network cross-supervised by camera-radar fused object 3D localization” (published in IEEE Journal of Selected Topics in Signal Processing, Vol.15, No.4, pp.954-967).

[0012] However, a drawback of these methods is that they require a considerable amount of memory to store multiple (N) consecutive frames.

[0013] Furthermore, object detection is often non-causal; in other words, it is necessary to acquire frames after an event has occurred in order to detect it. These limitations do not allow for real-time detection in in-vehicle systems. However, the real-time performance of detection is a crucial criterion for driver assistance systems or autonomous driving systems.

[0014] Therefore, the object of the present invention is to propose a system for detecting objects in a scene based on radar data frames of the scene using an artificial neural network. This system does not have the aforementioned disadvantages, namely, it does not require the storage of multiple consecutive frames, is causal, and allows for real-time object detection. Summary of the Invention

[0015] To achieve the above objectives, the following object detection system is proposed according to the present invention.

[0016] This system is for detecting objects in a scene based on radar data frames, which are sequentially arranged in time. Each radar data frame provides two or three types of information, including range, angle of arrival, and Doppler velocity. The system is characterized by comprising at least one spatiotemporal encoder and at least one decoder. The spatiotemporal encoder is designed to receive frames and includes multiple cascaded stages, each stage comprising a convolutional neural network (CNN) and a recurrent neural network (RNN) arranged in series. The encoder is configured to, based on its internal state, detect objects in a scene based on each new data frame r received at time k. k At this location, a set of spatial features F is provided. k The decoder updates its internal state based on the new frame. The internal state is formed by the internal states of the RNN network at each stage of the encoder, and the decoder uses a set of spatial features F at time k. k and the internal state H, which can be optionally updated by the encoder.k This provides output information related to objects present in the scene.

[0017] The frames received by the spatiotemporal encoder can be of different types.

[0018] These radar data frames are typically obtained by projecting radar measurement tensors into a two-dimensional space, or by constructing a three-dimensional tensor.

[0019] In the object detection system disclosed herein, the radar data frame provides (in particular, only provides) two or three types of information about the range, angle of arrival, and Doppler velocity.

[0020] In addition, the output information may include one or more basic information, which have different properties.

[0021] Therefore, in some embodiments, the output information is very simple and includes, for example, only one or more binary information and / or one or more information in numerical form (real numbers).

[0022] The output information may indicate, for example, whether a moving object was detected in the scene (binary information), and optionally the distance of the detected object (numerical information).

[0023] In some embodiments, the output information includes at least one mapping, specifically a mapping that gives the probability of the presence of an object at each point in the scene.

[0024] In some embodiments, the output information includes a semantic segmentation map constituting the scene, which is in the form of an image, and provides the category to which each pixel of the image belongs, the category being selected from a plurality of possible categories.

[0025] Furthermore, various architectures can be used for the spatiotemporal encoder and decoder.

[0026] In some embodiments, the convolutional network of a stage or each stage of the encoder's cascaded stages is a separable convolutional network, such as an inverted residual bottleneck type network.

[0027] A separable convolutional network is a network that includes at least one convolutional layer, wherein the convolutional operations are of the type "depthwise separable convolutions", in which operations in the depth direction ("depthwise convolution") and convolutional operations in the lateral direction ("pointwise convolution") are performed separately.

[0028] The “depth-separable convolution” type of convolution operation is described in detail in the following literature: Howard, AG, Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:17O4.O4861.

[0029] In some embodiments, the cascaded stages of the encoder, or the recurrent network of each stage, is a separable convolutional network, such as a ConvLSTM network, particularly a bottleneck LSTM type network.

[0030] In some embodiments, the encoder includes a final stage downstream of the cascaded stages, which consists of a convolutional network with separable convolutions, such as an inverted residual bottleneck type network.

[0031] In some embodiments, the decoder includes multiple cascaded stages, each stage of the decoder being configured to perform transposed convolutions.

[0032] Furthermore, the transmission of certain information from the encoder to the decoder can improve network performance.

[0033] Therefore, in some embodiments, at least one level of the decoder receives spatial features provided by the corresponding level of the encoder via a skip connection, and optionally receives the internal state of the recurrent neural network belonging to that level.

[0034] Furthermore, in some embodiments, the object detection system is configured to optimally utilize the information acquired by radar.

[0035] Therefore, in some embodiments, the object detection system includes multiple encoders, each encoder designed to receive radar measurement tensors (T0, T ...). RAD The radar data frames are obtained by projecting them onto different two-dimensional spaces. Each encoder is configured to, based on its internal state, process each new data frame r received at time k. k At time k, a set of spatial features is provided, and its internal state is updated according to the new frame; and at least one decoder is configured to provide output information using a set of spatial features at time k, and optionally using internal states updated by multiple encoders.

[0036] In some alternatives to these embodiments, at least one level of the decoder receives, via a skip connection, composite spatial features resulting from the splicing of spatial features provided by corresponding levels of the same level of multiple encoders, and optionally composite internal states obtained from the internal states of recurrent neural networks belonging to the same level of multiple encoders. Attached Figure Description

[0037] Other features and advantages of the invention will become apparent from reading the preferred embodiments of the invention with reference to the accompanying drawings, in which:

[0038] Figure 1 As already described, a radar system known in the prior art capable of generating radar data frames is schematically shown;

[0039] Figure 2 The operation timing of a recurrent convolutional neural network configured in an object detection system according to an embodiment of the present invention is illustrated schematically.

[0040] Figure 3 The encoder-decoder architecture of the object detection system according to a first embodiment of the present invention is illustrated schematically.

[0041] Figure 4A The illustration shows the learning phase of a recurrent convolutional neural network according to an example of a first embodiment of the present invention;

[0042] Figure 4B The illustration shows the inference phase of a recurrent convolutional neural network according to an example of a first embodiment of the present invention;

[0043] Figure 5 The architecture of an example object detection system according to a first embodiment of the present invention is illustrated schematically;

[0044] Figure 6 The encoder-decoder architecture of an object detection system according to a second embodiment of the present invention is illustrated schematically.

[0045] Figure 7 schematically shown Figure 6 A skip connection between the encoder and decoder of an object detection system. Detailed Implementation

[0046] The following considers a system for detecting objects in raw radar data frames that are sequentially generated at a predetermined repetition period T. Here, the raw radar data frame (hereinafter more simply referred to as a radar data frame or radar spectrum) refers to the projection of a radar measurement tensor into two-dimensional space, whose components are geometric coordinates or velocity values. For example, the measurement tensor may be the RAD tensor (range, angle, Doppler velocity) as defined above, and the radar data frame may be an RD (range, Doppler velocity), AD (angle, Doppler velocity), or RA (range, angle) frame. The raw radar data may be provided by a frequency-modulated continuous wave (FMCW) radar as described in the introduction or by another type (e.g., pulse radar).

[0047] This detection system aims to provide output information related to objects present in a given scene at each repetition cycle, particularly in the form of a mapping. Depending on the intended use, this mapping may be in the form of a confidence map, giving the probability of an object's presence at each point in the scene; or in the form of a semantic segmentation mask, giving the category to which each pixel of the image representing the scene belongs, selected from multiple available categories. Those skilled in the art can use other formats for the output information, particularly other types of mappings of objects present in the scene, without departing from the scope of this invention.

[0048] The object detection system according to the present invention uses a prediction model M, which includes a spatiotemporal encoder E and a decoder D. If we denote the sequence of N consecutive radar data frames as R... k =(r k-N+1 ,r k-N+2 ,..,r k ), where k is the time step index (e.g., RA or RD radar spectrum), then the prediction model is constructed as follows:

[0049] [Mathematical Expression 1]

[0050] M(R k )=(D ° E)(R k (1)

[0051] ° is a composite symbol for functions.

[0052] The spatiotemporal encoder E can be considered as a state machine: at each time k, it is based on the last received radar data frame r k and its current internal state H k-1 (Its memory) provides a set of spatial features F k And update its internal state, which can be represented as:

[0053] [Mathematical Expression 2]

[0054] E(r k Hk-1 ) = (F k H k (2)

[0055] Decoder D uses the internal state H updated by the spatiotemporal encoder. k and all spatial features F provided at time k k Provides information related to objects in the scene in the form of a mapping (e.g., a confidence map). k ,or:

[0056] [Mathematical Expression 3]

[0057] D(F k H k ) = P k (3)

[0058] Figure 2 The operation of the prediction model over time is illustrated schematically.

[0059] More precisely, the figure shows two consecutive time points indexed k-1 and k. At time k-1, the spatiotemporal encoder E receives radar data frame r. k-1 And according to its internal state H k-2 (its memory) provides a set of spatial features F to the decoder D. k-1 Furthermore, it is based on radar data frames r k-1 and its current state H k-2 This is used to update its internal state. The updated internal state is also provided to decoder D. Decoder D then processes the data based on spatial features F. k-1 and encoder update state H k-1 Predict the graph P of objects present in the scene. k-1 The process continues iteratively as shown at the next time step k.

[0060] The idea behind this invention is to combine at least one convolutional neural network (CNN) and one recurrent neural network (RNN) in a spatiotemporal encoder. The purpose of the convolutional neural network is to extract “spatial” information from radar data frames, where “spatial” refers to the variables represented in these frames (e.g., RA, AD, RD for the first layer), and the recurrent neural network operates on the spatial information thus extracted.

[0061] Advantageously, a spatiotemporal encoder can have multiple cascaded, alternating CNN and RNN networks. In practice, CNN networks can reduce the dimensionality of their inputs to retain only their relevant and concise representations, while RNN networks, by storing memories of past inputs in their internal states, can jointly manage spatial and temporal information. In this case, the RNN network is able to analyze the temporal dependencies between radar data frames.

[0062] Figure 3 The encoder-decoder architecture of an object detection system equipped with such a spatiotemporal encoder is illustrated schematically.

[0063] The spatiotemporal encoder E consists of Q ≥ 2 cascaded stages, each composed of a CNN network and an RNN network arranged in series. Therefore, the state of the spatiotemporal encoder at a given time is defined by the respective internal states of the RNN networks of each stage at that time. In other words:

[0064] [Mathematical Expression 4]

[0065]

[0066] in These are the internal states of the Q-level RNN networks belonging to the spatiotemporal encoder.

[0067] All spatial features F k and the state H of the space-time encoder k (In other words, the state of the RNN network it contains, i.e. its memory) is provided to the decoder D.

[0068] CNN neural networks can advantageously be separable convolutional networks.

[0069] In particular, CNN neural networks can be advantageously chosen as inverted residual bottlenecks (IR). A detailed description of this type of neural network can be found in the paper "MobileNetV2: Inverted residuals and linear bottlenecks" by M. Sandler et al., published in Proc. of the IEEE Conf. on computer vision and pattern recognition, 2018, pp. 4510-4520. Using IR networks reduces the number of parameters and the number of multiply-accumulate (MAC) operations compared to traditional CNN networks because IR networks use separable convolutions, meaning the convolutions consist of depthwise convolutions (i.e., one at a time on one channel) and pointwise convolutions (i.e., simultaneously on all channels).

[0070] Similarly, RNN neural networks can advantageously be separable convolutional networks.

[0071] Specifically, the RNN network can be chosen to be of the Long Short Term Memory (LSTM) type, and advantageously of the bottleneck LSTM type. A detailed description of such neural networks can be found in the paper entitled "Mobile video object detection with temporally-aware feature maps" by M. Liu et al., published in Proc. of the IEEE Conf. on computer vision and pattern recognition, 2018, pp. 5686-5695.

[0072] Advantageously, a normalization layer can be provided before the activation (sigmoid function) of the input gate, forget gate, and output gate of each LSTM network, which performs normalization on all spatial features.

[0073] Finally, as Figure 3 As shown, the first layer of the spatiotemporal encoder includes a conventional convolutional neural network to increase the number of channels, with each channel corresponding to a kernel for block convolution. Each stage of the CNN-RNN network operates on radar data frames that have previously undergone this block convolution step.

[0074] During training, the object detection system sequentially receives N data frames and their corresponding prediction results (i.e., mappings of the actual objects in the scene) at the input of the spatiotemporal encoder. Therefore, the model is trained to encode the sequence of N data frames to predict the corresponding mapping P at each time t. t Or for the decoder:

[0075] [Mathematical Expression 5]

[0076] D(F t H t ) = P t (4)

[0077] Where F t and H t These are all the features provided by the spatiotemporal encoder and their internal states after being updated at time t.

[0078] Figure 4A It shows according to Figure 3 This is a step in the training phase of an example object detection system. In the example shown, the spatiotemporal encoder consists of Q = 2 cascaded stages, each stage comprising a CNN network (IR network) and an RNN network set in series. Therefore, the state of the spatiotemporal encoder after updating at time k is determined by... The following is given. A block convolutional network is set up upstream of the CNN-RNN stage, and an IR network is set up downstream of the cascaded CNN-RNN stage. The spatiotemporal encoder sequentially receives radar data frames at the input end, and the decoder predicts P based on the last N=3 received frames. k The prediction is compared to a mapping of objects present in the scene. After calculating the loss function and performing gradient backpropagation to minimize the prediction error, the parameters of the individual IR and RNN neural networks are updated.

[0079] Once training is complete, the detection system can be used during the inference phase to predict information related to objects in the scene.

[0080] Figure 4B This illustrates a step in the inference phase of a previously trained object detection system. At each time k, the encoder updates its state based on the state it was in before the update. and the radar data frames it receives k Provide the decoder D with a set of spatial features and the updated state. Decoder D then infers a mapping of the objects detected in the scene.

[0081] Figure 5 The encoder-decoder architecture of an example object detection system according to a first embodiment of the present invention is illustrated schematically.

[0082] The architecture of this example embodiment is of the U-net type, and its description can be found in the article entitled "U-Net: convolutional networks for biomedical image segmentation" by O. Ronneberger et al., published in Proc. of Medical Image Computing and Computer-Assisted Intervention, MICCAI 2015, pp. 234-241.

[0083] The architecture includes a shrinking branch formed by the spatiotemporal encoder E and an expanding branch formed by the decoder D.

[0084] The spatiotemporal encoder E consists of two cascaded stages, 520 and 530 (in other words, Q=2). Each stage includes a cascaded CNN network (inverted residual bottleneck type) and an RNN network (bottleneck LSTM type). At each time k, the RNN network of the first stage, 520, transmits its internal state through the first temporal skip connection. Similarly, the second pair of 530 RNN networks provides its internal state to the decoder via a second time-hop layer connection. Provided to the decoder.

[0085] The spatiotemporal encoder E includes at least one block convolutional neural network 510 upstream of the first stage, whose output is a block of size C×H×W, where C is the number of channels (i.e. the number of convolutional kernels, and therefore the number of spatiotemporal features), and H and W are the height and width of the radar data frame, respectively.

[0086] Finally, downstream of stage 530, the spatiotemporal encoder includes a third and final stage, namely a CNN network (inverted residual bottleneck type) 540.

[0087] In consecutive levels of a CNN network, the number of channels is doubled, while the width and height of the blocks are halved. In other words, C = 2C1 = 4C2 = 8C3.

[0088] The decoder comprises three transposed 2D convolutional stages 550, 560, and 570, followed by a convolutional stage 580, to obtain an object graph (e.g., a confidence map or semantic segmentation of a scene image) as output information. Each transposed convolutional stage of the decoder corresponds to a stage of the encoder: stage 550 corresponds to CNN network stage 540, stage 560 corresponds to CNN-RNN stage 530, and stage 570 corresponds to CNN-RNN stage 520.

[0089] Each stage of the decoder receives spatial features from the corresponding stage (520, 530, 540) of the encoder via a skip connection, and, if applicable, receives the internal state of the RNN belonging to that stage (520, 530) via a temporal skip connection (TSC). This TSC connection avoids the loss of temporal information during the decoding path. (The internal state of the RNN belonging to a stage of the encoder is shown in the original text.) It is concatenated with the result of the transposed convolution provided by the previous stage of the corresponding stage of the decoder.

[0090] Therefore, for example, the internal state of the second-stage 530 RNN network of the encoder. The result of the transposed convolution of the first stage 550 of the decoder is concatenated or added. Similarly, the internal state of the first stage 520 of the encoder's RNN network is... The result of the transposed convolution of the second stage 560 of the decoder is concatenated or added.

[0091] Figure 6 The architecture of an object detection system according to a second embodiment of the present invention is illustrated schematically.

[0092] The second embodiment differs from the first embodiment in that it can utilize various dimensions such as distance, velocity, and azimuth / angle to detect objects present in the scene. Therefore, this embodiment can utilize all the information present in the radar signal.

[0093] According to the first alternative (not shown in the accompanying drawings), the spacetime encoder E RAD Receive continuous radar frames r in each time interval k .

[0094] In this alternative, radar frame r k It integrates three types of information: range, velocity, and angle (R, D, A). These radar frames can be in the form of tensors, for example, where two of the range, velocity, and angle (R, D, A) information are encoded according to the tensor's horizontal dimensions (height and width), while the third type of information is encoded according to the tensor's depth dimension ("channel").

[0095] Figure 6 A second alternative is shown. This alternative differs from the first in that it uses various views (or projections) of the radar measurement tensor, such as the RA, RD, and AD frames defined above, to detect objects present in the scene. Therefore, this alternative uses multiple spatiotemporal encoders to separate the dimensions of distance, velocity, and azimuth / angle pairwise.

[0096] The object detection system according to this alternative includes a view that measures tensors. Figure 1 Multiple encoders, that is, in the example shown, three encoders E RA E RD E AD Each encoder has the structure of the first embodiment. More precisely, each of these encoders includes the same number of Q stages, each stage consisting of a CNN network cascaded with an RNN network. At each time k, the spatial features from different encoders... The features are stitched together in the multi-view latent variable space to form a composite spatial feature. This feature is provided to one or more decoders, here being decoder D. RA and D RD Similarly, encoder E RA E RD E AD The internal states of the RNN network at each level q=1,...,Q are denoted as follows: They are pieced together to form a composite internal state. This state is transmitted to the corresponding level of each decoder, such as Figure 5 As shown. Therefore, in the example shown, each decoder D RA DRD Not only can it obtain data from the corresponding encoder (E) RA E RD The time information can also be obtained from three encoders (E) associated with three views of the measurement tensor. RA E RD E AD (Time information. Composite internal state) Transmission to the decoder is achieved through a shared temporal skip connection among all views, represented in the accompanying diagram as TMVSC (Temporal Multi-View Skip Connection).

[0097] In the example shown, decoder D RA It can predict the position of an object in RA (distance, azimuth) space, while the decoder D RD It can predict the location of these objects, or display them in RD (distance, velocity) space.

[0098] Figure 7 A more detailed description is provided by Figure 6 The TMVSC connection performs a composite internal state splicing operation.

[0099] Encoder E RD E AD E RA The internal states of the RNN network contained in the level q. If necessary, the data is downsampled or upsampled to make its dimensions consistent with C×H×W before being stitched together, thus obtaining a stitched result of size 3C×H×W. Then, a 1×1 two-dimensional convolution (channel pooling) is performed on it, thereby reducing the number of channels from 3C to C, and thus enabling them to be used as composite internal states in each stage of the decoder, as in the first embodiment.

Claims

1. A system for detecting objects in a scene based on radar data frames of the scene, the frames (r k The radar data frames (r) are sequential in time. k The information provided includes two types of information (RA, RD, AD) or three types of information (RAD) regarding distance (R), angle of arrival (A), and Doppler velocity (D), characterized in that... The system includes at least one spatiotemporal encoder (E, E) RD E AD E RA ) and at least one decoder (D, D) RA D RD The spatiotemporal encoder is designed to receive the frames and comprises multiple (Q) cascaded stages (520, 530), each stage comprising a concatenated convolutional neural network (CNN) and a recurrent neural network (RNN). The encoder is configured to, based on its internal state, process each new data frame r received at time k. k At this location, a set of spatial features F is provided. k The decoder (D, D) updates its internal state based on the new frame. This internal state is formed by the internal states of the RNN network at each stage of the encoder. RA D RD Using the set of spatial features F at time k k and the internal state H that can be optionally updated by the encoder. k To provide output information (P) related to the object present in the scene. k ).

2. The system for detecting objects in a scene according to claim 1, characterized in that, The radar data frame (r) k It is obtained by projecting the radar measurement tensor into a two-dimensional space, or by constructing a three-dimensional tensor.

3. The system for detecting objects in a scene according to claim 1 or 2, characterized in that, The output information (P) k It includes at least one mapping, specifically a mapping that gives the probability of the existence of the object at each point in the scene.

4. The system for detecting objects in a scene according to any one of claims 1 to 3, characterized in that, The output information includes a semantic segmentation map of the scene in the form of an image, and provides the category to which each pixel of the image belongs, the category being from multiple possible categories.

5. The system for detecting objects in a scene according to any one of the preceding claims, characterized in that, The convolutional network of a certain level or each level of the cascaded stages of the encoder is a separable convolutional network, such as an inverted residual bottleneck type network.

6. The system for detecting objects in a scene according to any one of the preceding claims, characterized in that, The cascaded stages of the encoder, or the recurrent network of each stage, is a separable convolutional network, such as a ConvLSTM network, especially a bottleneck LSTM type network.

7. The system for detecting objects in a scene according to claim 5 or 6, characterized in that, The encoder includes a final stage (540) downstream of the cascaded stages, the final stage being composed of a convolutional network with separable convolutions, such as an inverted residual bottleneck type network.

8. The system for detecting objects in a scene according to any one of the preceding claims, characterized in that, The decoder comprises multiple cascaded stages (550, 560, 570), each stage of which is configured to perform transposed convolution.

9. The system for detecting objects in a scene according to any one of the preceding claims, characterized in that, At least one level (550, 560, 570) of the decoder receives spatial features provided by the corresponding level of the encoder via a skip connection, and optionally receives the internal state of the recurrent neural network belonging to that corresponding level.

10. The system for detecting objects in a scene according to any one of claims 1 to 9, characterized in that, The system includes multiple encoders (E RD E AD E RA Each encoder is designed to receive radar measurement tensors (T) by means of the radar measurement tensor (T). RAD Radar data frames obtained by projecting them onto different two-dimensional spaces. Each encoder is configured to, based on its internal state, process each new data frame r received at time k. k At this location, a set of spatial features (F) is provided. k RA ;F k RD ; F k AD ), and update its internal state according to the new data frame; and The at least one decoder is configured to use the set of spatial features (F) at time k. k RA ;F k RD ; F k AD The output information (P) can be provided by optionally using internal states updated by the plurality of encoders. k ).

11. The system for detecting objects in a scene according to claim 10, characterized in that, At least one level of the decoder receives spatial features (F) provided by corresponding levels of the same level of the plurality of encoders via skip connections. k RA ;F k RD ;F k AD The composite spatial features (F) produced by splicing together k RAD The recurrent neural network can optionally receive the internal state of the corresponding level of the same level (q) belonging to the plurality of encoders. The obtained composite internal state

Images