Self-motion estimation method and device, and model training method and device

Abstract

A method and device for estimating self-motion, and a training device and method thereof are disclosed. The device for estimating self-motion can estimate self-motion information according to radar sensing data based on a motion recognition model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2018-0073709, filed on June 27, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference for all purposes. Technical Field

[0003] The following description relates to techniques for estimating ego-motion using a motion recognition model. Background Art

[0004] Recently, in order to solve the problem of classifying input patterns into specific groups, the application of effective pattern recognition methods of humans to real computers has been actively studied. One of the studies is about artificial neural networks modeled by mathematical expressions of the characteristics of human biological neurons. In order to solve the problem of classifying input patterns into specific groups, artificial neural networks use algorithms that mimic human learning abilities. Based on the algorithm, artificial neural networks can generate mappings between input patterns and output patterns, and the ability to generate mappings is expressed as the learning ability of artificial neural networks. In addition, artificial neural networks have a generalization ability that generates relatively correct outputs for input patterns that have not yet been used for learning based on learning results. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method of estimating self-motion includes generating input data based on radar sensing data collected by a radar sensor for each time frame, and estimating self-motion information of a self-motion estimation device from the input data based on a motion recognition model.

[0007] Estimating the ego-motion information may include extracting feature data from the input data based on a first model, and calculating the ego-motion information from the feature data based on a second model.

[0008] Estimating the self-motion information may include calculating at least one of a position and a posture of the self-motion estimation device as the self-motion information.

[0009] Estimating the ego-motion information may include inputting radar sensing data corresponding to at least two time frames for a layer corresponding to one time frame in a motion recognition model.

[0010] The motion recognition model may include a first model and a second model, the first model including a layer corresponding to each of a plurality of time frames, and the second model being connected to a plurality of layers of the first model. Estimating the self-motion information may include: extracting feature data of the corresponding time frame from input data of the corresponding time frame based on the layer corresponding to each of the plurality of time frames in the first model; and calculating the self-motion information of the corresponding time frame based on the extracted feature data based on the second model.

[0011] Estimation of self-motion information may include extracting current feature data from input data corresponding to a current frame based on a first model, loading prior feature data corresponding to a prior frame from a memory, and calculating self-motion information based on the prior feature data and current feature data based on a second model.

[0012] The method may further include storing, in a memory, feature data calculated in the current frame based on a first model included in the motion recognition model.

[0013] Generating the input data may include detecting a radar signal using at least one radar sensor arranged along an outer surface of the ego-motion estimation device, and generating the radar sensing data by pre-processing the detected radar signal.

[0014] Generating the input data may include selecting at least two items of radar sensing data corresponding to time frames different from each other by a preset time interval from among the plurality of radar sensing data items, and generating the input data based on the selected radar sensing data items.

[0015] Generating the input data may include excluding radar sensing data corresponding to a first frame of the plurality of time frames stacked in the input data in response to receiving radar sensing data corresponding to a subsequent frame.

[0016] Generating the input data may include generating radar sensing data from the radar signal, the radar sensing data indicating an angle and a distance of a point detected by the radar sensor for each quantized speed.

[0017] Generating the input data may include generating the input data from the radar sensing data, the input data indicating a horizontal angle and a distance of a point detected by the radar sensor for each quantized elevation angle.

[0018] Generating input data may include: classifying radar sensing data into static data about static points and dynamic data about dynamic points; generating static input data according to the static data, wherein the static input data indicates the horizontal angle and distance of the point detected by the radar sensor for each quantized elevation angle; and generating dynamic input data according to the dynamic data, wherein the dynamic input data indicates the horizontal angle and distance of the point detected by the radar sensor for each quantized elevation angle.

[0019] The motion recognition model can include convolutional neural network (CNN) and recurrent neural network.

[0020] The recurrent neural network can be a bidirectional neural network.

[0021] Estimating the self-motion information may include: in response to a stacking of a plurality of radar sensing data corresponding to a plurality of time frames in the input data, calculating the self-motion information for each of the plurality of time frames.

[0022] The method may further include detecting an object near the self-motion estimation device based on the estimated self-motion information.

[0023] In another general aspect, a model training method includes: generating reference input data related to multiple time frames and reference output data corresponding to the reference input data based on reference radar sensing data; and training a processor for parameters of the motion recognition model based on a motion recognition model so that the processor outputs reference output data based on the reference input data.

[0024] In yet another general aspect, a self-motion estimation device includes: a radar sensor configured to generate radar sensing data; and a processor configured to generate input data based on the radar sensing data for each time frame, and to estimate self-motion information of the self-motion estimation device based on the input data based on a motion recognition model.

[0025] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An example of estimating self-motion information from radar sensing data based on a motion recognition model is shown.

[0027] Figure 2 An example of the structure of a motion recognition model is shown.

[0028] Figure 3 An example of an ego-motion estimation method is shown.

[0029] Figure 4 An example of an ego-motion estimation device is shown.

[0030] Figure 5 An example of the process of ego-motion estimation is shown.

[0031] Figure 6 An example of the structure of a motion recognition model is shown.

[0032] Figure 7 and Figure 8An example of generating input data is shown.

[0033] Fig. 9 and Fig.10 An example of input data generated using stacked radar sensing data is shown.

[0034] Fig.11 and Fig.12 An example of a model training method is shown.

[0035] Fig.13 An example of a model training device is shown.

[0036] Fig.14 An example of the model training process is shown.

[0037] Fig.15 An example of an ego-motion estimation device using multiple radar sensors is shown.

[0038] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals should be understood to refer to the same elements, features, and structures. The drawings may not necessarily be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0039] The following detailed description is provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the method, device and / or system described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those order of operations set forth herein, but can be changed in a manner that will be clear after understanding the disclosure of the present application, except for the operations that must occur in a certain order. In addition, for greater clarity and brevity, the description of features known in the art may be omitted.

[0040] The features described herein may be embodied in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0041] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are intended to include plural forms. The terms "include", "comprise" and "have" indicate the presence of the described features, numbers, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. Terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning consistent with that in the context of the relevant technology and should not be interpreted as an ideal or overly formal meaning unless explicitly defined as such herein.

[0043] Regarding the reference numerals assigned to the elements in the drawings, it should be noted that, although the same elements are shown in different drawings, the same elements will be represented by the same reference numerals where possible. In addition, in the description of the embodiments, when it is considered that a detailed description of a known related structure or function will lead to an ambiguous interpretation of the present disclosure, such description will be omitted.

[0044] Figure 1 An example of estimating self-motion information from radar sensing data based on a motion recognition model is shown.

[0045] The motion recognition model may be a model designed to output the self-motion information 109 according to the radar sensing data 101. The motion recognition model may have a machine learning structure, but is not limited thereto. The self-motion estimation device generates the self-motion information 109 for each time frame at once according to the input data of the radar sensing data 101 in which a plurality of time frames are stacked, using the motion recognition model.

[0046] The radar sensing data 101 may be data sensed by a radar sensor. For example, the radar sensor receives a signal obtained by reflecting a signal transmitted from the radar sensor to a target point, and generates the radar sensing data 101 using the received signal. The radar sensing data 101 includes the distance from the radar sensor to the target point. Figure 1 As shown, radar sensing data 101 is input to the motion recognition model in a stacked form.

[0047] The self-motion information 109 is information associated with the motion of the device and includes, for example, the relative motion, coordinates, and posture of the device. Figure 1 In the example, the self-motion information 109 indicates the coordinates (x i ,y i), N is an integer greater than or equal to 1 and i is an integer between 1 and N. The time frame may be a unit of segmented time. The self-motion information 109 is not limited to the above examples, and thus may include various values, such as three-dimensional (3D) coordinates (e.g., x-coordinates, y-coordinates, and z-coordinates, distances, azimuths, and elevations), two-dimensional (2D) coordinates (e.g., x-coordinates and y-coordinates, distances, and angular coordinates), east, north, altitude, roll, yaw, pitch, speed, etc.

[0048] The motion recognition model includes a neural network 100. Based on the neural network 100, a method for performing self-motion recognition and a device for performing the method (hereinafter referred to as a "self-motion estimation device") are proposed, and a method for training the neural network 100 and a device for training the neural network 100 (hereinafter referred to as a "model training device") are proposed.

[0049] Before describing ego-motion recognition, the structure of the neural network 100 will be described below.

[0050] The neural network 100 includes a plurality of layers, each layer including a plurality of nodes. In addition, the neural network 100 includes connection weights connecting the plurality of nodes included in the plurality of layers to the nodes included in another layer. The model training device acquires the neural network 100 from an internal database stored in a memory, or receives the neural network 100 from an external server through a communicator.

[0051] The neural network 100 is, for example, a recognition model that uses a plurality of artificial nodes connected by connecting lines (eg, edges) to perform desired operations. The neural network 100 may be implemented by hardware or a combination of hardware and software. The neural network 100 may also be referred to as an artificial neural network.

[0052] The neural network 100 includes a plurality of nodes separated in each layer. The nodes are connected to each other through edges with connection weights. The connection weights may be predetermined values ​​of the edges and may also be referred to as connection strengths.

[0053] The neural network 100 includes a plurality of layers. The neural network 100 includes, for example, an input layer 110, a hidden layer 120, and an output layer 130. The input layer 110 receives an input for performing training or recognition, and transmits the input to the hidden layer 120. The output layer 130 generates an output of the neural network 100 based on a signal received from the hidden layer 120. The hidden layer 120 is inserted between the input layer 110 and the output layer 130, and changes the training input of the training data transmitted through the input layer 110 into an easily predictable value.

[0054] The input layer 110, the hidden layer 120, and the output layer 130 each include a plurality of nodes. The nodes included in the input layer 110 may be referred to as input nodes. The nodes included in the hidden layer 120 may be referred to as hidden nodes. The nodes included in the output layer 130 may be referred to as output nodes.

[0055] The input nodes included in the input layer 110 and the hidden nodes included in the hidden layer 120 are connected by edges having connection weights. The hidden nodes included in the hidden layer 120 and the output nodes included in the output layer 130 are connected by edges having connection weights.

[0056] Although not shown, the neural network includes a plurality of hidden layers. A neural network including a plurality of hidden layers may be referred to as a deep neural network. Training of a deep neural network may be referred to as deep learning. When the hidden layer 120 includes a first hidden layer, a second hidden layer, and a third hidden layer, the output of the hidden node included in the first hidden layer is connected to the hidden node included in the second hidden node. The output of the hidden node included in the second hidden layer is connected to the hidden node included in the third hidden layer.

[0057] The model training device and the self-motion estimation device input the output of the previous hidden node included in the previous hidden layer to each hidden layer using an edge with a connection weight. In addition, the model training device and the self-motion estimation device generate the output of the hidden node included in the hidden layer based on the activation function and the value obtained by applying the connection weight to the output of the previous hidden node. For example, in order to transmit an output to a subsequent hidden node, the result of the activation function may need to be greater than the threshold of the current hidden node. In this example, the node remains in an unactivated state and does not send a signal to the subsequent node until the threshold activation strength is reached using the input vector.

[0058] The model training device trains the neural network 100 by supervised learning. The model training device can be implemented by a hardware module or a combination of a software module and a hardware module. Supervised learning is a scheme in which training inputs of training data and training outputs corresponding to the training inputs are input to the neural network 100, and the connection weights of the edges are updated so that output data corresponding to the training outputs of the training data are output. Figure 1 A neural network having a node structure is shown, but the embodiment is not limited to the node structure. A neural network may be stored in a storage device using various data structures.

[0059] The training data is data used for training, and includes a pair consisting of a training input and a training output corresponding to the training input. The training input is, for example, pre-collected sample radar data. The training output is a given value to be output for a predetermined training input, and is, for example, self-motion information 109 given for the sample radar data.

[0060] The model training device determines the parameters of the nodes using a gradient descent scheme based on the output values ​​of the nodes included in the neural network and the losses back-propagated to the neural network. For example, the model training device updates the connection weights between the nodes by loss back-propagation learning. Loss back-propagation learning is a method in which the loss is estimated by performing forward calculations on given training data, the estimated loss is propagated in the reverse direction from the output layer 130 to the hidden layer 120 and the input layer 110, and the connection weights are updated to reduce the loss. The processing of the neural network 100 is performed in the direction from the input layer 110 to the hidden layer 120 and the output layer 130. In the loss back-propagation training, the update of the connection weights is performed in the direction from the output layer 130 to the hidden layer 120 and the input layer 110. One or more processes may use a buffer memory that stores a layer of calculation data or a series of calculation data to process the neural network in the desired direction.

[0061] The model training device defines an objective function for measuring the proximity of the currently set connection weights to the optimal value, continuously changes the connection weights based on the result of the objective function, and repeatedly performs training. For example, the objective function is a loss function for calculating the loss between the actual output value output by the neural network 100 based on the training input of the training data and the desired expected value to be output (e.g., the training output). The model training device updates the connection weights to reduce the value of the loss function.

[0062] The configuration of the motion recognition model will be described below.

[0063] Figure 2 An example of the structure of a motion recognition model is shown.

[0064] The motion recognition model 220 includes a first model 221 and a second model 222 .

[0065] The first model 221 is a model for extracting feature data from predetermined data. The self-motion estimation device extracts feature data from the input data 210 based on the first model 221. The first model 221 includes a convolutional neural network (CNN). The first model 221 includes a layer corresponding to each time frame of a plurality of time frames. Figure 2 As shown, the first model 221 includes a convolutional layer corresponding to each time frame of the input data 210.

[0066] The second model 222 is a model for generating the self-motion information 230 according to the feature data. The self-motion estimation device calculates the self-motion information 230 according to the feature data based on the second model 222. The second model 222 includes a recurrent neural network. For example, the second model 222 includes a bidirectional recurrent neural network as a recurrent neural network. Figure 2As shown, the second model 222 includes layers connected to the layers of the first model 221 corresponding to each time frame.

[0067] The input data 210 is data having a structure in which radar sensing data 201 corresponding to each of a plurality of time frames are stacked. Figure 2 In the example of , the input data 210 is data in which the radar sensing data 201 corresponding to the frame t-1, the frame t, and the frame t+1 are stacked, and t is an integer greater than or equal to 1.

[0068] Although Figure 2 The radar sensing data 201 is shown as a scanning image indicating a distance from a target point for each steering angle, but the radar sensing data 201 is not limited thereto. The radar sensing data 201 may include all data sensed by the radar sensor for each time frame.

[0069] Based on the layer corresponding to each of the plurality of time frames in the first model 221, the self-motion estimation device extracts feature data of the corresponding time frame from the input data 210 of the corresponding time frame. The self-motion estimation device calculates self-motion information 230 of the corresponding time frame according to the extracted feature data based on the second model 222. In addition, the self-motion estimation device inputs radar sensing data 201 corresponding to at least two time frames for the layer corresponding to one time frame in the motion recognition model 220. Figure 2 As shown, the self-motion estimation device inputs the radar sensing data 201 corresponding to the frame t-1 and the radar sensing data 201 corresponding to the frame t to the convolution layer corresponding to the frame t in the first model 221.

[0070] The self-motion estimation device estimates self-motion information 230 of a plurality of time frames from input data 210 of a plurality of time frames based on a motion recognition model 220. When a plurality of radar sensing data corresponding to a plurality of time frames are stacked in the input data 210, the self-motion estimation device obtains self-motion information 230 of each of the plurality of time frames. Figure 2 In the example of , the input data 210 indicates the radar sensing data 201 associated with the frame t-1, the frame t, and the frame t+1. The self-motion estimation device outputs the device coordinates in the frame t-1, the device coordinates in the frame t, and the device coordinates in the frame t+1 according to the input data 210.

[0071] Although Figure 2For simplicity of description, three time frames are shown, but the number of frames used in the self-motion estimation device is not limited thereto. For example, the self-motion estimation device generates input data 210 of M time frames. In this example, the input data 210 may be data in which radar sensing data 201 corresponding to frame t-M+1 to frame t are stacked, and M is an integer greater than or equal to 1. In addition, although Figure 2 It is shown that radar sensing data 201 corresponding to a plurality of time frames are stacked in the input data 210 , but the embodiment is not limited thereto. The input data 210 may include radar sensing data 201 corresponding to one time frame.

[0072] A method of estimating ego-motion using a motion recognition model will be described below.

[0073] Figure 3 An example of an ego-motion estimation method is shown.

[0074] In operation 310, the self-motion estimation device generates input data for each time frame based on radar sensing data collected by the radar sensor. For example, the radar signal is detected by at least one radar sensor arranged along the outer surface of the self-motion estimation device. The radar signal can be a signal emitted by the radar sensor, reflected from the target point, and then sensed by the radar sensor. The self-motion estimation device generates the radar sensing data by preprocessing the detected radar signal.

[0075] In operation 320, the ego-motion estimation apparatus estimates ego-motion information of the ego-motion estimation apparatus from input data based on a motion recognition model. The ego-motion estimation apparatus calculates at least one of a position and a posture of the ego-motion estimation apparatus as the ego-motion information.

[0076] Figure 4 An example of an ego-motion estimation device is shown.

[0077] The ego-motion estimation apparatus 400 includes a radar sensor 410 , a processor 420 , and a memory 430 .

[0078] The radar sensor 410 is a sensor that transmits and receives radar signals. The radar sensor 410 generates radar sensing data by preprocessing the radar signal. For example, the radar sensor 410 generates radar sensing data based on the frequency difference between the signal transmitted by the radar sensor 410 and the reflected signal.

[0079] The processor 420 generates input data according to the radar sensing data of each time frame. The processor 420 estimates the self-motion information of the self-motion estimation device 400 according to the input data based on the motion recognition model. Figures 5 to 15 The operation of processor 420 is further described.

[0080] The memory 430 temporarily or permanently stores information required for executing the self-motion estimation method. For example, the memory 430 stores a motion recognition model and trained parameters of the motion recognition model. In addition, the memory 430 stores radar sensing data and input data.

[0081] Figure 5 An example of the process of ego-motion estimation is shown.

[0082] The self-motion estimation device generates input data by performing a preprocessing operation 510 on the radar sensing data 501. The self-motion estimation device estimates self-motion information 509 from the input data based on a motion recognition model 520. However, the preprocessing operation 510 is not limited thereto. Through the preprocessing operation 510, the self-motion estimation device converts a radar signal corresponding to raw data acquired by the radar sensor into radar sensing data 501 indicating distance, angle, and speed information. The radar sensing data 501 may vary based on, for example, a desired resolution and a modulation scheme of the radar sensor.

[0083] The self-motion estimation device extracts current feature data from the input data corresponding to the current frame based on the first model 521. The self-motion estimation device loads the previous feature data 580 corresponding to the previous frame from the memory. After preprocessing, the radar sensing data 501 is represented as 3D information, such as distance, steering angle, and Doppler velocity, which can be a structure similar to an image, such as X coordinate, Y coordinate, and channel. When extracting features of the image, CNN and convolutional layers can achieve better performance. CNN can have good performance in extracting information related to the distance and angle axes (as position information), and can also have good performance for the Doppler velocity axis corresponding to the channel (e.g., RGB of the image).

[0084] The self-motion estimation device obtains self-motion information 509 based on the previous feature data 580 and the current feature data based on the second model 522. The radar sensor continuously acquires radar sensing data for each time frame. In this example, the radar sensing data acquired for adjacent time frames may have similar time correlation. Models with good performance in identifying data with time correlation may be recurrent neural networks and recurrent layers.

[0085] Figure 6 An example of the structure of a motion recognition model is shown.

[0086] When continuously acquiring continuous radar sensing data, the self-motion estimation device skips preprocessing and feature data extraction for the previous time frame because the preprocessing and feature data extraction for the previous time frame are redundant. For example, the self-motion estimation device stores feature data extracted from the CNN and performs preprocessing and feature data extraction on the radar sensing data acquired for each time frame to be sent to the recurrent neural network together with the previous feature data.

[0087] The self-motion estimation device calculates feature data corresponding to the current frame based on the convolution layer corresponding to the current frame of the first model 621 according to the input data 610 corresponding to the current frame. The self-motion estimation device stores the feature data corresponding to the current frame calculated based on the first model 621 included in the motion recognition model in the memory. The self-motion estimation device adds the feature data corresponding to the current frame to the feature database 680 stored in the memory.

[0088] The self-motion estimation device is based on input data 610 corresponding to a portion of the plurality of time frames (e.g., Figure 6 For example, the first model 621 can be implemented by outputting feature data of a corresponding time frame according to input data 610 corresponding to one time frame. The second model 622 can be implemented by outputting self-motion information 630 corresponding to M frames according to feature data corresponding to M frames. Figure 6 In the example of , M is "3". The self-motion estimation device extracts current feature data corresponding to frame t from input data 610 corresponding to frame t based on the first model 621, and loads previous feature data corresponding to frame t-M+1 to frame t-1. The self-motion estimation device generates self-motion information 630 corresponding to frame t-M+1 to frame t-1 based on the feature data corresponding to frame t-M+1 to frame t-1 based on the second model 622.

[0089] In this way, the ego-motion estimation device does not extract feature data corresponding to M time frames for each time frame, but extracts feature data corresponding to one time frame, thereby reducing the computational resources and time required for feature data extraction.

[0090] Figure 7 and Figure 8 An example of generating input data is shown.

[0091] The self-motion estimation device classifies a distance (e.g., a distance from the device to a target point) and a graph indicating a steering angle relative to a target point based on, for example, an elevation angle and a velocity at the target point. The self-motion estimation device generates input data by converting and classifying a radar sensing model into a form suitable for input in a motion recognition model.

[0092] The self-motion estimation device generates radar sensing data based on the radar signal, and the radar sensing data indicates the distance and angle of the point detected by the radar sensor for each quantized speed. The self-motion estimation device uses the generated radar sensing data as input data 710. For example, the speed verified based on the radar sensing data is the speed of the self-motion estimation device relative to the target point. For example, the angle verified based on the radar sensing data is the steering angle, which is the angular difference between the direction of travel of the device and the direction from the device toward the target point. Although the present disclosure describes the steering angle as a horizontal angle on a 2D plane sensed by the radar sensor, the embodiments are not limited thereto. The steering of the radar sensor can be performed vertically and horizontally. When the steering is performed vertically, the steering angle can be an elevation angle.

[0093] For the sake of simplicity, Figure 7 Input data 710 generated for one time frame is shown. Input data 710 generated for one time frame includes radar sensing data 711, which indicates the distance and angle corresponding to the target point detected by the radar sensor for each quantized speed. When radar sensing data corresponding to multiple time frames are stacked, the self-motion estimation device generates input data 710 for each time frame in the multiple time frames, and the input data 710 indicates the distance and angle corresponding to the target point for each quantized speed. The self-motion estimation device calculates self-motion information from the input data 710 based on the motion recognition model 720.

[0094] The self-motion estimation device generates input data based on the radar sensing data, which indicates the distance and horizontal angle corresponding to the point detected by the radar sensor for each elevation angle. The elevation angle can be the angle formed by the target point relative to the horizontal plane of the radar sensor relative to the vertical direction based on the radar sensor.

[0095] The self-motion estimation device classifies the radar sensing data into static data 810 associated with static points and dynamic data 820 associated with dynamic points. Static points can be stationary objects of target points and points with an absolute speed of zero. Dynamic points can be moving objects of target points and points with an absolute speed that is not zero. The self-motion estimation device generates static input data indicating the distance and horizontal angle of the point detected by the radar sensor for each quantization angle based on the static data 810. The self-motion estimation device generates dynamic input data indicating the distance and horizontal angle of the point detected by the radar sensor for each quantization angle based on the dynamic data 820. For the sake of simplicity of description, Figure 8The input data generated for one time frame is shown. The input data generated for one time frame includes static data 810 and dynamic data 820. The static data 810 of the predetermined time frame includes data 811, which indicates the distance and horizontal angle of the target point detected by the radar sensor for each quantized angle. The dynamic data 820 of the predetermined time frame includes data 821, which indicates the distance and horizontal angle of the target point detected by the radar sensor for each quantized angle.

[0096] The ego-motion estimation apparatus calculates ego-motion information from input data including static data 810 and dynamic data 820 based on a motion recognition model 830 .

[0097] Fig. 9 and Fig.10 An example of input data generated using stacked radar sensing data is shown.

[0098] The radar sensing data is aggregated along a spatial axis.

[0099] Fig. 9 The input data in which radar sensing data is stacked by skipping several time frames is shown. The self-motion estimation device selects at least two radar sensing data of time frames different by a preset time interval from among the plurality of radar sensing data. Fig. 9 In the example of , the preset time interval may be three frames. The self-motion estimation device collects the first frame to the tenth frame. The self-motion estimation device selects radar sensing data 911 corresponding to the first frame, the fourth frame, the seventh frame, and the tenth frame. In the eleventh frame, the self-motion estimation device selects radar sensing data 912 corresponding to the second frame, the fifth frame, the eighth frame, and the eleventh frame.

[0100] The self-motion estimation device generates input data based on the selected radar sensing data. In the tenth frame, the self-motion estimation device inputs the input data generated according to the radar sensing data 911 corresponding to the first frame, the fourth frame, the seventh frame and the tenth frame to the motion recognition model 920. The self-motion estimation device calculates the self-motion information corresponding to the first frame, the fourth frame, the seventh frame and the tenth frame based on the radar sensing data 911 based on the motion recognition model 920. In the eleventh frame, the self-motion estimation device inputs the input data generated according to the radar sensing data 912 corresponding to the second frame, the fifth frame, the eighth frame and the eleventh frame to the motion recognition model 920. The self-motion estimation device calculates the self-motion information corresponding to the second frame, the fifth frame, the eighth frame and the eleventh frame based on the radar sensing data 912 based on the motion recognition model 920. As an example, this order of the above-mentioned time frames is described, and the order of the time frames is not limited to this example.

[0101] Fig.10The generated input data corresponding to the predetermined time length of each time frame is shown. In response to receiving radar sensing data corresponding to a subsequent frame, the self-motion estimation device excludes radar sensing data corresponding to a first frame of the plurality of time frames stacked in the input data.

[0102] refer to Fig.10 , the self-motion estimation device collects radar sensing data corresponding to the first frame to the eighth frame. The self-motion estimation device generates input data 1011 corresponding to the eighth frame by stacking the radar sensing data corresponding to the first frame to the eighth frame. In the ninth frame, the self-motion estimation device excludes the radar sensing data corresponding to the first frame. The self-motion estimation device generates input data 1012 corresponding to the ninth frame by stacking the radar sensing data corresponding to the second frame to the ninth frame. The self-motion estimation device calculates the self-motion information corresponding to the first frame to the eighth frame based on the input data 1011 corresponding to the first frame to the eighth frame based on the motion recognition model 1020. The self-motion estimation device calculates the self-motion information corresponding to the second frame to the ninth frame based on the input data 1012 corresponding to the second frame to the ninth frame based on the motion recognition model 1020.

[0103] Fig.11 and Fig.12 An example of a model training method is shown.

[0104] Fig.11 The training process is shown.

[0105] In operation 1110, the model training device generates reference input data related to a plurality of time frames and reference output data corresponding to the reference input data based on the reference radar sensing data. The reference sensing data corresponds to the training input, and the reference output data corresponds to the training output. The training data includes, for example, a pair consisting of the reference radar sensing data and the reference output data. The reference output data may also be referred to as reference self-motion information.

[0106] In operation 1120, the model training device trains the processor for the parameters of the motion recognition model using the motion recognition model so that the processor outputs reference output data based on the reference input data. The model training device updates the parameters of the motion recognition model based on the motion recognition model until the loss between the temporary output calculated according to the reference input data and the reference output data is less than the threshold loss.

[0107] Reference Fig.12 Further description Fig.11 training.

[0108] In operation 1211, the model training device acquires reference radar sensing data. The model training device acquires the reference radar sensing data from a training database.

[0109] In operation 1212, the model training device generates reference input data and reference output data. Figures 7 to 10 The operations similar to the described operations generate reference input data according to the reference radar sensing data. The model training device obtains reference output data from the training database.

[0110] In operation 1221, the model training device calculates the output of the first model associated with the reference input data. Figure 1 As described, the model training device calculates temporary feature data by inputting reference input data into the first model and propagating the reference input data to each layer.

[0111] In operation 1222, the model training device calculates the output of the second model associated with the output of the first model. Figure 1 As described, the model training device calculates a temporary output by inputting feature data into a second model and propagating the feature data.

[0112] In operation 1223, the model training device updates the parameters based on the loss between the output of the second model and the reference output data. The model training device calculates the value of the objective function between the temporary output calculated based on the second model and the reference output data as the loss. The objective function can be, for example, mean square error (MSE), absolute error, and Huber loss. The objective function can be combined with a regularization term of the parameter for use. The model training device adjusts the parameters of the motion recognition model to reduce the loss. In the case where the motion recognition model is a neural network, the parameters of the motion recognition model can be (but are not limited to) connection weights.

[0113] In operation 1224, the model training device stores the updated parameters. The model training device stores the updated parameters in a memory. The model training device uploads the updated parameters to a server, or sends the updated parameters to the self-motion estimation device. The parameters uploaded to the server can be transmitted to the self-motion estimation device via a communication network or a data interface.

[0114] In operation 1225, the model training device determines whether the loss has converged. The model training device determines whether the calculated loss is less than the threshold loss. In response to the loss having converged, the model training device terminates the training. In response to the loss not converging, the model training device repeatedly performs operations 1211 to 1225.

[0115] Fig.13 An example of a model training device is shown.

[0116] The model training device 1300 includes a processor 1310 and a memory 1320 .

[0117] The processor 1310 generates reference input data 1391 related to a plurality of time frames and reference output data 1392 corresponding to the reference input data 1391 according to the reference radar sensing data. The processor 1310 trains the processor 1310 for the parameters 1322 of the motion recognition model 1321 based on the motion recognition model 1321, so that the processor 1310 outputs the reference output data 1392 based on the reference input data 1391. The operation of the processor 1310 is not limited thereto, but may be combined with the reference Fig.11 and Fig.12 To perform the described operation.

[0118] The memory 1320 temporarily or permanently stores information required for model training. The memory 1320 stores a motion recognition model 1321, parameters 1322, and training data. The memory 1320 stores reference input data 1391 and reference output data 1392 as training data. The reference output data 1392 may be a value given for the reference input data 1391, and may be self-motion information to be output in association with the reference input data 1391. In addition, the memory 1320 stores reference sensing data instead of the reference input data 1391.

[0119] The model training device 1300 may be implemented separately from the ego-motion estimation device, but is not limited thereto. The model training device 1300 and the ego-motion estimation device may also be implemented as a single device.

[0120] Fig.14 An example of the model training process is shown.

[0121] The model training device trains the motion recognition model to output reference self-motion information 1492 based on the reference radar sensing data 1491. The model training device uses N items in the continuous radar data for which preprocessing is completed. To perform training, the model training device uses N items in the continuous radar data. B The motion recognition model includes CNN and recurrent neural network. The self-motion information output by the motion recognition model includes the estimated value of the device's position and posture. Fig.14 As shown in , the self-motion information includes, for example, N sets of x-coordinates and y-coordinates for each time frame.

[0122] The model training apparatus acquires a radar signal as raw data, and generates reference radar sensing data 1491 through basic radar signal processing (RSP) preprocessing. The model training apparatus generates reference input data by temporarily accumulating the reference radar sensing data 1491.

[0123] To perform training, the model training device calculates temporary feature data for each time frame based on the first model 1421. Fig.14 In the example of , the model training device calculates temporary feature data for each of frame t-2, frame t-1, and frame t. The model training device calculates a temporary output for each time frame based on the second model 1422.

[0124] The model training device calculates an error value, such as a loss, by comparing the temporary output with the calibration true data (e.g., the reference self-motion information 1492). The model training device uses a loss function such as MSE.

[0125] The model training device propagates the calculated error value to the lower layer based on the back propagation through time (BPTT) scheme and updates the parameters between the layers of the motion recognition model.

[0126] For each training, the model training device generates a trained motion recognition model by storing the state values ​​and weights of the first model 1421 and the second model 1422. When the error value converges, the model training device terminates the training, otherwise retraining is performed.

[0127] When the model training device performs training, the first model 1421 includes the layer 1450 of the previous frame. When the first model 1421 is used for self-motion estimation, the layer 1450 of the previous frame is removed. The model training device uses a plurality of training data for training, thereby improving the training speed. In addition, the self-motion estimation device uses the radar sensing data corresponding to the current frame, thereby improving the operation speed of the self-motion estimation.

[0128] Fig.15 An example of an ego-motion estimation device using multiple radar sensors is shown.

[0129] refer to Fig.15 , the self-motion estimation device 1500 can be implemented as a vehicle, for example. The self-motion estimation device 1500 estimates the posture and movement path of the vehicle by positioning the radar sensor in the vehicle. The self-motion estimation device 1500 detects objects near the self-motion estimation device 1500 based on the estimated self-motion information. The self-motion estimation device 1500 assists the autonomous driving of the vehicle and various advanced driver assistance system (ADAS) functions. The self-motion estimation device 1500 acquires a wide-angle high-resolution image based on the self-position information accurately measured for each time frame.

[0130] like Fig.15As shown in , when the self-motion estimation device 1500 includes a plurality of radar sensors, the self-motion estimation device 1500 positions each of the radar sensors and fuses the positioning results, thereby improving the accuracy of the self-motion information estimation. The self-motion estimation device 1500 may combine the radar sensor with another sensor (e.g., a camera sensor, an ultrasonic sensor, and light detection and ranging (LiDAR)), or fuse the self-motion information with information such as a global navigation satellite system (GNSS), vehicle speed, and steering angle, thereby improving the accuracy. Fig.15 As shown in , when acquiring omnidirectional radar sensing data, the self-motion estimation device 1500 generates a 360-degree image around the vehicle or recognizes the surrounding environment of the vehicle.

[0131] The self-motion estimation apparatus 1500 can maintain accuracy regardless of the passage of time without requiring special actions such as loop closing. In addition, the self-motion estimation apparatus 1500 can maintain accuracy even when a moving object is in a capture scene.

[0132] The equipment, unit, module, device and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or combinations of any other devices or devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes (or is connected to) one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software.For the sake of brevity, the description of the examples described in this application can use the singular term "processor" or "computer", but in other examples, multiple processors or computers can be used, or processors or computers can include multiple processing elements, or multiple types of processing elements, or both.For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller.One or more processors or a processor and a controller can implement a single hardware component or two or more hardware components.Hardware components can have any one or more processing configurations in different processing configurations, and the example of processing configuration includes a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0133] The method for performing the operation described in the present application is performed by computing hardware, for example, by one or more processors or computers implemented as described above that execute instructions or software to perform the operations performed by these methods described in the present application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0134] The instructions or software for controlling the processor or computer to implement the hardware components and perform the method as described above are written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or special-purpose computer to perform the operations and methods performed by the hardware components as described above. In one example, the instructions or software include machine code directly executed by the processor or computer, such as machine code generated by a compiler. In another example, the instructions or software include higher-level codes executed by the processor or computer using an interpreter. An ordinary programmer in the art can easily write instructions or software based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, and the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing the operations performed by the hardware components and methods as described above.

[0135] The instructions or software that controls the processor or computer-implemented hardware components and performs the methods described above, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type memory (such as, multimedia card or micro card (for example, Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured as follows: to store instructions or software and any associated data, data files and data structures in a non-temporary manner, and to provide instructions or software and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0136] Although the present disclosure includes specific examples, it will be apparent to those skilled in the art that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered illustrative only and not for purposes of limitation. The description of the features or aspects in each example is considered applicable to similar features or aspects in other examples. Appropriate results may be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the present disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents are to be interpreted as being included in the present disclosure.

Claims

1. A method for estimating self-motion, the method comprising: generating input data based on radar sensing data collected by the radar sensor for each time frame; as well as estimating the self-motion information of the self-motion estimation device according to the input data based on the motion recognition model, The estimated self-motion information includes: Extracting current feature data from input data corresponding to a current time frame based on a first model, wherein the input data corresponding to the current time frame includes radar sensing data corresponding to the current time frame and radar sensing data corresponding to a time frame before the current time frame; loading from memory prior feature data corresponding to a prior time frame; calculating, based on the second model, self-motion information corresponding to the previous time frame and the current time frame respectively according to the previous feature data and the current feature data; and calculating at least one of the three-dimensional 3D coordinates, the two-dimensional 2D coordinates, the roll, the yaw or the pitch of the self-motion estimation device as the self-motion information, and Among them, the first model includes a convolutional neural network (CNN), and the second model includes a recurrent neural network.

2. The method according to claim 1, wherein: The motion recognition model includes: The first model includes a layer corresponding to each time frame of a plurality of time frames; and the second model is connected to a plurality of layers of the first model, and The estimated self-motion information includes: extracting feature data of the corresponding time frame from input data of the corresponding time frame based on the layer corresponding to each time frame in the first model; and The self-motion information of the corresponding time frame is calculated according to the extracted feature data based on the second model.

3. The method according to claim 1, further comprising: The feature data calculated in the current time frame based on the first model included in the motion recognition model is stored in the memory.

4. The method according to claim 1, wherein: Generating input data includes: detecting radar signals using at least one radar sensor arranged along an outer surface of the ego-motion estimation device; and The radar sensing data is generated by pre-processing the detected radar signals.

5. The method according to claim 1, wherein: Generating input data includes: selecting at least two items of radar sensing data corresponding to time frames different from each other by a preset time interval from the plurality of radar sensing data; and The input data is generated based on radar sensing data of the selected item.

6. The method according to claim 1, wherein: Generating input data includes: In response to receiving radar sensing data corresponding to a subsequent time frame, radar sensing data corresponding to a first time frame of the plurality of time frames that is stacked in the input data is excluded.

7. The method according to claim 1, wherein: Generating input data includes: Radar sensing data is generated from the radar signal, the radar sensing data indicating an angle and a distance relative to a point detected by the radar sensor for each quantized speed.

8. The method according to claim 1, wherein: Generating input data includes: Input data is generated from the radar sensing data, the input data indicating a horizontal angle and a distance of a point detected by the radar sensor for each quantized elevation angle.

9. The method according to claim 1, wherein: Generating input data includes: classifying the radar sensing data into static data about static points and dynamic data about dynamic points; generating static input data from the static data, the static input data indicating a horizontal angle and a distance of a point detected by the radar sensor for each quantized elevation angle; and Dynamic input data is generated from the dynamic data, the dynamic input data indicating a horizontal angle and a distance of a point detected by the radar sensor for each quantized elevation angle.

10. The method according to claim 1, wherein: The recurrent neural network is a bidirectional neural network.

11. The method according to claim 1, wherein: The estimated ego-motion information includes: In response to a stacking of multiple pieces of radar sensing data corresponding to a plurality of time frames in the input data, self-motion information is calculated for each of the plurality of time frames.

12. The method according to claim 1, further comprising: An object near the self-motion estimation device is detected based on the estimated self-motion information.

13. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method of claim 1.

14. A self-motion estimation device, comprising: a radar sensor configured to generate radar sensing data; as well as a processor configured to generate input data according to the radar sensing data for each time frame, and estimate the self-motion information of the self-motion estimation device according to the input data based on a motion recognition model, Wherein, the processor is further configured to: Extracting current feature data from input data corresponding to a current time frame based on a first model, wherein the input data corresponding to the current time frame includes radar sensing data corresponding to the current time frame and radar sensing data corresponding to a time frame before the current time frame; loading from memory prior feature data corresponding to a prior time frame; calculating, based on the second model, self-motion information corresponding to the previous time frame and the current time frame respectively according to the previous feature data and the current feature data; and calculating at least one of the three-dimensional 3D coordinates, the two-dimensional 2D coordinates, the roll, the yaw or the pitch of the self-motion estimation device as the self-motion information, and Among them, the first model includes a convolutional neural network (CNN), and the second model includes a recurrent neural network.

Images